TWiki . Dev . TestSuite

This page documents the testsuite engine for Nice. It also contains a list of wanted features that are not implemented yet.

Nice TestSuite Documentation

The Nice TestSuite is a framework to test the Nice compiler. For this aim, a proprietary file format for the testcases has been created. Here are some testcase examples:

/// COMMENT This testsuite tests the following ....
/// PASS
  int a = 1;

/// PASS
  /// TOPLEVEL
  /* A simple global var. */
  var int x = 0;


/// FAIL
  /// COMMENT this should fail
  int a = "";

/// PASS
  /// package a
  /// TOPLEVEL
  class A
  {
    int x = f();
  }
  int f() = 0;
  /// package b import a
  ;
TestCase? keywords are all prefixed with ///. On root level (without any whitespace indentation) currently following keywords are allowed

PASS and FAIL

The keywords PASS and FAIL indicate a testcase on its own, and whether the expected behaviour while compilation is pass or fail. Testcases cannot reference each other, they are all compiled and tested independently.

Known bugs

When a bug is found, I often write a testcase for it first, before the bug is fixed. In that case, it is useful to flag the testcase as being a known bug. This is done by adding bug at the end of the line. I such a testcase is indeed found to include an error, well we knew it and print nothing. But if it works, then print a victorious message! :-). In the summary:
  known bugs: 2
  fixed: 1

Testcases may contain the TOPLEVEL keyword that has the following meaning: everything in front of the TOPLEVEL keyword will be collected in the main() method. Statements behind the TOPLEVEL keyword are global to the package, like global variables, functions, methods and class definitions.

PACKAGE The PACKAGE keyword indicates which package the sources should belong to. More than one PACKAGE keyword can be declared to build a testcase with a certain package structure. Packages can be imported by the PACKAGE some IMPORT other keywords.

Global

After the GLOBAL keyword source code can be written that is accessable by all following testcases. Global sources can be placed at where (before, after or between testcases) in the testsuite file. The scope of global sources is restricted to the same testsuite file and to testcases that are behind the global source. Globals can appear more than one times in the file as the next example demonstrates
/// GLOBAL
  int g1 = 1
   
/// PASS
  /* access to g1 only */
  int a = 1;

/// GLOBAL
  int g2 = 2

/// PASS
  /* access to g1 and g2 */
  /// TOPLEVEL
  /* A simple global var. */
  var int x = 0;
Because globals are accessible only in the same file, placing global sources at the end of the file makes no sense.

Comments

Comments can be included, and if desired outputted, at any place in a testsuite. Comments have the following syntax /// COMMENT place your comments here

Comments are one-liners, if you want to write a long comment, write the whole comment in one line as described above or break it into more COMMENT keywords, each in a new line. The following example shows possible ways to comment

/// COMMENT This testsuite describes ....
/// GLOBAL
    ...
    
/// PASS
    /// COMMENT This testcase demonstartes ...
    /// COMMENT ... line two of testcase comment ...
    ...
    
/// COMMENT here we could comment the end of the testsuite :)
Activating comments in the output can be done by specifying the -comment flag in the command line

java -classpath /usr/share/java/nice.jar nice.tools.testsuite.TestNice ~/testsuite/ -comment or java -classpath /usr/share/java/nice.jar nice.tools.testsuite.TestNice -comment ~/testsuite/

While running the testcases temporary files and folders are created in "testsuite-temp-folder". Testcases that fail, in the sense of that they don't behave like expected (pass or fail) are collected in the folder "testsuite-fail-folder" for later investigation purposes. In this folder, each testcase is grouped in its own folder that is numbered from "1" on in ascending order. At the next testsuite run both folders are initially cleared. -- AlexGreif? - 27 Jun 2002

Specifying the location of failures

In a /// FAIL test, it is often desirable to specify where (which line, which column) the error should be reported. This allows for checking that the compiler gives the good location to the user, and also that the test does not fail "by accident", for an other unrelated reason (for instance a syntax error when writing the test).

This can be done, by embedding a comment of the form /* /// FAIL HERE */ just before the location of the desired error in the test source.

Warnings that result from wrong FAIL HERE positions are treated sort of failures. Thus the source code is shown and the compiler results.

Running the testsuite

To run the whole testsuite, just call make check in the main directory of Nice.

To run a specific subset, you can use:

java -Dassertions=true -classpath classes nice.tools.testsuite.TestNice testsuite/<directory>

If you use Java 1.4 or later, use java -ea instead of java -Dassertions=true (this should also be done in the Makefile, but how do we make it work for both 1.3 and 1.4?).

Feature requests

Saving output for failed tests

When a test fails, it would be useful if the output of the compilation (and the output of the run, with the stack trace of the exception thrown) was saved in a file, in the failure directory.

Saving successful tests

Sometimes I want to look at the code generated by succesful tests. It would be good to have a -save comment line option to say to save all tests, not just the failing ones.

Well, that's all for now! Quite a lot of new features :-) None is critical, so do what you can when you find the time. But all these features will make testing easier.

Compiler warnings

We need to track compiler warning, not just errors. I think by default a warning should be considered as an error. We could add a ///WARN tag to say that a test should (or can?) emit a warning. (Not urgent: I'm not sure if there is any interesting warning issued by the current compiler).

Failure positions

At the moment, the testsuite engine emits a warning if there was no error where you expected one. It should also warn if there was an error where you did not expect any. Probably, as an exception, a test case that has no explicit FAIL HERE is also OK.

The rationale is to test cases, like the one I just solved, where one error is expected, but the compiler reports two errors.

Macros

It could be useful to define some macros, in the form of /// keywords, for tests that are repeated many times with a similar pattern.

One example is:

/// TYPE Type1 SUBTYPE OF Type2

In the engine, this would be expanded to some Nice source that will compile if and only if Type1 is a subtype of Type2.

This is not urgent. If we really need something like this, perhaps it would be wiser (and more fun!) to transition the testsuite to a set of Nice functions, so that test cases could be written in Nice itself.

----- Revision r1.5 - 02 Aug 2003 - 15:40 GMT - DanielBonniot
Copyright © 1999-2003 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback.