TWiki > Dev > ToolsAndLibraries > TestSuite	TWiki webs: Dev | Doc | Main | TWiki | Sandbox
	Dev . { Changes | Index | Search | Go }

• Nice TestSuite Documentation
  • PASS and FAIL
    • Known bugs
  • Global
  • Comments
  • Specifying the location of failures
  • Running the testsuite
• Feature requests
  • Saving output for failed tests
  • Saving successful tests
  • Compiler warnings
  • Failure positions
  • Macros

Nice TestSuite Documentation

The Nice TestSuite is a framework to test the Nice compiler. For this aim, a proprietary file format for the testcases has been created. Here are some testcase examples:

/// COMMENT This testsuite tests the following ....
/// PASS
  int a = 1;

/// PASS
  /// TOPLEVEL
  /* A simple global var. */
  var int x = 0;


/// FAIL
  /// COMMENT this should fail
  int a = "";

/// PASS
  /// package a
  /// TOPLEVEL
  class A
  {
    int x = f();
  }
  int f() = 0;
  /// package b import a
  ;

PASS and FAIL

Known bugs

  known bugs: 2
  fixed: 1

Testcases may contain the TOPLEVEL keyword that has the following meaning: everything in front of the TOPLEVEL keyword will be collected in the main() method. Statements behind the TOPLEVEL keyword are global to the package, like global variables, functions, methods and class definitions.

PACKAGE The PACKAGE keyword indicates which package the sources should belong to. More than one PACKAGE keyword can be declared to build a testcase with a certain package structure. Packages can be imported by the PACKAGE some IMPORT other keywords.

Global

/// GLOBAL
  int g1 = 1
   
/// PASS
  /* access to g1 only */
  int a = 1;

/// GLOBAL
  int g2 = 2

/// PASS
  /* access to g1 and g2 */
  /// TOPLEVEL
  /* A simple global var. */
  var int x = 0;

Comments

Comments can be included, and if desired outputted, at any place in a testsuite. Comments have the following syntax /// COMMENT place your comments here

Comments are one-liners, if you want to write a long comment, write the whole comment in one line as described above or break it into more COMMENT keywords, each in a new line. The following example shows possible ways to comment

/// COMMENT This testsuite describes ....
/// GLOBAL
    ...
    
/// PASS
    /// COMMENT This testcase demonstartes ...
    /// COMMENT ... line two of testcase comment ...
    ...
    
/// COMMENT here we could comment the end of the testsuite :)

java -classpath /usr/share/java/nice.jar nice.tools.testsuite.TestNice ~/testsuite/ -comment or java -classpath /usr/share/java/nice.jar nice.tools.testsuite.TestNice -comment ~/testsuite/

While running the testcases temporary files and folders are created in "testsuite-temp-folder". Testcases that fail, in the sense of that they don't behave like expected (pass or fail) are collected in the folder "testsuite-fail-folder" for later investigation purposes. In this folder, each testcase is grouped in its own folder that is numbered from "1" on in ascending order. At the next testsuite run both folders are initially cleared. -- AlexGreif? - 27 Jun 2002

Specifying the location of failures

In a /// FAIL test, it is often desirable to specify where (which line, which column) the error should be reported. This allows for checking that the compiler gives the good location to the user, and also that the test does not fail "by accident", for an other unrelated reason (for instance a syntax error when writing the test).

This can be done, by embedding a comment of the form /* /// FAIL HERE */ just before the location of the desired error in the test source.

Warnings that result from wrong FAIL HERE positions are treated sort of failures. Thus the source code is shown and the compiler results.

Running the testsuite

To run a specific subset, you can use:

java -Dassertions=true -classpath classes nice.tools.testsuite.TestNice testsuite/<directory>

If you use Java 1.4 or later, use java -ea instead of java -Dassertions=true (this should also be done in the Makefile, but how do we make it work for both 1.3 and 1.4?).

Feature requests

Saving output for failed tests

Saving successful tests

Well, that's all for now! Quite a lot of new features :-) None is critical, so do what you can when you find the time. But all these features will make testing easier.

Compiler warnings

Failure positions

The rationale is to test cases, like the one I just solved, where one error is expected, but the compiler reports two errors.

Macros

It could be useful to define some macros, in the form of /// keywords, for tests that are repeated many times with a similar pattern.

One example is:

/// TYPE Type1 SUBTYPE OF Type2

In the engine, this would be expanded to some Nice source that will compile if and only if Type1 is a subtype of Type2.

This is not urgent. If we really need something like this, perhaps it would be wiser (and more fun!) to transition the testsuite to a set of Nice functions, so that test cases could be written in Nice itself.