Nice TWiki > Dev > NiceProjects? > TestSuite (r1.1) TWiki webs:
Dev | Doc | Main | TWiki | Sandbox
Dev . { Changes | Index | Search | Go }
This is the place where discussions around the nice testsuite can take place

Nice TestSuite Discussion

>We could also use other char sequences to signal the expected failure column
>or some other.
>I know you want to have something general in order to use a bunch of
>keyword in the middle of the code.
>My sugegstion would be to take
>/*/// FAIL HERE*/
>then I could scan /*...*/ and filter the keywords out like the others
>beginning with /// that are at the end of the line
>Then we would use the same format as in the past, and we would make it
>possible to use the same format for keywords ///, in the well known java
>comments /* and */.
>What do you think aboit it?
That sounds very reasonable.

>Why should we make a difference between the two type of markers.
>In my opinion both types a keywords used in the context we find them.
>I would appreciate /*/// FAIL HERE*/

>we could write it simpler:
>  /// TYPE Type1 SUBTYPE OF Type2
>"TYPE" and "SUBTYPE OF" are keywords , so we can leave the second ///
OK too.

Feature requests

Implementation of /* /// FAIL HERE */ as described above. FAIL HERE has been implemented, so the discussion can be removed, and the doc updated. I saw that a warning is issued when the error is not where expected. Since it is only a warning, it is easy to miss it. I think either it should be made an error, or the number of warnings should be given at the end, together with the number of tests and errors. -- DanielBonniot? - 12 Jul 2002

I think, supporting the number of errors at the end is the best solution -- AlexGreif? - 15 Jul 2002

OK. So it should end with something like:

number of testcases: 100
  succeeded: 97
  warnings :  2
  failed   :  1

I think it is better not to count warnings as succeeded, to stress that there is something to loot at. When the error location is wrong, it would also be nice to print both the expected and the actual location. This would help understanding the problem. I never know which one is printed! :-) -- DanielBonniot?

ok, but it gets complicated when the user expected one failure, but two failures occured in the same line, and none of the two matches with the expected one. Anyway is it possible that the compiler recognizes more than one failure in the same line? -- AlexGreif? - 15 Jul 2002

In theory it should be possible to have two errors on the same line. In the current implementation I think this can seldom happen (unless you write several methods on the same line! :-) But we should not rely on that to stay true forever. I think you could just list all the expected and all the actual locations. In most cases there will be only one anyway (I try to write small independant testcases, instead of longer ones). -- DanielBonniot? - 16 Jul 2002

Warnings that result of wrong FAIL_HERE positions are treated sort of failures. Thus the source code is shown and the compiler results. A new counter has been introduced, that counts the warnings independently of failures. -- AlexGreif? - 29 Jul 2002

Saving output for failed tests When a test fails, it would be useful if the output of the compilation (and the output of the run, with the stack trace of the exception thrown) was saved in a file, in the failure directory.

Saving successful tests Sometimes I want to look at the code generated by succesful tests. It would be good to have a -save comment line option to say to save all tests, not just the failing ones.

Known bugs When a bug is found, I often write a testcase for it first, before the bug is fixed. This means that an error is printed each time the testsuite is run, even if I work on something else. Moreover I don't want to commit the new case, because someone else might think they introduced a bug. So what I propose is to have a flag for test cases:/// PASS (BUG) and /// FAIL (BUG)to tell that the testcase is not working yet. This means that if an error if found, well we knew it and print nothing. But if it works, then print a vidtorious message! :-) and the source of the case. In the summary:

  known bugs: 2
  corrected: 1
Well, that's all for now! Quite a lot of new features :-) None is critical, so do what you can when you find the time. But all these features will make testing easier.

BTW, the testsuite is getting more and more powerful. Maybe it is getting time to put a copyright on it. Alex, I think it should be on your name. What about the GPL? :-)) -- DanielBonniot? - 29 Jul 2002

You mean, changing the headers in the source files? -- AlexGreif?

Yes. The files in n.t.testsuite.* have no copyright header (they have your name as the author). To apply the GPL, see (it could be similar to the compiler's headers, changing the name :-)

-- DanielBonniot? - 30 Jul 2002

Compiler warnings We need to track compiler warning, not just errors. I think by default a warning should be considered as an error. We could add a ///WARN tag to say that a test should (or can?) emit a warning.

Nice TestSuite Documentation The Nice TestSuite is a framework to test the Nice compiler. For this aim, a proprietary file format for the testcases has been created. Here are some testcase examples:

/// COMMENT This testsuite tests the following ....
/// PASS
  int a = 1;

/// PASS
  /* A simple global var. */
  var int x = 0;

/// FAIL
  /// COMMENT this should fail
  int a = "";

/// PASS
  /// package a
  class A
    int x = f();
  int f() = 0;
  /// package b import a
TestCase? keywords are all prefixed with ///. On root level (without any whitespace indentation) currently following keywords are allowed

PASS and FAIL GLOBAL COMMENT The keywords PASS and FAIL indicate a testcase on its own, and whether the expected behaviour while compilation is pass or fail. Testcases cannot reference each other, they are all compiled and tested independently.

Testcases may contain the TOPLEVEL keyword that has the following meaning: everything in front of the TOPLEVEL keyword will be collected in the main() method. Statements behind the TOPLEVEL keyword are global to the package, like global variables, functions or class definitions.

PACKAGE The PACKAGE keyword indicates which package the sources should belong to. More than one PACKAGE keyword can be declared to build a testcase with a certain package structure. Packages can be imported by the PACKAGE some IMPORT other keywords.

GLOBAL After the GLOBAL keyword source code can be written that is accessable by all following testcases. Global sources can be placed at where (before, after or between testcases) in the testsuite file. The scope of global sources is restricted to the same testsuite file and to testcases that are behind the global source. Globals can appear more than one times in the file as the next example demonstrates

  int g1 = 1
/// PASS
  /* access to g1 only */
  int a = 1;

  int g2 = 2

/// PASS
  /* access to g1 and g2 */
  /* A simple global var. */
  var int x = 0;
Because globals are accessible only in the same file, placing global sources at the end of the file makes no sense.

COMMENT Comments can be included, and if desired outputted, at any place in a testsuite. Comments have the following syntax /// COMMENT place your comments here

Comments are one-liners, if you want to write a long comment, write the whole comment in one line as described above or break it into more COMMENT keywords, each in a new line. The following example shows possible ways to comment

/// COMMENT This testsuite describes ....
/// PASS
    /// COMMENT This testcase demonstartes ...
    /// COMMENT ... line two of testcase comment ...
/// COMMENT here we could comment the end of the testsuite :)
Activating comments in the output can be done by specifying the -comment flag in the command line

java -classpath /usr/share/java/nice.jar ~/testsuite/ -comment or java -classpath /usr/share/java/nice.jar -comment ~/testsuite/

While running the testcases temporary files and folders are created in "testsuite-temp-folder". Testcases that fail, in the sense of that they don't behave like expected (pass or fail) are collected in the folder "testsuite-fail-folder" for later investigation purposes. In this folder, each testcase is grouped in its own folder that is numbered from "1" on in ascending order. At the next testsuite run both folders are initially cleared. -- AlexGreif? - 27 Jun 2002

Topic TestSuite . { Edit | Attach | Ref-By | Printable | Diffs | r1.5 | > | r1.4 | > | r1.3 | More }
Revision r1.1 - 12 Feb 2003 - 15:18 GMT - TWikiGuest
Parents: NiceProjects?
Copyright © 1999-2003 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback.