The regression tester is a program that allows you to do two things:
Since the running of the tests and the analysis of the output is separated you can easily do things like compare the results on multiple runs, extract various reports from a single output (like what tests have succeeded, which have failed, plus such information split by test groups). You can also extract some data from each test (such as the running time) and make simple reports.
Test cases can have comments associated with them (such as reminders of why it fails) and can be associated with zero or more test groups.
The regression tester is implemented in Perl as "testsafec.pl", which in turn contains simple wrappers for functions provided by the more generic RegTest.pm.
The regression tester uses relative paths so it must be run in the cil/test directory.
The basic command for running the tests is "testsafec –run". This runs all of the test cases, saving the log in the file "safec.log". Before creating this file, it renames previous versions of this file as "safec.log.<n>" where n is an integer starting from 1 to a maximum number that is configurable.
The following command line options are useful for running the tests (see "testsafec –help" for a complete list:
--one <testname> : runs only the named test
--gory : shows lots of details about the execution of the
test, such as the commands executed
--dryrun : only pretends to run the test. Useful to see what
would be run
--log : select the base name of the log file (default
"safec.log")
--logversions <n> : keep logs up to version <n>. Default is 5.
--noremake : runs the commands without trying to remake the safec
compiler before each test. Useful if you want to
work on the compiler while the tests are running
--safecdebug : uses the DEBUG version of the safec compiler and
uses the C compiler in debug mode. By default it
used the RELEASE version and the optimizing compiler.
--group <groupname> : adds all the tests in the named group to the list of
tests to be run or to participate in the analysis of
the log. (Right now we have groups: apache,
bad, cil, box, infer.) If no such option is
specified then all tests are selected. Multiple such
options can be given and are cumulative.
--nogroup <groupname> : excludes the tests in the named group from running
or from the analysis. Multiple such options can be
given and are cumulative. These options are
processed after all --group options have
been processed.
--listtests : list the tests that are enabled along with their
group membership. This is useful to find out what
tests and groups exist.
--stoponerror : stop at the first error
--showoutput : show the output on the console. Normally output is
saved in a file.
The basic command for analyzing log files is "testsafec". This will prompt the user to select one of the several log files that exist in the current directory and then (by default) it will print a list of the failed test cases, with a short (user provided) comment and the last error message detected in the output for that test case.
The following commands are useful during analysis:
--log : select the log file (see above)
--group, --nogroup : select groups (see above)
--listtests : list tests and groups (see above)
--param=<pnames> : show a report about the successes, with the columns
being the named parameters (separated by ,). Run
"testsafec --help" to see what parameters are
available. Use --param=ALL to make a report with
all available parameters.
--sort=<pnames> : sorts the report by the given parameters.
Furthermore, the reports that are generated can be compared. To compare the results of two runs (whose logs are say “safec.log.1” and “safec.log.2”, run
test/compare safec.log.2 safec.log.1 --group=slow
This will compare the results of safec.log.2 using as reference the results in log safec.log.1.
For this you have to edit testsafec.pl. You will see a large section in the middle of the file containing lines like:
\$TEST->add3Tests("test/array1");
(check out the definition of add3Tests at the bottom of the file). This adds three tests named "test/array1-cil", "test/array1-box" and test/array1-inferbox", each one containing one command that invokes "make test/array1 ...", where ... are appropriate parameters.
A second optional string parameter to add2Tests is something to be added to the command line.
A third optional array parameters is a list of patterns to be used in scanning the output of the test cases. This is an advanced feature and you are on your own.
To add just one test do (as in the body of add3Tests):
$TEST->newTest(Name => "mytestname",
Dir => "..",
Cmd => "make something",
Enabled => 1,
Comm => "Print this along with the test name",
Group => ["cil", "othergroup"],
Patterns => \%mypatterns);
Sometimes you might want to add just a comment or to add one group to a certain test. Use the following simple functions:
$TEST->addGroups("mytestname", "group1", "group2");
$TEST->addComment("mytestname", "Another line of comment");
There are some wrappers defined at the end of testsafec.pl:
$TEST->add3BadComment("test/scope3", "missing prototype");
(this one adds a comment and the group "bad" to all three test cases)
$TEST->addBadComment("li-box", "bug in box.ml");
(the same but just for one test)
$TEST->enable("li-box", 0); (disable the li-box test case)
For more advanced customization, read the Perl code. It is fairly easy to understand, especially the testsafec.pl.
A simpler (but somewhat less powerful) interface to add tests is provided by the smAddTest and smFailTest functions.
Suppose you want to add a test which runs “make sometarget someoptions”. Just say
smAddTest("sometarget someoptions");
somewhere after smAddTest is defined.
If at some point this test stops working for somereason (just a human-readable one-line description of what's wrong), and you want the regression tester to expect failure, change it to read
smFailTest("somereason", "sometarget someoptions");
There are also facilities for associating a diagnosis with a failure (for example, if failure of a particular test is often due to a known, fixable cause), and for running a shell script upon completion of a test. See the scott/tprintf and scott/ptrkinds tests for examples of each.
Finally, there is a script called regrtest in the base cil directory which runs testsafec.pl with a set of options which:
To use this, just say “./regrtest” in the cil directory. It accepts the “-help” option.
Every time you commit a change to the CIL or CCURED repositories you trigger two regression tests. The first is called the QUICK test and it takes about 5 minutes to complete and you should be receiving email with the results. If you don't then it must be that the automated regression tester is not running or it has encountered a problem that it cannot fix.
The automated regression tester is implemented as a script mk-reports.pl and lives in the home directory of user regtest on manju. To see whether the tester is running run ps -Af and look for a line perl mk-reports.pl -daemon. If you do not see any let me know.
Here are the operations that are performed by the tester:
The automated regression tester should be running continuously and monitoring the commits every 60 seconds. However, between midnight and 6am it alternates between the QUICK run described above and a SLOW run that processes those tests that the QUICK run does not do. A SLOW run might take anywhere between 15 and 30 minutes. The SLOW report is generated using the same checked-out repository as the QUICK run. Results of the regression test, reports and messages for the SLOW run are saved in similar files as for the QUICK run except that the word “quick” is replaced by “slow” in the name of those files.
If the QUICK report is zero-length then the SLOW run is not performed. This is at the moment the only way to prevent the SLOW run from happening.
If both the QUICK and SLOW reports exist then the directory containing the distribution can be deleted.
The reference report is in /home/regtest/cil.nightly/reference.report.quick.txt. You can edit it manually to change the reference for the messages but please do so only to improve it.
You can start regtest manually (though this shouldn't be necessary): “sudo become-regtest” (only certain users can do this), then “perl mk-reports.pl”.
For each of your commit, a copy of the repository is checked out. This takes about 150Mb on manju and 30 minutes worth of regression testing. If you want to disable the SLOW run (because, for example, you made a silly mistake and are committing a fix right away) you have to manually delete the contents (not the file) of the corresponding report.quick.