testrunner(1)
============
Nils Olav Selåsdal


NAME
----
testrunner - test harness driver for software development testing

SYNOPSIS
--------
testrunner [options] testsuite1 testsuite2 ...

DESCRIPTION
-----------
The testrunner(1) command reads test definitions from the given testsuite
files, executes the tests and reports the test results.

OPTIONS
-------
*-x,--exit_success*
    Make the testrunner(1) command always exit with success (exit code 0).
    Otherwise testrunner(1) exits with success if all the tests passes or
    exits with an error if any of the tests fails.

*-c.--clean*
    Clean any output artifacts left by the tests. The tests are not executed.
    This is intended to be used in a build system, e.g. as part of a 'clean'
    target.

*-C,--show-command*
    Show the command being run for each test.

*-e,--errexit*
    Terminate on the first test that fails

*-v,--verbose*
    Produce more verbose output. This e.g. shows the reason and output
    difference for tests that fails.

*-k KEYWORD,--keyword=KEYWORD*
    Run only tests names matching the given keyword. KEYWORD is a regular
    expression. This flag can be given multiple times.

*-s SUITE,--suite=SUITE*
    Run only tests in the given suite . SUITE is a regular
    expression. This flag can be given multiple times.

*-l,--list*
    List all the test cases present in the given testsuite definition files.

*-D DEFINE,--define=DEFINE*
    Define a variable available to the  test files. The variable is defined 
    with the form name=value. This flag can be given multiple times.

*-g, --generate*
    Run the defined test case commands and generate the output files. Use 
    with care, this overwritesexisting output files

*-t, --timeout*
    Override default individual test timeout in seconds.
    Does not affect tests that explicitly defined a timeout.



OVERVIEW
--------
The testrunner(1) is a test harness that executes commands/executables,
compares the output of the command with the desired output. Test cases fail
or passe depending on whether the actual output of the command matches 
the desired output, and a report of all the test cases and their pass/fail
status is produced.

The intended use of testrunner(1) is that developers write test programs
that exercise the code being tested as normal executables that produces 
text output.
Whether the test cases being executed are unit tests, or larger integration 
tests is up to the developer writing the tests.

Test results are printed out as the tests are being run, as well as saved
to a testsuite.log file.

While testrunner(1) was written to test C and C++ code, code written in 
any language can be tested as long as the tests can be executed as normal 
executables/commands and they write output to stdout or stderr.

The following criteria are used to fail or pass a test:

* Whether stdout of the executable matches the desired stdout output
* Whether stderr of the executable matches the desired stderr output
* Whether the exit code of the executable is any of the 
  desired exit codes, 0 by default.
* Whether the test command completes within the desired time period, 
  30 seconds by default.
 

DEFINING TEST CASES
-------------------
Test cases are loaded from separate testsuite files given as arguments to
testrunner(1). Several test cases can be defined in each testsuite file.

The output of the command from each test case will be compared with two 
files matching the test name with the added .stdout and .stderr suffix.
The test case output and the .stdout/.stderr files are assumed to be 
plain text. In the normal case the test case would only write to its 
stdout, leaving no output on its stderr, such that the corresponding
.stderr file doesn't need to exist.

The testsuite files are run as Python code, so the full python programming
environment is available in the testsuite files. However little to no  
Python programming knowledge is needed to define test cases.

In the testsuite files a test case is defined with the syntax of:

    DefTest('command', 'test_name')

**command** is the executable command to run, optionally with arguments. The
command is being run within a shell as /bin/sh -c 'command' , so full shell
syntax is available. However do note that only the exit status of the last 
command, if e.g. a pipeline is used as a command, is used for reporting back
the exit status to testrunner(1)

**test_name** is the name of the test, used in test reports and for the file names
of the desired output. As the test_name also names file names, it's desirable
if it does not contain spaces, directory separator or other characters that
makes file handling difficult. The test_name must be unique within all the
test suites being run.

e.g

    DefTest('echo Hello', 'hello_test')

This defines one test with the name hello_test that will run the 
command echo hello.

In addition to the test definition here, a file named
hello_test.stdout and hello_test.stderr should be created that contains the 
desired stdout and stderr output of running echo hello.

In our hello_test case, the hello_test.stdout would contain the text

    Hello

As that is what we expect the test to output. hello_test.stderr can be omitted
as we do not expect it to produce any output on stderr.

The testrunner can also take arguments that define Python variables that will
be available as global variables in the testsuite files. With a -Dfoo=bar
command line argument to the testrunner, the variable **foo** will be available
with the value **bar**.

GROUPING TESTS
--------------
The tests are also grouped in suites, and testrunner(1) can be instructed to
only run certain suites with the *-s* flag. If no test suites are defined, 
all test cases will belong to the test suite named 'default'.

A test suite can be defined with the syntax:
    
    DefSuite('suite_name')

**suite_name** is the name of the new suite. All the following test cases
defined with the 'DefTest()' command will belong to this new suite, until
another 'DefSuite()' command is issued.

MORE TEST CASE OPTIONS
----------------------
The DefTest() statement is just a Python function call, and it can take a few
optional parameters in addition to the mandatory parameters described in 
the previous chapter.

    DefTest('command', 'test_name', success_codes=[0], timeout=30)

**success_codes** is a list of exit codes of the command that will be treated
as a success. By default if the command exits with anything other than 
an exit code of 0, the test will be marked as failed. If e.g. an exit 
code of either 0 or 1 should indicate success, pass in 'success_codes=[0,1]'

If an empty list, 'success_codes=[]', is given, the exit code will be ignored
all together when determining if the test case fails or succeeds.

**timeout** is the max time in seconds the test case can run. If this timeout 
is reached, the command will be killed and the test case is marked as failed.
Note that the default timeout for a test is 30 seconds.

GENERATING INITIAL OUTPUT FILES
-------------------------------
In stead of performing the tests, testrunner(1) can run the commands defined 
in the test cases and create the .stdout/.stderr files. This can be used to
create the initial comparison files instead of generating them by hand and
can be helpful to re-generate the files when the test programs change.

The file generation mode is used when the -g or --generate flag is specified.
The flags for filtering tests or suites can be given to run only specific
test commands. 

Use this with care, as any existing .stdout/.stderr files will be overwritten.

If any of the generated .stdout/.stderr files end up being empty, the files
are removed.

FILE LOCATIONS
--------------
The files used by testrunner(1) are

* The testsuite files containing test case. 
* The command specified in a test case that is executed.
* The .stdout and .stderr files for each test case.

The test case commands are being run with working directory set to the location
of the testsuite file it is specified in, and the matching .stdout/.stderr files
for each test case are read from the same directory as the testsuite files.

EXAMPLES
--------
Run test definitions in the testsuite_foo file, that starts with the name xyz,
stop on the first failing test:
    
    testrunner.py -k ^xyz -e testsuite_foo

List test definitions in the testsuite_foo file that contains foobar in the
test name. This can be useful to check which tests matches the *-k* or *-s*
arguments that's given:

    testrunner.py -k foobar -l testsuite_foo

See the tests/ directory for some simple examples. These tests
can be run with:

    testrunner.py tests/testsuite_failures tests/testsuite_success

In the c_example/ directory is a small C code project that uses testrunner(1)
for testing. These tests can be compiled and run with:

    cd c_example && make check

LICENSE
-------
This software is licensed under the MIT license, a copy of the license is found
in the LICENSE file.

SEE ALSO
--------
Source code: http://github.com/noselasd/testrunner/