Parameterizing a test

A single test script can be parameterized into many tests using the "parameterize" directive. For example, this test script

#VVT: parameterize : MODEL = 1 2
print ( 'hello world' )

will produce two test executions, one with MODEL = 1 and another with MODEL = 2.

s968057% vvtest
Running these tests:
==================================================
params               NotRun   TestResults.Darwin/params.MODEL=1
params               NotRun   TestResults.Darwin/params.MODEL=2
==================================================
Summary: 0 pass, 0 timeout, 0 diff, 0 fail, 2 notrun, 0 notdone

Platform Darwin, num procs = 4, max procs = 4
Start time: Sun Jun 26 21:50:46 2016
Starting: TestResults.Darwin/params.MODEL=2
Starting: TestResults.Darwin/params.MODEL=1
Finished: params               Exit    pass     1s   TestResults.Darwin/params.MODEL=2
Finished: params               Exit    pass     1s   TestResults.Darwin/params.MODEL=1
Progress: 2/2 = %100.0, time = 1s

==================================================
params               Exit    pass     1s   06/26 21:50:46 TestResults.Darwin/params.MODEL=1
params               Exit    pass     1s   06/26 21:50:46 TestResults.Darwin/params.MODEL=2
==================================================
Summary: 2 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

Finish date: Sun Jun 26 21:50:47 2016 (elapsed time 1s)
Test directory: TestResults.Darwin

So the same test script is executed for each parameter value but in a separate directory. To be useful, however, the script needs to get hold of the parameter values during execution. This is done by importing the test helper script, called "vvtest_util.py", which is written into the execution directory for each test. Here is the same script with the helper:

#VVT: parameterize : MODEL = 1 2
import vvtest_util as vvt
print ( 'running test with MODEL = ' + vvt.MODEL )

The same tests run as before, but now the test prints the parameter value:

s968057% cd TestResults.Darwin
s968057% ls params.MODEL=1
execute.log     params.vvt      vvtest_util.py  vvtest_util.pyc

s968057% cat params.MODEL=1/execute.log
Starting test: params
Directory    : /Users/rrdrake/vvtdoc/ex3/TestResults.Darwin/params.MODEL=1
Command      : /usr/bin/env python params.vvt
Timeout      : 3600

Cleaning execute directory...
Linking and copying working files...
ln -s /Users/rrdrake/vvtdoc/ex3/params.vvt params.vvt

running test with MODEL = 1

s968057% cat params.MODEL=2/execute.log
Starting test: params
Directory    : /Users/rrdrake/vvtdoc/ex3/TestResults.Darwin/params.MODEL=2
Command      : /usr/bin/env python params.vvt
Timeout      : 3600

Cleaning execute directory...
Linking and copying working files...
ln -s /Users/rrdrake/vvtdoc/ex3/params.vvt params.vvt

running test with MODEL = 2

A test script would utilize the parameter name(s) and value(s) to run variations of the test. Note that the helper script is detailed in a separate section.

More than one parameterize directive can be specified. In this case, a cartesian product is performed. That is, all parameter values of the first parameter name are matched with all parameter values of the second parameter name. For example, consider this test script

#VVT: parameterize : MODEL = 1  2
#VVT: parameterize : YIELD = 1.e5  1.e6  1.e7
import vvtest_util as vvt
print ( 'running test with MODEL = ' + vvt.MODEL + ' YIELD = ' + vvt.YIELD )

This script would result in 2*3 = 6 tests:

s968057% vvtest
<snip>
==================================================
params               Exit    pass     1s   06/27 06:19:31 TestResults.Darwin/params.MODEL=1.YIELD=1.e5
params               Exit    pass     1s   06/27 06:19:31 TestResults.Darwin/params.MODEL=1.YIELD=1.e6
params               Exit    pass     1s   06/27 06:19:31 TestResults.Darwin/params.MODEL=1.YIELD=1.e7
params               Exit    pass     1s   06/27 06:19:32 TestResults.Darwin/params.MODEL=2.YIELD=1.e5
params               Exit    pass     1s   06/27 06:19:32 TestResults.Darwin/params.MODEL=2.YIELD=1.e6
params               Exit    pass     1s   06/27 06:19:31 TestResults.Darwin/params.MODEL=2.YIELD=1.e7
==================================================
Summary: 6 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

If a third "parameterize" was specified, then all combinations of MODEL and YIELD would combine with each value of the third parameter. And so on.

Instead of a cartesian product, you can manually match up parameter names and their values in list form. For example,

#VVT: parameterize : MODEL,YIELD = 1,1.e5 2,1.e6 3,1.e7
import vvtest_util as vvt
print ( 'running test with MODEL = ' + vvt.MODEL + ' YIELD = ' + vvt.YIELD )

This would result in three tests:

s968057% vvtest
<snip>
==================================================
params               Exit    pass     1s   06/27 06:29:36 TestResults.Darwin/params.MODEL=1.YIELD=1.e5
params               Exit    pass     1s   06/27 06:29:36 TestResults.Darwin/params.MODEL=2.YIELD=1.e6
params               Exit    pass     1s   06/27 06:29:36 TestResults.Darwin/params.MODEL=3.YIELD=1.e7
==================================================
Summary: 3 pass, 0 timeout, 0 diff, 0 fail, 0 notrun, 0 notdone

Optional attributes can be given to the "parameterize" directive. The allowable attributes are "testname", "platforms", and "options".

If an attribute expression is True, then the "parameterize" directive is applied. If False, then it is ignored.

Setting the parameter types

By default, the parameter values in the vvtest_util.py file are strings. They can be made to be ints or floats by adding the "autotype" attribute. For example, if this is in the test header

#VVT: parameterize (autotype) : np = 1 32 256
#VVT: parameterize (autotype) : dx = 0.1  0.01  0.001

then the np parameter would be of type int and the dx parameter would be of type float. Only int and float are currently supported.

Note that if autotype is added, but all values cannot be cast to an int or a float, then the parameter type will fall back to a string.