Skip to content

The parameterize directive

Jump to bottom
Richard R. Drake edited this page Apr 14, 2024 · 3 revisions

This section is for reference; see parameterizing a test for more detail on how to use parameters.

To parameterize on a single parameter name:

#VVT: parameterize : name = value1 value2 value3

This will create three test instances, one with name=value1, one with name=value2, and one with name=value3.

A parameter name must be a valid shell or Python variable.

To parameterize on grouped parameter names:

#VVT: parameterize : nameA,nameB = valueA1,valueB1 valueA2,valueB2 valueA3,valueB3

This will create three test instances:

  1. nameA=valueA1 and nameB=valueB1
  2. nameA=valueA2 and nameB=valueB2
  3. nameA=valueA3 and nameB=valueB3

Line continuation is frequently used with parameterized grouping to make it more readable:

#VVT: parameterize : nameA, nameB = valueA1, valueB1
#VVT::                              valueA2, valueB2
#VVT::                              valueA3, valueB3

Multiple parameterize directives

When multiple parameterize directives are given, the Cartesian product of each is taken to form the set of test instances. For example,

#VVT: parameterize : nameA = valueA1 valueA2
#VVT: parameterize : nameB = valueB1 valueB2 valueB3

will produce 2*3=6 test instances:

  1. nameA=valueA1 and nameB=valueB1
  2. nameA=valueA1 and nameB=valueB2
  3. nameA=valueA1 and nameB=valueB3
  4. nameA=valueA2 and nameB=valueB1
  5. nameA=valueA2 and nameB=valueB2
  6. nameA=valueA2 and nameB=valueB3

Allowable attributes

The allowable common attributes are "testname", "platform", and "option". These decide whether the directive is processed or not.

The staged attribute

The "staged" attribute causes the test instances corresponding to the parameter values to be run sequentially and in the same execution directory. See the section on Using staging in parameterized tests for more information, but an example syntax is

#VVT: parameterize (staged) : name = value1 value2 value3

The autotype attribute

By default, the type of each parameter value in the vvtest_util.py file is a string. The autotype attribute tries to determine the type automatically from the values of a parameter. Consider

#VVT: parameterize (autotype) : np = 1 4 16
#VVT: parameterize (autotype) : foo,bar = a,0.1 b,0.01 c,0.001

import vvtest_util as vvt
print ( 'type of np '+str(type(vvt.np)) )
print ( 'type of foo '+str(type(vvt.foo)) )
print ( 'type of bar '+str(type(vvt.bar)) )

In this case, the Python type of the np variable will be int, foo will be str, and bar will be float.

The autotype logic does this:

  1. Try casting each parameter value to an int. If success, use type int
  2. Try casting each parameter value to a float. If success, use type float
  3. If neither of those, then use type str

The str, int, and float attributes

These attributes provide precise control over the types of each parameter. They cannot be mixed with the autotype attribute for any single parameterize directive.

If one type attribute is specified, then all parameters will take on that variable type. In this example, the np parameter will be an int and both dt and dh will be float:

#VVT: parameterize (int) : np = 1 4 16
#VVT: parameterize (float) : dt,dh = 1,2 0.1,0.2

When more than one variable type is specified, then the number of type specifiers must match the number of variable names in a group, such as

#VVT: parameterize (int,float,int) : np,dt,N = 1,0.1,10 2,0.2,20

Using a parameter generator

The parameter names and values can be specified from an external script using the generator attribute, such as

#VVT: parameterize (generator) : myscript --my-args

A generator script must print to "stdout" a JSON formatted list of dictionaries with the parameter names and values. For example

from json import dumps
print ( dumps( [ {'np':1, 'dt':0.1, 'N':10}, {'np':4, 'dt':0.05, 'N':20} ] ) )

If the script is executable, it will execute it. Otherwise it is assumed to be Python and the current interpreter executable is used to run the script. Note that the script name can be the test file itself and command line arguments used to branch to a function for printing the parameters.

Some constraints:

  1. Only the json.dumps() output should be written to "stdout" - it must fit on a single line (no pretty printing!)
  2. The format is reconstructable using Python json.loads()
  3. The format is a list of dictionaries
  4. Each dictionary must be the same size and have the same keys (the parameter names)
  5. The parameter values must be int, float, or str
  6. Currently, the names and values must be the same for multiple calls

The types are taken from the JSON reconstruction (but can only be int, float, and str).

The repeatable or deterministic constraint implies that any "random" values must use a seed so that the same values are generated.

The generator command has ${NAME} and ${PLATFORM} replaced with the test name and the current platform name before the command is executed.

Specifying dependencies with a parameter generator

The parameter generator script prints a line specifying the parameter names and values of each test. An optional second line can be printed by the generator script to specify a test dependency for each test. The line must be a single JSON line of the form

[ {'name':{'param1':val1, 'param2':val2}}, ... ]

It must be a list of the same length as the first line. Each entry is dictionary mapping the dependency test name to the dependency test parameter names and values.

This will define a single test dependency for each test specified in the first line.

Actually, each dependency list entry can be one of

  1. A dictionary mapping test names to a dictionary of the parameter names and values
  2. None (a JSON null), which means do not add a dependency
  3. A string containing a shell-style glob pattern

Example generator script output with a dependency line:

[{"A": 1, "B": "b1"}, {"A": 2, "B": "b2"}, {"A": 3, "B": "b3"}]
[null, {"C": {"A": 2, "B": "b2"}}, "dep.A=*"]

The second entry in this example means the current test with parameters A=2 & B='b2' depends on test "C" with parameters A=2 & B='b2'.

Currently, multiple dependencies for a test can be given as long as they have different test names. A multiple dependency example is

{"C": {"A": 2}, "D": {"A": 2}}

which means the test depends on two tests:

  1. test name "C" with param A=2
  2. test name "D" with param A=2
Clone this wiki locally