-
Notifications
You must be signed in to change notification settings - Fork 2
options
The -o command line option can be used to pass arbitrary option strings into each test. A test can use the options for any number of reasons, such as using -o debug to select and run with a debug executable.
Also, tests can be enabled or disabled based on option values, as described below.
The option strings are used to name the TestResults runtime directory, which allows a separation of multiple vvtest invocations with different options. For example, consider this test file named "opts.vvt":
import vvtest_util as vvt
print ( 'the options are ' + str(vvt.OPTIONS) )
Running this test without -o options results in
s974534$ vvtest -v
...
==================================================
pass 0:01 01/01 16:59:06 TestResults.iDarwin/opts
==================================================
Summary:
completed: 1
1 pass
total: 1
Finish date: Fri Jan 1 16:59:07 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin
then do
s974534$ tail -n 2 TestResults.iDarwin/opts/execute.log
the options are []
Note the empty option list in the test output. Now run again but with an option:
s974534$ vvtest -v -o debug
...
==================================================
pass 0:01 01/01 17:03:50 TestResults.iDarwin.ON=debug/opts
==================================================
Summary:
completed: 1
1 pass
total: 1
Finish date: Fri Jan 1 17:03:51 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin.ON=debug
Note that the execution directory contains the -o value. The test output is:
s974534$ tail -n 2 TestResults.iDarwin.ON\=debug/opts/execute.log
the options are ['debug']
Multiple -o options are allowed. For example,
s968057% vvtest -v -o gpu -o debug
...
Finish date: Fri Jan 1 17:08:27 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin.ON=debug_gpu
For the test results name, the options are sorted then concatenated with underscores.
The options given on the command line are used to process "options=" attributes in test directives. This allows enabling/disabling a test, determining test parameterizations, and more.
For example, consider the test file "opts.vvt":
#VVT: enable (options="not debug")
pass
When this is run without a -o option, the test will be run:
s974534$ vvtest -v
...
==================================================
pass 0:01 01/01 19:41:55 TestResults.iDarwin/opts
==================================================
Summary:
completed: 1
1 pass
total: 1
Finish date: Fri Jan 1 19:41:56 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin
But when -o debug is given, the test is not run:
s974534$ vvtest -v -o debug
Test list:
completed: 0
skipped: 1
1 due to "excluded by option expression"
total: 1
Finish date: Fri Jan 1 19:44:46 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin.ON=debug
Consider an example using option values to control test parameterizations:
#VVT: parameterize (options=debug) : np = 1
#VVT: parameterize (options=not debug) : np = 4
import vvtest_util as vvt
print ( 'np = ' + vvt.np )
Run this without a -o option and look at the output:
s974534$ vvtest -v
...
Finish date: Fri Jan 1 17:20:09 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin
s974534$ tail -n 2 TestResults.iDarwin/opts.np\=4/execute.log
np = 4
Then run it with -o debug
...
Finish date: Fri Jan 1 17:23:52 2021 (elapsed time 1s)
Test directory: TestResults.iDarwin.ON=debug
s974534$ tail -n 2 TestResults.iDarwin.ON\=debug/opts.np\=1/execute.log
np = 1
Note how the "np" parameter took on different values depending on the -o option given.
In general, option expressions in a test file are evaluated against the list of option values given on the command line. Most directives accept option expressions, including these:
directive example | notes |
---|---|
#VVT: enable (options="debug") | run the test if -o debug is given |
#VVT: parameterize (options="debug") : np = 1 | parameterize only if -o debug is given |
#VVT: link (options="intel or debug") : file.txt | link file.txt if intel or debug options are given |
#VVT: copy (options="intel-*") : file.txt | copy file.txt if any -o options start with "intel-" |
#VVT: analyze (options="not intel") : --analyze | don't analyze for intel option |
#VVT: timeout (options="debug") : 3600 | set timeout to 3600 seconds for debug |
#VVT: baseline (options="not debug") : --baseline | don't rebaseline for debug |
#VVT: depends on (options="CUDA") : cuda-* | add a dependency for CUDA option |
#VVT: preload (options="debug") : alabel | apply plugin preload label for debug |
Note that the value can be any expression, including expressions with shell style wildcard pattern matching.