-
Notifications
You must be signed in to change notification settings - Fork 9
CTF JSON Test Script Description
Integrated tests are described using JSON test scripts. If you are unfamiliar with JSON, you can find a great introduction here: https://www.w3schools.com/js/js_json_syntax.asp.
The JSON CTF input file, or Test Script, is comprised of a number of properties that describe the tests that will be executed. Each Test Script can have multiple tests, with each each testing multiple requirements. Example test scripts can be found under example_scripts and functional_tests in the repo.
The Test Script has the following properties:
- test_script_number: Unique identifier for the test script. Format Should be a descriptive test ID that would allow the review to understand what is being tested.
- test_script_name: A short descriptive name for the tests in the file.
- requirements: A dictionary where the key is the SRS and the value is a string that states if the test script fully or partially verifies the SRS.
- description: Detailed description of the script. Start with the RTM test for the requirements you are testing. Then include the information that would capture what environments the tests can execute within. The procedures for executing the script, if special steps are needed. The init point to be used for the tests.
- test_setup: Any steps that should be taken prior to running this test script, such as another script to be run first, or a CTF config file to be used.
- owner: The name of the test owner.
- ctf_options: CTF properties that may be overridden for this test script.
- import: Specify functions (described below) that can be imported from other JSON files.
- functions: User-defined functions that can be used by the test script, described below.
- tests: an array of objects, where each object describes a single Test. The Test objects are described below.
Example:
{
"test_script_number": "Basic-CTF-Plugin-Test",
"test_script_name": "Test_CTF_Basic_Example.json",
"requirements": {
"MyRequirement": "N/A"
},
"description": "Basic CTF Example Script Showing Simple Commands/Telemetry Verification",
"owner": "CTF",
"test_setup": "",
"ctf_options": {
"verify_timeout": 4
},
"import": {},
"tests": [
{
"test_number": "Basic-CTF-Plugin-Test-1",
"description": "Start CFS, Send TO NOOP command",
"instructions": [
{
"instruction": "StartCfs",
"data": {
"target": ""
},
"wait": 1
},
{
"instruction": "EnableCfsOutput",
"data": {
"target": ""
},
"wait": 1,
"description": "Need this to enable the telemetry thread. Enable Output counts as 1 TO cmd"
},
{
"instruction": "CheckEvent",
"data": {
"args": [
{
"app_name": "TO",
"event_id": "0x3",
"event_str": "TO - ENABLE_OUTPUT cmd succesful for routeMask:0x00000001",
"event_str_args": ""
}
],
"target": ""
},
"wait": 1,
"description": "Need this to enable the telemetry thread. Enable Output counts as 1 TO cmd"
},
{
"instruction": "SendCfsCommand",
"data": {
"target": "",
"mid": "TO_CMD_MID",
"cc": "TO_NOOP_CC",
"args": {}
},
"wait": 1
},
{
"instruction": "SendCfsCommand",
"data": {
"target": "",
"mid": "TO_CMD_MID",
"cc": "TO_RESET_CC",
"args": {}
},
"wait": 1
},
{
"instruction": "CheckTlmValue",
"data": {
"target": "",
"mid": "TO_HK_TLM_MID",
"args": [
{
"compare": "==",
"variable": "usCmdCnt",
"value": 0
}
]
},
"wait": 1
}
]
}
]
}The Function objects are collections of CTF Commands that can be executed by each test case. These functions can be defined within the same Test Script or in a separate file and be imported (see functional_tests/plugin_tests/cfe_6_7_tests/ for examples) The object has the following properties:
- description: Describe purpose of the function including the steps the function with take and the results the function will provide. Intended use details should also be included.
- varlist: List of arguments required by the function
Example:
"functions":{
"UserDefinedFunction":{
"description":"<Explain purpose, results, steps of this user-defined functions>",
"varlist":["_arg1", "_arg2"],
"instructions":[
//CTF instruction objects go here
]
}
}The Test objects each describe a single test, which can verify any number of requirements. Each Test contains the requirements verified by the test, a description of the test, and a number of Command objects that define the actions to take during the test. The object has the following properties:
- test_number: Unique identifier of this test in the script. Use the test script name with a unique number.
- description: Describes what actions this test is performing. Include Preconditions, the expected actions/commands, and the expected results that are going to be observed.
- instructions: An array of instruction objects that define the actions taken by the test (described below)
Example:
{
"test_number":"ExampleTest-1",
"description":"<Explain preconditions, actions, results, steps of test case>",
"instructions":[
//instruction objects go here
]
}The CTF instruction object defines what action CTF is supposed to take. Each instruction object has the following base structure:
- instruction: (string) The CTF instruction to execute. Must be one of the instructions provided by a loaded plugin.
- data: The arguments for the CTF instruction. Each instruction is going to have a different data object, described below.
- wait: The amount of time to wait from the completion of the last command before executing the current instruction.
- verify_timeout: (Optional) The amount of time for verification instructions to wait for verification to succeed before timing out.
Instructions are implemented in their respective CTF plugin. Currently CTF provides the following plugins, and users may implement their own. See the respective plugin ReadMe files for details on their instructions and usage.
The CFS Plugin provides CFS command/telemetry support for CTF.
The CCSDS Plugin provides interfaces and utilities for CCSDS messages. It is responsible for parsing message structures and constructing messages with the correct header and payload formats.
The SSH Plugin provides remote and local shell command execution capability for CTF.
The Variable Plugin provides CTF users with the capability to Set / Check / Update user-defined variables from json test scripts.
The Control-Flow Plugin provides the functionality of CTF control flow statement at instruction level.
The UserIO Plugin handles user input/output operations, including pausing test for safety critical operations.
The Validation Plugin provides functionality to interpret a cFE binary event log to a human-readable text file, and search for a text string in a file. It also allows the user to delete files/folders and copy files/folders on the local host machine.
The Example Plugin shows a simple CTF plugin that can perform a single test instruction and a single verification instruction. It is provided as a template for users to create their own plugins.