diff --git a/docs/Everything you wanted to know about Mbed TLS unit tests.pdf b/docs/Everything you wanted to know about Mbed TLS unit tests.pdf new file mode 100644 index 0000000..af118a7 Binary files /dev/null and b/docs/Everything you wanted to know about Mbed TLS unit tests.pdf differ diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..3cb6e42 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,12 @@ +# Presentations + +## For contributors + +### Coding + +[Everything you wanted to know about Mbed TLS unit tests]() + +### Reviews + +[How to be an effective Mbed TLS reviewer](<../reviews/How to be an effective Mbed TLS reviewer.pdf>) + diff --git a/index.md b/index.md index 1ab9468..8a36ebf 100644 --- a/index.md +++ b/index.md @@ -40,5 +40,6 @@ project/index.md reviews/index.md security-advisories/index.md CONTRIBUTING.md +docs/index.md kb/index.md ``` diff --git a/kb/development/debugging_tips.md b/kb/development/debugging_tips.md new file mode 100644 index 0000000..282a6b3 --- /dev/null +++ b/kb/development/debugging_tips.md @@ -0,0 +1,168 @@ +# Tips for debugging Mbed TLS + +This is a collection of tips for debugging TF-PSA-Crypto or Mbed TLS. +It may also be useful for debugging applications using these projects, but that is not this document's main purpose. + +This document assumes some familiarity with the project, e.g. that you already know how to build and test it. + +This document is written primarily with Linux in mind. Similar platforms such as macOS will require few adaptations. Windows (except WSL) is out of scope. + +## Reproducing CI builds with debugging + +### Getting the build products from `all.sh` + +Normally, `all.sh` cleans up after itself. However, it will leave build products around if a compilation or runtime step fails. If you want to see build products from a passing component, add the command `false` after the build steps. + +If you have a wrapper around `all.sh`, note that passing `--keep-going` (`-k`) makes it clean up on errors as well. + +Cancelling `all.sh` with `Ctrl+C` (SIGINT) makes it clean up. But using `Ctrl+\\` (SIGQUIT) bypassing the cleanup. Also, you can use `Ctrl+Z` to inspect an intermediate step. + +### Editing `all.sh` for debugging + +To reproduce an `all.sh` component locally, but with debugging enabled: + +* For most builds using `make` (without CMake), in particular including all driver builds: add `ASAN_CFLAGS='-Og -g3'` or `ASAN_CFLAGS='-O0 -g3'` before the build step. +* For builds using CMake: add or change the build type to `Debug` or `ASanDbg`, e.g. `cmake -DCMAKE_BUILD_TYPE=Debug`. + +After changing the source, you'll need to re-run `all.sh`, including its initial cleanup state which is not trivial to bypass. To speed this up, enable [ccache](https://ccache.dev/). In most `all.sh` components, you can enable ccache by setting +``` +CC="ccache ${CC:cc}" ASAN_CC="ccache clang" +``` + +## Sanitizers + +### Sanitizers used in test scripts + +#### ASan: AddressSanitizer + +* Documentation: https://github.com/google/sanitizers/wiki/addresssanitizer +* Detects: buffer overflows, use after free, memory leaks +* Compilers: GCC, Clang +* Compiler flags: `-fsanitize=address -fno-sanitize-recover=all` (in both `CFLAGS` and `LDFLAGS`) +* CMake build types: `ASan`, `ASanDbg` +* Used in: most builds in `all.sh` + +#### MSan: MemorySanitizer + +* Documentation: https://github.com/google/sanitizers/wiki/memorysanitizer +* Detects: uninitialized memory +* Compilers: GCC, Clang +* Compiler flags: `-fsanitize=memory` (in both `CFLAGS` and `LDFLAGS`) +* CMake build types: `MemSan`, `MemSanDbg` +* Used in: `component_test_memsan*` + +#### TSan: ThreadSanitizer + +* Documentation: https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual +* Detects: race conditions +* Compilers: GCC, Clang +* Compiler flags: `-fsanitize=thread` (in both `CFLAGS` and `LDFLAGS`) +* CMake build types: `TSan`, `TSanDbg` +* Used in: `component_test_tsan*` + +#### UBSan: UndefinedBehaviorSanitizer + +* Documentation: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html +* Detects: null pointer misuse, bitwise shift amount out of range, signed integer overflow, … +* Compilers: GCC, Clang +* Compiler flags: `-fsanitize=undefined` (in both `CFLAGS` and `LDFLAGS`) +* CMake build types: `ASan`, `ASanDbg` +* Used in: most builds in `all.sh` + +### Valgrind + +Valgrind mostly duplicates Asan+Msan, but very occasionally finds something that they don't. + +* Documentation: https://valgrind.org/docs/manual/manual.html +* Detects: buffer overflows, use after free, memory leaks, uninitialized memory + * We don't currently use it for race conditions. +* Compilers: any +* Compiler flags: N/A — runtime instrumentation only +* CMake target: `make memcheck` +* Run with: + ``` + valgrind -q --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=50 --log-file=myprogram.MemoryChecker.log myprogram + grep . myprogram.MemoryChecker.log myprogram + ``` +* Used in: `component_release_test_valgrind*` + +### Getting symbolic backtraces from symbolizers + +By default, ASan/MSan/TSan/UBSan display traces without symbolic information. For traces with symbol names, you need to set environment variables: + +``` +export ASAN_OPTIONS=symbolize=1 +export MSAN_OPTIONS=symbolize=1 +export TSAN_OPTIONS=symbolize=1 +export UBSAN_OPTIONS=print_stacktrace=1 +``` + +With Clang, depending on how it's installed, you may need to specify the path to the correct version of `llvm-symbolizer` in `ASAN_SYMBOLIZER_PATH`, `MSAN_SYMBOLIZER_PATH` and `TSAN_SYMBOLIZER_PATH`. For example: + +``` +if ASAN_SYMBOLIZER_PATH=$(readlink -f "$(command -v clang)") && + ASAN_SYMBOLIZER_PATH="${ASAN_SYMBOLIZER_PATH%/*}/llvm-symbolizer" +then + export ASAN_SYMBOLIZER_PATH + export MSAN_SYMBOLIZER_PATH="$ASAN_SYMBOLIZER_PATH" + export TSAN_SYMBOLIZER_PATH="$ASAN_SYMBOLIZER_PATH" +fi +``` + +See [SanitizerCommonFlags](https://github.com/google/sanitizers/wiki/SanitizerCommonFlags) for more flags you can use in `$xxSAN_OPTIONS`. + +### Sanitizers for constant-time testing + +See “[Mbed TLS test guidelines — Constant-flow testing](../development/test_suites.md#constant-flow-testing)”. + +## Reverse debugging + +### What is reverse debugging? + +Also known as back-in-time debugging or time travel debugging. + +Reverse debugging allows you to go backward in time when stepping through a program. For example, a reverse single step after returning from a function goes back to the function's `return` statement. + +### Tools for reverse debugging + +* Gdb supports reverse debugging, but not out of the box, it requires some complex setup. +* LLDB does not support reverse debugging as of 2025. +* Visual Studio (under Windows) supports reverse debugging since 2017. + +Reverse debugging works by taking snapshots of a program and recording its inputs and outputs. It may or may not work when the program interacts with its environment in complex ways, since the environment does not roll back when the program does. + +### Replay debuggers + +A replay debugger records one execution of the program. It then replays this same execution, simulating all inputs and outputs. + +#### Replay debugging on Linux with rr + +Install the Mozilla Record and Replay framework (rr) from https://rr-project.org/ or e.g. `apt install rr`.​ + +If needed, give yourself debugging permission: + +``` +# The Ubuntu default is 4 which is too paranoid. +sudo sysctl kernel.perf_event_paranoid=1.​ +# Make this persistent across reboots. +echo 'kernel.perf_event_paranoid = 1' >>/etc/sysctl.d/zz-local.conf​ +``` + +To debug a program​, build it with debugging symbols as usual (`-O0 –g3` or `–Og -g3`).​ Then run it once to save a full trace of the execution: + +``` +rr record tests/test_suite_ssl +``` + +Then `rr replay` gives you a gdb interface where reverse execution actually works.​ You can use [`reverse-xxx` commands​](https://sourceware.org/gdb/current/onlinedocs/gdb.html/Reverse-Execution.html) such as: + +* `rs` (`reverse-step`) steps into functions​. +* `rn` (`reverse-next`) steps over function calls​. +* `reverse-finish` goes back to where the current function was called​. +* `set exec-direction reverse` changes `step`, `next`, etc. to go backwards. Switch this off with `set exec-direction forward`. + +If you use a frontend, configure it to run `rr replay` instead of `gdb myprogram`.​ If the frontend uses gdb's machine interface, use `rr replay -i=mi …` instead of `gdb -i=mi …`. + +#### Replay debugging on macOS with warpspeed + +Try [warpspeed](https://github.com/kallsyms/warpspeed). diff --git a/kb/development/test_suites.md b/kb/development/test_suites.md index 3611cf9..42d5ab7 100644 --- a/kb/development/test_suites.md +++ b/kb/development/test_suites.md @@ -14,7 +14,10 @@ Each paragraph describes one test case and must consist of: 1. One line, which is the test case name. 1. An optional line starting with the 11-character prefix `depends_on:`. This line consists of a list of compile-time options separated by the character ':', with no whitespace. The test case is executed only if all of these configuration options are enabled in `mbedtls_config.h`. Note that this filtering is done at run time. -1. A line containing the test case function to execute and its parameters. This last line contains a test function name and a list of parameters separated by the character ':'. Each parameter can be any C expression of the correct type (only `int` or `char *` are allowed as parameters). +1. A line containing the test case function to execute and its parameters. This last line contains a test function name and a list of parameters separated by the character ':'. The parameter must be valid for the function type: + * `int` or other integral type: an integer-valued C expression, evaluated in a separate function in the same C source file as the test code. So this expression has access to macros, types and even global variables defined in the header of the `.function` file, but not to local variables of the test function. + * `[const] char *`: a string between double quotes. A backslash escapes the next character (needed for `\":`). + * `[const] data_t *`: a byte string written in hexadecimal, between double quotes. For example: @@ -31,7 +34,7 @@ Code file that contains the actual test functions. The file contains a series of * `BEGIN_HEADER` / `END_HEADER` - Code that will be added to the header of the generated `.c` file. It could contain include directives, global variables, type definitions and static functions. * `BEGIN_DEPENDENCIES` / `END_DEPENDENCIES` - A list of configuration options that this test suite depends on. The test suite will only be generated if all of these options are enabled in `mbedtls_config.h`. * `BEGIN_SUITE_HELPERS` / `END_SUITE_HELPERS` - Similar to `XXXX_HEADER` sequence, except that this code will be added after the header sequence, in the generated `.c` file. -* `BEGIN_CASE` / `END_CASE` - The test case functions in the test suite. Between each of these pairs, you should write *exactly* one function that is used to create the dispatch code. Between the `BEGIN_CASE` directive and the function definition, you shouldn't add anything, not even a comment. +* `BEGIN_CASE` / `END_CASE` - The test case functions in the test suite. Between each of these pairs, you should write *exactly* one function that is used to create the dispatch code. The function must return `void` and may only take supported parameter types. Comments are allowed before and inside the function's prototype. An optional addition `depends_on:` has same usage as in the `.data` files. The section with this annotation will only be generated if all of the specified options are enabled in `mbedtls_config.h`. It can be added to the following delimiters: @@ -52,18 +55,49 @@ An optional addition `depends_on:` has same usage as in the `.data` files. The s /* BEGIN_CASE depends_on:MBEDTLS_AES_C */ ``` -## `helpers.function` file +## Building and running tests -This file, as its name indicates, contains useful common helper functions that can be used in the test functions. There are several functions, which are described in [`helpers.function`](https://github.com/Mbed-TLS/mbedtls/blob/development/tests/suites/helpers.function) itself. Following are a few common functions: +### Building your test suites -* `hexify()` - A function converting binary data into a null-terminated string. You can be use it to convert a binary output to a string buffer, to be compared with expected output given as a string parameter. -* `unhexify()` - A function converting a null-terminated string buffer into a binary buffer, returning the length of the data in the buffer. You can use it to convert the input string parameters to binary output for the function you are calling. -* `TEST_ASSERT(condition)` - A macro that prints failure output and finishes the test function (`goto exit`) if the `condition` is false. -* Different `rnd` functions that output different data, that you should use according to your test case. `rnd_std_rand()`, `rnd_zero_rand()`, `rnd_buffer_rand()`, `rnd_pseudo_rand()`. For more information on what each random function does, refer to their description in the `helpers.function` file. +The test suite `.c` files are auto generated with the `generate_test_code.py` script. You could either use this script directly, or run `make` in the `tests/` folder, as the [`Makefile`](https://github.com/Mbed-TLS/mbedtls/blob/development/tests/Makefile) utilizes this script. Once the `.c` files are generated, you could build the test suite executables running `make` again. Running `make` from the Mbed TLS root folder will also generate the test suite source code, and build the test suite executables. -## Building your test suites +### Running unit tests -The test suite `.c` files are auto generated with the `generate_code.pl` script. You could either use this script directly, or run `make` in the `tests/` folder, as the [`Makefile`](https://github.com/Mbed-TLS/mbedtls/blob/development/tests/Makefile) utilizes this script. Once the `.c` files are generated, you could build the test suite executables running `make` again. Running `make` from the Mbed TLS root folder will also generate the test suite source code, and build the test suite executables. +You can run a single test suite individually. To run all the test suites: + +* Run `make test` from the top-level directory of the build tree. +* When building with Make, this runs `tests/scripts/run-test-suites.pl`, which you can call directly. +* When building with CMake, this uses `ctest`. + +To skip a few test suites: + +* With Make, set the environment variable `SKIP_TEST_SUITES` to a comma-separated list of short names, e.g. + ``` + SKIP_TEST_SUITES=constant_time_hmac,lmots,lms,gcm,psa_crypto.pbkdf2,ssl_decrypt make test + ``` +* With CMake, set the CMake parameter `SKIP_TEST_SUITES` to a comma or semicolon-separated list of short names, e.g. + ``` + cmake -B build-debug -DCMAKE_BUILD_TYPE=Debug -DSKIP_TEST_SUITES=constant_time_hmac,lmots,lms,gcm,psa_crypto.pbkdf2,ssl_decrypt + ``` + You have to re-run `cmake` if you want to change the set of skipped suites. + +### Running only one test case + +If you just want to see information about failing test cases: + +``` +tests/test_suite_foo |& grep -Ev '(PASS|SKIP|----)' +``` + +But sometimes you want to set a breakpoint in a debugger and not have it trigger on “boring” test cases. At the time of writing, there is no way to skip individual test cases. Various kludges are possible, such as: + +* Edit the `.data` file to remove or comment out the boring test cases, and rebuild the test suite. Remember not to commit this change! +* Copy the interesting test case to the top of the `.data` file. Remember to update the test case in its “true” location if you modify it, and not to commit the copy. +* Copy the `.datax` file, remove boring test cases from the copy and pass it to the executable. + ``` + awk -vRS= -vORS='\n\n' '/test case description regex/' my.datax + tests/test_suite_foo my.datax + ``` ## Introducing new tests @@ -72,20 +106,15 @@ When you want to introduce a new test, if the test function: * Already exists and it only missing the test data, then update the .data file with the additional test data. If required, you can add a resource file to the `data_files/` subfolder. * Doesn't exist, you can implement a new test function in the relevant `.function` file following the guidelines mentioned above and add test cases to the .data file to test your new feature. -If you need to define a new test suite, for example when you introduce a new cryptography module, update the [`Makefile`](https://github.com/Mbed-TLS/mbedtls/blob/development/tests/Makefile) to build your test suite. - You should write your test code in the same platform abstraction as the library, and should not assume the existence of platform-specific functions. -Note that SSL is tested differently, with sample programs under the `programs/ssl/` folder. These are executed when you run the scripts `tests/ssl-opt.sh` and `tests/compat.sh`. - +Note that historically, most of SSL was tested differently, with sample programs under the `programs/ssl/` folder. These are executed when you run the scripts `tests/ssl-opt.sh` and `tests/compat.sh`. However, for new code, we prefer to have unit tests as well. ## `.function` example ```c /* BEGIN_HEADER */ #include "mbedtls/some_module.h" - -#define MAX_SIZE 256 /* END_HEADER */ /* BEGIN_DEPENDENCIES @@ -93,42 +122,72 @@ Note that SSL is tested differently, with sample programs under the `programs/ss * END_DEPENDENCIES */ -/* BEGIN_CASE depends_on:MBEDTLS_DEPENDENT_MODULE */ -void test_function_example( char *input, char *expected_output, int expected_ret ) +/* BEGIN_CASE depends_on:MBEDTLS_MODULE_OPTIONAL_PART */ +void test_function_example(data_t *input, data_t *expected_output, int expected_ret) { - int ilen, olen; - unsigned char buf[MAX_SIZE]; - unsigned char output[MAX_SIZE], output_str[MAX_SIZE]; - - memset( buf, 0, sizeof( buf ) ); + unsigned char *output = NULL; + size_t output_size = expected_output->len; + size_t output_length = SIZE_MAX; - ilen = unhexify( buf, input ); + TEST_CALLOC(output, output_size); - TEST_ASSERT( mbedtls_module_tested_function( buf, len, output ) == expected_ret ); + TEST_EQUAL(mbedtls_module_tested_function(input->x, input->len, + expected_output->x, output_size, + &output_length), + expected_ret); - if( ret == 0 ) - { - hexify( output_str, output, olen ); - TEST_ASSERT( strcasecmp( (char *) output_str, output ) == 0 ); + if (ret == 0) { + TEST_MEMORY_COMPARE(expected_output->x, expected_output->len, + output, output_len); } + +exit: + mbedtls_free(output); } /* END_CASE */ ``` ## Guidance on writing unit test code +Many helper macros and functions are available in [the `tests` directory of the framework repository](https://github.com/Mbed-TLS/mbedtls-framework/blob/main/tests) (location since Mbed TLS 3.6.0, also applying to TF-PSA-Crypto). They are declared in [`` header files](https://github.com/Mbed-TLS/mbedtls-framework/blob/main/tests/include/test). + ### Testing expected results Calls to library functions in test code should always check the function's return status. Fail the test if anything is unexpected. -The header file [`tests/include/test/macros.h`](https://github.com/Mbed-TLS/mbedtls/blob/development/tests/include/test/macros.h) declares several useful macros, including: +The header file [``](https://github.com/Mbed-TLS/mbedtls-framework/blob/development/tests/include/test/macros.h) declares several useful macros, including: * `TEST_EQUAL(x, y)` when two integer values are expected to be equal, for example `TEST_EQUAL(mbedtls_library_function(), 0)` when expecting a success or `TEST_EQUAL(mbedtls_library_function(), MBEDTLS_ERR_xxx)` when expecting an error. * `TEST_LE_U(x, y)` to test that the unsigned integers `x` and `y` satisfy `x <= y`, and `TEST_LE_S(x, y)` when `x` and `y` are signed integers. -* `ASSERT_COMPARE(buffer1, size1, buffer2, size2)` to compare the actual output from a function with the expected output. +* `TEST_MEMORY_COMPARE(buffer1, size1, buffer2, size2)` to compare the actual output from a function with the expected output. * `PSA_ASSERT(psa_function_call())` when calling a function that returns a `psa_status_t` and is expected to return `PSA_SUCCESS`. +* `TEST_FAIL("explanation of why this shouldn't happen")` for code that should be unreachable. * `TEST_ASSERT(condition)` for a condition that doesn't fit any of the special cases. - * In rare cases where a part of the test code shouldn't be reached, the convention is to use `TEST_ASSERT(!"explanation of why this shouldn't be reached")`. + +Older test code only had `TEST_ASSERT`. But in new test code, please use higher-level macros where applicable, as they have additional conveniences. + +These macros can be used in the `.function` file, but also in auxiliary functions. If the assertion fails, in addition to marking the test case as failed, the macros cause `goto exit` to happen, thus the function must have an `exit` label. Often you'll need to write some code after the `exit:` label, but as a convenience, if you have no cleanup code, the test framework will add `exit:;` to test entry points that don't have an `exit:` label. + +### Output on failure + +If a test fails, the location of the error is displayed, as well as the failed assertion. + +If the test code runs into more than one failed assertion, only information about the first one is displayed. This is usually the right thing because as soon as one assertion has failed, the data is probably in a bad state anyway. + +When a test assertion is in a loop, or in an auxiliary function that is called multiple times, the location is not enough to know exactly where the failure happened. You can call `mbedtls_test_set_step()` to declare a “step number” which is displayed together with the location on failure. For example: + +``` +for (int i = 0; i < max; i++) { + mbedtls_test_set_step(i); + one_iteration(i); + TEST_ASSERT(intermediate_check()); +} + +mbedtls_test_set_step(max); +final_checks(left_output); +mbedtls_test_set_step(max + 1); +final_checks(right_output); +``` ### Buffer allocation @@ -139,52 +198,52 @@ For output buffers, it's usually desirable to also check that the function works Here is an example of a test function that checks that a library function has the desired output for a given input. ```c /* BEGIN_CASE */ -void test_function( data_t *input, data_t *expected_output ) +void test_function(data_t *input, data_t *expected_output) { -// must be set to NULL both for ASSERT_ALLOC and so that mbedtls_free(actual_output) is safe +// must be set to NULL both for TEST_CALLOC and so that mbedtls_free(actual_output) is safe unsigned char *actual_output = NULL; size_t output_size; size_t output_length; /* Good case: exact-size output buffer */ output_size = expected_output->len; - ASSERT_ALLOC( actual_output, output_size ); + TEST_CALLOC(actual_output, output_size); // set output_length to a bad value to ensure mbedtls_library_function updates it output_length = 0xdeadbeef; - TEST_EQUAL( mbedtls_library_function( input->x, input->len, - actual_output, output_size, - &output_length ), 0 ); + TEST_EQUAL(mbedtls_library_function(input->x, input->len, + actual_output, output_size, + &output_length), 0); // Check both the output length and the buffer contents - ASSERT_COMPARE( expected_output->x, expected_output->len, - actual_output, output_length ); + TEST_MEMORY_COMPARE(expected_output->x, expected_output->len, + actual_output, output_length); // Free the output buffer to prepare it for the next subtest - mbedtls_free( actual_output ); + mbedtls_free(actual_output); actual_output = NULL; /* Good case: larger output buffer */ output_size = expected_output->len + 1; - ASSERT_ALLOC( actual_output, output_size ); + TEST_CALLOC(actual_output, output_size); output_length = 0xdeadbeef; - TEST_EQUAL( mbedtls_library_function( input->x, input->len, - actual_output, output_size, - &output_length ), 0 ); - ASSERT_COMPARE( expected_output->x, expected_output->len, - actual_output, output_length ); - mbedtls_free( actual_output ); + TEST_EQUAL(mbedtls_library_function(input->x, input->len, + actual_output, output_size, + &output_length), 0); + TEST_MEMORY_COMPARE(expected_output->x, expected_output->len, + actual_output, output_length); + mbedtls_free(actual_output); actual_output = NULL; /* Bad case: output buffer too small */ output_size = expected_output->len - 1; - ASSERT_ALLOC( actual_output, output_size ); - TEST_EQUAL( mbedtls_library_function( input->x, input->len, - actual_output, output_size, - &output_length ), - MBEDTLS_ERR_XXX_BUFFER_TOO_SMALL ); - mbedtls_free( actual_output ); + TEST_CALLOC(actual_output, output_size); + TEST_EQUAL(mbedtls_library_function(input->x, input->len, + actual_output, output_size, + &output_length), + MBEDTLS_ERR_XXX_BUFFER_TOO_SMALL); + mbedtls_free(actual_output); actual_output = NULL; exit: - mbedtls_free( actual_output ); + mbedtls_free(actual_output); } /* END_CASE */ ``` @@ -195,7 +254,20 @@ In a test case that always uses PSA crypto, call `PSA_INIT()` at the beginning a In a test case that uses PSA crypto only when building with `MBEDTLS_USE_PSA_CRYPTO`, call `USE_PSA_INIT()` at the beginning and `USE_PSA_DONE()` at the end. -See [`tests/include/test/psa_crypto_helpers.h`](https://github.com/Mbed-TLS/mbedtls/blob/development/tests/include/test/macros.h) for more complex cases. +See [``](https://github.com/Mbed-TLS/mbedtls-framework/blob/development/tests/include/test/psa_crypto_helpers.h) for more complex cases. + +### Constant-flow testing + +We run some tests with [MemorySanitizer (MSan)](https://github.com/google/sanitizers/wiki/memorysanitizer) and [Valgrind](https://valgrind.org/docs/manual/mc-manual.html) configured to detect secret-dependent control flow: branches or memory addresses computed from secret data. These tests detect library code that could leak secret data through timing side channels to local attackers via shared hardware components such as a memory cache or a branch predictor. We refer to such tests as “constant-time” or more accurately “constant-flow” testing. + +Constant-flow testing was added relatively recently in the history of the project, and many functions that should be constant-flow are not tested. However, constant-flow testing is preferred when writing new code that claims to be constant-flow, and especially when fixing a timing side channel. + +In unit tests, use the following macros, from [``](https://github.com/Mbed-TLS/mbedtls-framework/blob/main/tests/include/test/constant_flow.h): + +* `TEST_CF_SECRET(buffer, size)`: marks the given buffer as secret. Call this on keys, plaintext and other confidential data before passing it to library functions. +* `TEST_CF_PUBLIC(buffer, size)`: marks the given buffer as public. Call this on outputs before testing their content. + +Note that you need to call `TEST_CF_PUBLIC` before `TEST_MEMORY_COMPARE`. However, it is not needed with scalar comparison assertions (`TEST_EQUAL`, etc.), which make a public copy of its argument before comparing them. ## Guidance on writing unit test data diff --git a/kb/testing/testing-constant-flow.md b/kb/testing/testing-constant-flow.md index 7280f16..2d659e8 100644 --- a/kb/testing/testing-constant-flow.md +++ b/kb/testing/testing-constant-flow.md @@ -1,5 +1,7 @@ # Tools for testing constant-flow code +*This document is an investigation into test tooling. For usage in Mbed TLS and TF-PSA-Crypto unit tests, see “[Mbed TLS test guidelines — Constant-flow testing](../development/test_suites.md#constant-flow-testing)”.* + Code that manipulates secret values (private keys, etc.) needs to be constant-flow (often called constant-time, though the requirements are actually stricter than "the total running time is a constant"), that is contain no branches that depend on secret values, and no memory accesses at addresses depending on a secret value, in order to avoid leaking the secret value through side channels. Ideally, this should not only be enforced by code review, but also tested or checked by tools. This pages list some available options.