`ort compare` should be able to ignore tool versions

[pepper-jk]

What is the functionality you want to propose?

The ort compare command allows a TEXT_DIFF between two ORT result files. It also offers multiple arguments to ignore ever changing metadata, which is seldomly relevant to compare actual scan results, such as --ignore-time, --ignore-environment, and --ignore-tmp-dir.

When passing --ignore-environment the command will ignore the scanner/environment section of the ORT result file, which also includes the tool versions. Tools such as ScanCode.
However, a little later in the file, the ScanCode version gets repeated under scanner/scan_results/provenance/scanner, see result snippets at the end. This version does not get ignored, when the -e flag is active, but it should.

From my understanding the version is listed there for reference while looking at the configuration, which is passed to ScanCode.
So although this version is not part of the environment section, it could be ignored with no harm done.
It would also make the flag more complete in my opinion.

What is the use-case for your enhancement?

We use the ort compare command as part of our integration test for our ORT config.

For this we use the flags mentioned above in order to only see actual changes to the configs behavior or sometimes scancode's license scores on a version upgrade. This also helps avoid high maintenance on the integration test CI pipeline, as the test should not fail on different tool versions, timestamps, or the runner's temp directories.

With --ignore-environment only excluding the ScanCode version in some places, but not in others, every ScanCode update requires manual maintenance on the integration tests.

Alternatives you have considered

Another way to implement it would be to add yet another argument --ignore-tool-versions with sole focused on tool versions anywhere instead of forcing it into the --ignore-environemt.
An argument for the scan results / tool config section might run into the issue to overlap with things most users do not want to exclude or do not intuit to tool versions.

We are aware there are ways to automate a merge of the ort compare output into the test-data.yml, but while convenient this would increase the risk of missing actual changes or issues. So right now we are doing the updates manually.

To my shame I have to admit, I have not tested the SEMANTIC_DIFF for this use case yet.
But even if that does not have the same issue, I think ignoring this ScanCode version should be addressed for the TEXT_DIFF.

Additional context

scanner/environment:

scanner:
  start_time: "2025-08-14T14:32:49.971223719Z"
  end_time: "2025-08-14T14:35:27.972145581Z"
  environment:
    ort_version: "66.0.3"
    build_jdk: "21.0.8+9-LTS"
    java_version: "21.0.8"
    os: "Linux"
    processors: 8
    max_memory: 4294967296
    variables:
      ORT_CONFIG_DIR: "/builds/foss-compliance/ort-config"
      HOME: "/home/ort"
      JAVA_HOME: "/opt/java/openjdk"
      ANDROID_HOME: "/opt/android-sdk"
    tool_versions:
      ScanCode: "32.4.1"

scanner/scan_results/provenance/scanner

  scan_results:
  - provenance:
      vcs_info:
        type: "Git"
        url: "https://github.com/OpenVPN/openvpn"
        revision: "c9540130121bfc21902a54c8e4c9bc1108695068"
        path: ""
      resolved_revision: "c9540130121bfc21902a54c8e4c9bc1108695068"
    scanner:
      name: "ScanCode"
      version: "32.4.1"
      configuration: "--copyright --license --info --strip-root --timeout 300 --license-score\
        \ 80 --json"

Relevant ort compare output:

$ ort compare -t -e -d evaluator-results.yml test-data.yml
Hoplite is configured to infer which sealed type to choose by inspecting the config values at runtime. This behaviour is now deprecated in favour of explicitly specifying the type through a discriminator field. In 3.0 this new behavior will become the default. To enable this behavior now (and disable this warning), invoke withExplicitSealedTypes() on the ConfigLoaderBuilder.
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 66.0.3,
|    |   | |       _/ |    |    built with JDK 21.0.8+9-LTS, running under Java
|    |   | |    |   \ |    |    Executing 'compare' as 'ort' on Linux
\________/ |____|___/ |____|    with 1 CPUs and a maximum of 247 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /builds/foss-compliance/ort-config
HOME = /home/ort
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk

Looking for ORT configuration in the following file:
        /builds/foss-compliance/ort-config/config.yml
The ORT results differ:
--- a/ort/evaluator/evaluation-result.yml
+++ b/tests/openvpn/evaluation-result.yml
@@ -115,15 +115,15 @@
         type: "Git"
         url: "https://github.com/OpenVPN/openvpn"
         revision: "c9540130121bfc21902a54c8e4c9bc1108695068"
         path: ""
       resolved_revision: "c9540130121bfc21902a54c8e4c9bc1108695068"
     scanner:
       name: "ScanCode"
-      version: "32.4.1"
+      version: "32.4.0"
       configuration: "--copyright --license --info --strip-root --timeout 300 --license-score\
         \ 80 --json"
     summary:
       start_time: "1970-01-01T00:00:00Z"
       end_time: "1970-01-01T00:00:00Z"
       licenses:
       - license: "Apache-2.0"

[sschuberth]

When passing --ignore-environment the command will ignore the scanner/environment section of the ORT result file, which also includes the tool versions. Tools such as ScanCode.
However, a little later in the file, the ScanCode version gets repeated under scanner/scan_results/provenance/scanner, see result snippets at the end. This version does not get ignored, when the -e flag is active, but it should.

Actually, I think it should be the opposite: The --ignore-environment flag is only supposed to ignore environmental settings, like the amount of memory etc., that have no influence of the results, usually. It's a not unfortunate that toolVersions is part of Environment... it probably should have rather been a sibling of the latter.

From my understanding the version is listed there for reference while looking at the configuration, which is passed to ScanCode.
So although this version is not part of the environment section, it could be ignored with no harm done.

Again, it's more like the opposite: The scanner version listed as part of toolVersions does not really have any meaning, it's just listed there for reference, in a central place. The version of the scanner listed as part of the scan results, however is the version used for matching scan results when looking them up in a scan storage.

Another way to implement it would be to add yet another argument --ignore-tool-versions with sole focused on tool versions anywhere instead of forcing it into the --ignore-environemt.

Yes, from a first glance I like that better. For consistency, we could make --ignore-environment ignore all environment properties except tool versions, and then introduce --ignore-tool-versions to really ignore tool versions everywhere, in the environment and in the scan results.

What do you think?

[sschuberth]

Actually, in the context of #10676 I'm thinking about dropping toolVersions from Environment completely...

[pepper-jk]

Understandable. That would also make adding the --ignore-tool-versions argument cleaner, as it would not need to "mess" with the environment section.

[sschuberth]

@pepper-jk, @jens-erdmann, please check whether #10796 would work for you.

[jens-erdmann]

@pepper-jk, @jens-erdmann, please check whether #10796 would work for you.

I will test the changes on our end.