Allow curations to be applied without rerunning the analyzer

[sschuberth]

Currently, the turn around-times to test (technical) curations (also see #6187) locally or on CI are rather high as curations are "baked" into Analyzer results. This means one needs to rerun the analyzer, even if the previous analysis was successful, just in order to get updated curation data applied to its results.

Just brainstorming some ideas how to address this (without any ordering implied):

• Do not store curations as part of Analyzer results at all, but create a new "Curator" tool that takes the Analyzer result as input and adds curations data to a separate section in the ORT result file, similar to like any other tool except for the Analyzer works.
• Create a helper command that can patch-up analyzer results with updates / different curations. Actually, there already is the PackageCurationsCommand's SetCommand which could probably be used for this.

[sschuberth]

BTW, depending on the implementation, this could also solve #5637.

[mnonnenmacher]

Suggested name for the new ORT result section that could contain package curations, package configurations and resolutions: "corrections"

[mnonnenmacher]

To ensure consistency within a pipeline of the ORT tools it is important that configuration is only applied or replaced before it is used as input for a tool. For example, replacing package curations after running the scanner could lead to inconsistencies if the provenance metadata of a package is changed and as a result the scan result for that package does not match the provenance anymore.

A summary of the order in which tools should be run and configuration should be added to the result file to ensure consistency in later tools in the pipeline (package curations are split into technical "metadata corrections" and "legal curations" based on the idea from #6187):

Analyzer
-> package metadata corrections
-> Scanner (uses provenance corrections), Advisor (potentially uses identifier corrections, e.g. purl)
-> package configurations, issue resolutions, vulnerability resolutions, legal curations
-> Evaluator
-> rule violation resolutions
-> Reporter

[mnonnenmacher]

Based on my above comment I propose that we extend the OrtResult model to contain the configuration used by each tool. I see two options:

• Extend the section for each tool to contain the configuration consumed by this tool. For example, the AnalyzerRun could contain the list of curations, or the EvaluatorRun the lists of PackageConfigurations and resolutions used during the run.
• Extend the OrtResult to contain a separate section which contains all of the configuration values. For example, the analyzer adds curations to this model and later tools like the scanner or advisor only use curations from there but not from any other sources.

For both solutions it would be possible to add commands that replace the existing configuration, for example one command could update the curations in an OrtResult and also check if it already contains data that depends on them, like a scan result, and in this case either fail or print a warning that the result might be inconsistent.

@oss-review-toolkit/core-devs Any preference which direction we should take?

[sschuberth]

I'm having a hard time to decide / make up an opinion:

Extend the section for each tool to contain the configuration consumed by this tool.

On the one hand I like this as things that seemingly belong together are stored together, and we already have AnalyzerConfiguration etc. However, the question rises how to deal with configuration that might might be consumed by multiple tools.

Extend the OrtResult to contain a separate section which contains all of the configuration values.

This would work around the question how to deal with configuration used by different tools, and better match the also already existing RepositoryConfiguration (which in turn also contains an AnalyzerConfiguration). But what about the existing "top-level" AnalyzerConfiguration etc. then?

So we already have a mix of (primarily) storing configuration by origin, or by consumer. Maybe we should in this context also consider to make the "origin" of configuration a property of the configuration itself.

[mnonnenmacher]

I'm having a hard time to decide / make up an opinion:

Extend the section for each tool to contain the configuration consumed by this tool.

On the one hand I like this as things that seemingly belong together are stored together, and we already have AnalyzerConfiguration etc. However, the question rises how to deal with configuration that might might be consumed by multiple tools.

Extend the OrtResult to contain a separate section which contains all of the configuration values.

This would work around the question how to deal with configuration used by different tools, and better match the also already existing RepositoryConfiguration (which in turn also contains an AnalyzerConfiguration). But what about the existing "top-level" AnalyzerConfiguration etc. then?

I have a tendency towards a top-level section for those properties, but I think we need a better name than "configuration" to separate it from the "tool configuration" in config.yml and .ort.yml. Above "corrections" was suggested, this works for curations and package configurations but not for resolutions. So maybe two top-level sections corrections and resolutions?

So we already have a mix of (primarily) storing configuration by origin, or by consumer. Maybe we should in this context also consider to make the "origin" of configuration a property of the configuration itself.

I agree, at least for corrections and resolutions. For example, for curations we should store the provider. But for the tool configurations this might not be possible, because it can come from multiple sources like environment variables, command line parameters or config.yml, and this could be different for each property.

To make a draft:

ort:
  ...
  corrections:
    packageCurations:
      ...
    packageConfigurations:
      ...
  resolutions:
    issues:
      ...
    ruleViolations:
      ...
    vulnerabilities:
      ...

[fviernau]

I want to bring to attention the following use case relevant for support workflows, e.g. when using orthw:

1. Make changes of curations in your config repository
2. Recompute the reports

This IMO should be as fast as possible, so I fear if we

1. Add a separate command to patch-up curations the turn-around time significantly increases
   for large ORT result files
2. Only have the possibility to replace all curations including querying remote curation providers if one uses them.
   But to keep turn-around time low, one should be able to replace only the local curations but keep the ones queried from
   remote.

What do you think?

edit: I've decided to write this down to an issue, see oss-review-toolkit/orthw-shell#60.

[mnonnenmacher]

I want to bring to attention the following use case relevant for support workflows, e.g. when using orthw:

1. Make changes of curations in your config repository

2. Recompute the reports

The problem with that workflow is that depending on the curation changes it could not be sufficient to only re-run the reporter, but it might be required to also re-run the advisor, scanner, and evaluator to get correct results.

This IMO should be as fast as possible, so I fear if we

1. Add a separate command to patch-up curations the turn-around time significantly increases
   for large ORT result files

I think the curations should still be fetched by the analyzer command to not always require an extra step to fetch curations. The benefit of the separate command is that can be run independent of which other tool needs to be run afterwards, so it could be used to fix a purl before running the advisor, to fix a VCS URL before running the scanner, and so on. But yes, it introduces the overhead of an additional serialization and deserialization.

2. Only have the possibility to replace all curations including querying remote curation providers if one uses them.
   But to keep turn-around time low, one should be able to replace only the local curations but keep the ones queried from
   remote.

To implement this correctly #5668 needs to be done first.