FileProvenanceFileStorage cannot find file listings

[MarcelBochtler]

Since #6970, we're seeing the following error in our scans:

ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from 66eab51c0401c5041aa975dbf9c112cecfb9dfdd/file_list.xz: FileNotFoundException: /root/.ort/scanner/file-lists/66eab51c0401c5041aa975dbf9c112cecfb9dfdd/file_list.xz (No such file or directory)

We did not yet configure the postgres storage for the file listings, but from my understanding not configuring that, shouldn't lead to errors in the scan.

Edit: We're using ORT's legacy docker image for these scans.

[sschuberth]

@MarcelBochtler can this still be reproduced after the recent PRs #7119, #7121 and #7129 in that area?

[MarcelBochtler]

In the meantime we configured Postgres for the file lists, which solves this issue.

As a test I disabled it again, and yes, this can still be reproduced.

[alpianon]

I'm experiencing the same bug and using same workaround. ORT 19.1.0, built from sources with OpenJDK 11 on Ubuntu 22.04

[sschuberth]

In any case, glad you're taking a look at ORT, @alpianon. Note that BitBake-related stuff is still in a fork only. Please ensure to ask question rather sooner than later, e.g. in Slack, as we're aware that ORT is not trivial to setup and configure, esp. with performance in mind.

[alpianon]

In any case, glad you're taking a look at ORT, @alpianon. Note that BitBake-related stuff is still in a fork only. Please ensure to ask question rather sooner than later, e.g. in Slack, as we're aware that ORT is not trivial to setup and configure, esp. with performance in mind.

I'm not working only on bitbake-based projects, now I have a java+npm project to scan :)

Glad that you started developing a bitbake analyzer. Maybe we can talk about it on slack

[sschuberth]

Maybe we can talk about it on slack

We should absolutely talk! But I guess a dedicated meeting makes sense. I'll reach out to you privately.

[rettichschnidi]

I am seeing this error too when doing the steps described in the tutorial:

$ ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results/
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.0.0-049.sha.ed7fdbe,
|    |   | |       _/ |    |    built with JDK 11.0.22+7, running under Java 17.0.11.
|    |   | |    |   \ |    |    Executing 'scan' as 'username' on Linux
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 24128 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/username/.ort/config
ORT_DATA_DIR = /home/username/.ort
HOME = /home/username
SHELL = /bin/bash
TERM = xterm-256color
GOPATH = /home/username/opt/go

Looking for ORT configuration in the following file:
        /home/username/.ort/config/config.yml (does not exist)

Scanning projects with:
        ScanCode (version 32.1.0)
Scanning packages with:
        ScanCode (version 32.1.0)
01:24:09.973 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from 39f76b06552989aaa626341d3c3936645bad36b3/file_list: FileNotFoundException: /home/username/.ort/scanner/file-lists/39f76b06552989aaa626341d3c3936645bad36b3/file_list (No such file or directory)
Caused by: FileNotFoundException: /home/username/.ort/scanner/file-lists/39f76b06552989aaa626341d3c3936645bad36b3/file_list.xz (No such file or directory)
01:24:31.163 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from c643dce0af603a22b7be53afb0844ce302fa93e6/file_list: FileNotFoundException: /home/username/.ort/scanner/file-lists/c643dce0af603a22b7be53afb0844ce302fa93e6/file_list (No such file or directory)
Caused by: FileNotFoundException: /home/username/.ort/scanner/file-lists/c643dce0af603a22b7be53afb0844ce302fa93e6/file_list.xz (No such file or directory)
Wrote scan result to '/home/username/some-path/ort-example-mime-types/mime-types-ort-scanner-results/scan-result.yml' (0.42 MiB) in 133.975115ms.
The scan took 39.348319185s.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.

But when running the same command a 2nd time the errors are gone, as the mentioned files have been created during the first run:

$ ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results/
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.0.0-049.sha.ed7fdbe,
|    |   | |       _/ |    |    built with JDK 11.0.22+7, running under Java 17.0.11.
|    |   | |    |   \ |    |    Executing 'scan' as 'username' on Linux
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 24128 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/username/.ort/config
ORT_DATA_DIR = /home/username/.ort
HOME = /home/username
SHELL = /bin/bash
TERM = xterm-256color
GOPATH = /home/username/opt/go

Looking for ORT configuration in the following file:
        /home/username/.ort/config/config.yml (does not exist)

Scanning projects with:
        ScanCode (version 32.1.0)
Scanning packages with:
        ScanCode (version 32.1.0)
Wrote scan result to '/home/username/some-path/ort-example-mime-types/mime-types-ort-scanner-results/scan-result.yml' (0.42 MiB) in 209.344576ms.
The scan took 3.591540482s.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.

[sschuberth]

Are there any files present below /home/username/.ort/scanner/file-lists? I'm curious whether files or only being looked up under the wrong location maybe, or whether they're not being created in the first place.

[rettichschnidi]

Are there any files present below /home/username/.ort/scanner/file-lists?

Yes, but not the ones ort scan was not able to find. Those files however exist after ort scan ran (and complained about them being missed).

I can reproduce the error like this:

1. rm -rf ~/.ort/
2. ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results

Running ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results a 2nd time will then be much faster and does no longer show this error.

[reuvenperetz]

Hi, is there any update about this issue? I'm experiencing the same problem...