Inconsistent License cache: rebuilding index.

[Gator8]

Description

Recently upgraded to 31.2.1 from 30.1, this has started happening on a much more frequent basis. So much so that I've had to disable these scans until the issue is resolved. Looks like the file scan runs fine, but when it tries to scan licenses, I get this error:

14:53:16 Inconsistent License cache: rebuilding index.
14:53:16 ERROR: Failed to load license cache (the file may be corrupted ?).
14:53:16 Please delete "C:\tools\scancode-toolkit-v31.2.1\venv\lib\site-packages\licensedcode\data\cache\license_index\index_cache" and retry.
14:53:16 If the problem persists, copy this error message and submit a bug report at https://github.com/nexB/scancode-toolkit/issues/
14:53:16 Traceback (most recent call last):
14:53:16 File "C:\tools\scancode-toolkit-v31.2.1\venv\lib\site-packages\licensedcode\cache.py", line 338, in load_cache_file
14:53:16 return pickle.load(lfc)
14:53:16 File "C:\tools\scancode-toolkit-v31.2.1\venv\lib\site-packages\attr_make.py", line 958, in slots_setstate
14:53:16 def slots_setstate(self, state):
14:53:16 Exception
14:53:16
14:53:16 The above exception was the direct cause of the following exception:
14:53:16
14:53:16 Traceback (most recent call last):
14:53:16 File "C:\tools\scancode-toolkit-v31.2.1\venv\lib\site-packages\licensedcode\cache.py", line 84, in load_or_build
14:53:16 return load_cache_file(cache_file)
14:53:16 File "C:\tools\scancode-toolkit-v31.2.1\venv\lib\site-packages\licensedcode\cache.py", line 346, in load_cache_file
14:53:16 raise Exception(msg) from e
14:53:16 Exception: ERROR: Failed to load license cache (the file may be corrupted ?).
14:53:16 Please delete "C:\tools\scancode-toolkit-v31.2.1\venv\lib\site-packages\licensedcode\data\cache\license_index\index_cache" and retry.
14:53:16 If the problem persists, copy this error message and submit a bug report at https://github.com/nexB/scancode-toolkit/issues/

Judging by the timestamp on the left and the fact that it takes 10 minutes to rebuild the cache from the command line, there is never any attempt to rebuild the cache.

How To Reproduce

I'm not sure how easy it is to reproduce.

C:\tools\scancode-toolkit-v31.2.1\scancode -clpi -n 8 [BUNCH OF IGNORES] --strip-root --only-findings --verbose --json-pp %1.json --spdx-rdf %2.spdx %3

System configuration

Windows 10
Scancode 31.2.1
zip file downloaded from github.

[pombredanne]

@Gator8 how did you install scancode? Which exact zip file did you use?
The wheel and application comes now with a pre-built index.

[Gator8]

[pombredanne]

@Gator8 Thanks. That's really weird and this should never happen in this case unless there is some filesystem level file corruption.
Is this c:\tools a shared drive somehow?
Can you reproduce this on a fresh scancode zip extract?

[Gator8]

the tools dir is not shared, and yes, I was able to reproduce it right after a fresh install on more than one machine.

[pombredanne]

@Gator8 Thank you. Where these all on Windows?

[Gator8]

Yes. All on Windows (10 and server 2019)

[pombredanne]

I think this is related to #3179 ... and some subtle pickling bug in Python.

[RabeeaEgbareia]

I'm facing the same issue (installed on mac with docker - scanCode.io)

[AyanSinhaMahapatra]

@RabeeaEgbareia can you tell us which scancode-toolkit version you are using btw? Are you using the latest https://github.com/nexB/scancode-toolkit/releases/tag/v31.2.4 which had fixes for this? On the scancode.io side, can you confirm you are using the latest develop from https://github.com/nexB/scancode.io/ which has the updated https://github.com/nexB/scancode.io/blob/main/setup.cfg#L75 scancode-toolkit version 32.2.4?

[RabeeaEgbareia]

@AyanSinhaMahapatra
scancodeio version = 32.0.1
scancode-toolkit[packages]==31.2.4

[RabeeaEgbareia]

@AyanSinhaMahapatra
Is it good enough ?

[AyanSinhaMahapatra]

@RabeeaEgbareia yup! We fixed the issue by #3193 for 31.2.4 which was solved. But seems like this has resurfaced again, I'm looking into this.

[AyanSinhaMahapatra]

I could not reproduce the issue with a fresh docker install of scancodeio version = 32.0.1, or a fresh pip install of scancode-toolkit 31.2.4 on a ubuntu 20.04 LTS. 🤔

@tdruez can you check if this happens on your side in mac?
@pombredanne FYI ^

[tdruez]

@AyanSinhaMahapatra sure, what exactly do you need me to run?