purl-next: Update PurlDB to support on-demand - golang

[pombredanne]

Update PurlDB to support on-demand, live query of upstream package registries for the PURL types documented in the PURL specification to validate the existence of PURLs for one ecosystem

A good start is this issue to finish:

• Add on-demand package data collection for golang #596

[chinyeungli]

The #596 was finished but we should/need to modify the code to use the latest code from packageurl-python (0.17.2) to get the go packages' metadata especially for the download url. However, I am not able to build purldb using the latest packageurl-python. I am consulting with @TG1999 about it.
Following is the background,

In PurlDB, after upgrading to packageurl-python==0.17.2, running make clean followed by make dev results in a normal build time, but triggers the following error:

ERROR: Cannot install None, aboutcode.federatedcode, purl2vcs, purl2vcs[testing]==2.0.0, purldb, purldb[dev]==6.0.0, scancode-toolkit and scancode-toolkit[packages]==32.3.3 because these package versions have conflicting dependencies.

The conflict is caused by:
    purldb 6.0.0 depends on packageurl-python>=0.17.1
    purl2vcs 2.0.0 depends on packageurl_python>=0.9.0
    purl2vcs[testing] 2.0.0 depends on packageurl_python>=0.9.0
    purldb[dev] 6.0.0 depends on packageurl-python>=0.17.1
    aboutcode-federatedcode 0.1.0 depends on packageurl_python>=0.16.0
    aboutcode-hashid 0.2.0 depends on packageurl_python>=0.15.6
    aboutcode-toolkit 11.1.1 depends on packageurl_python>=0.9.0
    fetchcode 0.6.0 depends on packageurl-python
    scancode-toolkit 32.3.3 depends on packageurl-python>=0.9.0
    scancode-toolkit[packages] 32.3.3 depends on packageurl-python>=0.9.0
    scancodeio 35.0.0 depends on packageurl-python==0.17.1
    The user requested (constraint) packageurl-python==0.17.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

When using packageurl-python==0.17.1, running make dev completes without any errors. However, after upgrading to packageurl-python==0.17.2 and re-running make dev without executing make clean, the build process becomes significantly slower, eventually resulting in the following error after approximately 10 minutes.

Collecting pycodestyle>=2.8.0 (from purl2vcs==2.0.0)
  Using cached pycodestyle-2.14.0-py2.py3-none-any.whl.metadata (4.5 kB)
error: resolution-too-deep

× Dependency resolution exceeded maximum depth
╰─> Pip cannot resolve the current dependencies as the dependency graph is too complex for pip to solve efficiently.

hint: Try adding lower bounds to constrain your dependencies, for example: 'package>=2.0.0' instead of just 'package'.

Link: https://pip.pypa.io/en/stable/topics/dependency-resolution/#handling-resolution-too-deep-errors
make: *** [Makefile:50: dev] Error 1

My hunch is the issue might be related to scancodeio 35.0.0 relying on packageurl-python==0.17.1 while purldb requires packageurl-python==0.17.2, although I'm not entirely certain.

Note: if I remove the scancodeio in the requirements.txt and setup.cfg, the make dev run fine with packageurl-python==0.17.2

(The build fails with the same error on azure: https://dev.azure.com/nexB/purldb/_build/results?buildId=16854&view=logs&j=42e6f090-fbfe-5dad-0b95-813514edce20&t=a2549dd8-a900-5e63-68b6-206b46d779b3)

[chinyeungli]

as it relies on packagepurl-python's build_golang_download_url() and there is an issue (See #596 (comment)). I am moving it to the next iteration as this is fixed once the packagepurl-python has the issue fixed.