add on demand package data collection for golang, gitlab and bitbucket #596 #608

chinyeungli · 2025-04-15T22:40:21Z

Fixes Add on-demand package data collection for golang #596

… golang Signed-off-by: Chin Yeung Li <[email protected]>

… of test code that will need to be removed). Signed-off-by: Chin Yeung Li <[email protected]>

``` pkg:golang/github.com/* pkg:golang/gitlab.com/* pkg:golang/bitbucket.org/* ``` Signed-off-by: Chin Yeung Li <[email protected]>

Signed-off-by: Chin Yeung Li <[email protected]>

* Collect metadata from API for the following "namespace" ``` pkg:golang/github.com/* pkg:golang/gitlab.com/* pkg:golang/bitbucket.org/* ``` * Add tests * Add "golang" in the "supported_ecosystems" list in the api.py Signed-off-by: Chin Yeung Li <[email protected]>

JonoYang

@chinyeungli I am looking at https://github.com/package-url/purl-spec/blob/main/PURL-SPECIFICATION.rst#rules-for-each-purl-component and I am not sure if we can add gitlab.com in the package namespace otherwise, the code looks good.

JonoYang · 2025-04-15T23:57:13Z

minecode/collectors/generic.py


+    if from_go_lang:
+        packages[0].type = "golang"
+        packages[0].namespace = "github.com/" + packages[0].namespace


@chinyeungli could there be golang packages not from github?

Yes. Only golang packages from github use this map_fetchcode_supported_package function.
Others will use map_golang_package()

JonoYang · 2025-04-16T00:17:25Z

minecode/collectors/golang.py

+    version = ""
+    if "@" in purl_str:
+        version = purl_str.rpartition("@")[2]
+    subset = purl_str.partition("pkg:golang/gitlab.com/")[2].partition("@")[0]


@pombredanne https://github.com/package-url/purl-spec/blob/main/PURL-SPECIFICATION.rst#rules-for-each-purl-component

Does this mean we cannot have things like gitlab.com in the namespace field?

FYI, this is about getting the subset and version from a purl str that then pass to https://github.com/aboutcode-org/purldb/blob/596_add_on-demand_package_data_collection_for_golang/minecode/collectors/golang.py#L253

minecode/collectors/golang.py

Signed-off-by: Chin Yeung Li <[email protected]> Co-authored-by: Jono Yang <[email protected]>

…r_golang

Signed-off-by: Chin Yeung Li <[email protected]>

…r_golang

TG1999 · 2025-07-29T05:40:26Z

minecode/miners/golang.py

+    license_text = package_data.get("licenses")
+    extracted_license_statement = [license_text]
+
+    download_url = (


Go has some weird rules to encode upper case in these strings. See https://github.com/aboutcode-org/go-inspector/blob/442bc5b83d5aeff2b7a27937ec82b63277bc8f7c/src/go_inspector/utils.py#L211

We are adding support for getting golang download URL in PURL library. @pombredanne @chinyeungli I think we can reuse that here ?

package-url/packageurl-python#195 here is PR for same.

@TG1999 Thanks. We can use the code there once it's merged.

Signed-off-by: Chin Yeung Li <[email protected]>

…ages #596 Signed-off-by: Chin Yeung Li <[email protected]>

* This is so we can use the updated packageurl-python library Signed-off-by: Jono Yang <[email protected]>

* purldb depends on scancodeio which depends on sctk 32.4.0 (scancodeio 35.1.0 depends on scancode-toolkit==32.4.0) Signed-off-by: Chin Yeung Li <[email protected]>

Signed-off-by: Chin Yeung Li <[email protected]>

…r_golang

Signed-off-by: Chin Yeung Li <[email protected]>

TG1999 · 2025-09-08T11:40:58Z

minecode/collectors/bitbucket.py

+                for item in data["values"]:
+                    version = item["name"]
+                    author = ""
+                    if "target" in item and item["target"]:


What about using .get here

target = item.get("target") or {} author = target.get("author") or {} if author.get("type") == "author": user = author.get("user") or {} author_display_name = user.get("author")

TG1999 · 2025-09-08T11:41:52Z

minecode/collectors/github.py

+            break
+
+        for tag in data:
+            version_list.append(tag["name"])


If "name" property does not exits on a tag it should not crash, We should log and continue

TG1999 · 2025-09-08T11:42:17Z

minecode/collectors/gitlab.py

+        data = response.json()
+        version_author_list = []
+        # Get all available versions
+        for item in data:


IMO .get will be a more better option

TG1999 · 2025-09-08T11:49:04Z

minecode/miners/bitbucket.py

+            ]
+            data = response.json()
+            # Search for license files in the root directory
+            for item in data["values"]:


IMO a .get will be a better choice

TG1999

@chinyeungli majorly looks good to me, IMO we should use .get instead of directly getting an item using dict"foo"], so we can log that and know whenever the contract changes from upstream.

Signed-off-by: Chin Yeung Li <[email protected]>

TG1999

LGTM! Thanks

chinyeungli added 5 commits April 9, 2025 18:30

#596 - Added code for golang collector and updated the miner code for…

0882289

… golang Signed-off-by: Chin Yeung Li <[email protected]>

#596 - Working in progress to handle bitbucket.org (It contains a lot…

1c1fbd7

… of test code that will need to be removed). Signed-off-by: Chin Yeung Li <[email protected]>

#596 - Added the following data collection for golang

b0f1a22

``` pkg:golang/github.com/* pkg:golang/gitlab.com/* pkg:golang/bitbucket.org/* ``` Signed-off-by: Chin Yeung Li <[email protected]>

Correct typo

82cb1fb

Signed-off-by: Chin Yeung Li <[email protected]>

chinyeungli requested a review from JonoYang April 15, 2025 22:40

chinyeungli changed the title ~~596 add on demand package data collection for golang~~ add on demand package data collection for golang #596 Apr 15, 2025

JonoYang requested changes Apr 16, 2025

View reviewed changes

chinyeungli mentioned this pull request Apr 16, 2025

Enhance License Detection and Filename Handling for Github /api/collect/ #609

Open

chinyeungli and others added 6 commits April 16, 2025 15:04

Update minecode/collectors/golang.py

905195a

Signed-off-by: Chin Yeung Li <[email protected]> Co-authored-by: Jono Yang <[email protected]>

Merge branch 'main' into 596_add_on-demand_package_data_collection_fo…

16e9fac

…r_golang

change from web scrapping to fetch from deps.dev #596

cb18cb3

Signed-off-by: Chin Yeung Li <[email protected]>

Better error handling and remove test code #596

1b5034e

Signed-off-by: Chin Yeung Li <[email protected]>

Correct ci_code_style #596

7adce31

Signed-off-by: Chin Yeung Li <[email protected]>

Merge branch 'main' into 596_add_on-demand_package_data_collection_fo…

e3d973a

…r_golang

pombredanne mentioned this pull request Jul 28, 2025

Add on-demand package data collection for golang #596

Closed

TG1999 reviewed Jul 29, 2025

View reviewed changes

chinyeungli and others added 5 commits July 29, 2025 15:30

Upgrade to use packageurl-python 0.17.2 #596

fae4ff0

Signed-off-by: Chin Yeung Li <[email protected]>

Use "build_golang_download_url" to build the download url for go pack…

140de5b

…ages #596 Signed-off-by: Chin Yeung Li <[email protected]>

Use branch of scancode.io #596

674691d

* This is so we can use the updated packageurl-python library Signed-off-by: Jono Yang <[email protected]>

update to use scancode-toolkit 32.4.0 #596

dbfac90

* purldb depends on scancodeio which depends on sctk 32.4.0 (scancodeio 35.1.0 depends on scancode-toolkit==32.4.0) Signed-off-by: Chin Yeung Li <[email protected]>

Update packageurl-python to 0.17.3 #596

fdd3b51

Signed-off-by: Chin Yeung Li <[email protected]>

pombredanne changed the title ~~add on demand package data collection for golang #596~~ add on demand package data collection for golang, gitlab and bitbucket #596 Sep 2, 2025

chinyeungli added 3 commits September 3, 2025 14:09

Update packages dependencies

1e204a6

Signed-off-by: Chin Yeung Li <[email protected]>

Merge branch 'main' into 596_add_on-demand_package_data_collection_fo…

4a8b4cf

…r_golang

Add error handling for failing to fetch in fetchcode #596

711b97c

Signed-off-by: Chin Yeung Li <[email protected]>

TG1999 reviewed Sep 8, 2025

View reviewed changes

chinyeungli added 3 commits September 9, 2025 15:18

Update code to use .get instead of using dict['item'] #596

27d08de

Signed-off-by: Chin Yeung Li <[email protected]>

Revert code changes for build_bitbucket_packages()

e196df3

Signed-off-by: Chin Yeung Li <[email protected]>

Properly to use .get instead of using dict['item'] #596

2497a53

Signed-off-by: Chin Yeung Li <[email protected]>

TG1999 approved these changes Sep 9, 2025

View reviewed changes

TG1999 merged commit edff9e1 into main Sep 9, 2025
3 of 6 checks passed

TG1999 mentioned this pull request Sep 9, 2025

purl-next: Update PurlDB to support on-demand - golang #644

Closed

Uh oh!

add on demand package data collection for golang, gitlab and bitbucket #596 #608

add on demand package data collection for golang, gitlab and bitbucket #596 #608

Uh oh!

Conversation

chinyeungli commented Apr 15, 2025 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonoYang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TG1999 left a comment

Choose a reason for hiding this comment

Uh oh!

TG1999 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chinyeungli commented Apr 15, 2025 •

edited by pombredanne

Loading