[Ideas] Submodule Pinning

[ChaosEternal]

Description

Several submodules are currently pulling from main or an active
branch, this could lead to non-reproducibility over time.
Pinning the submodules is a solution to this problem.

However, this method doesn't grant immunity to the scenarios that the upstream deleted the repo or the history containing the pinned SHA. To solve this problem, an alternative solution is using subtree instead of submodule. subtree can keep a snapshot of the upstream repo and track the commit SHA of the upstream repo in history of our repo.

My plan is to pin the commit sha of the submodules as the first stage, and experiment the conversion to subtree.

Use case/motivation

Submodules involved:

• gpcontrib/gpcloud/test/googletest
• contrib/pax_storage/src/cpp/contrib/googletest
• contrib/pax_storage/src/cpp/contrib/tabulate
• contrib/pax_storage/src/cpp/contrib/googlebench
• contrib/pax_storage/src/cpp/contrib/cpp-stub
• dependency/yyjson

These submodules are pinned to main or some active branch, eg. v1.15x, but as the upstream evolves, the content of these submodules will change unexpectedly and the whole project will become non-reproducible.

Pinning the submodules to a specific commit SHA can somehow solve this problem, as the content of the submodules are referenced by SHA, which is immutable.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[edespino]

While reviewing the implementation, I suggested we consider using release tags instead of raw commit SHAs for the submodules, where available. Tags offer better visibility into the upstream version being used and reduce ambiguity for future maintenance. The full rationale is included in the PR #1084 comments, but I wanted to surface the suggestion here as well so others can weigh in.

Happy to discuss pros/cons of commit SHAs vs. tags here if there's broader interest before we finalize an approach.

[yjhjstz]

vote for use tags.

[ChaosEternal]

Let's use this discuss as our single source of truth.

| Improved traceability: Tags often correspond to stable releases

True, but Git itself doesn't have a mechanism to prevent a tag being changed. If our goal is to have reproducibility, I don't think using tag is a good trade off for readability. I prefer log the tag information of the corresponding commit SHA in the .gitmodules file as a comment or record the tag in the commit message. (I can make this change)

| Community alignment: Staying in sync with upstream’s versioning
Yes, this needs a little bit manual work. when upstream has a new version, we modify the .gitmodules file to point to the SHA of the new version, make our test. and the process is tracked in our commit history.

| Reduced churn: Upstream tags are less likely to be force-pushed or removed compared to commits on active branches.

The both companies (name starts with G) have history of shutdown/remove projects. And it is not necessary to risk our project's stability and reproducibility on the assumptions that "upstream tags are less likely to be force-pushed or removed"

Summary:

Using tag is convenient and eye-friendly.
However, since we want to have reproducibility, directly putting the commit SHA in the .gitmodules file is better.
It is not worth to pin to a tag.

[tuhaihe]

Hey @ChaosEternal when reviewing the PR, I found it hard to compare the commit SHA with the corresponding branch, which will not be friendly for future review and maintenance. When referring to the main/master branch in the .gitmodules file, I saw that the latest commit SHA has changed for most projects.

Here is my proposal for this:

• If the projects have tags, we can use the tags as a mark to have an eye-friendly view, and if needed, can add the commit SHA and commit date as a comment for tag tracking to prevent the tags from being forced removed in the upstream projects.
• If the projects have no tags with only the main branch, we can specify one commit in the main branch as the mark, and can add the commit date as a comment for note.

For the concern on upstream projects, I think if the upstream projects have been removed or shut down, then we need to seek an active alternative to them, but for now, at the very lowest possibility. So let's get started with an easy way to proceed.

[ChaosEternal]

Hi Dianjin, If we care about eye friendly, as I suggested, we can put the branch or tag in the comment of the .gitmodules file and in the commit message. For checking, it is very hard to find out if an upstream has some updated tags(impossible for human eyes). What we need to do is to have a CI task to check that, and having commit SHA is much easier for scripts.

…

On Wed, May 7, 2025 at 2:37 PM Dianjin Wang ***@***.***> wrote: Hey @ChaosEternal <https://github.com/ChaosEternal> when reviewing the PR, I found it hard to compare the commit SHA with the corresponding branch, which will not be friendly for future review and maintenance. When referring to the main/master branch in the .gitmodules file, I saw that the latest commit SHA has changed for most projects. Here is my proposal for this: - If the projects have tags, we can use the tags as a mark to have an eye-friendly view, and if needed, can add the commit SHA and commit date as a comment for tag tracking to prevent the tags from being forced removed in the upstream projects. - If the projects have no tags with only the main branch, we can specify one commit in the main branch as the mark, and can add the commit date as a comment for note. For the concern on upstream projects, I think if the upstream projects have been removed or shut down, then we need to seek an active alternative to them, but for now, at the very lowest possibility. So let's get started with an easy way to proceed. — Reply to this email directly, view it on GitHub <#1083 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALMFGYFJVL2PVOCDFHKRYL25GEX7AVCNFSM6AAAAAB4LE4DLKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMBVHA2DQMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tuhaihe]

I recently did a test following Ed's suggestions from PR #1084 comments. It worked, as I can track versions more effectively on my test branch: https://github.com/tuhaihe/cloudberrydb/commits/submodule-update05/, which also incorporates the commit SHA as a reliable reference.

I also took a look at the manual on .gitmodules, where I found a helpful example for upgrading a submodule version, similar to Ed's suggestion: https://git-scm.com/docs/gitsubmodules#_workflow_for_a_third_party_library. By using this method, we can avoid updating the .gitmodules file manually.

Of course, we can specify the branch in the .gitmodules file as well like your way! Here needs a tradeoff.

Anyway, let's move forward.

[tuhaihe]

As we’re approaching the 2.0.0 release, I noticed that there hasn’t been significant progress on PR #1084. If the author and reviewers are unable to reach a consensus in the near term, I’d suggest we consider deferring this PR to a future release so it doesn’t block the current release schedule.

Let me know what you think.