GCS bucket to store the KEPs metadata

Hi team,

We have PR: kubernetes/contributor-site#222

1. We want to generate the metadata file called keps.json file where we will be having all the details about the KEPs from the repo: https://github.com/kubernetes/enhancements through the command kepctl query
2. This generated metadata file (during the CI/CD pipeline) will be pushed to GCS bucket, once the new PR is merged in kubernetes/enhancements repo.
3. Now this metadata file will be used in site : https://deploy-preview-222--kubernetes-contributor.netlify.app/keps/ where we can search the KEPs by PR number, release, SIG or keywords. Since these are 2 different repo we want to push it into GCS bucket.

Slack thread: https://kubernetes.slack.com/archives/CCK68P2Q2/p1628494118056300

[spiffxp]

I'm not opposed to making a new bucket for y'all, but can I ask why this couldn't live in-repo? It feels mildly weird to have two different sources of truth about the state of KEPs, one a git repo, one a GCS bucket.

[spiffxp]

For example, we have some workflows where contributors must run a hack/update-foo.sh locally when contributing a PR, and the CI runs a hack/verify-foo.sh to make sure the file(s) are kept up to date. This is how https://go.k8s.io/github-labels is kept in sync with https://git.k8s.io/test-infra/label_sync/labels.yaml

That places the burden contributor-side though, which isn't super fun.

Another option would be to have a presubmit responsible for adding a commit to the branch that does the update-foo if verify-foo fails. That would need to be done via a github account that has signed the CNCF CLA, and require that contributors allow maintainers to edit their PR branches.

Another option would be to have a bot auto-PR a change to the well-known file in the repo, and use a specific tide query to have it automatically merged. This is the workflow that prow currently uses to auto-update itself.

[sftim]

What I'd imagined is that pushing to the k/enhancements default branch would render / generate a machine-readable KEP list as an artifact, and push that into object storage.

Then the consumer end fetches the JSON from object storage (eg to render a list of KEPs for https://k8s.dev/)

No need for contributors to do the build locally or to commit the generated file. To make things slick we ought to test pull requests as well, making sure that any change that breaks the generation fails some test.

Another option would be to have a bot auto-PR a change to the well-known file in the repo, and use a specific tide query to have it automatically merged. This is the workflow that prow currently uses to auto-update itself.

That works too. A third option is to dedicate an otherwise-empty branch to the generated files. You can then serve the generated files via Git and, if you want to, also via GitHub Pages.

My usual preference for this is to build artifacts from a clone of the default branch, then store those somewhere that isn't Git (this is similar to how you'd handle compiled code, container images, etc).

[sftim]

We might also want to consider reliability. If the generated KEP list is served from something that has good availability guarantees, we can consider fetching it using AJAX. If the primary storage is as a file in Git source control on GitHub, then to me that isn't available enough - GitHub does have outages and doesn't make strong promises about service availability.

[spiffxp]

OK, so I'm cool with creating a bucket, and setting up a service account that can write to that bucket.

For support/debugging purposes, who else should have write access to this bucket? Anything prow touches we are ensuring k8s-infra-prow-oncall@ has full permissions over. But for specific jobs or pieces of infra, we also want a more focused group of people with the domain expertise needed to troubleshoot. I'm also fine if it the answer is "nobody should have access to this, to keep us honest."

I'll have a sample PR up by tomorrow.

[spiffxp]

Or... today. But not deploying today.

• terraform/kubernetes-public: add k8s-keps #2570

[spiffxp]

/priority important-soon
/milestone v1.23