RFC-0040 - Proposal: Bring your own Stack

[FloThinksPi]

RFC-0040

Branched out Discussion to review RFC-0040 - Proposal: Bring your own Stack

[FloThinksPi]

Continuing the conversation concerning custom stacks from @rkoster #1220 (comment)

I understand the desire to fix this, but maybe a compromise could be found in making stack management an admin only feature (using the cf cli). Ideally using a blobstore first approach so it works in airgapped environments.

We could extend the feature flag to have 3 modes - off, admin-only and on.
Also we could allow an admin to set a docker://URL stack as system stack in the stacks table. This one can only be done by admins.
So then this feature to consume a stack from the outside could actually be an admin only thing. However not sure how much value this adds for an operator/admin. Note that by default the feature-flag anyway will be disabled so an operator firstly has to decide to provide this function in his foundation deliberately. In an airgapped environment an operator would not enable this feature i guess since its pointless. When they would run their own container registry in that environment they could also just use the usual bosh mechanism to deploy the stack onto the diego cells anyway.

On a higher level I'm also worried about handing responsibility for stacks in the hands of app developers. One of the great features of CF is that platform operators can fix CVEs in for example OpenSSL with a single bosh deploy.

I also thought intensively over that, in the end we actually already allowed this when we introduced "bring your own buildpack" to some degree. If you run your own buildpack - or even a system one - you anyway have to regularly restage to consume patches to your buildpack or languages libs. In case your application does not do dependency version pinning at all or does just pin direct dependencies but not transitive ones. E.g. patched log4j libs as an example everyone might know of. Even though CF gives you some things automatically in my experience users didn't know that sometimes they have to restage to consume new libs as they (naively)thought the system takes care if it. For example take the python buildpack - you only get a CVE patched in the python interpreter if you restage! Same goes for the Go Compiler, Ruby or Java JRE.

What may thus be desirable when allowing freedom like custom buildpacks or custom stacks is to explain the boundary conditions very well and make them transparent to the user. Can be maybe added to the RFC/a new one that we make this more transparent than today either in the API with a special flag(e.g. security-lifecycle-status: or something like that that shows you a restage is required) visible in cf apps or as Documentation.

• System Buildpack ➕ System Stack ➡️ You regularly have to restage to deploy new updates of the buildpack or you apps dependencies(if not pinned)
• Custom Buildpack ➕ System Stack ➡️ You regularly have to bump the buildpack version AND have to restage to deploy new updates to the buildpack or you apps dependencies(if not pinned)
• Custom Buildpack ➕ Custom Stack ➡️ You regularly have to bump the buildpack version AND the Stack version AND have to restage to deploy new updates of the buildpack or you apps dependencies(if not pinned)

Deliberately in the RFC only Custom Stack + Custom buildpack shall be supported so bring your own everything. And firstly the operator has to allow its users with the feature flag to use this feature. Secondly a user has to willingly move away from the system buildpack and system stack - this cannot be an accidental thing. So if better documented i think if a user wants to deliberately care for his buildpack and stack an operator may be enabled to allow him that. As an operator of a foundation compliance can still be validated with a scan over the CF API programmatically in case the operator is responsible for the apps on the foundation. Or decide to not enable that feature at all in case he as concerns.

Last but not least I'm wondering about the roll out mechanisms for handeling custom stack updates, given these changes are not orchestrated through bosh, how do we ensure we don't over we Diego? There needs to be some sort of global max_in_flight setting to control the total number of apps that are being restarted at the same time.

Since we run at large scale with the docker feature flag enabled on our Foundations we got quite some experience already. In CF the custom stack proposal from a diego point of view is nothing else than a docker app. Diego is - as outlined in the RFC - unware of lifecycles thats a CF API concept. It just knows LRPs and Tasks with either a baseLayer beeing on the disk or from a container registry see https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md maybe i missed something here but that means we can derive the behaviour of custom stacks for diego from the already known behaviour of docker apps withing CF since from a diego point of view they are identical.
Also the update behaviour of the stack is the same as with lifecycle docker. E.g. if you specify a container with tag latest on a app restart you run with the newest stack similar as when we bump the system stack currently.
We simply just allow to combine lifecycle docker and buildpack in the sense that the CF API schedules a staging task with a baseLayer from a container registry.

Thinking about that now - maybe it makes sense for the feature flag for custom stacks to be only activate-able when diego_docker https://docs.cloudfoundry.org/adminguide/docker.html feature flag is also set. Opinions on that ?