How to list bob layers?

[mvAtWork]

Hi,

I'm currently working on a project that still uses git submodules as layers. The switch to managed layers is planned but still a ways off.

My use case is CI-related. When building this project in the CI, I handle the initial git checkout manually because the current build may require changes that are spread over several SCMs, e.g. a merge request in the main repo which can only build successfully with the changes from two other merge requests, one from a repo in a layer and one from a repo that a recipe depends on.

Assume I already have a method to define these dependencies and can process them in the CI. I then need to distinguish between the SCMs that will be checked out by bob, for which I will write the appropriate scmOverrides, and the layers that I need to check out manually, i.e. do a submodule update. I can of course try to parse the layers from the config.yaml, but with possibly present nested layers and/or overrides this becomes very complicated very quickly.

So my main question is: is there a way to list the layers that bob knows about incl. nested layers? Ideally, this would work for managed as well as unmanaged layers. With managed layers enabled, bob layers status shows no output for the unmanaged layers. I think showing the mere existence of the layers with their path would suffice for my purpose.

I would greatly appreciate any insights and pointers, thanks very much. :)

(Side note: in the future this distinction between layers and other SCMs will probably still be necessary to figure out whether to add an entry to scmOverrides or layersScmOverrides.)

Versions

• Bob: 0.25.0
• Python: 3.11.5

[jkloetzke]

I think there are different aspects to this question.

First of all, you're right that there is currently no nice way to list layers. I guess this can be improved rather easily. The main question is, what information should be listed is addition to the layer name and in which format. Some machine-readable format(s) are probably advisable.

But for your use case there might be a chicken-and-egg problem. Bob iteratively checks out nested layers. So it is not always possible to list all layers before having fetched the first batch of layers. Could that be a problem? Do we need to make this "interruptible" so that you get the chance to amend the layersScmOverrides?

[mvAtWork]

Sorry, I forgot to mention, this is currently a work in progress, everything I wrote includes some guesswork from me.

Yes to machine-readable format, I will have to parse the output. For the unmanaged layers, the layer name and its location in the file system should be enough. I think it would be asking a bit much for bob to figure out the SCM for a layer that it didn't put there itself. ;)
For managed layers, I assume I would need the layer name and location as well as some information about the attached SCM. At least the repo url would be necessary, current branch and/or commit SHA would be nice-to-have.

I see what you mean with the chicken-and-egg problem. It is basically the same problem I'm facing right now, not knowing what branch/commit to check out before everything was checked out. Here's the way I'm currently trying to do this:

• check out the main repo with --recurse-submodules
• get the dependencies for the merge request
• go through all submodules and, if there's a match in the merge request dependencies, check out the new branch/commit
• all remaining merge request dependencies are written into scmOverrides

So I think it should be fine for bob to do basically the same: do the full checkout of all layers, then report what layers it found (managed as well as unmanaged). I would then generate the appropriate layersScmOverrides and have bob check out the (possibly overridden) layers again.

I realize this approach might fall short, if the changes in the merge requests include the addition (or removal?) of a layer, but I assume this case would be so rare that we can ignore this for now.

[jkloetzke]

For managed layers, I assume I would need the layer name and location as well as some information about the attached SCM. At least the repo url would be necessary, current branch and/or commit SHA would be nice-to-have.

That should not be too hard.

So I think it should be fine for bob to do basically the same: do the full checkout of all layers, then report what layers it found (managed as well as unmanaged). I would then generate the appropriate layersScmOverrides and have bob check out the (possibly overridden) layers again.

I realize this approach might fall short, if the changes in the merge requests include the addition (or removal?) of a layer, but I assume this case would be so rare that we can ignore this for now.

I guess it boils down to make a loop between the invocation of Bob and the update of the layersScmOverrides. You do that until the output of bob layers ls does not change any more. That should catch all the cases.

I've created #631 to track the missing feature and marked it for the next release...

[mvAtWork]

Thank you very much for the quick work. This looks very promising. 😃

I've checked the new layers ls command on a project with unmanaged nested layers and get this output:

second_layer:
    disposition: unmanaged
    path: layers\second_layer
first_layer:
    disposition: unmanaged
    path: layers\first_layer

Would it be possible to show the path relative to the root project? The path of the second layer is actually <root dir>\layers\first_layer\layers\second_layer. => see edit

Also, what does the -lc LAYERCONFIG option do? Can I use this for (iteratively) adding the layersScmOverrides?

---

Edit: I'm not sure how all this (just listing the layers as well as showing the path relative to root) would work for layers that appear multiple times. In the example project I've just now added the second layer to the root project at <root dir>\layers\second_layer and I still get the same output as shown above.

Note that this is a bit of a theoretical example. I'm not sure if we even have such projects where a layer is referenced twice (directly and indirectly) or if such projects would even be able to build without errors.

[jkloetzke]

Also, what does the -lc LAYERCONFIG option do? Can I use this for (iteratively) adding the layersScmOverrides?

Yes. To override layer SCMs, you have to provide the layersScmOverrides through the -lc options. The reason is that the "normal" configuration file (default.yaml) can only be parsed after all layers have been fetched. To solve this chicken-and-egg problem, there is a dedicated configuration file for layers.

Edit: I'm not sure how all this (just listing the layers as well as showing the path relative to root) would work for layers that appear multiple times. In the example project I've just now added the second layer to the root project at <root dir>\layers\second_layer and I still get the same output as shown above.

I guess you have enabled the managedLayers policy without being fully aware of the implications. Hopefully, the documentation gives some insight. In short: historically, such nested layers were supported but they proved to be useless. With the introduction of managed layers, we additionally converted them to a directed graph and they are checked out flat next to each other in the layers directory.

But I just realised that the layers ls and layers status command ignore missing unmanaged layers. Not sure if this is a good idea. It for sure looks like a bug for the layers status command.