[feature] Install modules with their dependency

[iglov]

Maybe we can do it? ^^
P.S. I'm not a programmer >>

[andersonmills]

This is in the long term wishes and desires for r10k (and the PE product, code-manager). There are some significant issues with it, unfortunately. Whether this work will be prioritized at Puppet is currently in review.

[bittner]

@andersonmills Any news on this?

[andersonmills]

I'm sorry to say, nothing recent.

There are some significant issues with it

What are the significant issues? If they could be listed here, perhaps someone would be willing to tackle them and send an MR?

Would there be any reason not to use librarian to manage modules and their dependencies, the way librarian-puppet does?

[bittner]

Re "significant issues", this would be interesting to have explained. I'd say this is fairly simple, the existing tools know how to do it.

I could demonstrate the rather stable, albeit slow, deployment process for a control-repo we have implemented that combines r10k and puppet, and allows the Puppetfile to contain only the essential dependencies.

• Everything that is dependent from those dependencies is pulled in by puppet.
• Everything that is not required (anymore) is cleaned out by r10k beforehand.

The Process Outlined

1. Run r10k to deploy the environment(s). -- This will remove all dependencies except the ones explicitly listed in the Puppetfile
2. Compute missing dependencies (via puppet module list --tree), and reinstall them (via puppet module install ...)
3. Verify whether there are any packages marked "invalid" (via puppet module list --tree again), and force-reinstall them (via puppet module uninstall --force ... && puppet module install ...)
4. Verify that all packages are installed correctly (via puppet module list --tree, e.g. by grepping for UNMET DEPENDENCY)

That's pretty easy and works super-stable! For months now.

It's a bit slow though, as I said, because - clearly - there is some unneeded uninstallation and re-installation of r10k and puppet module going on, which could be avoided in a combined approach.

@andersonmills So, what are the "significant issues" you're talking about?

[andersonmills]

I apologize for the ambiguous "significant issues." I'm stuck a bit, in that, as a Puppet employee, I don't want to expose deficiencies we want to correct with our products, even the open source ones. I also don't want to make any promises, because, though I'm an engineer and want to solve problems, I'm not the deciding voice in where the company puts its effort This just hasn't been getting attention, as is probably clear.

That said, I think the biggest issue is obvious. r10k needs a dependency resolution engine of some sort. Unfortunately, we can't use the one from puppet module (tool) (PMT). Librarian might be the answer, but I haven't looked into the options.

[bittner]

BTW, some other attempt to resolve the issue is is @rnelson0's puppet-generate-puppetfile. You need to version control all your resolved dependencies with this though, which entails a dependency management overhead.

This shows, however, that the solutions are out there. Puppet Labs only needs to decide where to do the copy+paste from (kind of).

[rnelson0]

@bittner

Re "significant issues", this would be interesting to have explained. I'd say this is fairly simple, the existing tools know how to do it.

It is not unheard of for librarian-puppet to get stuck in a dependency loop chain and never complete, which is what r10k is most often compared to. It's a tricky problem - especially if two modules have different requirements for the version of their dependent modules.

There's also the issue that a user may not WANT a dependent module, because there's no concept of Hard and Soft Dependencies (or Mandatory and Optional, etc). For instance, I run entirely EL-based linux VMs, I don't manage any Debian, Gentoo, or Arch based systems. Yet some of my modules' metadata lists requirements for apt and portage modules. Without a hard/soft flag for those, automatic dependency trackers will import all of those modules when a user has no need for them. Any solution will need to address this, too, which probably requires updates to the metadata.json schema in addition to improved tooling.

Also, I think your listed process makes some assumptions about access to the dependent modules. In an offline environment, those modules may not be available directly or on a forge mirror, so the solution would a way to signal no dependency resolution is desired.

BTW, some other attempt to resolve the issue is is @rnelson0's puppet-generate-puppetfile. You need to version control all your resolved dependencies with this though, which entails a dependency management overhead.

My tool relies on PMT to resolve dependencies, and that is ... not optimal. First, it actually involves running puppet module install $module over and over, which is very slow. Second, it iterates through the modules in the listed order. Let's pretend your Puppetfile lists the modules alphabetically. If you have a module A with a dependency on module X version >2.0.0, and a module Z with dependency X version >2.0.0 and <2.5.0, it can cause a problem if X version 2.5.0 or greater is available. Module A will install X version 2.5.0 and then installation of Z fails because it cannot meet the dependency (PMT does not automatically downgrade dependencies). Does the user rely on functionality for X v2.5.0, or would v2.4.0 work just as well? Only the user can tell.

This is a huge reason why pinning is important, and emphasized with my tool. Yes, it's a PITA. No, it's not fun. Yes, it helps prevent accidental regressions and conflicts. No, it's not perfect. While I really hope that people find the tool helpful, and I really do recommend it, I think it's only a stop gap on the way to a better tooling chain. Please use it, please pin your modules, but we need to look at something completely different for integration with r10k and other tools.

I think it's more likely that the algorithms and libraries used by the PMT need abstracted out into a tool kit of their own. If I understand the PMT code better - and had more time to work on it - I would absolutely use it to replace the guts of puppet-generate-puppetfile. At that point, it might be a good solution for integration with r10k, but I really presume that someone else will do this work before me (PRs always welcome of course!)

[github-actions]

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

[iglov]

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

bump.

[github-actions]

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.