Use a conda lock to create the env only once a week

[valeriupredoi]

@zklaus pointed me to this idea (cheers, dude!) the iris guys are implementing in SciTools/iris-esmf-regrid#68 and with the original and nice discussion in SciTools/iris#4108 - basically use a lockfile to generate and store the deps env in and do this on a periodic basis, but less than daily. Discussion point: do we want such a feature as well? I'd argue that it is useful to have it only as a lifeboat if the env in CI tests/user source installs fails to solve, but otherwise don't rely on it on a regular basis coz a week of old deps can bring us havoc the next week - or can it? I'd argue this would be needed in the case of ESMValTool where the env is hairy, and not Core but I'm opening the issue here for a technical discussion @ESMValGroup/esmvaltool-coreteam

[zklaus]

There is a nice overview of the advantages in SciTools/iris-esmf-regrid#68. Basically, by using a lock file, you completely avoid the solving stage. This is opposed to merely specifying all versions in an environment file, where conda will still perform a solve to create a consistent environment with all dependencies of dependencies.

Regardless of how often the update would happen, having it as a discrete, atomic thing is advantageous because the problems that do crop up are not mixed in with other changes in an unrelated PR. That also makes it possible to follow the changes in dependencies through a series of clear commits.

If we find that the changes over one week regularly are too numerous to deal with, we can increase the frequency. There will probably be no need to decrease the frequency because no change happens if there is no change in dependencies.

[valeriupredoi]

sounds good, even though I don't like atomic stuff (failed my Adv Quantum Mechanics in college 😁 ) - I'll have a start at implementing this sooner than later 👍

[bouweandela]

I'm not sure we need this, because we use Docker containers built from the main branch with already complete environments to run our tests on regular pull requests. The only time we need to solve an environment is for the nightly tests or when dependencies change.

[valeriupredoi]

yeah @bouweandela - that's exactly where locks come in hand 👍

[bouweandela]

You can't use the lock file when the dependencies change, you'll need to regenerate it then. The nightly tests are also meant as a way to spot problems with updated versions of packages early, if you run them from an environment lock file there may not be much added value left. But maybe I'm overlooking something? Of course they do test if a development installation works fine..