Conflicting environments are handled incorrectly

[timholy]

I'm reporting this to julia itself (rather than in Pkg.jl) because key components of the fix seemingly must be made in base/loading.jl. No objections, though, if folks want to transfer this to Pkg.jl.

This bug comes in two closely-related flavors.

Flavor 1: switching between incompatible environments should be disallowed

Suppose I have two projects with incompatible [compat] requirements:

Proj1.toml:

name = "Proj1"
uuid = "61c9ae6d-79d8-47c4-9a39-97bb6365db78"
authors = ["Tim Holy <tim.holy@gmail.com>"]
version = "0.1.0"

[deps]
ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"

[compat]
ColorTypes = "0.8"

Proj2.toml:

name = "Proj2"
uuid = "7bf079ca-24d6-42d5-9ea6-67ba343487f2"
authors = ["Tim Holy <tim.holy@gmail.com>"]
version = "0.1.0"

[deps]
ColorTypes = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"

[compat]
ColorTypes = "0.10"
Colors = "0.12"

Now let's use these in a Julia session:

tim@diva:/tmp/envs/Proj1$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.2-pre.0 (2020-04-15)
 _/ |\__'_|_|_|\__'_|  |  release-1.4/ef4fe83698* (fork: 122 commits, 121 days)
|__/                   |

julia> using ColorTypes

shell> cd ../Proj2/
/tmp/envs/Proj2

(Proj1) pkg> activate .
 Activating environment at `/tmp/envs/Proj2/Project.toml`

julia> using Colors
[ Info: Precompiling Colors [5ae59095-9a9b-59fe-a467-6f913c188581]
ERROR: LoadError: UndefVarError: XRGB not defined
Stacktrace:
 [1] top-level scope at /home/tim/.julia/packages/Colors/k4h4b/src/Colors.jl:7 (repeats 2 times)
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] top-level scope at none:2
 [4] eval at ./boot.jl:331 [inlined]
 [5] eval(::Expr) at ./client.jl:449
 [6] top-level scope at ./none:3
in expression starting at /home/tim/.julia/packages/Colors/k4h4b/src/Colors.jl:7
ERROR: Failed to precompile Colors [5ae59095-9a9b-59fe-a467-6f913c188581] to /home/tim/.julia/compiled/v1.4/Colors/NKjaT_sPqJx.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029
 [4] require(::Base.PkgId) at ./loading.jl:927
 [5] require(::Module, ::Symbol) at ./loading.jl:922
 [6] eval(::Module, ::Any) at ./boot.jl:331
 [7] eval_user_input(::Any, ::REPL.REPLBackend) at /home/tim/src/julia-1/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
 [8] run_backend(::REPL.REPLBackend) at /home/tim/.julia/packages/Revise/WkyNB/src/Revise.jl:1023
 [9] top-level scope at none:0

julia> 

That error is very confusing for the developer. XRGB was defined in ColorTypes v0.10, and I've declared that Colors v0.12 depends on it. So how can XRGB be undefined? The key information---that I've switched environments---is rarely reported by users. (timholy/Revise.jl#468, JuliaIO/QuartzImageIO.jl#62)

To me, it seems that the right way to handle it is to throw

(Proj1) pkg> activate .
 Activating environment at `/tmp/envs/Proj2/Project.toml`
ERROR: `/tmp/envs/Proj2/Project.toml` depends on ColorTypes v0.10, but ColorTypes v0.8.1 is already loaded.
Consider starting a fresh Julia session in this environment.

AFAICT, currently we don't save any information about the version of loaded packages. This is why I suspect we need to fix this partially in base/loading.jl.

Flavor 2: creation of new environments must preserve session compatibility

If you've loaded one version of a package, creation of new environments should probably write compatible version info into the Manifest of a newly-created environment.

(@v1.4) pkg> generate Proj1
 Generating  project Proj1:
    Proj1/Project.toml
    Proj1/src/Proj1.jl

shell> cd Proj1/
/tmp/envs/Proj1

(@v1.4) pkg> activate .
 Activating environment at `/tmp/envs/Proj1/Project.toml`

(Proj1) pkg> add ColorTypes@0.8
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `git@github.com:JuliaRegistries/General.git`
   Updating registry at `~/.julia/registries/HolyLabRegistry`
   Updating git-repo `git@github.com:HolyLab/HolyLabRegistry.git`
  Resolving package versions...
   Updating `/tmp/envs/Proj1/Project.toml`
  [3da002f7] + ColorTypes v0.8.1
   Updating `/tmp/envs/Proj1/Manifest.toml`
  [3da002f7] + ColorTypes v0.8.1
  [53c48c17] + FixedPointNumbers v0.7.1
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 

julia> using ColorTypes
[ Info: Precompiling ColorTypes [3da002f7-5984-5a60-b8a6-cbb66c0b333f]

shell> cd ..
/tmp/envs

(Proj1) pkg> generate Proj2
 Generating  project Proj2:
    Proj2/Project.toml
    Proj2/src/Proj2.jl

shell> cd Proj2/
/tmp/envs/Proj2

(Proj1) pkg> activate .
 Activating environment at `/tmp/envs/Proj2/Project.toml`

(Proj2) pkg> add ColorTypes
  Resolving package versions...
   Updating `/tmp/envs/Proj2/Project.toml`
  [3da002f7] + ColorTypes v0.10.2
   Updating `/tmp/envs/Proj2/Manifest.toml`
  [3da002f7] + ColorTypes v0.10.2
  [53c48c17] + FixedPointNumbers v0.8.0
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 

(Proj2) pkg> add Colors
  Resolving package versions...
   Updating `/tmp/envs/Proj2/Project.toml`
  [5ae59095] + Colors v0.12.0
   Updating `/tmp/envs/Proj2/Manifest.toml`
  [5ae59095] + Colors v0.12.0
  [189a3867] + Reexport v0.2.0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [ea8e919c] + SHA 
  [6462fe0b] + Sockets 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

julia> using Colors
[ Info: Precompiling Colors [5ae59095-9a9b-59fe-a467-6f913c188581]
ERROR: LoadError: UndefVarError: XRGB not defined
Stacktrace:
 [1] top-level scope at /home/tim/.julia/packages/Colors/k4h4b/src/Colors.jl:7 (repeats 2 times)
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] top-level scope at none:2
 [4] eval at ./boot.jl:331 [inlined]
 [5] eval(::Expr) at ./client.jl:449
 [6] top-level scope at ./none:3
in expression starting at /home/tim/.julia/packages/Colors/k4h4b/src/Colors.jl:7
ERROR: Failed to precompile Colors [5ae59095-9a9b-59fe-a467-6f913c188581] to /home/tim/.julia/compiled/v1.4/Colors/NKjaT_sPqJx.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029
 [4] require(::Base.PkgId) at ./loading.jl:927
 [5] require(::Module, ::Symbol) at ./loading.jl:922

It would be a little weird that a new project gets affected by the environment of an unrelated package, but until we can load multiple versions of the same module I don't see a good alternative.

[KristofferC]

Just to give a third flavor where you get a compat problem without switching project after loading a package:

export JULIA_DEPOT_PATH="/tmp/.julia"
julia -q
(v1.4) pkg> add Colors@0.12
...
   Updating `/private/tmp/.julia/environments/v1.4/Project.toml`
  [5ae59095] + Colors v0.12.0
   Updating `/private/tmp/.julia/environments/v1.4/Manifest.toml`
  [3da002f7] + ColorTypes v0.10.2
  [5ae59095] + Colors v0.12.0
...

(v1.4) pkg> activate CustomProject
 Activating new environment at `/private/tmp/envs/CustomProject/Project.toml`

(CustomProject) pkg> add ColorTypes@0.8
...
  [3da002f7] + ColorTypes v0.8.1 # Colors is incompatible with this
   Updating `/private/tmp/envs/CustomProject/Manifest.toml`
  [3da002f7] + ColorTypes v0.8.1
  [53c48c17] + FixedPointNumbers v0.7.1

julia> using Colors # this is the first package load
[ Info: Precompiling Colors [5ae59095-9a9b-59fe-a467-6f913c188581]
ERROR: LoadError: UndefVarError: XRGB not defined

The reason for the error here is that while Colors is loaded from the global project, it pulls in dependencies from the manifest of the active project and these might be incompatible.

[KristofferC]

Note that the behavior in my last comment is explicitly documented https://docs.julialang.org/en/v1/manual/code-loading/#Environment-stacks-1Th but I thought it was worth pointing out anyway since it is related.

• Packages in non-primary environments can end up using incompatible versions of their dependencies even if their own environments are entirely compatible. This can happen when one of their dependencies is shadowed by a version in an earlier environment in the stack (either by graph or path, or both).

Since the primary environment is typically the environment of a project you're working on, while environments later in the stack contain additional tools, this is the right trade-off: it's better to break your development tools but keep the project working. When such incompatibilities occur, you'll typically want to upgrade your dev tools to versions that are compatible with the main project.

[timholy]

Interesting. One could modify my suggestion in the OP to warn users about the incompatibility only if a package operation fails. OTOH, this would fail to catch cases where there is no compile-fatal mismatch, only a functionality-correctness-fatal mismatch. I worry the last class of problems is really hard to debug properly.

[KristofferC]

I think it would at least make sense to warn when loading a package and that package is already loaded from a different path than where it would be otherwise be loaded from.

[StefanKarpinski]

I think that warning upon activating a different environment (i.e. Flavor 1) is the most viable option. I'm don't think that putting a warning in the code loading path when an earlier stack environment shadows the dependency of a package from later in the stack since that is fine 95% of the time and just works. That seems like it's going to be a very common warning that people will just learn to ignore and will become a pointless annoyance that makes Julia look janky—like method ambiguity warnings of old.

[StefanKarpinski]

While warning when a dependency is already loaded from somewhere other than where it would be loaded if a later environment was used in isolation would be too fiddly, it might be ok if we recorded compat info in manifests when resolving them and checked if the already-loaded version is within the compat bounds or not. Recording in the manifest is helpful since otherwise you have to look in potentially a lot of random places to figure out what the compat bounds are, whereas you already had to find a dependency in the manifest in order to know what to load, so we're already doing that work mostly.

[tkf]

How about manually editing LOAD_PATH and DEPOT_PATH? Isn't it easier to deal with it if you warn while loading?

[StefanKarpinski]

Code loading needs to be pretty simple: we are already pushing the limits of what it's reasonable to do while loading code. Anything that avoids complicating code loading further is better.

[tkf]

Isn't it impossible to detect LOAD_PATH and DEPOT_PAT manually edited by the user other than during code loading? Or do you mean to create a custom vector type so that it can be detected when, e.g., a new element is pushfirst!-ed? Or just not support conflict detection in such case?