Interference between startup file dependencies and extension triggers

[MilesCranmer]

Copying FluxML/Zygote.jl#1427 here as it seems to be a deeper problem. This issue may be solved by #49701, but it appears to be rooted in the mechanism of Julia reading the Manifest.toml file when checking extensions, so I am not sure if it solves it. I can try to build Julia from source to test it, but I will copy this issue here for now.

---

It seems like there is some interference between packages that load Colors.jl in my startup.jl file, and extensions that rely on Colors.jl - in my case, ProfileView.jl in my startup file, and Zygote.jl's ZygoteColorsExt in my local project.

In particular, if my startup.jl file looks like this:

atreplinit() do repl
    @eval begin
        using ProfileView
    end
end

(ProfileView.jl loads Colors.jl)

and I activate my package DynamicExpressions.jl with ]activate DynamicExpressions.jl, which has Zygote.jl as a dependency (which has ZygoteColorsExt as an extension), but not Colors.jl as a direct dependency, and then run

julia> using DynamicExpressions

I will always see the following output:

[ Info: Precompiling ZygoteColorsExt [e68c091a-8ea5-5ca7-be4f-380657d4ad79]
ERROR: LoadError: ArgumentError: Package Colors [5ae59095-9a9b-59fe-a467-6f913c188581] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1739
 [2] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1625
 [3] macro expansion
   @ ./loading.jl:1613 [inlined]
 [4] macro expansion
   @ ./lock.jl:267 [inlined]
 [5] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1576
 [6] top-level scope
   @ ~/.julia/packages/Zygote/HTsWj/ext/ZygoteColorsExt.jl:5
 [7] include
   @ ./Base.jl:457 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
   @ Base ./loading.jl:2010
 [9] top-level scope
   @ stdin:2
in expression starting at /Users/mcranmer/.julia/packages/Zygote/HTsWj/ext/ZygoteColorsExt.jl:1
in expression starting at stdin:2
┌ Error: Error during loading of extension ZygoteColorsExt of Zygote, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile ZygoteColorsExt [e68c091a-8ea5-5ca7-be4f-380657d4ad79] to "/Users/mcranmer/.julia/compiled/v1.9/ZygoteColorsExt/jl_GgjWAy".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2260
│      [3] compilecache
│        @ ./loading.jl:2127 [inlined]
│      [4] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1770
│      [5] _require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1625
│      [6] _require_prelocked(uuidkey::Base.PkgId)
│        @ Base ./loading.jl:1623
│      [7] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1198
│      [8] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1255
│      [9] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1083
│     [10] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, ocachepath::String, sourcepath::String, depmods::Vector{Any})
│        @ Base ./loading.jl:1363
│     [11] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
│        @ Base ./loading.jl:1459
│     [12] _require(pkg::Base.PkgId, env::String)
│        @ Base ./loading.jl:1748
│     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1625
│     [14] macro expansion
│        @ ./loading.jl:1613 [inlined]
│     [15] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [16] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1576
│     [17] eval
│        @ ./boot.jl:370 [inlined]
│     [18] eval_user_input(ast::Any, backend::REPL.REPLBackend, mod::Module)
│        @ REPL ~/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:153
│     [19] repl_backend_loop(backend::REPL.REPLBackend, get_module::Function)
│        @ REPL ~/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:249
│     [20] start_repl_backend(backend::REPL.REPLBackend, consumer::Any; get_module::Function)
│        @ REPL ~/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:234
│     [21] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool, backend::Any)
│        @ REPL ~/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
│     [22] run_repl(repl::REPL.AbstractREPL, consumer::Any)
│        @ REPL ~/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
│     [23] (::Base.var"#1017#1019"{Bool, Bool, Bool})(REPL::Module)
│        @ Base ./client.jl:421
│     [24] #invokelatest#2
│        @ ./essentials.jl:816 [inlined]
│     [25] invokelatest
│        @ ./essentials.jl:813 [inlined]
│     [26] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
│        @ Base ./client.jl:405
│     [27] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:322
│     [28] _start()
│        @ Base ./client.jl:522
└ @ Base loading.jl:1204

I have tried removing my entire Julia directory, and reinstalling from scratch, but I get the same error. I haven't been able to build a smaller MWE. This issue seems to be with the fact that Colors.jl is loaded by my startup file (via ProfileView.jl), but is not in my project's Manifest.toml, which might be interfering with the extension...

More clues:

1. If I manually add Colors.jl to my package, then everything works and the ZygoteColorsExt compiles without errors.
2. If I launch Julia with --startup-file=no, then the error goes away.
3. If I do successfully compile this extension, even by temporarily adding Colors.jl to my project's Project.toml/Manifest.toml, then the error will not show up again. This happens even if I remove Colors.jl from my project's Project.toml/Manifest.toml. It is only after I delete ~/.julia/compiled/v1.9/ZygoteColorsExt does the error come back.

---

julia> versioninfo()
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 6 on 6 virtual cores
Environment:
  JULIA_NUM_THREADS = auto
  JULIA_FORMATTER_SO = /Users/mcranmer/julia_formatter.so

[KristofferC]

Can you try on this branch as well #49680?

[MilesCranmer]

Awesome, fixed on backports-release-1.9! Thanks