run `pkg>` operations in subprocess?

[StefanKarpinski]

@timholy has noted that a lot of Pkg code gets invalided by new methods. That got me thinking: we don't need to load Pkg most of the time at all and when we run commands in the pkg> repl mode, we don't even need to do that in the main Julia process: we could have a separate Julia+Pkg sysimg just for running Pkg commands.

[KristofferC]

FWIW, I think quite a lot of the invalidations have gotten fixed. Also, you need Pkg in the sysimage for things to run quickly in a new process but the fact that Pkg is in the sysimage means that it is already loaded when you start Julia.

[StefanKarpinski]

we could have a separate Julia+Pkg sysimg just for running Pkg commands.

I.e. normal sysimg does not include Pkg whereas pkg> commands start a Julia process with a separate sysimg that includes Pkg but in which you never load any other packages.

[KristofferC]

Oh missed that. That sysimage would have to include the dependencies as well. So we would have some "mini-Pkg" in the normal sysimage that keeps the state that is kept between calls and start-up new processes that pass over that state and get results back (exceptions etc) and handle those? I think this would complicate the code and make it hard to debug quite a bit.

And after all, Pkg is just a normal package so even if we do all this, the same problem will happen for all other packages, and rewriting the world to run in separate processes (á la microservices) is of course not feasible. So probably just have to deal with the invalidation problem.

[timholy]

Invalidations from Revise got entirely eliminated. Invalidations from FixedPointNumbers and probably others have not. But there may be systematic changes to the invalidation logic that will eliminate or reduce the invalidations from those packages.

[StefanKarpinski]

It's entirely possible this isn't a good idea but it came up on Slack and I didn't want to lose it into the Slackhole, so I opened this issue to discuss.

[c42f]

Another rather speculative option would be to run pkg in a fixed world - I've recently been adding a builtin for that in the context of the parser work (JuliaLang/julia#35243). But I will say that this has consequences we might not like or expect. (As one example, it doesn't interact well with installing alternative loggers, which I think is one of the big reasons for method invalidations recently worked around by Tim. But there's other unexpected things which can break; I've become so used to being able to extend anything that turning this off can lead to various surprises.)

[KristofferC]

Revise also doesn't work then :)

[c42f]

Revise also doesn't work then :)

Not necessarily. The world age is dynamically scoped, so running in a fixed world would only happen if the Pkg code was entered through an entry point which fixes the world age. For example, you could declare that pkg REPL mode does this, but not when using Pkg via the API.

[c42f]

PR for _apply_in_world is up: JuliaLang/julia#35844. Would be interesting to see whether this helps with pkg repl mode.

[KristofferC]

I don't think we need to do much here. Most of the invalidations are fixed and Pkg is not really any different from any other stdlib w.r.t this issue.