Workers should inherit Pkg environment

[rened]

(Moved over from JuliaLang/Pkg.jl#675)

If I have a project where I installed MyPackage and I can say

julia --project -e "using MyPackage"

the following does only work when I manually activate the environment:

using Distributed
using MyPackage   # works
addprocs(3)
@everywhere
  using Pkg; Pkg.activate(".")  # required
  using MyPackage
  ...
end

I would suggest that workers inherit the environment that is active when they are added with addprocs. Should there be any calls to Pkg.activate after addprocs they have to be manually performed by the user on all workers.

[affans]

Can you explain a little bit more to whats happening? I have a large codebase I want to port to 1.0 that uses HPC/addprocs quite a bit. If I know better what is happening, maybe I can submit a PR

[JLTastet]

I just got bitten by this issue while I was porting a small project from 0.6 to 1.0. As soon as I started using a separate environment for the project, worker processes started to throw Package ... is required but does not seem to be installed exceptions.

In addition to @rened's suggestion, I would also suggest that (MyProject) activate . activates the environment on worker processes as well, or at least issues a warning instead of silently ignoring them. This would be useful when starting Julia with the -p <N> option and then manually activating the environment.

[JLTastet]

@affans It seems that (MyProject) activate . and Pkg.activate(".") only activate the environment on the master process. To activate it on all processes, you would need to use the following code:

using Distributed
@everywhere using Pkg
@everywhere Pkg.activate(".")

You can then use @everywhere using MyPackage as usual.

[StefanKarpinski]

This requires some careful thinking about how best to do this. One possibility is to send a manifest from the master node to the workers and insist that the manifest be usable on all nodes. That's fine for non-dev packages and even for dev packages with relative paths but dev packages with absolute paths may be a bit of an issue. We could rewrite dev paths to use a commit instead and then send that over; of course, it requires that the workers know about the tree hash that's being used...

[matteoacrossi]

I ran into the same problem and it took me a while to understand that the problem was with activate. So I believe there should at least be a warning when using activate with worker processes.

[simonbyrne]

Another work around is to use the JULIA_PROJECT environment variable instead of --project, as that will be passed to subprocesses.

[StefanKarpinski]

For now, that's the recommended way to do this.

[oxinabox]

One thing one wants to be clear on is which enviroment a worker should be getting.
If a Package is startng a worker, then the worker should probably have that packages enviroment.
not the enviroment that the main process is running in.

As in: malmaud/TensorFlow.jl#493

[oxinabox]

Oh in the case of a package that owns a worker and want it it use it's environment it is even more complicated. Since even if the package created the worker,
It will still exist globally, and so when any using is run in the main process, the worker will attempt to load the module -- which will be from the main environment.

So workers need both the packages environment and the main environment.
I think this can be done with LOAD_PATH for nested environments right?