`] build PyCall` does not work on NixOS

[samuela]

I'm running into some unexpected hiccups building PyCall.jl on NixOS. I have successfully used PyCall.jl on NixOS in the past, so I'm a bit stumped as to why it's borked now. I get the following failure:

(@v1.7) pkg> build PyCall
    Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6cdc8832ba11c7695f494c9d9a1c31e90959ce0f/build.log`
    Building PyCall → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/71fd4022ecd0c6d20180e23ff1b3e05a143959c2/build.log`
ERROR: Error building `PyCall`:
PREFIX=/home/skainswo/.julia/conda/3
Unpacking payload ...
cannot execute /home/skainswo/.julia/conda/3/conda.exe: NIX_LD or NIX_LD_x86_64-linux is not set
cannot execute /home/skainswo/.julia/conda/3/conda.exe: NIX_LD or NIX_LD_x86_64-linux is not set
┌ Info: No system-wide Python was found; got the following error:
│ Base.IOError("could not spawn setenv(`/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\"`,[\"CXX=g++\", \"PATH=/nix/store/79zvzq15mga6bgk5phnmw0fd2pnpim9v-bash-interactive-5.1-p8/bin:/nix/store/f1n2zyq47sjn0qfhxvrdawyzlgk4sgax-patchelf-0.13/bin:/nix/store/7wn7xx8faqirzkvh5iwj26b024jzg09p-gcc-wrapper-10.3.0/bin:/nix/store/v819nrv8d33ns36gm4v9vqydq1v95axi-gcc-10.3.0/bin:/nix/store/dhj23pp1sld19shjslkzw3y9pr3qpi61-glibc-2.33-59-bin/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/gxd0gvw88wyl9bqjc03ypm5zs7qbaf74-binutils-wrapper-2.35.2/bin:/nix/store/8d30avp4xj1w8krq7zax1mlxgs28663y-binutils-2.35.2/bin:/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin:/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/r87rxnys39173q5nrlqq46vpa2203rmk-findutils-4.8.0/bin:/nix/store/xgqrjhz9sx8bllvayz18sql2dfd89q4g-diffutils-3.8/bin:/nix/store/aw3vgxn4fdrjqgmc94fr582hjhr2z3p3-gnused-4.8/bin:/nix/store/rvdb7057bmg3rasbqq8lzl10dffy3p31-gnugrep-3.7/bin:/nix/store/3qmgmnz3msrk0cf0qgcyja0wzyfhpxc2-gawk-5.1.1/bin:/nix/store/xc1dbg8mc8hd6bq7zizxf5nyrljk8az0-gnutar-1.34/bin:/nix/store/1wads5y31dky89hxnn5zmiri3c72kdpc-gzip-1.11/bin:/nix/store/l8d7zz3ninmgg71738q485anfjh5yd6q-bzip2-1.0.6.0.2-bin/bin:/nix/store/ba439awj33pw4l4bj95k4dy58nx69b8v-gnumake-4.3/bin:/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin:/nix/store/22avws4xw0chqniqym3q0w4nhdcr0hh2-patch-2.7.6/bin:/nix/store/aq0v28n1ka8m1ayg6wrpbjy00bb7fj3k-xz-5.2.5-bin/bin\", \"HOST_PATH=/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin:/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/r87rxnys39173q5nrlqq46vpa2203rmk-findutils-4.8.0/bin:/nix/store/xgqrjhz9sx8bllvayz18sql2dfd89q4g-diffutils-3.8/bin:/nix/store/aw3vgxn4fdrjqgmc94fr582hjhr2z3p3-gnused-4.8/bin:/nix/store/rvdb7057bmg3rasbqq8lzl10dffy3p31-gnugrep-3.7/bin:/nix/store/3qmgmnz3msrk0cf0qgcyja0wzyfhpxc2-gawk-5.1.1/bin:/nix/store/xc1dbg8mc8hd6bq7zizxf5nyrljk8az0-gnutar-1.34/bin:/nix/store/1wads5y31dky89hxnn5zmiri3c72kdpc-gzip-1.11/bin:/nix/store/l8d7zz3ninmgg71738q485anfjh5yd6q-bzip2-1.0.6.0.2-bin/bin:/nix/store/ba439awj33pw4l4bj95k4dy58nx69b8v-gnumake-4.3/bin:/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin:/nix/store/22avws4xw0chqniqym3q0w4nhdcr0hh2-patch-2.7.6/bin:/nix/store/aq0v28n1ka8m1ayg6wrpbjy00bb7fj3k-xz-5.2.5-bin/bin\", \"doInstallCheck=\", \"CC=gcc\", \"_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu\", \"outputs=out\", \"TMP=/run/user/1000\", \"OBJCOPY=objcopy\", \"nobuildPhase=echo\\necho \\\"This derivation is not meant to be built, aborting\\\";\\necho\\nexit 1\\n\", \"SSL_CERT_FILE=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71/etc/ssl/certs/ca-bundle.crt\", \"USER=skainswo\", \"NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1\", \"SYSTEM_CERTIFICATE_PATH=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71/etc/ssl/certs/ca-bundle.crt\", \"depsBuildTargetPropagated=\", \"LD=ld\", \"PYTHONHASHSEED=0\", \"NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1\", \"IN_NIX_SHELL=pure\", \"NM=nm\", \"STRIP=strip\", \"OBJDUMP=objdump\", \"XDG_DATA_DIRS=/nix/store/f1n2zyq47sjn0qfhxvrdawyzlgk4sgax-patchelf-0.13/share\", \"nativeBuildInputs=\", \"SHELL=/nix/store/79zvzq15mga6bgk5phnmw0fd2pnpim9v-bash-interactive-5.1-p8/bin/bash\", \"system=x86_64-linux\", \"depsBuildBuild=\", \"name=nix-shell\", \"RANLIB=ranlib\", \"TMPDIR=/run/user/1000\", \"NIX_SSL_CERT_FILE=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71/etc/ssl/certs/ca-bundle.crt\", \"buildInputs=/nix/store/ggqkxkqjn8hf7h1kgv5qz8lwas9i5z6k-nss-cacert-3.71 /nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1 /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9\", \"NIX_BUILD_TOP=/run/user/1000\", \"depsHostHostPropagated=\", \"depsTargetTargetPropagated=\", \"doCheck=\", \"depsBuildTarget=\", \"SOURCE_DATE_EPOCH=315532800\", \"STRINGS=strings\", \"HOME=/home/skainswo\", \"TERM=xterm-256color\", \"TEMP=/run/user/1000\", \"PYTHONIOENCODING=UTF-8\", \"PYTHONNOUSERSITE=1\", \"_PYTHON_HOST_PLATFORM=linux-x86_64\", \"NIX_CFLAGS_COMPILE= -frandom-seed=7q7jlhbyfg -isystem /nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/include -isystem /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/include -isystem /nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/include -isystem /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/include\", \"DETERMINISTIC_BUILD=1\", \"patches=\", \"READELF=readelf\", \"AS=as\", \"__ETC_PROFILE_SOURCED=1\", \"depsHostHost=\", \"TEMPDIR=/run/user/1000\", \"NIX_INDENT_MAKE=1\", \"strictDeps=\", \"shell=/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/bash\", \"NIX_BINTOOLS=/nix/store/gxd0gvw88wyl9bqjc03ypm5zs7qbaf74-binutils-wrapper-2.35.2\", \"SIZE=size\", \"LOGNAME=skainswo\", \"SHLVL=2\", \"NIX_CC=/nix/store/7wn7xx8faqirzkvh5iwj26b024jzg09p-gcc-wrapper-10.3.0\", \"_=/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin/julia\", \"PAGER=less\", \"configureFlags=\", \"depsTargetTarget=\", \"AR=ar\", \"NIX_ENFORCE_NO_NATIVE=1\", \"PWD=/home/skainswo/dev/research/julia/permuted_sgd\", \"NIX_STORE=/nix/store\", \"NIX_BUILD_CORES=2\", \"PYTHONPATH=/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib/python3.9/site-packages\", \"out=/nix/store/7q7jlhbyfgim8i3w055lihzwxqlidxh6-nix-shell\", \"builder=/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/bash\", \"propagatedBuildInputs=\", \"depsBuildBuildPropagated=\", \"shellHook=\", \"phases=nobuildPhase\", \"CONFIG_SHELL=/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/bash\", \"NIX_LDFLAGS=-rpath /nix/store/7q7jlhbyfgim8i3w055lihzwxqlidxh6-nix-shell/lib64 -rpath /nix/store/7q7jlhbyfgim8i3w055lihzwxqlidxh6-nix-shell/lib  -L/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/lib -L/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib -L/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/lib -L/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib\", \"NIX_HARDENING_ENABLE=fortify stackprotector pic strictoverflow format relro bindnow\", \"JULIA_LOAD_PATH=@:/run/user/1000/jl_h44YES\", \"propagatedNativeBuildInputs=\", \"stdenv=/nix/store/lwcfw5vyjlzrs35k1hanv21j1q2s5c5w-stdenv-linux\", \"LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:\", \"OPENBLAS_MAIN_FREE=1\"]): no such file or directory (ENOENT)", -2)
└ using the Python distribution in the Conda package
[ Info: Downloading miniconda installer ...
[ Info: Installing miniconda ...
ERROR: LoadError: failed process: Process(`/home/skainswo/.julia/conda/3/installer.sh -b -f -p /home/skainswo/.julia/conda/3`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:531 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:446
  [3] run
    @ ./process.jl:444 [inlined]
  [4] _install_conda(env::String, force::Bool)
    @ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:207
  [5] _install_conda(env::String)
    @ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:193
  [6] runconda(args::Cmd, env::String)
    @ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:128
  [7] add(pkg::String, env::String; channel::String)
    @ Conda ~/.julia/packages/Conda/1403Y/src/Conda.jl:223
  [8] add (repeats 2 times)
    @ ~/.julia/packages/Conda/1403Y/src/Conda.jl:222 [inlined]
  [9] top-level scope
    @ ~/.julia/packages/PyCall/L0fLP/deps/build.jl:79
 [10] include(fname::String)
    @ Base.MainInclude ./client.jl:451
 [11] top-level scope
    @ none:5
in expression starting at /home/skainswo/.julia/packages/PyCall/L0fLP/deps/build.jl:43

It appears to be hung up on some aspect of the conda installation. However, building Conda.jl works without any issues:

(@v1.7) pkg> build Conda
    Building Conda → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6cdc8832ba11c7695f494c9d9a1c31e90959ce0f/build.log`
(@v1.7) pkg> 

A few things are weird to me about this situation:

• Why is PyCall.jl failing to build due to issues with Conda while building Conda.jl works just fine?
• cannot execute /home/skainswo/.julia/conda/3/conda.exe: NIX_LD or NIX_LD_x86_64-linux is not set suggests to me that Conda.jl is smart enough to use nix-ld when it's available, but I haven't been able to find anything about this in the docs or in the sources for Conda.jl or PyCall.jl. Where is this warning coming from?
• The error suggests that it's trying and failing to run /nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\", but this is very odd since I don't even have that version of python installed! I have python 3.9.9 with a different hash entirely! How is it hallucinating this python install that does not exist?

The main thing that has changed since the last time this worked for me is that I upgraded NixOS from 21.05 to 21.11.

Here's my shell.nix for reference:

let
  # Last updated: 1/3/22. From status.nixos.org.
  pkgs = import (fetchTarball ("https://github.com/NixOS/nixpkgs/archive/adf7f03d3bfceaba64788e1e846191025283b60d.tar.gz")) { };
in
pkgs.mkShell {
  buildInputs = with pkgs; [
    # See https://github.com/NixOS/nixpkgs/issues/66716. Necessary for julia to
    # be able to download packages.
    cacert

    julia_17-bin

    # Necessary for PyPlot.jl
    python3
  ];
}

and here's the version of python that's actually available:

[nix-shell:~/dev/research/julia/permuted_sgd]$ which python
/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin/python

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)

[samuela]

Update this is reproducible even after deleting ~/.julia/conda

[stevengj]

Installing Conda.jl does not actually install the distro — that only happens when a user, or a package like PyCall, first requests that something be installed. e.g. try:

import Conda
Conda.add("numpy")

That being said, it shouldn't need to install Conda at all — by default on Unix, PyCall tries to use the system Python, and apparently that is failing. What happens if you try

python3 -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))"

in your terminal?

[samuela]

I get

[nix-shell:~/dev/research/julia/permuted_sgd]$ python3 -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))"
3.9

I think the weird thing is that it's trying to run the wrong python3: /nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3 whereas what's installed is

[nix-shell:~/dev/research/julia/permuted_sgd]$ which python3
/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin/python3

[samuela]

FWIW part of the magic of nix-shell is that I'm freed from having a global python install across the whole system. That may be part of what's tripping it up.

[stevengj]

PyCall just searches the $PATH environment variable for python3. If nix-shell uses some other mechanism for which, that won't work.

The workaround is simply to set

ENV["PYTHON"]="/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin/python3"
Pkg.build("PyCall")

to specify the particular path that you want. (You only need to do this once and PyCall will remember the path subsequently.)

[samuela]

Here's my $PATH:

[nix-shell:~/dev/research/julia/permuted_sgd]$ echo $PATH
/nix/store/79zvzq15mga6bgk5phnmw0fd2pnpim9v-bash-interactive-5.1-p8/bin:/nix/store/f1n2zyq47sjn0qfhxvrdawyzlgk4sgax-patchelf-0.13/bin:/nix/store/7wn7xx8faqirzkvh5iwj26b024jzg09p-gcc-wrapper-10.3.0/bin:/nix/store/v819nrv8d33ns36gm4v9vqydq1v95axi-gcc-10.3.0/bin:/nix/store/dhj23pp1sld19shjslkzw3y9pr3qpi61-glibc-2.33-59-bin/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/gxd0gvw88wyl9bqjc03ypm5zs7qbaf74-binutils-wrapper-2.35.2/bin:/nix/store/8d30avp4xj1w8krq7zax1mlxgs28663y-binutils-2.35.2/bin:/nix/store/qmlbjq46nhn1v08nnb23y2myrvra8hm7-julia-bin-1.7.1/bin:/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin:/nix/store/abc8hz0v47jzsf048j1qx7m94bxfhpl7-coreutils-9.0/bin:/nix/store/r87rxnys39173q5nrlqq46vpa2203rmk-findutils-4.8.0/bin:/nix/store/xgqrjhz9sx8bllvayz18sql2dfd89q4g-diffutils-3.8/bin:/nix/store/aw3vgxn4fdrjqgmc94fr582hjhr2z3p3-gnused-4.8/bin:/nix/store/rvdb7057bmg3rasbqq8lzl10dffy3p31-gnugrep-3.7/bin:/nix/store/3qmgmnz3msrk0cf0qgcyja0wzyfhpxc2-gawk-5.1.1/bin:/nix/store/xc1dbg8mc8hd6bq7zizxf5nyrljk8az0-gnutar-1.34/bin:/nix/store/1wads5y31dky89hxnn5zmiri3c72kdpc-gzip-1.11/bin:/nix/store/l8d7zz3ninmgg71738q485anfjh5yd6q-bzip2-1.0.6.0.2-bin/bin:/nix/store/ba439awj33pw4l4bj95k4dy58nx69b8v-gnumake-4.3/bin:/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin:/nix/store/22avws4xw0chqniqym3q0w4nhdcr0hh2-patch-2.7.6/bin:/nix/store/aq0v28n1ka8m1ayg6wrpbjy00bb7fj3k-xz-5.2.5-bin/bin:/run/wrappers/bin:/home/skainswo/.nix-profile/bin:/etc/profiles/per-user/skainswo/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin

which includes /nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin and no other python installations.

(You only need to do this once and PyCall will remember the path subsequently.)

How does PyCall remember this path? Is it remembered per Julia session or forever or something else?

Perhaps it's remembering an incorrect path?

[samuela]

I did a little more poking around and I think I better understand the issue now. The problem is that part of PyCall's ] build process involves the generation of a file ~/.julia/packages/PyCall/L0fLP/deps/deps.jl:

[nix-shell:~/dev/research/julia/pumpkin]$ cat /home/skainswo/.julia/packages/PyCall/BD546/deps/deps.jl
const python = "/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3"
const libpython = "/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/lib/libpython3.9.so.1.0"
const pyprogramname = "/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6/bin/python3"
const pyversion_build = v"3.9.6"
const PYTHONHOME = "/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6:/nix/store/nki9ywqzbvz68vr75kn2r7g1q84f5agy-python3-3.9.6"

"True if we are using the Python distribution in the Conda package."
const conda = false

Because this file lives "forever" in ~/.julia, it becomes outdated after changing the python version (upgrading, etc). Setting the PYTHON env var works as a workaround. Still I think it'd be nice if it just pulled the python path out of PATH instead of generating this deps.jl cruft. What is the purpose in creating deps.jl as part of the build process?

[usmcamp0811]

I just recently wanted to make a Julia project using a Nix shell and ran into this problem. I was able to solve it by just hacking the PYTHONPATH and everything just seemd to work. This is my shell for reference.

{ lib, writeText, writeShellApplication, substituteAll, gum, inputs, pkgs
, hosts ? { }, ... }:
let
  inherit (lib) mapAttrsToList concatStringsSep;
  inherit (lib.campground) override-meta;
  src = ./.;
  julia-env = pkgs.julia.withPackages.override { extraLibs = [ python-env ]; } [
    "IJulia"
    "CSV"
    "DataFrames"
    "PythonCall"
  ];

  pypkgs-build-requirements = { redfin = [ "setuptools" ]; };

  p2n-overrides = pkgs.poetry2nix.defaultPoetryOverrides.extend (self: super:
    builtins.mapAttrs (package: build-requirements:
      (builtins.getAttr package super).overridePythonAttrs (old: {
        buildInputs = (old.buildInputs or [ ]) ++ (builtins.map (pkg:
          if builtins.isString pkg then builtins.getAttr pkg super else pkg)
          build-requirements);
      })) pypkgs-build-requirements);

  python-env = pkgs.poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    overrides = p2n-overrides;
    python = pkgs.python311;
  };
in pkgs.mkShell {
  buildInputs = [
    pkgs.poetry
    julia-env
    python-env
  ];
  env = {
    PYTHONPATH = "${python-env}/lib/python3.11/site-packages:${python-env}/lib/site-packages:$PYTHONPATH";
  };
  shellHook = ''
    echo "Poetry environment and Julia project set up."
  '';
}

[yurrriq]

Thanks, @usmcamp0811! Here's my shell for https://github.com/lambdaclass/julia_koans in case it helps someone else.

let
  python-env = python3.buildEnv.override {
    extraLibs = [
      python3Packages.numpy
    ];
  };
  julia-env = julia.withPackages.override { extraLibs = [ python-env ]; } [
    "PythonCall"
    "Test"
  ];
in
mkShell {
  env = {
    PYTHONPATH = "${python-env}/lib/python${lib.versions.majorMinor (lib.getVersion python3)}/site-packages";
  };
  nativeBuildInputs = [
    julia-env
    python-env
  ];
}