ENH: Hacking Python's import system to find the installed package even in the source directory

[lithomas1]

Background

In pandas, we have a source directory structure that looks like this.

pandas
    pandas
        __init__.py
        ...
        _libs
            Extension modules
        ...

Usually, with setuptools, we build the extension modules inplace, and importing pandas from the source directory works fine because of this.

However, with meson-python, we build/install pandas to the site-packages directory(with pip install . -v --no-build-isolation --config-settings="builddir=builddir" ). Since the build is now out-of-tree, import pandas within the source directory, fails (expectedly), since we can't find the extension modules in the source directory anymore.

This is a massive PITA, since now you can't work/debug on pandas using the Python interpreter in the base of the source directory anymore.

Solution

(Disclaimer: I'm still very confused by how the Python import system works internally, so I'm probably wrong/missing something somewhere).

The problem is mainly caused by python pre-pending the current working dir to the front of sys.path. To fix this, we'd need to not load our pandas from there, and instead from the one installed in site-packages.

In order to execute code/modify sys.path before the interpreter initializes the module search path, we have two options which are the .pth files and sitecustomize.py.

Unfortunately, the .pth files and sitecustomize.py seem to be run before Python prepends the current working directory to the front of sys.path, making it impossible to delete.

A workaround to this would be to inject a new MetaPathFinder(maybe can inherit from PathFinder?) into sys.meta_path, to detect when e.g. pandas is being imported from its source directory. Since .pth files don't support executing arbritrary code well(I think it's going to be deprecated sometime), sitecustomize.py is probably the place to put this code.

This would require meson-python to be responsible for writing/appending to this sitecustomize.py file, which is why I'm asking for feedback.

cc @mesonbuild/meson-python-devs and @stefanv (this might be related to the devpy python shell not working on < 3.11)

[stefanv]

IPython does not have this problem, so with devpy you can do ./dev.py ipython and have a working shell that points to your installed copy.

Python 3.11 added the -P flag, which allows you to run Python without the current path being added to the PYTHONPATH.

You can also do:

PYTHONPATH='$PWD/builddir/usr/lib64/python3.10/site-packages' python -i -c "import sys; del(sys.path[0])"

Only problem is that that then prevents calling -c "import pandas" (since the first -c terminates the argument list), so it wasn't obvious how to include this in ./dev.py python.

[FFY00]

I believe your assessment is correct, but I think I have a simpler solution for #87 😊

Our custom finder needs to be added via a .pth file, but we could delay inserting the path into sys.path until the finder is used, at which time the current directory has already been prepended to sys.path.

[lithomas1]

I believe your assessment is correct, but I think I have a simpler solution for #87 😊

Our custom finder needs to be added via a .pth file, but we could delay inserting the path into sys.path until the finder is used, at which the current directory has already been prepended to sys.path.

Awesome!

Is #87 ready for me to take for a spin then? The editable method still seems to be returning NotImplementedError.

[FFY00]

I am still missing the logic to generate the editable wheel. Let me see if I am still able to finish that up today.

[FFY00]

This should be fixed now that editable installs are implemented. I think all workflows issues are fixed by using editable installs, which do not have this issue.

Python 3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> import sys
>>> sys.path
['/home/anubis/git/pandas/.mesonpy/editable/install/usr/lib/python3.10/site-packages', '', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/anubis/.virtualenvs/mesonpy-editable/lib/python3.10/site-packages']

Or is still something else I am missing?