`AttributeError: module 'importlib.resources' has no attribute 'path'` on Python 3.13

[domdfcoding]

Describe the bug
On Python 3.13.0a1, while building numpy (directly from GitHub) meson fails first with AttributeError: module 'importlib.resources' has no attribute 'path' and then ../../meson.build:40:22: ERROR: Unhandled python exception.

importlib.resources.path was removed in Python 3.13

To Reproduce

1. Build Python 3.13.0a1 from source (https://github.com/python/cpython/releases/tag/v3.13.0a1)
2. Run python3.13 -m pip wheel git+https://github.com/numpy/numpy

Expected behavior
Meson correctly compiles numpy

system parameters

• Native build
• Ubuntu 20.04
• Python 3.13
• Meson 1.2.99

[rgommers]

Thanks @domdfcoding.

Using numpy isn't an ideal way to report this issue, because it has a vendored Meson version that trails behind Meson's master branch.

However, the bug report seems valid still - https://docs.python.org/3/library/importlib.resources.html#importlib.resources.path indeed says that it was deprecated in Python 3.11, and if it was removed in 3.13.0a1, that breaks the usage at

meson/mesonbuild/dependencies/python.py

Line 116 in bf9314e

with importlib.resources.path('mesonbuild.scripts', 'python_info.py') as f:

The fix is to use importlib.resources.as_file for Python >= 3.9 >3.12

[eli-schwartz]

Sadly, that introduces conditional code which looks really ugly and doing so makes me feel sad. :(

I actually do not understand either the API evolution of importlib.resources -- the as_file approach looks terrible and non-idiomatic and feels like users of the stdlib are being punished for mysterious reasons -- or the removal of the straightforward wrappers that were intuitive and always did the right thing and caused zero harm but a lot of good, in addition to being broadly portable across python versions and even transparently gaining new features as time went by.

@jaraco would it be possible to back out this removal and declare the previously deprecated functions to be long-term stable, idiomatic, intended uses -- precisely as they were in 3.12, as high level usability glue atop the internal plumbing of as_file?

[nanonyme]

I wonder if it would clutter functional code less if the compat code was done at the beginning of the module assuming it needs workaround in meson.

[jaraco]

Sadly, that introduces conditional code which looks really ugly and doing so makes me feel sad. :(

A perhaps preferable way to avoid the conditional code is to unconditionally depend on importlib_resources and use that approach on all Python versions. It has as_file support going back to Python 2.7... until your project drops support for versions of Python that don't have the functionality you require, at which point you can switch from importlib_resources to importlib.resources. If you wish to more aggressively depend on importlib.resources, you can conditionally depend on it only for older Python versions and selectively import the compatible APIs based on Python version. If you can't have dependencies, then you may be stuck with conditionally implementing the different behaviors.

I actually do not understand either the API evolution of importlib.resources -- the as_file approach looks terrible and non-idiomatic and feels like users of the stdlib are being punished for mysterious reasons -- or the removal of the straightforward wrappers that were intuitive and always did the right thing and caused zero harm but a lot of good, in addition to being broadly portable across python versions and even transparently gaining new features as time went by.

I'm happy to shed some light on the motivations. In python/importlib_resources#58, it was discovered that the original assumption that "all resources are files in Python packages" was too constraining and ran against the reasonable expectations of packages to supply subdirectories of data. It was not possible to break this assumption while retaining the existing APIs; that is, the path and read_text and other functions had the assumption baked in; hence why a new, more capable API was envisioned. It's taken a lot of work to implement that interface, adapt the previous API to work off that old interface, and ultimately deprecate the old interface (including running the deprecation by the community and the Python steering committee). The reason for deprecating and removing the old interface is because it presents a maintenance burden and encourages cargo-cult adoption.

Case in point, 3bbfd4d was authored this year, well after the deprecation and many years after the preferred interface was published. The recommended replacement has been present in CPython since 3.9 and available in the importlib_resources backport for Python 2.7+, giving a great deal of flexibility for adoption.

We wish to give users "preferably one" way to access resources and not leave inadequate interfaces to proliferate. The harm comes from leaving the deprecated interfaces in place, encouraging users to adopt these inadequate interfaces without understanding the limitations and requiring all readers to understand the tradeoffs of one interface over the other (or naïvely selecting one without the understanding).

You might also observe that by the time Python 3.13 is released, Python 3.8 will be EOL, so even with the removal, there will be no time when the actively supported set of Python versions doesn't have a single supported API for resources in the stdlib. I do see how that's little comfort for projects that support EOL Pythons or wish to test against prereleases.

@jaraco would it be possible to back out this removal and declare the previously deprecated functions to be long-term stable, idiomatic, intended uses -- precisely as they were in 3.12, as high level usability glue atop the internal plumbing of as_file?

It would be possible, but doing so would reintroduce the harm described above and erase the difficult work it's taken to get here. At this stage, I'd say there would need to be a very compelling case, much more than a concern of (temporary) ugliness.

I appreciate you giving me the opportunity to shed some light on the issue and I hope this message helped alleviate some of the frustration that this change has brought. Please let me know if there may be another way I can help.

[eli-schwartz]

It was not possible to break this assumption while retaining the existing APIs; that is, the path and read_text and other functions had the assumption baked in; hence why a new, more capable API was envisioned.

Sure, but reimplementing the older API on top of the newer API should still make everything work fine. importlib.resources.path() internally "grew more capable". If you wrote code which was valid on python 3.8, it was still valid on python 3.12. If you wrote code which was valid on python 3.12, it would not be guaranteed to work on python 3.8.

It's pretty common and straightforward to add new features to software, so importlib.resources has not innovated in the theory of software innovation in that respect.

The reason for deprecating and removing the old interface is because it presents a maintenance burden and encourages cargo-cult adoption.

In general, deprecating an old interface is not something I object to. Removing it is a bit more iffy. Encouraging cargo-cult adoption is a problem solved by deprecation warnings, not removal.

I'm not certain I understand your reference to a maintenance burden. Given the old interface was a wrapper on top of the new interface, there's effectively no code to maintain.

In fact, the old interface is a wrapper whose function body is identical to the documentation of importlib.resources:

Deprecated since version 3.11: Calls to this function can be replaced using as_file():

as_file(files(package).joinpath(resource))

This very much feels like the polar opposite of a maintenance burden.

Case in point, 3bbfd4d was authored this year, well after the deprecation and many years after the preferred interface was published. The recommended replacement has been present in CPython since 3.9 and available in the importlib_resources backport for Python 2.7+, giving a great deal of flexibility for adoption.

I agree that commit 3bbfd4d was authored well after the deprecation, using the deprecated version. I did this for premeditated reasons: because I required an API that existed on python 3.7 and this project does not use non-stdlib code (backports are not in the stdlib).

[eli-schwartz]

It would be possible, but doing so would reintroduce the harm described above and erase the difficult work it's taken to get here. At this stage, I'd say there would need to be a very compelling case, much more than a concern of (temporary) ugliness.

As far as I can tell, the only harm is a perceived harm of messaging, that it is harmful to people to be able to use deprecated functions because deprecated functions are deprecated and supposed to not be used.

Given that, and in combination with this:

You might also observe that by the time Python 3.13 is released, Python 3.8 will be EOL, so even with the removal, there will be no time when the actively supported set of Python versions doesn't have a single supported API for resources in the stdlib. I do see how that's little comfort for projects that support EOL Pythons or wish to test against prereleases.

I believe that it is an undue burden on projects that wish to test against prereleases while supporting all non-EOL versions of python. It's also a problem when those projects don't just want to test against prereleases, but want to be used by other projects that need to test against prereleases because it is often quite complex to get them working in time. The PyData ecosystem is not a trivial one. And meson running on python 3.12 could build numpy/scipy for python 3.13 -- but we don't yet have a meson-internal wheel creator which could build the dist-info and package up a wheel for python 3.13 while internally running on top of python 3.12. We'd like one, it simply exists in the nebulous future.

Side note: meson supports all non-EOL versions of python, which in this case means python 3.8 (~11 more months of support). Additionally, we do not bump the python_requires if it would be untruthful to do so, which means that we support python 3.7 until someone submits a PR that adds python 3.8-specific features and points out how wonderful and amazing those features are which we should totally use in order to enhance our experience developing on the codebase. For dropping python 3.6 and moving on to python 3.7, there was a nice list of really exciting features and I was quite happy to push and push and push us onwards. I didn't really feel like python 3.8 had any killer features, or even much of anything in the way of features that I especially wanted for this codebase in particular, and that required code changes to take advantage of... so I haven't pushed for that and was thinking probably to just drop 3.7 and 3.8 simultaneously in 11 months from now. Which means we could drop our support for EOL python today, if it helped with importlib.resources usage... but. It doesn't help. We would need to drop support for 3.8 as well, which we cannot do right now.

tl;dr It's not the EOL python that is affecting us here, it's the prerelease python.

[eli-schwartz]

I'd like to return back to a previous comment I made:

high level usability glue atop the internal plumbing of as_file

I, personally, find the "new and improved" importlib.resources API to be awkward to use. More or less for one single reason: having to chain together files() with pathlib-style overloaded division and then pass the result to an as_file function.

It feels very, very awkward to me as compared to the possibility of automatically having the support of a context manager when you need one. The importlib.resources module has, in my eyes, ceased to Do The Right Thing for you after this change. I would expect a function like as_file to be a little-used "expert API" primarily of interest to people seeking to extend importlib.resources with custom traversable types.

What I'd really like to see is that in python 3.13 I could do this:

import importlib.resources

with importlib.resources.files('mypackage') / 'datafile.txt' as f:
    do_something_with_filesystem_file(f)

It's the same thing I'm already doing with the legacy API, except that you port to the new API like this:

-with importlib.resources.path('mypackage', 'datafile.txt') as f:
+with importlib.resources.files('mypackage') / 'datafile.txt' as f:

[jaraco]

I've never loved the fact that one needs to files(module) / 'filename' for the most basic usage. I often find myself using files(module).joinpath('filename') in order to have an expression on which I can invoke a method (e.g. files(module).joinpath('filename').read_text().

What I'd really like to see is that in python 3.13 I could do this:

The main reason it wasn't implemented this way was because for the common case where packages are installed into the file system, it was nice for files() to return a pathlib.Path object, which may have (or develop) its own behavior when entered as a context manager, but more importantly, would need to be wrapped in order to supply behavior that's not intrinsic to a path.

I suppose what you're proposing could be possible, but I'm not sure the ergonomics benefits would outweigh the complication that would ensue from wrapping pathlib.Path and replacing as_file with an integrated context manager.

It would require updating this protocol and then probably wrap the return value from files in an adapter that provides the context manager. The real challenge is going to be writing an adapter that's resilient across traversal.

And now that I think about it, it won't work to change Traversable. Traversable is what the providers are meant to supply. We'll have to change the files return type from a Traversable to this new enterable type.

Would you be willing to draft an implementation and tell me if after experimenting with the above approach or perhaps an alternative if you think the approach would be worth the trouble?

If so, I think the next best step would be to file a bug with importlib_resources.