DLL load error while importing scipy.sparse.linalg module on Windows

[terem42]

My bug report is about DLL load error on running scipy-1.3.2 library for Windows 10 64bit

Reproducing code example:

import scipy.sparse.linalg
print("loaded")

After small research I've identified a culput, there is no PATH env setup for windows platform inside __config__.py of scipy package, or it's not get executed correctly.

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir

Henceforth, the path to .libs folder is not added to Windows path and libraries never ever gets found.

Error message:

Traceback (most recent call last):
  File "test_lib.py", line 1, in <module>
    import scipy.sparse.linalg
  File "C:\apps\Python37\lib\site-packages\scipy\sparse\linalg\__init__.py", line 113, in <module>
    from .isolve import *
  File "C:\apps\Python37\lib\site-packages\scipy\sparse\linalg\isolve\__init__.py", line 6, in <module>
    from .iterative import *
  File "C:\apps\Python37\lib\site-packages\scipy\sparse\linalg\isolve\iterative.py", line 10, in <module>
    from . import _iterative
ImportError: DLL load failed

Scipy/Numpy/Python version information:

Python 3.7.5 win64
scipy-1.3.2
numpy-1.17.4

Temporary workaround I'm using for the moment being, is a small update within __init__.py repeating the missing search path addition.

import os
import sys

extra_dll_dir = os.path.join(os.path.dirname(__file__), '.libs')
if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):  
  os.environ['PATH'] = extra_dll_dir + os.pathsep + os.environ['PATH']

[ilayn]

That's because you haven't clicked add python to path box when you are installing python.

Instead add libs to path

[terem42]

Normally this should happen either at config or init lib stage, because I can't know in advance whether library uses DLLs or not. Hence, I propose to add aforementioned path settings to __init__.py

[rgommers]

I think this has never come up, because there are many reasons you'll want to set PATH on Windows (e.g. https://docs.python.org/2/using/windows.html#finding-the-python-executable). Anaconda et al. do this automatically. How did you install Python?

[ilayn]

We can't dictate where or how the python access should reside. I don't think we can do anything about it. It's beyond scipy's jurisdiction.

[clemextravisci]

We have a similar issue happening in scipy.interpolate. Not sure if related, but the content of the .libs folder in 1.3.2 is missing the file msvcp140.dll compared to the extra-dll folder in 1.3.1. Copying the file in the folder fixes the issue.

[ilayn]

That should be fixed in #10628

[clemextravisci]

Just to make sure I understand correctly, a new PR needs to be created to modify the destination folder of the msvcp140.dll file?

Similar to what was done here, but .libs instead of extra-dll:
https://github.com/MacPython/scipy-wheels/pull/50/files#diff-180360612c6b8c4ed830919bbb4dd459R169-R172

So, it can work seamlessly with numpy configuration tool:
https://github.com/numpy/numpy/blob/master/numpy/distutils/misc_util.py#L2329

[ilayn]

After switching to Python3.8 I can confirm its reproducibility. So I was wrong above. This is also affecting the numpy builds. My apologies for the quick and false response. I'll take this to NumPy maintainers.

[pv]

The problem I think is this: https://github.com/MacPython/scipy-wheels/blob/master/appveyor.yml#L167

We used a pinned version of numpy.distutils with the correct Windows patches, but for Python 3.8 this was changed to use Numpy upstream, which has a different name for the DLL directory.

EDIT: somehow also the scipy-1.3.2-cp37-cp37m-win_amd64.whl has libraries in the .libs directory, and not in extra-dll.

[ilayn]

This is also happening on my local NumPy and SciPy builds.

[pv]

So there seem to be now several different issues:

(i) MacPython/scipy-wheels#55 changed the pinned Numpy distutils version, which changed the name of the DLL directory. However, msvcp140.dll copying was not updated to match, so the file now goes to the wrong place. I opened a separate ticket for this: MacPython/scipy-wheels#57

(ii) There is something strange with the behavior of PATH and DLL loading. Those who can test it, could you try give information on whether the

extra_dll_dir = os.path.join(os.path.dirname(__file__), '.libs')

if sys.platform == 'win32' and os.path.isdir(extra_dll_dir):
    os.environ.setdefault('PATH', '')
    os.environ['PATH'] += os.pathsep + extra_dll_dir

fail to change PATH at all, or, whether the issue is some DLL loading problem that for some reason appending the load directory does not work any more, but prepending works.

[ilayn]

isdir is the False branch somehow. My guess is that this is run on the src.win... folder and not in the lib.win... folder during the install. But I am really speculating now.

[pv]

The __config__.py code is run just when importing the installed package (see scipy/__init__.py), not at build or install time, so the contents of the build directory (src.* lib.*) should play no role and only what gets installed (or put into the .whl) should matter.