Extras not getting installed when the package that includes the extras has already been installed

[eeshangarg]

• Pip version: 9.0.1
• Python version: 3.5.2
• Operating system: Ubuntu 16.04.3 LTS

Description:

Note: I initially reported this here pypa/setuptools#1247, but was advised to report it here instead.

A similar issue was reported and fixed here for pip.

In our case:

1. We have a repo with multiple packages in it.
2. We have the zulip_bots package that depends on requests and simple_salesforce. simple_salesforce further depends on requests[security].
3. The zulip_bots package also depends on the zulip package in the same repo which also depends on/installs requests. So, it is clear that requests does get installed whether you install zulip or zulip_bots.
4. We usually use our provision script to set up a dev env and install the zulip and zulip_bots packages.
5. But recently, when we ran this executable (part of the zulip_bots package), we ran into:

pkg_resources.DistributionNotFound: The 'cryptography>=1.3.4; extra == "security"' distribution was not found and is required by requests

The above error led us to conclude that for some reason, requests[security] was not getting satisfied as expected.

It would mean a lot to us if someone could please shed some light on whether this is a re-manifestation of #3189 and #3198 or simply a quirk in our understanding of how extra dependencies are resolved! Thanks! :)

[mitar]

I think I have a similar case. Reproduction is as follows. If I have a requirements.txt file like:

git+https://gitlab.com/datadrivendiscovery/metadata.git@v2018.1.5
scikit-learn

and I do:

pip install --process-dependency-links -r requirements.txt

(--process-dependency-links has nothing with this report, this is just because the package I am demonstrating with requires it)

Then scipy is not installed, despite being and extra dependency alldeps in scikit-learn[alldeps] from d3m_metadata package (installed from the git repo above). scipy installs correctly if there is no scikit-learn entry in requirements.txt. It seems extras do not get combined as one would expect. This is really tricky because one cannot really know all dependencies packages can have. So If in my project I need d3m_metadata package and just basic scikit-learn, I would like to list only those. But now what happens i that d3m_metadata package does not get all expected dependencies and it fails.

[nickwilliams-eventbrite]

I am having the following problem (commented on #3516). Is this issue the cause of that problem?

I am also still having this problem in Pip 9.0.3 and 10.0.1. I'll provide a specific example. PySOA provides a PyTest plugin, and that plugin has a set of requirements, represented in the extra pysoa[pytest]. Projects that use PySOA would put pysoa in their normal requirements (so that the PyTest plugin extras aren't included in the distribution), but would put pysoa[pytest] in their testing extras so that the PyTest plugin extras are installed for testing:

   install_requires={ ... 'pysoa' ...},
   tests_require={ ... 'pysoa[pytest]' ... },
   extras_require={
       'testing': { ... 'pysoa[pytest]' ...},
   },

In our CI environment, when we run python setup.py test, it works perfectly. The pysoa[pytest] requirement, with extras, is installed, and tests pass. However, locally, we install it with pip install -e .[testing], and that approach does not install the extras from pysoa[pytest], so our tests fail locally unless we manually install the extra requirements.