Proposal: `pipenv` patterns and antipatterns for python library project

[vlcinsky]

Hacking maya I learned few lessons which resulted in my following proposal of recommended usage of pipenv in python libraries. I expect others to review the proposal and if we reach agreement, the (updated) text could end up in pipenv docs.

pipenv patterns and antipatterns for python library project

EDIT
Following is best applicable for general (mostly Open Source) python libraries, which are supposed to run on different python versions and OSes. Libraries developed in strict Enterprise environment may be different case (be sure to review all the Problems sections anyway).

END OF EDIT

TL;DR: Adding pipenv files into python library project is likely to introduce extra complexity and can hide some errors while not adding anything to library security. For this reason, keep Pipfile, Pipfile.lock and .env out of library source control.

You will be able to use full power of pipenv regardless of it's files living in .gitignore.

Python library versus python application

By python library I mean a project, typically having setup.py, being targeted for distribution and usage on various platform differing in python version and/or OS.

Examples being maya, requests, flask etc.

On the other side (not python library) there are applications targeted for specific python interpreter, OS and often being deployed in strictly consistent environment.

pipfile describes these differences very well in it's Pipfile vs setup.py.

What is pipenv (deployment tool)

I completely agree on the statement, that pipenv is deployment tool as it allows to:

• define strict requirements (Pipfile.lock) for deployment of virtual environment
• apply those strict requirements in reproducible manner on different machines

It helps when one has to deploy an application or develop in python environment very consistent across multiple developers.

To call pipenv packaging tool is misleading if one expects it to create python libraries or to be deeply involved in creation of them. Yes, pipenv can help a lot (in local development of libraries) but can possibly harm (often in CI tests when used without deeper thought).

Applying "security reasons" in wrong context

TL;DR: pipenv provides secure environment via applying approved concrete dependencies described in Pipfile.lock file and python library is only allowed to define abstract dependencies (thus cannot provide Pipfile.lock).

pipenv shines in deployment scenarios following these steps:

• define abstract dependencies (via Pipfile)
• generate from it concrete dependencies resulting in Pipfile.lock
• create (virtual) python environment reflecting those concrete dependencies
• run tests to make sure, given environment works as expected and is secure
• release the tested "golden" Pipfile.lock as definition of approved python environment
• others can use pipenv sync to apply "the golden" Pipfile.lock elsewhere getting identical python environment.

With development of python library one cannot achieve such security, because libraries must not define concrete dependencies. Breaking this rule (thus trying to declare concrete dependencies by python library) results in problems such as:

• problems to find satisfying version of shared libraries (each strict package defines exact version of shared library and it is very likely the versions will differ and prevent finding commonly acceptable version)
• concrete dependencies may depend on python version, OS or other environment markers and trying to install the package in diferent context can easily fail to satisfy some of rules defined in original abstract dependencies.

Problem: Hiding broken setup.py defined dependencies

setup.py shall define all abstract dependencies via install_requires.

If Pipfile defines those dependencies too, it may easily hide problems such as:

• missing dependency in install_requires
• Pipfile defines specific rules (version ranges etc.) for a dependency and install_requires does not.

To prevent it, follow these rules:

• library defined dependencies must not appear in Pipfile
• the [packages] section in Pipfile shall be either empty or define only single dependency on the library itself.

Problem: Pipfile.lock in repository

Keeping Pipfile.lock (typically for "security reasons") in library repository is wrong, because:

• described dependencies are likely to be invalid for different python versions or in another OS
• developers are forced to update the file not only when they add/remove some dependency, but also when other libraries are updated and may be usable within the library.

To prevent it, one should:

• remove Pipfile.lock from repository and add it into .gitignore

Problem: Competing with tox (hiding usedevelop)

If tox.ini contains in it's commands section entries such as:

• pipenv install
• pipenv install --dev
• pipenv lock

it is often a problem, because:

• pipenv install shall install only the library itself, and tox is (by default) doing it too. Apart from duplicity it also prevents of usedevelop=True and usedevelop=False in tox.ini because Pipenv is able to express it only in one variant (and tox.ini allows differencies in different environments).

To prevent it, one should:

• refrain from using pipenv in tox.ini. See requests tox.ini

Problem: Breaking builds, if pipenv fails

pipenv is under heavy development and things break sometime. If such issue breaks your CI build, there is a failure which could be prevented by not using pipenv and using traditional tools (which are often a bit more mature).

To prevent it, one should:

• think twice before adding pipenv into a CI build script, tox.ini or similar place. Do you know what value you get from adding it? Could be the job done with existing tooling?
• do not add it "just for security reasons" or because "everybody does".

Summary

Key questions regarding pipenv role in development of python library are:

• What value pipenv really brings? A: Virtualenv management tool.
• What is relevent use case for pipenv? A: Manage virtualenv.
• Shall it appear in the library repository? A: NO.

Few more details and tricks follow.

pipenv will not add any security to your package

Do not push it into project just because everybody does it or because you expect extra security. It will disappoint you.

Securing by using concrete (and approved) dependencies shall take place in later phase in the application going to use your library.

Keep Pipfile, Pipfile.lock and .env files out of repository

Put the files into .gitignore.

Pipfile is easy to recreate as demonstrated below as most or all requirements are already defined in your setup.py. And the .env file probably contains private information, which shall not be shared.

Keeping these files out of repository will prevent all the problems, which may happen with CI builds when using pipenv in situations, which are not appropriate.

pipenv as developer's private toolbox

pipenv may simplify developer's work as virtualenv management tool.

The trick is to learn, how to quickly recreate your (private) pipenv related files, e.g.:

$ cd <project_repository>
$ # your library will bring the dependencies (via install_requires in setup.py)
$ pipenv install -e .   
$ # add more dev tools you preffer 
$ pipenv install --dev ipython pdbpp
$ # start hacking
$ pipenv shell
...

Use .env file if you need convenient method for setting up environment variables.

Remember: Keep pipenv usage out of your CI builds and your life will be simpler.

Trick: Use setup.py ability to declare extras dependencies

In your setup.py use the extras_requires section:

from setuptools import setup

setup(
    name='mypackage',
    ....,
    install_requires=["jinja2", "simplejson"],
    extras_require={
        'tests': ['pytest', 'pyyaml'],
        'pg': ['psycopg2'],
    },
    ....
)

To install all dependencies declared for tests extra:

$ pipenv install -e .[tests]

Note, that it will always include the install_requires dependencies.

This method does not allow spliting dependencies into default and dev sections, but this shall not be real problem in expected scenarios.

[techalchemy]

This is very impressive, thanks a ton for compiling. Will definitely review in more detail in a bit

/cc @uranusjr @jtratner @ncoghlan

[vlcinsky]

Some references to maya issues:

• Remove Pipfile.lock from repository kennethreitz/maya#138 (RemovePipfile.lock from repository)
• Skip pipenv run in tox.ini, as the virtualenv is already activated kennethreitz/maya#139 (Skip pipenv run in tox.ini ...)
• fix pendulum>=1.0 in setup.py (was missing version) kennethreitz/maya#145 (fix pendulum>=1.0 in setup.py: version was in Pipfile but was missing in setup.py)
• Clean Pipfile, remove requirements already stated in setup.py kennethreitz/maya#143 (PR showing how pipenv issue broke whole Travis run)
• Refactor pipenv usage according to semi-official best practices. kennethreitz/maya#144 (PR Refactor pipenv usage according to semi-official best practices)

[uranusjr]

I love this too. Maybe we should add this to Pipenv’s documentation, or even the Python Packaging User Guide.

[tsiq-oliver]

The corollary of the above advice appears to be "forego deterministic/reproducible CI builds", which strikes me as a very large anti-pattern.

What are you proposing as an alternative which would still allow for determinism?

[vlcinsky]

@tsiq-oliverc Deterministic builds have their place at the moment, an application is to be built.

Imagine following attempt to perform really deterministic builds of python library:

• builds have to be based on Pipfile.lock
• each execution context (combination of each python and OS variant) may have different Pipfile.lock resulting from library abstract dependencies defined in Pipfile
• Repository would have to provide separate Pipfile.lock instances defined in the repository. Note, that building Pipfile.lock automatically during CI build does not add any determinism

This is a lot of extra effort. And what you get is a library, which will be installed in different context (e.g. a week later standard installation will pick up upgraded dependency or two) and which will not get anything from the fact, you used Pipfile.lock, which is at the moment obsolete.

The conflict is in the fact the library must never define strict dependencies inside.

If you think, there is another alternative to gain deterministic builds for python library - describe it.

[tsiq-oliver]

@vlcinsky - If a consumer of your library uses different versions of dependencies, etc. then that's out of your control. So I agree there's no feasible way for a library maintainer to manage that.

But the goal here is presumably much smaller scope. In particular, I'd see the goals for a library maintainer as the following (which are roughly equivalences):

1. If you run your CI twice, you're guaranteed to get the same result (network issues notwithstanding!).
2. You can locally recreate (and thus debug) behaviour you observe on CI, even if that means running Docker/etc. locally.
3. You can confidently say "My library behaves as expected with dependency versions X, Y, Z" to your consumers.

If any of those three things don't hold, it strikes me as antithetical to quality control.

So yes, I'd say that if you guarantee to support Python variants A, B and C to your consumer, and they behave differently enough that one lockfile (etc.) doesn't cut it, then you should have three lockfiles (or whatever).

I haven't used Pipenv enough to know how easy that would be in practice, though.

[Moritz90]

I'm currently considering adding Pipfiles to a few library projects for the CI system as well.

I absolutely need the dependency locking (+hashing) for complying with company-wide security guidelines and I currently don't need to test with different Python versions, since there's only one that's officially supported. And the fact that pipenv simplifies setting up a local development environment, including the virtualenv, is a nice side-effect.

And what you get is a library, which will be installed in different context (e.g. a week later standard installation will pick up upgraded dependency or two) and which will not get anything from the fact, you used Pipfile.lock, which is at the moment obsolete.

This is not universally true. In the world of enterprise software, you still have very specific environments that are officially supported and a security issue in a dependency results in your product being updated rather than the customer updating the dependency themselves.

(Yes, I'm talking about a library, not an application here...)

[vlcinsky]

@Moritz90 your scenario is for python library in enterprise environment and there pipenv may help because it is much more deterministic environment.

My description is aiming at general python libraries such as flask, request, maya etc. where the context is much more variable. Trying to fix couple of things in maya I got frustrated learning, that in many cases usage of pipenv introduced real problems (typically hiding problems which would be normally detected) while not providing much or any added value.

Getting deterministic builds is good thing, but it incurs costs. And if done wrong, you may pay extra for lower quality result - and this is what I wanted to prevent.

[uranusjr]

I’d argue this is one of the instances we don’t want the builds to be absolutely deterministic. If you don’t pin your dependencies with ==, you’re committing to maintain support to multiple versions by default, and should design the library that way. A dependency upgrade breaking the build on CI is actually a good thing because it exposes a bug in the library. Completely deterministic dependencies (as managed by Pipenv) would mask that. It would still be beneficial to be able to be determinitic when you want it, that is generally not the best.

[tsiq-oliver]

@uranusjr - Sure. I agree that if the desire is "non-deterministic builds", then the advice up top may well make sense. In fact, it's almost a logical equivalence, and could be stated much more succinctly: "If you don't want deterministic builds, then don't use a tool (pipenv) whose purpose is to ensure deterministic builds" 😄.

But that's certainly not a desirable goal in general.

[vlcinsky]

@tsiq-oliverc nice scope definition - it supports focused discussion. I would add one more requirement: The CI determinism shall not hide possible issues within tested library.

If we use Pipenv.lock, create virtualenv based on that and run CI test of the library, we did part of the functionality the library is supposed to do - installing proper dependencies. If the library is somehow broken in this regard - preinstalled environment would hide this problem.

To me it seems more important to detect issues within a library than to run CI in deterministic way. If there is a way to do both (e.g. running the test behind private pypi index, which could also support determinism) I have no problem, but if there is a conflict, I have my priorities.

Do not take me wrong: there is no desire to run non-deterministic builds, my desire is to run CI builds, which will detect as much issues as possible.

[Moritz90]

@vlcinsky Sure, I just wanted to share my experience to make sure that the updated documentation reflects it as well. The current documentation does a great job at explaining the tradeoffs:

For libraries, define abstract dependencies via install_requires in setup.py. [...]
For applications, define dependencies and where to get them in the Pipfile and use this file to update the set of concrete dependencies in Pipfile.lock. [...]
Of course, Pipfile and pipenv are still useful for library developers, as they can be used to define a development or test environment.
And, of course, there are projects for which the distinction between library and application isn’t that clear. In that case, use install_requires alongside pipenv and Pipfile.

(Highlighted the part that applies in my case.)

I just want to make sure it stays that way. I think your original post contains too many blanket statements without a disclaimer that you're talking about an open-source project that's going to be published on PyPI.

[vlcinsky]

@Moritz90 I completely agree. I was trying to highlight that focus but I can make it even more visible.