Roadmap update for TUF support

[LucidOne]

Is it possible to get an update on the development roadmap about when TUF or other encryption support might be deployed? Thanks!

Also, it appears that tomorrow will be 6 years since January 5, 2013.

[nealmcb]

Thanks - good question.
I note that TUF is one of the options noted under Add support for API keys · Issue #994 · pypa/warehouse which appears under the "Cool but not urgent" milestone.
I agree that TUF addresses many of the nicely described PyPI-specific concerns that @dstufft wrote way back on 23 July 2013: Why Package Signing is not the Holy Grail

[LucidOne]

I believe that the package signing issues stated can be resolved and in 2019 our internet depends on the security of infrastructure such as PyPI. If there is not a roadmap for TUF support, I'm going to look at solving the problem.

[MyNameIsCosmo]

Perhaps we can evaluate integration with third-party services for verification?
Keybase.io helps create trust by building a verified profile attached to a GPG key across multiple services.
GitHub handles gpg signature verification for commits, refs, releases, etc. (although, your GH pgp key is kept in your GH settings... 2fa could help mitigate unauthorized account access)
Falling back, you can have a PGP key on MIT's PGP keybase which forwards a public key to key servers around the world.

Now, this does assume putting trust in third-party services themselves (and the author of the actual PGP key), but the chances are very slim that your PGP key would be replaced on multiple services at once. The biggest issue is an author being unable to secure their private key (no password, weak password, bad private key handling, etc...), or their account becoming compromised (e.g. GitHub).

This doesn't solve the problem on PyPi's side of verifying someone, although PyPi shouldn't have to verify anyone. PyPi should host the package. It should display information about the package being signed (or unsigned), the package's origins and contents, and whoever is downloading the package should check that key against their trusted sources.
In the past, PyPi has hosted malicious code. Of course, this happens. It is expected. Other package managers suffer as well.

---

I'm sure we all can go on about this.
Sure, PGP signing doesn't guarantee security. It doesn't guarantee verification. It doesn't even guarantee safe, robust, reliable, ... code. PGP is just a part of a solution to of a complex problem.
As signing becomes supported by major services (package maintainers, source code repositories, document services, ...), a trust network (keybase, key servers, ...) can be utilized to help security-minded users track where their packages are coming from.

If a user or a maintainer doesn't care about signing, it doesn't have to be enforced at the beginning, If the feature is there, people will use it.

Edit:
A great talk - How Much Do You Trust That Package?

[LucidOne]

I am convinced that this problem is tractable and I think the key here is that EigenTrust can provide us a mathematical model to get started.

We already have some data in the git commit history. When someone signs a git commit there is an eigentrust metric that can be calculated for all of the previous signed commits. There is reason to believe that causality and techniques like Bayesian inference may also be useful here where multiple commits are signed by a key we do not yet trust, but are temporally followed by a key we do trust. We can also federate our memories of git history and checksums to detect anomalies and attacks on infrastructure.

Sites like github and sr.ht can already provide trust metrics by validating that an email address is connected to a GPG key. We can formalize and automate our existing manual heuristics for validating packages.

Further I think we should start thinking about metadata for developers and organizations that produce software. Organizations can maintain a META-DEV repository that contains the PGP signing keys of active developers, key revocation information, and per repository release manager designation. Developers can also maintain a META-DEV repository that contains yaml (or whatever) metadata linking to a developers twitter / blog / instagram / keybase / identi.ca / matrix / Mastodon / xmpp / whatever where the PGP fingerprint can be posted.

Perhaps we need to start building keyrings in OS native packaging formats (.deb, .rpm, etc) so that trust can at least be established for the most critical python packages.

This is a complex problem but we need to take what small steps forward that we can instead of waiting another year to secure PyPI or even bother to figure out certificate pinning.

[brainwane]

Thank you to everyone who's raised this issue and shared their thoughts and useful resources! And sorry for the slow response!

Short answer: we'll be discussing TUF & Warehouse much more in April.

Longer answer:

The folks working on Warehouse have gotten funding to concentrate on improving Warehouse's security, and have kicked off work (funded by the Open Technology Fund) towards multi-factor auth, API keys, and an audit trail. And -- to quote the blog post --

Facebook... has provided the Python Software Foundation with a monetary gift that will be used to fund the development and deployment of enhanced security features to PyPI....

The PSF Packaging Working Group plans to use these funds to implement highly requested security features in PyPI such as cryptographic signing and verification of files uploaded and installed from the index. Additionally, systems for the automated detection of malicious uploads will lower the time to response and improve the resiliency of PyPI against attacks such as "pytosquatting".

This work will be undertaken in the second half of 2019 but planning will begin in the second quarter of the year.

We anticipate that in mid-April (so, basically within about a month) we'll be announcing a formal Request For Information to ask people to tell us about their interest in being contracted to do this work, and that part of that discussion will be further, more detailed conversations about whether TUF is the right tool for this job. So please watch for that, on this issue and on https://discuss.python.org/c/packaging .

(cc @pradyunsg since I think you're interested in this.)

[JustinCappos]

From the TUF side, we're very interested in moving this forward. Let us know what we can do to help!

[LucidOne]

https://motherboard.vice.com/en_us/article/pan9wn/hackers-hijacked-asus-software-updates-to-install-backdoors-on-thousands-of-computers

[trishankatdatadog]

Same, happy to help with this, just let us know how.

[westurner]

"PyPI security work: multifactor auth progress & help needed"
https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-help-needed/1042/10

[brainwane]

At PyCon sprints several people spoke about the potential future of TUF in Warehouse and Python packaging, and put notes at https://docs.google.com/document/d/1Wz2-ECkicJgAmQDxMFivWmU2ZunKvPZ2UfQ59zDGj7g/ .

[trishankatdatadog]

Thanks, Sumana! I'm happy to help with the design and coding for this project @lukpueh @awwad

[ofek]

I'll also devote whatever time is necessary to get this done

[lukpueh]

Thanks for putting together our notes, @brainwane! It was a pleasure meeting you guys at PyCon. @ewdurbin, any news on the RFI?

@ofek and @trishankatdatadog, your help will be very much appreciated.