RSS feed for new file uploads to existing packages

[edmorley]

Hi!

Currently there are two RSS feeds available (added by #990):

• https://pypi.org/rss/updates.xml -> "40 latest updates"
• https://pypi.org/rss/packages.xml -> "40 newest packages"

The former allows for monitoring of when a new version of a package is released, however doesn't show when new uploads to an existing package occur - which is a problem, since it's not unusual for files to be uploaded to a package in a staggered manner.

For example:

• twine uploading sdist and multiple wheels sequentially over a slow connection.
• projects using a combined Travis+AppVeyor CI solution to build the sdist and manylinux/Windows/OS X wheels (since no one CI solution covers all platforms), where the two CI systems get out of sync due to backlogs (see 2.3.1 pypi broken aio-libs/aiohttp#2347 and Large lag between Windows and Linux/OS X PyPI uploads aio-libs/multidict#182).
• projects uploading a wheel for the first time to an existing release that was previously sdist only.

This causes issues for projects using pip's --require-hashes mode along with tooling that automatically updates dependencies in requirements files (such as https://pyup.io/ ), since:

1. pyup.io sees the updates.xml RSS feed saying a new package version has been published
2. a PR is opened updating the requirements file with the new package version and hashes of whatever file uploads exist at that point in time
3. the tests may or may not pass at that point (depending on if the sdist or an appropriate wheel exists; in the aiohttp case the sdist wasn't uploaded for hours later)
4. even if the tests pass then, they may break later after the PR was merged, if an appropriate wheel becomes available, since pip prefers wheels over the sdist, and the requirements file will only contain the hash of the sdist

If an RSS feed were available of "40 newest file uploads" or even "40 newest changes of any type, whether that be metadata or uploads", then tools like pyup.io could use it to automatically update the requirements files hashes as uploads occurred (presumably with a slight delay to batch changes).

See:
pyupio/pyup#250

Note: This is different from #1683 / #2165, since they are about a feed for just one package (and I think still only at the releases level, not the uploads level), rather than a feed for all packages site-wide.

CC @jayfk

[di]

I think this makes sense and is doable. We occasionally have the same issue when using pyup as well, so we feel your pain. 🙂

[brainwane]

Folks who need this might want to check whether the Libraries.io API for https://libraries.io/pypi might suit their needs in the short term.

[alexbecker]

This is also a problem for python mirroring at PyDist--there's no way to know whether a new distribution has been added, so the only real options are to:

• ignore new distributions (perhaps re-query each release a short time after the first distribution for it is published)
• periodically re-query every release of every package to find any new distributions

The former is not great if package maintainers are going back and publishing new wheels when new versions of Python are released, or when they start building for more platforms. But the latter seems like a lot of load for both the mirror and PyPI.