Changing Infohash from Unsupported Info-Fields

[da2ce7]

Parent: #275

Hi @da2ce7, I've been thinking about changing the infohash again, and I do not like the side effect. It has a similar effect to the "source" file. As we change the infohash:

• All the trackers listed on the torrent file are not going to work because the infohash has changed.
• Peer can only download the torrent from other peers using the same tracker.

What we are doing is kind of creating a new "normalized" torrent but somehow splitting the swarm.

In a public index/tracker, I think we should generate the same torrent, only adding the linked/associated tracker in the announce list.

Originally posted by @josecelano in #260 (comment)

[josecelano]

Hi @da2ce7, I'm working on a sample application to import torrents from an external source into the Torrust Index.

Repo: https://github.com/torrust/academic-torrents-importer (still private).

The process is:

1. Check if the torrent already exists in the Index using the API and the current infohash
2. Upload the torrent if it does not exist.

As the info-hash might change if I retry the request, the script is going to try to upload the same torrent again because checking whether the torrent exists or not will return a Not Found 404 (because we are using the original infohash).

I'm using the API endpoint GET /v1/torrent/{info_hash} to check if the torrent exists. I would need to check if a torrent with a given original info-hash exists (not the new info-hash). Currently, that's not possible either with the search or torrent details endpoint.

If we decide to continue changing the info-hash after uploading, we should provide a way for clients to know if a torrent with an original info-hash was already uploaded. It should tell clients the current info-hash for the torrent. Otherwise, we force people who want to migrate torrents to our Index to track info-hash changes, making migration tools more complex. And I think we should make migrating torrents from other indexes as easy as possible. For the time being, I'm going to re-upload the torrent and do not make the migration tool fail if the torrent already exists (current behavior), but that's an expensive action in comparison to just checking if the original info-hash was already uploaded with a current or new endpoint.

[josecelano]

By the way, @da2ce7 the API is returning a 400 status code if you try to POST a torrent with an info-hash that already exists (or title). Maybe we should return a 409 status code. See https://stackoverflow.com/a/70371989/3012842

[josecelano]

@da2ce7 and I have been discussing it today. @da2ce7 proposed the following approach:

Current solution

The torrust_torrents table contains an original_info_hash field. It was added in the PR.

If you upload a torrent and the infohash changes (because we discard non-standard fields in the info dictionary), we store the original infohash in that field in the database.

When you try to get the torrent info from the API endpoint GET /v1/torrent/{info_hash}, you get a Not Found 404 response if there is no torrent in the torrust_torrents matching the info_hash column, but it does not check if there is a record with the columns original_info_hash matching that info-hash.

Cameron's proposal

We should create a new table: torrust_torrent_info_hashes.

CREATE TABLE "torrust_torrent_info_hashes" (
	"torrent_id"	INTEGER NOT NULL,
	"original_info_hash"	TEXT NOT NULL,
	"canonical_info_hash"	TEXT NOT NULL,
	PRIMARY KEY("original_info_hash","canonical_info_hash"),
	FOREIGN KEY("torrent_id") REFERENCES "torrust_torrents"("torrent_id")
)

• canonical_info_hash is the info-hash we assign to the torrent after uploading it to the Index.
• original_info_hash is the original torrent infohash. It's the same value we are string in the torrust_torrents::original_info_hash field.

The relationship is one-to-many between the torrent and all its original info-hashes.

We should migrate the data from the torrust_torrents::original_info_hash column to the new table. And delete the torrust_torrents::original_info_hash column field after the migration.

The new table should also contain a record for the canonical info-hash. That means there should be at least one record for each torrent where the original_info_hash and the canonical_info_hash are the same.

Regarding the API ...

In the GET /v1/torrent/{info_hash} endpoint, we should

• Look up in the new table to check if the torrent exists.
• If the torrent exists and the request uses the canonical info-hash the response should be the same as the current behavior.
• If the torrent exists but the info-hash is not the canonical one, we should return a redirection 302 pointing to the URL using the canonical info-hash.

In the POST /v1/torrent/upload endpoint, we should:

• Look up in the new table to check if the torrent exists.
• Use the new table to check if the torrent already exists.
• If the torrent exists, it should return a [409 Conflict](409 Conflict) status code. If the info-hash is not the canonical one, it should make it clear with an error message like "the torrent was already uploaded with infohash {ORIGINAL_INFOHASH} generating the new canonical infohash {CANONICAL_INFOHASH}".

Is that OK, @da2ce7, or am I missing something?