Store all torrent fields in the database

[josecelano]

Relates to: #285

Currently, we have these fields when we parse/decode the torrent file:

#[derive(PartialEq, Debug, Clone, Serialize, Deserialize)]
pub struct Torrent {
    pub info: TorrentInfo, //
    #[serde(default)]
    pub announce: Option<String>,
    #[serde(default)]
    pub nodes: Option<Vec<TorrentNode>>,
    #[serde(default)]
    pub encoding: Option<String>,
    #[serde(default)]
    pub httpseeds: Option<Vec<String>>,
    #[serde(default)]
    #[serde(rename = "announce-list")]
    pub announce_list: Option<Vec<Vec<String>>>,
    #[serde(default)]
    #[serde(rename = "creation date")]
    pub creation_date: Option<i64>,
    #[serde(rename = "comment")]
    pub comment: Option<String>,
    #[serde(default)]
    #[serde(rename = "created by")]
    pub created_by: Option<String>,
}

Some are not persisted in the database. We could persist all fields in the Torrent struct. Some fields are missing in the database like:

• nodes
• encoding
• httpseeds
• creation_date
• comment
• created_by

CREATE TABLE "torrust_torrents" (
	"torrent_id"	INTEGER NOT NULL,
	"uploader_id"	INTEGER NOT NULL,
	"category_id"	INTEGER,
	"info_hash"	TEXT NOT NULL UNIQUE,
	"size"	INTEGER NOT NULL,
	"name"	TEXT NOT NULL,
	"pieces"	TEXT NOT NULL,
	"piece_length"	INTEGER NOT NULL,
	"private"	BOOLEAN DEFAULT NULL,
	"root_hash"	INT NOT NULL DEFAULT 0,
	"date_uploaded"	TEXT NOT NULL,
	"source"	TEXT DEFAULT NULL,
	FOREIGN KEY("uploader_id") REFERENCES "torrust_users"("user_id") ON DELETE CASCADE,
	FOREIGN KEY("category_id") REFERENCES "torrust_categories"("category_id") ON DELETE SET NULL,
	PRIMARY KEY("torrent_id" AUTOINCREMENT)
);

Subtasks

• Store in the database the torrent fields: creation_date, comment, created_by, encoding #296
• Store in the database the torrent field: httpseeds #438. BEP 17.
• Store in the database the torrent field: nodes #437. BEP 5.

[josecelano]

Should the creation date be the original, upload or download torrent date? @da2ce7

[da2ce7]

Should the creation date be the original, upload or download torrent date? @da2ce7

I think that it should be the original. Since it is closest to the real date. However dose the GUI show this date?

[josecelano]

Should the creation date be the original, upload or download torrent date? @da2ce7

I think that it should be the original. Since it is closest to the real date. However dose the GUI show this date?

No, It doesn't.

It only shows the upload date and the canonical info-hash

[da2ce7]

We have the problem that there is a many-to-one relationship with the original creation date, and our torrent in the DB... 😮‍💨

[josecelano]

We have the problem that there is a many-to-one relationship with the original creation date, and our torrent in the DB... 😮‍💨

Maybe we should keep the original creation date is the info-hash does not change, and update the creation date to the upload date if the canonical info-hash is different. As I see it we are creating a new torrent because the torrent identity is defined by its info-hash.

[da2ce7]

So the torrent creation date is canonical. Then that is easy: it is a one-to-one relationship. 😄

[josecelano]

Hi @da2ce7, Maybe some fields were not included intentionally because they are described in BEPs we do not support or because they are not officially defined in any BEP. Anyway, i would include them.

Original unofficial specification

creation_date:

BEP: There's no specific BEP for "creation_date".

Description: This field is an optional key that contains the creation time of the torrent, in standard UNIX epoch format (seconds since 1-Jan-1970 00:00:00 UTC).

comment:

BEP: Again, there's no specific BEP for "comment". It's part of the original unofficial specification.

Description: An optional field that contains free-form comments for the torrent. It's essentially a text field where the creator of the torrent can put any desired information.

created_by:

BEP: Like "comment" and "creation_date", there's no specific BEP for "created_by". It's part of the original unofficial specification.

Description: Another optional field that identifies the software used to create the .torrent file.

encoding:

BEP: There's no specific BEP that defines the "encoding" field. It's part of the original unofficial specification.

Description: It specifies the character encoding used for various strings within the torrent meta-info, such as the "comment" or "created by" fields. The most common encoding is UTF-8.

These fields are from the original unofficial specification of the BitTorrent protocol, all of them are widely recognized and used by most modern BitTorrent clients.

BEP 5: DHT Protocol

nodes:

• BEP 5

Description: This BEP defines the Distributed Hash Table (DHT) protocol that BitTorrent clients use to find peers without using a central tracker. The "nodes" field in a .torrent file provides an initial list of nodes for bootstrapping into the DHT.

BEP 32: IPv6 extension for DHT

nodes6:

• BEP 32

BEP 17: HTTP Seeding (Hoffman-style)

httpseeds:

• BEP 17

In the main area of the metadata file and not part of the "info" section, will be a new key, "httpseeds". This key will refer to a list of URLs, and will contain a list of web addresses where torrent data can be retrieved. This key may be safely ignored if the client is not capable of using it.

BEP 19: HTTP/FTP Seeding (GetRight-style)

url-list:

• BEP 19

Using HTTP or FTP servers as seeds for BitTorrent downloads.

[da2ce7]

Hello @josecelano

I like idea of including these unofficial fields. However we should be careful that the uploaded torrents may not have valid or even malicious data that is uploaded into these fields.

[josecelano]

Hello @josecelano

I like idea of including these unofficial fields. However we should be careful that the uploaded torrents may not have valid or even malicious data that is uploaded into these fields.

Hi @da2ce7 Maybe we should draft a new BEP collecting and describing all the unofficial fields in order to make them official. I think that would be a really great contribution.

[da2ce7]

@josecelano I would love to do that once I find time. I would use our https://github.com/torrust/teps repo for the draft, then once we are happy with it; we can submit it to https://www.bittorrent.org/

[josecelano]

@josecelano I would love to do that once I find time. I would use our https://github.com/torrust/teps repo for the draft, then once we are happy with it; we can submit it to https://www.bittorrent.org/

OK, then I think I can start implementing this issue so we have more info for the TEP/BEP.

Besides, I have the database with the 2400 torrents from academictorrenst, so we can check what values those fields have.