Endpoint to create random torrents #237

josecelano · 2023-07-27T17:18:58Z

Relates to: torrust/torrust-index-gui#185

Sometimes we need to generate a random torrent for testing purposes.

We need to generate test torrents for the frontend application. With this new endpoint, you can get a random torrent:

http://0.0.0.0:3001/v1/torrents/random

The torrent is a single-file torrent using a UUID for its name and the original data file contents.

In this repo, we use the imdl command to generate random torrents but in the frontend we are using cypress, and it seems the best option is to make a request to an endpoint to obtain dynamic fixtures like random (or customized in the future) torrents.

torrust/torrust-index-gui#185

TODO

Generate the random ID
Calculate the correct value for the "pieces" field in the torrent file (sha1 hash of the contents).
Refactor: extract service. Remove the domain code from the handler.
Refactor: add a new named the constructor for the Torrent instead of using the from_db_info_files_and_announce_urls. We do not need the torrent_id, for example.

Other things that we could implement:

Add an env var to enable this endpoint only for development/testing environments.
Add an URL parameter with the UUDI: v1/torrents/random/:uuid. We use the UUID for the torrent name (torrent file name: name.torrent).

josecelano · 2023-07-27T17:28:57Z

Hi @da2ce7, before continuing with this PR I'm going to check if the solution fits well with the frontend. I can use this new endpoint in a cypress test.

josecelano · 2023-07-28T20:23:30Z

Hi @da2ce7, this is the current status:

There is a new endpoint: GET http://localhost:3001/torrents/random that returns a random torrent.
That endpoint is always available: development and production
It generates a random UUID. There is no option yet to define a custom test torrent (customs torrent fields or custom torrent data from which the torrent file is generated).

I have tested the solution in the frontend, and it works. The code is still a draft version. Maybe It would look better if we pre-generated the UUID and then we requested the test torrent with a predefined UUID. The endpoint could be like GET http://localhost:3001/test-torrent/:uuid

This is the Cypress E2E test in the fronted (pending refactoring);

describe("A registered user", () => {
  let registration_form: RegistrationForm;

  before(() => {
    registration_form = random_user_registration_data();

    cy.visit("/");
    cy.visit("/signup");
    cy.register(registration_form);
    cy.login(registration_form.username, registration_form.password);
  });

  it("should be able to upload a torrent", () => {
    let torrentId = "";

    cy.log("download a random torrent for the test");

    cy.request({
      url: "http://localhost:3001/v1/torrents/random",
      encoding: "binary"
    }).then((response) => {
      cy.log("random torrent downloaded");

      const header = response.headers["content-disposition"];

      // todo: extract
      let contentDisposition: string;
      if (typeof header === "string") {
        contentDisposition = header;
      } else {
        contentDisposition = header[0];
      }

      const filename = extractFilename(contentDisposition);
      const torrentPath = `cypress/fixtures/torrents/${filename}`;
      torrentId = extractTorrentIdFromFilename(filename);

      // Alias the torrentId for later use
      // We do not need this part if we generate the UUDI here in the frontend
      cy.wrap(torrentId).as("torrentId");

      cy.log("torrent ID: ", torrentId);
      cy.log("random torrent downloaded: ", torrentPath);

      cy.writeFile(torrentPath, response.body, "binary");
    });

    cy.visit("/upload");

    cy.get("@torrentId").then((torrentId) => {
      cy.get("input[data-cy=\"upload-form-title\"]").type(`title-${torrentId}`);
      cy.get("textarea[data-cy=\"upload-form-description\"]").type(`description-${torrentId}`);
    });
    cy.get("select[data-cy=\"upload-form-category\"]").select("software");

    // todo: add tags.
    // cy.get("input[data-cy=\"upload-form-tags\"]").select('fractals');

    cy.get("@torrentId").then((torrentId) => {
      cy.get("input[data-cy=\"upload-form-torrent-upload\"]").selectFile(
        {
          contents: `cypress/fixtures/torrents/file-${torrentId}.txt.torrent`,
          fileName: `file-${torrentId}.torrent`,
          mimeType: "application/x-bittorrent"
        }, { force: true });
    });

    cy.get("button[data-cy=\"upload-form-submit\"]").click();

    cy.url().should("include", "/torrent/");
  });
});

Questions

We can leave the endpoint for production too. The only problem I see is a DDoS attack. Still, we already have another public endpoint (for example, to download a torrent file), so we need to solve that problem with a rate limit or other solution anyway.
I will implement the other approach: GET http://localhost:3001/test-torrent/:uuid. In the future, we could add another endpoint where we can POST the full description of the test torrent. We could POST something like this:

{
   "announce-list": [],
   "info": {
      "length": 37,
      "name": "file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt",
      "piece length": 16384,
      "pieces": "<hex>3E E7 F3 33 EA A5 CC D1 33 84 A3 85 F9 32 6B 2E 18 0F FB 20</hex>",
      "private": 0
   }
}

This is the data we need:

    let torrent_info = DbTorrentInfo {
        torrent_id: 1,
        info_hash: String::new(),
        name: format!("file-{id}.txt"),
        pieces: sha1(&file_contents),
        piece_length: 16384,
        private: None,
        root_hash: 0,
    };

    let torrent_files: Vec<TorrentFile> = vec![TorrentFile {
        path: vec![String::new()],
        length: 37, // Number of bytes for the UUID plus one char for line break (`0a`).
        md5sum: None,
    }];

    let torrent_announce_urls: Vec<Vec<String>> = vec![];

How the Test Torrents are generated

The process to create a random torrent is as follows:

Generate a random UUID.
Create a text file whose name contains the ID generated in the previous step.
Write the ID into the text file.
Create the torrent file for the text file.

NOTICE: I'm only using single-file torrents.

That's what I'm doing in the Axum handler. You can do the same manually with the following steps:

Generate a random UUID: d6170378-2c14-4ccc-870d-2a8e15195e23 (you can use an online generator).

Create a random text file: file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt

Open the file with a text editor and write the UUID.

This should be the file contents:

$ xxd file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt 
00000000: 6436 3137 3033 3738 2d32 6331 342d 3463  d6170378-2c14-4c
00000010: 6363 2d38 3730 642d 3261 3865 3135 3139  cc-870d-2a8e1519
00000020: 3565 3233 0a                             5e23.

NOTICE: 0a is the line break char.

Then you can create the torrent file with the intermodal console command.

$ imdl torrent create file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt
[1/3] 🧿 Searching `file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt` for files…
[2/3] 🧮 Hashing pieces…
[3/3] 💾 Writing metainfo to `file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt.torrent`…
✨✨ Done! ✨✨

The torrent file contents are:

$ xxd file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt.torrent 
00000000: 6431 303a 6372 6561 7465 6420 6279 3131  d10:created by11
00000010: 3a69 6d64 6c2f 302e 312e 3132 3133 3a63  :imdl/0.1.1213:c
00000020: 7265 6174 696f 6e20 6461 7465 6931 3639  reation datei169
00000030: 3035 3337 3332 3365 383a 656e 636f 6469  0537323e8:encodi
00000040: 6e67 353a 5554 462d 3834 3a69 6e66 6f64  ng5:UTF-84:infod
00000050: 363a 6c65 6e67 7468 6933 3765 343a 6e61  6:lengthi37e4:na
00000060: 6d65 3435 3a66 696c 652d 6436 3137 3033  me45:file-d61703
00000070: 3738 2d32 6331 342d 3463 6363 2d38 3730  78-2c14-4ccc-870
00000080: 642d 3261 3865 3135 3139 3565 3233 2e74  d-2a8e15195e23.t
00000090: 7874 3132 3a70 6965 6365 206c 656e 6774  xt12:piece lengt
000000a0: 6869 3136 3338 3465 363a 7069 6563 6573  hi16384e6:pieces
000000b0: 3230 3a3e e7f3 33ea a5cc d133 84a3 85f9  20:>..3....3....
000000c0: 326b 2e18 0ffb 2065 65                   2k.... ee

You can also get the info using imdl with:

$ imdl torrent show -j file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt.torrent | jq

The output is this JSON:

{
  "name": "file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt",
  "comment": null,
  "creation_date": 1690537323,
  "created_by": "imdl/0.1.12",
  "source": null,
  "info_hash": "a366418d2ac082bcdd57ddae3449ab4ad52f6a84",
  "torrent_size": 201,
  "content_size": 37,
  "private": false,
  "tracker": null,
  "announce_list": [],
  "update_url": null,
  "dht_nodes": [],
  "piece_size": 16384,
  "piece_count": 1,
  "file_count": 1,
  "files": [
    "file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt"
  ]
}

You can also use other tools to extract the torrent info like:

https://chocobo1.github.io/bencode_online/

{
   "created by": "imdl/0.1.12",
   "creation date": 1690537323,
   "encoding": "UTF-8",
   "info": {
      "length": 37,
      "name": "file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt",
      "piece length": 16384,
      "pieces": "<hex>3E E7 F3 33 EA A5 CC D1 33 84 A3 85 F9 32 6B 2E 18 0F FB 20</hex>"
   }
}

If you upload the test torrent to the Turrsut index and then you download the torrent file again, you get this torrent file:

{
   "announce-list": [],
   "info": {
      "length": 37,
      "name": "file-d6170378-2c14-4ccc-870d-2a8e15195e23.txt",
      "piece length": 16384,
      "pieces": "<hex>3E E7 F3 33 EA A5 CC D1 33 84 A3 85 F9 32 6B 2E 18 0F FB 20</hex>",
      "private": 0
   }
}

NOTICE: we are not including some fields like created by, creation date, etc.

And finally, you can also generate the "pieces" field in the torrent file with:

$ echo -n -e "d6170378-2c14-4ccc-870d-2a8e15195e23\n" | openssl dgst -sha1 -binary | xxd -p
3ee7f333eaa5ccd13384a385f9326b2e180ffb20

In our database:

We could write a blog post on torrust.com about generating test torrents.

Other considerations

Now that we know how to quickly build a test torrent, we could remove the development dependency on the imdl console command to run the tests. We can extract a module for the generation of test torrents. In fact, that's a refactoring I have to do before merging this PR. I think test are going to be faster. In the current implementation to generate random torrents we first create the torrent file and then we read the contents from the disk:

#[derive(Clone)]
pub struct TestTorrent {
    /// Parsed info from torrent file.
    pub file_info: TorrentFileInfo,
    /// Torrent info needed to add the torrent to the index.
    pub index_info: TorrentIndexInfo,
}

impl TestTorrent {
    pub fn random() -> Self {
        let temp_dir = temp_dir();

        let torrents_dir_path = temp_dir.path().to_owned();

        // Random ID to identify all the torrent related entities: files, fields, ...
        // That makes easier to debug the tests outputs.
        let id = Uuid::new_v4();

        // Create a random torrent file
        let torrent_path = random_torrent_file(&torrents_dir_path, &id);

        // Load torrent binary file
        let torrent_file = BinaryFile::from_file_at_path(&torrent_path);

        // Load torrent file metadata
        let torrent_info = parse_torrent(&torrent_path);

        let torrent_to_index = TorrentIndexInfo {
            title: format!("title-{id}"),
            description: format!("description-{id}"),
            category: software_predefined_category_name(),
            torrent_file,
        };

        TestTorrent {
            file_info: torrent_info,
            index_info: torrent_to_index,
        }
    }

    pub fn info_hash(&self) -> InfoHash {
        self.file_info.info_hash.clone()
    }

With this solution, we do not new the file system.

We need them to generate random torrents. We use an UUID for the torrent name and contents and we nned the hex package to generate the torrent file "pieces" field.

For now, it will be used only for testing purposes. We need to generate random torrent in Cypress in the Index Frontend app.

The new enpopint is: `http://0.0.0.0:3001/v1/torrent/meta-info/random/:uuid` The segments have changed to differenciate the indexed torrent from the torrent file (meta-indo). An indexed torrent is a torrent file (meta-info file) with some extra classification metadata: title, description, category and tags. There is also a new PATH param `:uuid` which is an UUID to identify the generated torrent file. The UUID is used for: - The torrent file name - The sample contents for a text file ffrom which we generate the torrent file.

codecov-commenter · 2023-07-31T15:11:40Z

Codecov Report

Merging #237 (4b6f25c) into develop (4415144) will increase coverage by 1.12%.
The diff coverage is 80.00%.

@@             Coverage Diff             @@
##           develop     #237      +/-   ##
===========================================
+ Coverage    57.50%   58.62%   +1.12%     
===========================================
  Files          126      128       +2     
  Lines         7377     7490     +113     
===========================================
+ Hits          4242     4391     +149     
+ Misses        3135     3099      -36

Flag	Coverage Δ
rust	`58.62% <80.00%> (+1.12%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
src/models/info_hash.rs	`88.46% <0.00%> (ø)`
src/web/api/v1/contexts/torrent/responses.rs	`0.00% <0.00%> (ø)`
src/web/api/v1/contexts/torrent/handlers.rs	`52.80% <13.33%> (-3.82%)`	⬇️
src/models/torrent_file.rs	`75.71% <100.00%> (+44.46%)`	⬆️
src/services/hasher.rs	`100.00% <100.00%> (ø)`
src/services/torrent_file.rs	`100.00% <100.00%> (ø)`
src/web/api/v1/contexts/torrent/routes.rs	`100.00% <100.00%> (ø)`

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

to generate random torrent files. It was moved from the handler.

josecelano · 2023-07-31T15:26:08Z

Hi @da2ce7, This is ready. Finally, the endpoint is like this:

GET http://0.0.0.0:3001/v1/torrent/meta-info/random/d6170378-2c14-4ccc-870d-2a8e15195e23

And it's public and always enabled (in production too).

As I explained above, the endpoint generates a torrent file (meta-info info) for a single text file whose contents are the UUID.

The test in the frontend is now much cleaner:

  it("should be able to upload a torrent", () => {
    const torrent_info = generateRandomTestTorrentInfo();

    cy.request({
      url: `http://localhost:3001/v1/torrent/meta-info/random/${torrent_info.id}`,
      encoding: "binary"
    }).then((response) => {
      cy.log("random torrent downloaded to: ", torrent_info.path);
      cy.writeFile(torrent_info.path, response.body, "binary");
    });

    cy.visit("/upload");

    cy.get("input[data-cy=\"upload-form-title\"]").type(torrent_info.title);
    cy.get("textarea[data-cy=\"upload-form-description\"]").type(torrent_info.description);
    cy.get("select[data-cy=\"upload-form-category\"]").select("software");
    cy.get("input[data-cy=\"upload-form-torrent-upload\"]").selectFile(
      {
        contents: torrent_info.path,
        fileName: torrent_info.filename,
        mimeType: "application/x-bittorrent"
      }, { force: true });

    cy.get("button[data-cy=\"upload-form-submit\"]").click();

    // It should redirect to the torrent detail page.
    cy.url().should("include", "/torrent/");

    cy.exec(`rm ${torrent_info.path}`);
  });

da2ce7

Looks Simple and Clear, Good Work

The constructor to hidrate the object from the database should depeden on the other to create a new Torrent from scracth. In fact, the `torrent_id` and `info_hash` in the `DbTorrentInfo` are not needed.

josecelano · 2023-08-01T09:09:39Z

I won't add tests for the new endpoint or write documentation because it's an endpoint we are only using internally for the time being.

josecelano requested a review from da2ce7 July 27, 2023 17:19

josecelano added 4 commits July 31, 2023 13:10

chore: add dependencies: hex, uuid

dd1dc0c

We need them to generate random torrents. We use an UUID for the torrent name and contents and we nned the hex package to generate the torrent file "pieces" field.

feat: new endpoint to generate random torrents

30bf79e

For now, it will be used only for testing purposes. We need to generate random torrent in Cypress in the Index Frontend app.

fix: clippy warnings alter updating clippy to clippy 0.1.73

dfa260e

refactor: extract torrent file service

b2870b9

to generate random torrent files. It was moved from the handler.

josecelano marked this pull request as ready for review July 31, 2023 15:20

josecelano added 2 commits July 31, 2023 16:36

refactor: extract hasher service

40c4df0

test: add test for random torrent file generator service

4b6f25c

josecelano marked this pull request as draft July 31, 2023 16:03

josecelano marked this pull request as ready for review July 31, 2023 16:24

josecelano mentioned this pull request Jul 31, 2023

E2E test: torrent upload torrust/torrust-index-gui#185

Merged

da2ce7 approved these changes Jul 31, 2023

View reviewed changes

refactor: invert the dependency between Torrent named constructors

0f163cf

The constructor to hidrate the object from the database should depeden on the other to create a new Torrent from scracth. In fact, the `torrent_id` and `info_hash` in the `DbTorrentInfo` are not needed.

josecelano merged commit e2a0ed4 into torrust:develop Aug 1, 2023

josecelano mentioned this pull request Aug 1, 2023

Remove dependency on imdl to run tests? #241

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Endpoint to create random torrents #237

Endpoint to create random torrents #237

Uh oh!

josecelano commented Jul 27, 2023 •

edited

Loading

Uh oh!

josecelano commented Jul 27, 2023

Uh oh!

josecelano commented Jul 28, 2023 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 31, 2023 •

edited

Loading

Uh oh!

josecelano commented Jul 31, 2023

Uh oh!

da2ce7 left a comment

Uh oh!

josecelano commented Aug 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Endpoint to create random torrents #237

Endpoint to create random torrents #237

Uh oh!

Conversation

josecelano commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

josecelano commented Jul 27, 2023

Uh oh!

josecelano commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Questions

How the Test Torrents are generated

Other considerations

Uh oh!

codecov-commenter commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

josecelano commented Jul 31, 2023

Uh oh!

da2ce7 left a comment

Choose a reason for hiding this comment

Uh oh!

josecelano commented Aug 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

josecelano commented Jul 27, 2023 •

edited

Loading

josecelano commented Jul 28, 2023 •

edited

Loading

codecov-commenter commented Jul 31, 2023 •

edited

Loading