Skip to content

Conversation

@ardaatahan
Copy link
Contributor

@ardaatahan ardaatahan commented Feb 11, 2025

This PR aims to bring over huggingface_hub Python library's resumable downloads logic to swift-transformers:

  • Implements chunked transfer and retry logic for more reliable downloads
  • Adds support for resumable downloads through range headers
  • Adds configurable parameters for timeout, retries, and chunk size
  • Improves error handling and size verification for downloads
  • Uses temporary files during download to prevent corruption of existing files
  • Adds new tests to verify functionality of resumable downloads
  • Fixes testDeepSeekQwenChatTemplate function which was previously validating an outdated chat template output. The test now includes the correct tokens, referenced from the tokenizer repo

@pcuenca
Copy link
Member

pcuenca commented Feb 12, 2025

cc @greenrazer, would you like to give it a go at testing / reviewing this? or @FL33TW00D if you'd be up for it.

@FL33TW00D
Copy link
Contributor

FL33TW00D commented Feb 13, 2025

Just tested this out, looks great thank you for the contribution!

EDIT: seems the hub is giving transient 504s or the build would pass :)
EDIT2: Build is now passing, happy to merge this @pcuenca

@FL33TW00D FL33TW00D merged commit 4f97f98 into huggingface:main Feb 18, 2025
1 check passed
@DePasqualeOrg
Copy link
Contributor

@FL33TW00D, just a suggestion, but maybe squash the commits before merging next time, so that the repo's commit history doesn't get cluttered. I think you can also set this as a setting on the repo.

@FL33TW00D
Copy link
Contributor

@DePasqualeOrg Whoops! Used to my repos being set to squash by default. I don't seem to have the power to change it @pcuenca

@pcuenca
Copy link
Member

pcuenca commented Feb 20, 2025

Changed! Sorry, it used to be just me merging PRs :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants