Description
# HF Transfer Speed up file transfers with the Hub. # DISCLAIMER This library is a power user tool, to go beyond `~500MB/s` on very high bandwidth network, where Python cannot cap out the available bandwidth. This is *not* meant to be a general usability tool. It purposefully lacks progressbars and comes generally as-is. Please file issues *only* if there's an issue on the underlying downloaded file. ## Contributing ```sh python3 -m venv ~/.venv/hf_transfer source ~/.venv/hf_transfer/bin/activate pip install maturin maturin develop ``` ### `huggingface_hub` If you are working on changes with `huggingface_hub` ```sh git clone git@github.com:huggingface/huggingface_hub.git # git clone https://github.com/huggingface/huggingface_hub.git cd huggingface_hub python3 -m pip install -e ".[quality]" ``` You can use the following test script: ```py import os # os.environ["HF_ENDPOINT"] = "http://localhost:5564" os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" from huggingface_hub import HfApi, logging logging.set_verbosity_debug() hf = HfApi() hf.upload_file(path_or_fileobj="/path/to/my/repo/some_file", path_in_repo="some_file", repo_id="my/repo", repo_type="model") ```
Release History
| Version | Changes | Urgency | Date |
|---|---|---|---|
| 0.1.9 | Imported from PyPI (0.1.9) | Low | 4/21/2026 |
| v0.1.9 | ## What's Changed * feat: bump `hf_transfer` dev version to `0.1.9` by @McPatate in https://github.com/huggingface/hf_transfer/pull/41 * Add license file to pyproject.toml by @mcaccin in https://github.com/huggingface/hf_transfer/pull/50 * Refreshing the lib slightly. by @Narsil in https://github.com/huggingface/hf_transfer/pull/52 ## New Contributors * @mcaccin made their first contribution in https://github.com/huggingface/hf_transfer/pull/50 **Full Changelog**: https://github.com/hu | Low | 1/7/2025 |
| v0.1.9rc0 | ## What's Changed * feat: bump `hf_transfer` dev version to `0.1.9` by @McPatate in https://github.com/huggingface/hf_transfer/pull/41 * Add license file to pyproject.toml by @mcaccin in https://github.com/huggingface/hf_transfer/pull/50 * Refreshing the lib slightly. by @Narsil in https://github.com/huggingface/hf_transfer/pull/52 ## New Contributors * @mcaccin made their first contribution in https://github.com/huggingface/hf_transfer/pull/50 **Full Changelog**: https://github.com/hu | Low | 12/30/2024 |
| v0.1.8 | ## What's Changed * fix: forward all headers on identical host by @McPatate in https://github.com/huggingface/hf_transfer/pull/45 **Full Changelog**: https://github.com/huggingface/hf_transfer/compare/v0.1.7...v0.1.8 | Low | 7/23/2024 |
| v0.1.7 | ## What's Changed * feat: use final redirect URL for chunk download by @McPatate in https://github.com/huggingface/hf_transfer/pull/39 * feat: display a better error msg on status error by @McPatate in https://github.com/huggingface/hf_transfer/pull/44 ## New Contributors * @dtrifiro made their first contribution in https://github.com/huggingface/hf_transfer/pull/38 * @akx made their first contribution in https://github.com/huggingface/hf_transfer/pull/43 **Full Changelog**: https://gi | Low | 7/23/2024 |
| v0.1.6 | ## What's Changed * Update dev version after release. by @Narsil in https://github.com/huggingface/hf_transfer/pull/24 * reduce default (90s) keepalive to 15s by @XciD in https://github.com/huggingface/hf_transfer/pull/25 * Upgrading dependencies. by @Narsil in https://github.com/huggingface/hf_transfer/pull/26 **Full Changelog**: https://github.com/huggingface/hf_transfer/compare/v0.1.5...v0.1.6 | Low | 2/29/2024 |
| v0.1.5 | ## What's Changed * feat: add upload to s3 example by @McPatate in https://github.com/huggingface/hf_transfer/pull/8 * Create LICENSE by @Narsil in https://github.com/huggingface/hf_transfer/pull/11 * Make it clearer that the library is provided as-is and is not meant by @Narsil in https://github.com/huggingface/hf_transfer/pull/13 * Download callback by @cbensimon in https://github.com/huggingface/hf_transfer/pull/14 * Adding Version to hf_transfer. by @Narsil in https://github.com/hugging | Low | 1/23/2024 |
| v0.1.4 | Release v0.1.4 | Low | 11/6/2023 |
| v0.1.4rc1 | ## What's Changed * feat: add upload to s3 example by @McPatate in https://github.com/huggingface/hf_transfer/pull/8 * Create LICENSE by @Narsil in https://github.com/huggingface/hf_transfer/pull/11 * Make it clearer that the library is provided as-is and is not meant by @Narsil in https://github.com/huggingface/hf_transfer/pull/13 * Download callback by @cbensimon in https://github.com/huggingface/hf_transfer/pull/14 * Adding Version to hf_transfer. by @Narsil in https://github.com/hugging | Low | 10/31/2023 |
| v0.1.3 | ## What's Changed * Other retry implem. by @Narsil in https://github.com/huggingface/hf_transfer/pull/3 * Document how to tune the number of threads one can use in hf_transfer by @thomasw21 in https://github.com/huggingface/hf_transfer/pull/6 * Fixing headers for private repos. by @Narsil in https://github.com/huggingface/hf_transfer/pull/5 * Multipart upload by @McPatate in https://github.com/huggingface/hf_transfer/pull/7 ## New Contributors * @thomasw21 made their first contribution i | Low | 3/24/2023 |
| v0.1.2 | **Full Changelog**: https://github.com/huggingface/hf_transfer/compare/v0.1.1...v0.1.2 | Low | 3/3/2023 |
| v0.1.1 | ## What's Changed * With S3 signed URL, we can't do an HEAD. by @XciD in https://github.com/huggingface/hf_transfer/pull/1 ## New Contributors * @XciD made their first contribution in https://github.com/huggingface/hf_transfer/pull/1 **Full Changelog**: https://github.com/huggingface/hf_transfer/compare/v0.1.0...v0.1.1 | Low | 2/20/2023 |
| v0.1.0 | Initial release. **Full Changelog**: https://github.com/huggingface/hf_transfer/commits/v0.1.0 | Low | 12/16/2022 |