trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
Why this rank:Strong adoptionRelease freshnessHealthy release cadence
Description
# Trafilatura: Discover and Extract Text Data on the Web <br/> <img alt="Trafilatura Logo" src="https://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-logo.png" align="center" width="60%"/> <br/> [](https://pypi.python.org/pypi/trafilatura) [](https://pypi.python.org/pypi/trafilatura) [](http://trafilatura.readthedocs.org/en/latest/?badge=latest) [](https://codecov.io/gh/adbar/trafilatura) [](https://pepy.tech/project/trafilatura) [](https://aclanthology.org/2021.acl-demo.15/) <br/> <img alt="Demo as GIF image" src="https://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-demo.gif" align="center" width="80%"/> <br/> ## Introduction Trafilatura is a cutting-edge **Python package and command-line tool** designed to **gather text on the Web and simplify the process of turning raw HTML into structured, meaningful data**. It includes all necessary discovery and text processing components to perform **web crawling, downloads, scraping, and extraction** of main texts, metadata and comments. It aims at staying **handy and modular**: no database is required, the output can be converted to commonly used formats. Going from HTML bulk to essential parts can alleviate many problems related to text quality, by **focusing on the actual content**, **avoiding the noise** caused by recurring elements like headers and footers and by **making sense of the data and metadata** with selected information. The extractor strikes a balance between limiting noise (precision) and including all valid parts (recall). It is **robust and reasonably fast**. Trafilatura is [widely used](https://trafilatura.readthedocs.io/en/latest/used-by.html) and integrated into [thousands of projects](https://github.com/adbar/trafilatura/network/dependents>) by companies like HuggingFace, IBM, and Microsoft Research as well as institutions like the Allen Institute, Stanford, the Tokyo Institute of Technology, and the University of Munich. ### Features - Advanced web crawling and text discovery: - Support for sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS) - Smart crawling and URL management (filtering and deduplication) - Parallel processing of online and offline input: - Live URLs, efficient and polite processing of download queues - Previously downloaded HTML files and parsed HTML trees - Robust and configurable extraction of key elements: - Main text (common patterns and generic algorithms like jusText and readability) - Metadata (title, author, date, site name, categories and tags) - Formatting and structure: paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting - Optional elements: comments, links, images, tables - Multiple output formats: - TXT and Markdown - CSV - JSON - HTML, XML and [XML-TEI](https://tei-c.org/) - Optional add-ons: - Language detection on extracted content - Speed optimizations - Actively maintained with support from the open-source community: - Regular updates, feature additions, and optimizations - Comprehensive documentation ### Evaluation and alternatives Trafilatura consistently outperforms other open-source libraries in text extraction benchmarks, showcasing its efficiency and accuracy in extracting web content. The extractor tries to strike a balance between limiting noise and including all valid parts. For more information see the [benchmark section](https://trafilatura.readthedocs.io/en/latest/evaluation.html) and the [evaluation readme](https://github.com/adbar/trafilatura/blob/master/tests/README.rst) to run the evaluation with the latest data and packages. #### Other evaluations: - Most efficient open-source library in *ScrapingHub*'s [article extraction benchmark](https://github.com/scrapinghub/article-extraction-benchmark) - Best overall tool according to [Bien choisir son outil d'extraction de contenu à partir du Web](https://hal.archives-ouvertes.fr/hal-02768510v3/document) (Lejeune & Barbaresi 2020) - Best single tool by ROUGE-LSum Mean F1 Page Scores in [An Empirical Comparison of Web Content Extraction Algorithms](https://webis.de/downloads/publications/papers/bevendorff_2023b.pdf) (Bevendorff et al. 2023) ## Usage and documentation [Getting started with Trafilatura](https://trafilatura.readthedocs.io/en/latest/quickstart.html) is straightforward. For more information and detailed guides, visit [Trafilatura's documentation](https://trafilatura.readthedocs.io/): - [Installation](https://t
Release History
| Version | Changes | Urgency | Date |
|---|---|---|---|
| v2.2.0 | Major changes: - Revamped text recovery and extraction sequence: better recall overall and better extraction on forums by @adbar (#881) - More targeted boilerplate pruning to reduce over-extraction by @adbar (#887) - More robust tests and better processing for nested elements by @adbar (#866) - Better handling of text in sub-elements by @adbar (#869) Fixes: - Improve title extraction by @novucs (#848) - Restore text inside nobr-tags by @kod-kristoff (#826) - Keep space after inline for | High | 7/31/2026 |
| v2.1.0 | Major changes: - Dependencies updated, lxml in particular (with minimal changes in the code) - Faster XPath performance using XSLT extensions by @Honesty-of-the-Cavernous-Tissue (#793) - More deprecation warnings - More robust code Fixes: - Fix node pruning raising an AttributeError in certain cases by @PLPeeters (#761) - Refine img src url and fix table extraction bugs by @unsleepy22 (#762) - Fix eager doctype regex matching by @UVMvmfee (#764) - Add extract_with_metadata method and | High | 6/7/2026 |
| 2.0.0 | Imported from PyPI (2.0.0) | Low | 4/21/2026 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v2.0.0 | Breaking changes: - Python 3.6 and 3.7 deprecated (#709) - `bare_extraction()`: - now returns an instance of the `Document` class by default - `as_dict` deprecation warning â use `.as_dict()` method on return value (#730) - `bare_extraction()` and `extract()`: `no_fallback` deprecation warning â use `fast` instead (#730) - downloads: remove `decode` argument in `fetch_url()` â use `fetch_response` instead (#724) - deprecated graphical user interface now removed (#713) - extraction: | Low | 12/3/2024 |
| v1.12.2 | - downloads: add support for SOCKS proxies with @gremid (#682) - extraction fix: ValueError in table spans (#685) - spider: `prune_xpath` parameter added by @felipehertzer (#684) - spider: relax strict parameter for link extraction (#687) - sitemaps: `max_sitemaps` parameter added by @felipehertzer (#690) - maintenance: make compression libraries optional (#691) - metadata: review and lint code (#694) | Low | 9/10/2024 |
| v1.12.1 | Navigation: - spider: restrict search to sections containing URL path (#673) - crawler: add parameter class and types, **breaking change** for undocumented functions (#675) - maintenance: simplify link discovery and extend tests (#674) - CLI: review code, add types and tests (#677) Bugfixes: - fix `AttributeError` in element deletion (#668) - fix `MemoryError` in table header columns (#665) Docs: - docs: fix variable name for extract_metadata in quickstart by @jpigla in #678 | Low | 8/20/2024 |
| v1.12.0 | Breaking change: - enforce fixed list of output formats, deprecate `-out` on the CLI (#647) Faster, more accurate extraction: - review link and structure checks (#653) - improve justext fallback (#652) - baseline: prevent LXML error in JSON-LD (#643), do not use as backup extraction (#646) - review XPaths for undesirable content (#645) Bugfixes and maintenance: - CLI fix: markdown format should trigger `include_formatting` (#649) - images fix: use a length threshold on src attribute | Low | 7/30/2024 |
| v1.11.0 | Breaking change: - metadata now skipped by default (#613), to trigger inclusion in all output formats: - `with_metadata=True` (Python) - `--with-metadata` (CLI) Extraction: - add HTML as output format (#614) - better and faster baseline extraction (#619) - better handling of HTML/XML elements (#628) - XPath rules added with @felipehertzer (#540) - fix: avoid faulty readability_lxml content (#635) Evaluation: - new scripts and data with @LydiaKoerber (#606, #615) - additio | Low | 6/27/2024 |
| v1.10.0 | Breaking changes: - raise errors on deprecated CLI and function arguments (#581) - regroup classes and functions linked to deduplication (#582) ``trafilatura.hashing`` â ``trafilatura.deduplication`` Extraction: - port of is_probably_readerable from readability.js by @zirkelc in #587 - Markdown table fixes by @naktinis in #601 - fix list spacing in TXT output (#598) - CLI fixes: file processing options, mtime, and tests (#605) - CLI fix: read standard input as binary (#607) Downloa | Low | 5/30/2024 |
| v1.9.0 | Extraction: - add markdown as explicit output (#550) - improve recall preset (#571) - speedup for readability-lxml (#547) - add global options object for extraction and use it in CLI (#552) - fix: better encoding detection (#548) - recall: fix for lists inside tables with @mikhainin (#534) - add symbol to preserve vertical spacing in Markdown (#499) - fix: table cell separators in non-XML output (#563) - slightly better accuracy and execution speed overall Metadata: - add file creat | Low | 5/2/2024 |
| v1.8.1 | Maintenance: - Pin LXML to prevent broken dependency (#535) Extraction: - Improve extraction accuracy for major news outlets (#530) - Fix formatting by correcting order of element generation and space handling with @dlwh (#528) - Fix: prevent tail insertion before children in nested elements by @knit-bee (#536) | Low | 4/3/2024 |
| v1.8.0 | Extraction: - Better precision by @felipehertzer (#509, #520) - Code formatting in TXT/Markdown output added (#498) - Improved CSV output (#496) - LXML: compile XPath expressions (#504) - Overall speedup about +5% Downloads and Navigation: - More robust scans with `is_live_page()` (#501) - Better sitemap start and safeguards (#503, #506) - Fix for headers in response object (#513) Maintenance: - License changed to Apache 2.0 - `Response` class: convenience functions added (#497) | Low | 3/20/2024 |
| v1.7.0 | Extraction: - improved `html2txt()` function (#483) Downloads: - add advanced `fetch_response()` function â pending deprecation for `fetch_url(decode=False)` Maintenance: - support for LXML v5+ (#484 by @knit-bee, #485) - update [htmldate](https://github.com/adbar/htmldate/releases/tag/v1.7.0) | Low | 1/25/2024 |
| v1.6.4 | Maintenance: - MacOS: fix setup, update htmldate and add tests (#460) - drop invalid XML element attributes with @vbarbaresi in #462 - remove cyclic imports (#458) Navigation: - introduce `MAX_REDIRECTS` config setting and fix urllib3 redirect handling by @vbarbaresi in #461 - improve feed detection (#457) Documentation: - enhancements to documentation and testing with @Maddesea in #456 | Low | 1/8/2024 |
| v1.6.3 | Extraction: - preserve space in certain elements with @idoshamun (#429) - optional list of xPaths to prune by @HeLehm (#414) Metadata: - more precise date extraction (see [htmldate](https://github.com/adbar/htmldate/releases/tag/v1.6.0)) - new `htmldate` extensive search parameter in config (#434) - changes in URLs: normalization, trackers removed (see [courlan](https://github.com/adbar/courlan/releases/tag/v0.9.5)) Navigation: - reviewed code for feeds (#443) - new config option: e | Low | 11/29/2023 |
| v1.6.2 | Extraction: - more lenient HTML parsing (#370) - improved code block support with @idoshamun (#372, #401) - convertion of relative links to absolute by @feltcat (#377) - remove use of signal from core functions (#384) Metadata: - JSON-LD fix for sitenames by @felipehertzer (#383) Command-line interface: - more robust batch processing (#381) - added `--probe` option to CLI to check for extractable content (#378, #392) Maintenance: - simplified code (#408) - support for Python 3. | Low | 9/6/2023 |
| v1.6.1 | Extraction: - minor fixes: tables in figures (#301), headings (#354) and lists (#318) Metadata: - simplify and fully test JSON parsing code, with @felipehertzer (#352, #368) - authors, JSON and unicode fixes by @felipehertzer in #365 - fix for authors without `additionalName` by @awwitecki in #363 Navigation: - reviewed link processing in feeds and sitemaps (#340, #350) - more robust spider (#359) - updated underlying courlan package (#360) Full Changelog: https://github.com/adba | Low | 6/15/2023 |
| v1.6.0 | Extraction: - new content hashes and default file names (#314) - fix deprecation warning with @sdondley in #321 - fix for metadata image by @andremacola in #328 - fix potential unicode issue in third-party extraction with @Korben00 in #331 - review logging levels (#347) Command-line interface: - more efficient sitemap processing (#326) - more efficient downloads (#338) - fix for single URL processing (#324) and URL blacklisting (#339) Navigation - additional safety check on domain | Low | 5/11/2023 |
| v1.5.0 | Extraction: - fixes for metadata extraction with @felipehertzer (#295, #296), @andremacola (#282, #310), and @edkrueger (#303) - pagetype and image urls added to metadata by @andremacola (#282, #310) - add as_dict method to Document class with @edkrueger in #306 - XML output fix with @knit-bee in #315 - various smaller fixes: lists (#309), XPaths, metadata hardening Navigation: - transfer URL management to courlan.UrlStore (#232, #312) - fixes for spider module Maintenance: - simp | Low | 3/30/2023 |
| v1.4.1 | Extraction: - extraction bugs fixed (#263, #266), more robust HTML doctype parsing - XML output improvements by @knit-bee (#273, #274) - adjust thresholds for link density in paragraphs Metadata: - improved title and sitename detection (#284) - faster author, categories, domain name, and tags extraction - fixes to author emoji regexes by @felipehertzer (#269) Command-line interface: - review argument consistency and add deprecation warnings (#261) Setup: - make download timeout | Low | 1/19/2023 |
| v1.4.0 | Impact on extraction and output format: - better extraction (#233, #243 & #250 with @knit-bee, #246 with @mrienstra, #258) - XML: preserve list type as attribute (#229) - XML TEI: better conformity with @knit-bee (#238, #242, #253, #254) - faster text cleaning and shorter code (#237 with @deedy5, #245) - metadata: add language when detector is activated (#224) - metadata: extend fallbacks and test coverage for json_metadata functions by @felipehertzer (#235) - TXT: change markdown formatt | Low | 10/18/2022 |
| v1.3.0 | - fast and robust `html2txt()` function added (#221) - more robust parsing (#228) - fixed bugs in metadata extraction, with @felipehertzer in #213 & #226 - extraction about 10-20% faster, slightly better recall - partial fixes for memory leaks (#216) - docs extended and updated (#217, #225) - prepared deprecation of old `process_record()` function - more stable processing with updated dependencies **Full Changelog**: https://github.com/adbar/trafilatura/compare/v1.2.2...v1.3.0 | Low | 7/29/2022 |
| v1.2.2 | - more efficient rules for extraction - metadata: further attributes used (with @felipehertzer) - better baseline extraction - issues fixed: #202, #204, #205 - evaluation updated Full Changelog: https://github.com/adbar/trafilatura/compare/v1.2.1...v1.2.2 | Low | 5/18/2022 |
| v1.2.1 | ## What's Changed - ``--precision`` and ``--recall`` arguments added to the CLI - better text cleaning: paywalls and comments - improvements for Chinese websites (with @glacierck & @immortal-autumn): #186, #187, #188 - further bugs fixed: #189, #192 (with @felipehertzer), #200 - efficiency: faster module loading and improved RAM footprint **Full Changelog**: https://github.com/adbar/trafilatura/compare/v1.2.0...v1.2.1 | Low | 5/2/2022 |
| v1.2.0 | - efficiency: replaced module readability-lxml by trimmed fork - bugs fixed: (#179, #180, #183, #184) - improved baseline extraction - cleaner metadata (with @felipehertzer) **Full Changelog**: https://github.com/adbar/trafilatura/compare/v1.1.0...v1.2.0 | Low | 3/7/2022 |
| v1.1.0 | - encodings: better detection, output NFC-normalized Unicode - maintenance and performance: more efficient code - bugs fixed (#119, #136, #147, #160, #161, #162, #164, #167 and others) - prepare compatibility with upcoming Python 3.11 - changed default settings - extended documentation **Full Changelog**: https://github.com/adbar/trafilatura/compare/v1.0.0...v1.1.0 | Low | 2/21/2022 |
| v1.0.0 | - compress HTML backup files & seamlessly open .gz files - support JSON web feeds - graphical user interface integrated into main package - faster downloads: reviewed backoff, compressed data - optional modules: downloads with `pycurl`, language identification with `py3langid` - bugs fixed (#111, #125, #132, #136, #140) - minor optimizations and fixes by @vbarbaresi in [#124](https://github.com/adbar/trafilatura/pull/124) & [#130](https://github.com/adbar/trafilatura/pull/130) - fixed arr | Low | 11/30/2021 |
| v0.9.3 | - better, faster encoding detection: replaced chardet with charset_normalizer - faster execution: updated justext to 3.0 - better extraction of sub-elements in tables (#78, #90) - more robust web feed parsing - further defined precision- and recall-oriented settings - license extraction in footers (#118) **Full Changelog**: https://github.com/adbar/trafilatura/compare/v0.9.2...v0.9.3 | Low | 10/21/2021 |
| v0.9.2 | - first precision- and recall-oriented presets defined - improvements in authorship extraction (thanks @felipehertzer) - requesting TXT output with formatting now results in Markdown format - bugs fixed: notably extraction robustness and consistency (#109, #111, #113) - setting for cookies in request headers (thanks @muellermartin) - better date extraction thanks to htmldate update | Low | 10/6/2021 |
| v0.9.1 | - improved author extraction (thanks @felipehertzer!) - bugs fixed: HTML element handling, HTML meta attributes, spider, CLI, ... - docs updated and extended - CLI: option names normalized (heed deprecation warnings), new option `explore` | Low | 8/2/2021 |
| v0.9.0 | - focused crawling functions including politeness rules - more efficient multi-threaded downloads + use as Python functions - documentation extended - bugs fixed: extraction and URL handling - removed support for Python 3.4 | Low | 6/15/2021 |
| v0.8.2 | - better handling of formatting, links and images, title type as attribute in XML formats - more robust sitemaps and feeds processing - more accurate extraction - further consolidation: code simplified and bugs fixed | Low | 4/21/2021 |
| v0.8.1 | - extraction trade-off: slightly better recall - code robustness: requests, configuration and navigation - bugfixes: image data extraction | Low | 3/11/2021 |
| v0.8.0 | - improved link discovery and handling - fixes in metadata extraction, feeds and sitemaps processing - breaking change: the `extract` function now reads target format from `output_format` argument only - new extraction option: preserve links, CLI options re-ordered - more opportunistic backup extraction | Low | 2/19/2021 |
| v0.7.0 | - customizable configuration file to parametrize extraction and downloads - better handling of feeds and sitemaps - additional CLI options: crytographic hash for file name, use Internet Archive as backup - more precise extraction - faster downloads: `requests` replaced with bare `urllib3` and custom decoding - consolidation: bug fixes and improvements, many thanks to the issues reporters! | Low | 1/4/2021 |
| v0.6.1 | - added `bare_extraction` function returning Python variables - improved link discovery in feeds and sitemaps - option to preserve image info - fixes (many thanks to bug reporters!) | Low | 12/2/2020 |
| v0.6.0 | - link discovery in sitemaps - compatibility with Python 3.9 - extraction coverage improved - deduplication now optional - bug fixes | Low | 11/6/2020 |
| v0.5.2 | - optional language detector changed: `langid` â `pycld3` - helper function `bare_extraction()` - optional deduplication off by default - better URL handling (`courlan`), more complete metadata - code consolidation (cleaner and shorter) | Low | 9/22/2020 |
| v0.5.1 | - extended and more convenient command-line options - output in JSON format - bug fixes | Low | 7/15/2020 |
| v0.5.0 | - faster and more robust text and metadata extraction - more efficient batch processing (parallel processing, URL queues) - support for ATOM/RSS feeds - complete command-line tool with corresponding options | Low | 6/2/2020 |
| v0.4.1 | - better metadata extraction and integration (XML & XML-TEI) - more efficient processing - output directory as CLI-option | Low | 4/24/2020 |
| v0.1.0 | First release used in production and meant to be archived on Zenodo for reproducibility and citability. | Low | 9/25/2019 |
Dependencies & License Audit
Loading dependencies...
Similar Packages
More from pypi
markitdownUtility tool for converting various files to Markdown
fastapiFastAPI framework, high performance, easy to learn, fast to code, ready for production
djangoA high-level Python web framework that encourages rapid development and clean, pragmatic design.
flaskA simple framework for building complex web applications.
More in Databases
milvusMilvus is a high-performance, cloud-native vector database built for scalable vector ANN search
onyxOpen Source AI Platform - AI Chat with advanced features that works with every LLM
memgraphHigh-performance open-source in-memory graph database for GraphRAG, AI memory, agentic AI, and real-time graph analytics. Cypher-compatible, built in C++.
sentry-sdkPython client for Sentry (https://sentry.io)