RDTextract
- Python
- NLP
- LLM
- PyPI
- HTML→Markdown
- Open Source
HTML→Markdown extractor designed for LLM training corpora. Eliminates all noise (navigation, ads, repeated widgets) with an integrated quality scoring system and a JSON-LD fallback for SPA pages. Score of 99.3/100 on the Tranco top-1,000 .fr benchmark — zero artifacts on all 4 measured metrics. Multi-language stub detection (FR/EN/ES/DE/IT). One dependency: beautifulsoup4.
Installation
pip install RDTextractKey figures
- 99,3/100
- Quality score
- 0
- Artefacts
- 672
- Pages tested
- 5
- Languages
- MIT
- Licence
Comparative benchmark
| Extractor | Score | 2×bullet | Orphans | HTTP dump | Para dup |
|---|---|---|---|---|---|
| 🏆RDTextract | 99,3 | 0 | 0 | 0 | 0 |
| html2text | 96,2 | 1 | 148 | 70 | 2 373 |
| trafilatura | 96,1 | 18 | 780 | 10 | 309 |
| markdownify | 85,5 | 0 | 246 | 107 | 5 042 |
Benchmark on 672 Tranco top-1,000 .fr pages