AG-BPE (Attention-Guided Byte-Pair Encoding)
Featured- NLP
- Tokenization
- Python
- Transformers
Tokenizer combining BPE with weighted multi-layer Transformer attention (v4) for semantically coherent merge decisions. 3.85x compression, 0.03ms decoding, zero hard OOV on multilingual text (French, Korean, math symbols, emojis).
Key figures
- 3
- Stars
- 1K
- Views
- 1K+
- Downloads
- 0
- Citations
- 3,85x
- Compression
- 0,03ms
- Decoding
Technical details
Weighted multi-layer attention
A 6-layer Transformer ContextAnalyzer (12 heads, 768 hidden dim) computes a per-character-pair attention score at every layer, aggregated with learned weights favoring deeper, more semantically rich layers. The final merge score blends raw frequency with weighted attention (λ = 1000).
Perfect multilingual robustness
Zero "hard" out-of-vocabulary tokens on a test corpus mixing French, Korean (안녕하세요), math symbols and emojis — despite training only on French text. Example: AG-BPE correctly segments "cor|éen" where GPT-2 breaks the word into "cor|é|en".
Benchmarked against GPT-2, BERT, T5 and tiktoken
With an 18,000-token vocabulary, AG-BPE v4 achieves the best effectiveness-per-KB ratio (0.0149) among the production tokenizers tested, and decodes 20-30x faster than GPT-2, BERT-base, or T5-base.
Built for production
Advanced Unicode cleaning (NFKC normalization, Unicode-category filtering), automatic checkpoints every 1000 merges with crash recovery, adaptive memory management (100K-context sampling to avoid OOM), and dual-format saves (binary + human-readable JSON).