Cadence, Nomade & Focal — LLM français 15M
Featured- NLP
- LLM
- From-Scratch
- Adaptive Computation
- Research
Family of three 15M-parameter French LLMs trained from scratch (no pretrained weights, no dependency on 🤗 Transformers) on a single GTX 1080 Ti, exploring looped (recurrent-depth) transformers and per-token adaptive computation. Cadence applies the 8 transformer blocks 4 times (effective depth 32) with no early exit; Focal and Nomade add per-token early exit, using an absolute entropy threshold and a size-invariant percentile threshold respectively. Result: -7% validation perplexity for Cadence (28.9 vs 31.2 baseline), but no single model wins on every axis.
Key figures
- 15M ×3
- Parameters
- 28,9 (-7%)
- Perplexity (Cadence)
- 425M tokens
- Corpus
- 1× GTX 1080 Ti
- Training GPU
Research deep-dive
Three ways to loop the same transformer
The 8 LLaMA-style Transformer blocks (RoPE, RMSNorm, SwiGLU, QK-Norm) are applied 4 times — effective depth 32 — with zero added parameters beyond a small per-iteration depth embedding. Cadence loops with no early exit; Focal halts a token once its output entropy drops below a fixed threshold (1.0 nat); Nomade uses a percentile threshold (the 30% most confident tokens freeze each iteration) — the only variant whose criterion transfers to a different model size.
No model wins everything
Cadence cuts validation perplexity by 7% (28.9 vs 31.2 baseline) and is more coherent in-domain. Focal pushes in-domain coherence even further (44.1) but collapses out-of-domain (29.1). Nomade trades away the perplexity edge (31.0, near baseline) to become the only looped model that beats the baseline out-of-domain (41.8) and invents the fewest names (0.094). The in-domain/out-of-domain trade-off is the actual finding, not a single winner.
Fast research protocol, not a production run
Trained from scratch (no 🤗 Transformers weights) on a single GTX 1080 Ti, using only 20% of the training corpus (425M tokens total) over 2 epochs, evaluated on a single seed (50 prompts × 200 tokens). Directions are consistent but magnitudes aren't statistically validated — treated as a lab notebook, not a definitive benchmark.
Built on known shoulders, no novelty claimed
The family explicitly builds on ACT (Graves, 2016), the Universal Transformer (Dehghani et al., 2018), Recurrent-Depth Transformer / Huginn-3.5B (Geiping et al., 2025), CALM (Schuster et al., 2022), and LoopViT (Shu et al., 2026) — with two from-scratch looped studies reproduced in French as the direct basis (Kohli et al. 2026, COLM; Chen 2026). Every model card says it plainly: "no claim of novelty."