RDTRL
- NLP
- Reinforcement Learning
- Research
- Language Models
Experimental testbed for whether a neural network can learn to produce language from a scalar reward alone, with random initial weights, no pretraining, and no input/output pairs. Search spaces are small enough to enumerate exhaustively, so valid-mass, effective modes, the Gibbs optimum, and conditional structure are computed exactly rather than estimated. Test 1 (copying a fixed sentence) shows success comes from reward decomposability, not search itself. Test 2 (a hand-written grammar judged by a deterministic parser) reaches 99.9% grammaticality with zero learned rules — but both "successes" turn out to be empty on inspection. Test 3 (a two-agent referential game) concludes that only the network's parametrization produces compositionality, never selected by training itself. v0.5.0, five rounds of external review by Dipankar Sarkar.
Key figures
- 99,9%
- Grammaticality (Test 2)
- 3
- Tests
- 5
- Sarkar review rounds
- v0.5.0
- Version
How it works
Scalar reward, zero supervision
Random initial weights, no pretraining, no input/output pairs: the network learns only from a scalar reward. Search spaces remain small enough to enumerate exhaustively, so valid-mass, effective modes, the Gibbs optimum, and conditional structure are computed exactly — not estimated.
Test 1: copying a sentence, to understand why it works
Copying a fixed sentence shows that success comes from the decomposability of the reward, not from the search itself.
Test 2: a grammar, a deterministic parser, 99.9%
A hand-written grammar judged by a deterministic parser reaches 99.9% grammaticality with zero learned rules, and isolates the cause of mode collapse — a "success" that turns out to be empty once you look at how it's achieved: the network hides in a single-number sublanguage where the agreement constraint is vacuous.
v0.3.0 → v0.4.0: a bound never exceeded, and it survives a change of grammar
A policy with no determiner-noun coupling has product support, so at full validity it is capped by the largest fully-valid product in the sub-language it reaches. This bound is never exceeded across 70 runs, hit exactly by half of them — a plateau, not a basin. v0.4.0 confirms the bound survives a change of grammar: a three-gender lexicon gives a ratio of 3 instead of 2, predicted before the run and verified without exception across 70 seeds.
Test 3: a two-agent referential game
27 referents, 27 messages, a sender and a receiver trained by pure reinforcement learning. Result: compositionality was never selected by training — only the network's parametrization made it possible or impossible. Eight hypotheses refuted along the way, including the theoretical certificate imported from Test 2, which doesn't survive the move to two agents.
External review: Dipankar Sarkar
The product bound is due to Dipankar Sarkar (independent researcher, ACM & IEEE member, 138+ citations), who independently derived the order-1 marginals from the published paper, then showed that the two degenerate corners contain the same number of valid sentences but not the same maximal product. Five rounds of his critiques also fixed several methodological errors, from determiner coverage to Test 3's significance threshold.