01 Why this project exists
Kallini et al. (2024) asked whether language models find "impossible languages" — English deterministically perturbed to violate the regularities of natural language — harder to learn. They do. This project asks the inverse question: given a GPT-2 that was pre-trained from scratch on an impossible language and has never seen a word of natural English, can it learn to translate back into the possible language?
Recovery, not acquisition. If information locality governs what models can do with linguistic structure, then the difficulty of undoing a locality violation should scale with the severity of the violation — a graded prediction we can test metric by metric.
02 The core idea: information locality
Natural languages keep words that depend on each other close together in linear order. This tendency — information locality — is one of the most robustly documented regularities across the world's languages, and it is the ruler by which this entire project measures its perturbations. So it is worth slowing down and building the idea up properly: what a dependency is, how its length is measured, why distance costs anything, and why the same logic might apply to a transformer that has no working memory at all.
What counts as a dependency, and how long is it?
In dependency grammar, every word in a sentence (except the root) is linked to exactly one head — the word it modifies or completes. An article depends on its noun, a subject on its verb, a preposition's object on the preposition. Each of these links is a dependency arc, and its length is the number of word positions separating dependent from head. The best way to build intuition is to read some trees. In each figure below, arcs point from dependent to head, the label gives the grammatical relation and the arc's length, and color encodes distance: short (1–2), medium (3–4), long (5+). The bold word is the root.
The locality story: this is what natural word order buys you. Four of the five arcs connect immediate or near neighbors; the longest ("mat" back to the root "sat") spans just three positions. A comprehender — human or model — never has to hold a dependent in memory for long before its head arrives. Averaging the arc lengths gives the metric used throughout this project:
The locality story: even natural English tolerates the occasional long arc. The subject "dog" must wait five positions for its verb "barked" while the relative clause intervenes — that is the red arc, and it is exactly where psycholinguistics finds elevated reading times. But look at the rest of the tree: the grammar spends distance only where it must, keeping every other arc at length 1–2 and holding the average to 1.86. Locality is a pressure on the average, not a ban on long dependencies. That is why ADL — an average — is the right measuring stick, and why a perturbation that lengthens arcs everywhere is so unlike anything natural.
The locality story: compare this tree to Figure 2 arc by arc. Not a single word was added or removed, and the grammatical relations are identical — but every green arc save two has turned orange or red, the tree is now a thicket of crossing dependencies, and the flip that matters most is "cat → chased": an object that used to sit two positions from its verb must now be linked across six positions of unrelated material, with no grammatical cue for where to look. The scramble is lexically invisible but metrically loud: ADL nearly doubles, from 1.86 to 3.29. This is what our perturbations do to every sentence, in controlled doses.
The locality story: this pair is the key to the beyond-paper results in §08. The two shaded phrases swapped places — a global reordering at the token level — yet five of the six arcs are completely untouched, because they live inside phrases and phrases moved as blocks. Only the single arc that crosses a phrase boundary ("station → walked") stretched, and only by one position. ADL creeps from 1.67 to 1.83 while a token-level view would call this sentence heavily perturbed. Locality can be preserved at one granularity (within phrases) while being violated at another (between them) — hold that thought for the PhraseShuffle experiments below.
Why distance costs: Dependency Locality Theory
Why should distance matter at all? Gibson's Dependency Locality Theory (1998, 2000) gives the classic processing answer. Comprehension is incremental: as each word arrives, the parser must either store an expectation (a subject is waiting for its verb) or integrate the new word with an earlier one. Both operations get more expensive with distance — the longer a dependent waits for its head, the more intervening material must be held in, and retrieved from, working memory. DLT predicts measurable processing difficulty exactly where long arcs pile up, and decades of reading-time and neuroimaging evidence agree: object relative clauses (which stack two long dependencies) are reliably harder than subject relatives, and difficulty tracks the amount of intervening material.
Figure 3 above is DLT's nightmare scenario: "cat" must be linked to "chased" across six positions of unrelated words, with no grammatical cue for where to look. The lexical content is fully intact, but the sentence has become cognitively inaccessible — which is precisely what makes such strings good candidates for "impossible" languages.
Dependency length minimization: the typological evidence
If long dependencies are expensive, languages should evolve word orders that avoid them — and they do. Liu (2008) measured dependency lengths across 20 languages; Futrell, Mahowald & Gibson (2015) extended this to 37, comparing each real corpus against random reorderings of the same dependency trees. In every language, observed dependency lengths sit far below the random baseline. The effect shows up in the grammar itself: heavy constituents get shifted to the edge ("I gave the book to Mary" vs "I gave to Mary the book that my grandmother left me"), head-initial and head-final languages mirror each other's ordering preferences, and Hawkins' processing typology and Temperley & Gildea's corpus work trace dozens of word-order universals back to this single pressure. Dependency length minimization (DLM) is about as close as linguistics gets to a law-like generalization.
From memory to information theory — and to transformers
There is a deeper way to state the principle that never mentions human memory. Futrell's information-theoretic reformulation says: in natural language, statistical dependencies are local too — the mutual information between two words decays with the distance between them. Any predictor operating with bounded or lossy memory of the past does measurably better when the information it needs is nearby; this is the memory–surprisal tradeoff. Stated that way, the principle applies to any sequential processor: a human parser with limited working memory, or a transformer whose attention is soft, trained, and empirically sharpest over recent tokens (Khandelwal et al., 2018: "sharp nearby, fuzzy far away").
This substrate-independent formulation is what licenses the central question of the paper. If locality were only a fact about human working memory, there would be no reason to expect GPT-2 to care about it. But if locality is an efficiency condition on any bounded sequential predictor, then a transformer should show the same graded sensitivity — harder recovery as dependencies stretch — for its own architectural reasons. That is exactly the hypothesis the experiments test.
Locality as the ruler for our impossible languages
Because ADL is computable per sentence, we can put an exact number on how much locality damage each perturbation inflicts — before any model is trained. Measured on the actual test data:
| Text condition | Input ADL, bnc_spoken (≈12-token sentences) | Input ADL, Gutenberg (≈40-token sentences) | Behavior as sentences grow |
|---|---|---|---|
| Natural English | 2.49 | 3.16 | Nearly flat — DLM at work |
| LocalShuffle (w=3) | — | 3.59 | Bounded: no token moves more than 2 positions |
| PartialReverse | 3.32 | 6.90 | Grows with the reversed suffix |
| FullShuffle | 3.34 | 15.26 | Unbounded: grows in proportion to sentence length |
| WordHop (§08) | ≈ natural — only the verb–marker arc stretches, always by 4 | Constant: one fixed-length arc per verb | |
| PhraseShuffle (§08) | — | 3.50 | Bounded at phrase level: within-phrase arcs never stretch |
Three things to notice. First, natural English barely moves between 12-token and 40-token sentences (2.49 → 3.16) — that near-flatness across a 3× length increase is dependency length minimization, visible in our own data. Second, the perturbations differ not just in how much they stretch dependencies but in how that stretch scales: LocalShuffle's damage is constant by construction, PartialReverse's grows moderately, and FullShuffle's explodes from 3.34 to 15.26 as sentences get longer. Third — and this only becomes visible with the two extra perturbations we ran beyond the paper — raw ADL is not the only ruler. PhraseShuffle reorders a sentence globally yet its ADL barely rises above natural (3.50 vs 3.16), because the arcs that matter most live inside phrases and phrases move as intact blocks. There is a second, complementary measure of damage: the fraction of dependency arcs that survive perturbation untouched (the perturbed-input Triple F1 baseline). Section 08 shows that this second ruler, not ADL, is what best predicts recovery.
All perturbations in this project preserve lexical content entirely. Only linear order — and hence dependency length — is manipulated. Whatever the model recovers, it recovers from structure, not from missing-word guessing.
03 Three impossible languages, with examples
The paper studies three perturbations from Kallini et al., chosen because they span a spectrum of locality violation while all preserving vocabulary. All operate on GPT-2 token IDs, not whitespace words — matching pre-training exactly.
LocalShuffle (window = 3)
Tokens are shuffled randomly within non-overlapping windows of three. Every token stays within two positions of home, so the damage is severe locally but strictly bounded: dependency distances cannot grow with sentence length.
PartialReverse
A random point is chosen, a marker token R is inserted, and everything after it is reversed. The prefix survives intact and the marker is an explicit structural clue — but the model still has to reconstruct a long reversed suffix.
FullShuffle
All tokens are shuffled globally. No local structure survives, and — crucially — the expected dependency distance grows with sentence length. This is the only perturbation whose locality damage is unbounded, and that single property ends up explaining the most dramatic result in the paper.
| Perturbation | Operation | Locality damage | Surviving arcs (bnc_spoken) |
|---|---|---|---|
| PartialReverse | Reverse suffix after marker R | Moderate — prefix intact, one strong cue | 0.48 |
| LocalShuffle | Shuffle within windows of 3 | Bounded — every token stays within 2 positions | 0.36 |
| FullShuffle | Shuffle all tokens globally | Unbounded — grows with sentence length | 0.18 |
The initial segment survives intact and the reverse marker leaves one strong structural clue the model can exploit.
Short-range syntax is perturbed, but every token stays within its window — the damage never grows with sentence length.
Global scrambling scatters dependents arbitrarily far from their heads. Longer sentence, longer dependencies, harder recovery.
04 Data, models, and training setup
The project uses two BabyLM-derived sources because sentence length is part of the story: bnc_spoken, which is shorter and noisier, and Gutenberg, which is longer and more syntactically complete.
| Dataset split | Size | Average sentence length | Why it is useful |
|---|---|---|---|
| bnc_spoken train | 100K | 12.77 | Short, spoken-style sentences with fewer structural cues. |
| bnc_spoken test | 1K | 11.72 | Measures recovery under short-context conditions. |
| Gutenberg train | 100K | 40.61 | Longer, better-formed written sentences with richer dependencies. |
| Gutenberg test | 1K | 46.78 | Measures whether longer contexts help or hurt recovery. |
The most important design choice is the starting point. We do not start from an English-pretrained GPT-2. That would let the model fall back on natural-English priors and confound recovery with retrieval of a known target distribution. Instead, we fine-tune the GPT-2 checkpoints that Kallini et al. pre-trained from scratch on each impossible language — one model per perturbation, each inverting the language it grew up in. Acquisition difficulty is already paid for; what remains is pure reconstruction.
Training details, for the record: effective batch size 512, AdamW at 2e-5 with cosine schedule and 500 warmup steps, label smoothing 0.1, bfloat16 on H100s, 5,000 steps on the 100K datasets. Every fine-tuning run was repeated with five random seeds (1337, 2025, 314159, 42, 8675309); all reported curves are seed averages with standard-deviation error bars.
05 Experiments and hypotheses
The paper is organized around three experiments, each tied to a concrete hypothesis about information locality.
| Experiment | Question | Hypothesis |
|---|---|---|
| Training sample size | How does recovery improve over training checkpoints? | All models improve, but the rate tracks locality damage: PartialReverse fastest, FullShuffle slowest, LocalShuffle in between. |
| Sentence length | Do longer sentences help or hurt recovery? | Longer sentences give richer context, but also stretch dependencies. Where damage is bounded, context should win; where damage grows with length, recovery should break. |
| Perplexity / naturalness | Does the generated text move back toward ordinary English? | If recovery is genuine, predictions should converge toward the natural-English baseline under a standard GPT-2. |
Four evaluation lenses: Exact Match for strict reconstruction, BLEU for partial overlap, Average Dependency Length (ADL) for the locality profile of the output itself, and Dependency Triple F1 for recovered syntactic relations (spaCy en_core_web_trf parses; a triple counts only if dependent, relation, and head all match). The division of labor matters: ADL is reference-free — a property of the output string — while Triple F1 is reference-bound — faithfulness to the source. Their dissociation under FullShuffle turns out to be one of the most interesting results.
06 What the results actually show
Result 1: recovery difficulty tracks locality disruption
On bnc_spoken, PartialReverse learns fastest and tops out highest. LocalShuffle and FullShuffle trail — but the metrics disagree in an informative way. FullShuffle edges out LocalShuffle on exact match (0.23 vs 0.17), yet LocalShuffle wins on Dependency Triple F1 (0.53 vs 0.51). Exact match penalizes any positional error regardless of structural correctness; LocalShuffle's errors are locally bounded and structurally recoverable, so surface metrics underestimate its recovery.
Result 2: sentence length is the stress test
Moving from short spoken sentences to long Gutenberg prose, the bounded perturbations improve — PartialReverse EM goes 0.29 → 0.52, LocalShuffle 0.17 → 0.25 — while FullShuffle collapses: EM 0.23 → 0.01, Triple F1 down to 0.14. The mechanism is visible in the input ADL: globally shuffling a 40-token sentence stretches average dependency length to 15.26, versus 3.34 for short sentences, while LocalShuffle's window keeps input ADL nearly constant (3.59). When dependency distance grows with sentence length, richer context cannot compensate; when it is bounded, longer sentences just mean more evidence.
| Perturbation | Surface recovery | Structural recovery | Effect of longer sentences | Interpretation |
|---|---|---|---|---|
| PartialReverse | Best overallEM 0.29 → 0.52 from short to long sentences; BLEU up to 79%. | StrongTriple F1 0.67 on bnc_spoken, 0.84 on Gutenberg. | HelpsLonger preserved prefix, more anchors. | One good structural cue makes a hard transformation learnable. |
| LocalShuffle | ModestEM stays low because order must be rebuilt from weak local cues. | Stronger than it looksTriple F1 0.53 → 0.77 with longer sentences, well above the 0.36 baseline. | HelpsDamage is bounded, so context is pure signal. | The model rebuilds the syntactic skeleton before the exact surface form. |
| FullShuffle | SplitComparable to others on short sentences (EM 0.23), near zero on long ones. | CollapsesTriple F1 falls to 0.14 on Gutenberg. | Breaks itInput ADL 3.34 → 15.26; damage grows with length. | Not merely harder — a categorical break where locality damage becomes unbounded. |
Result 3: the model learns structure before it learns perfect strings
Exact match makes the task look harsher than it is. Evaluated on dependency structure directly, the model reconstructs correct grammatical relationships well before — and well beyond — exact string reconstruction.
| Condition | Exact Match | Dependency Triple F1 | perturbed-input baseline |
|---|---|---|---|
| PartialReverse, bnc_spoken | 0.29 | 0.67 | 0.48 |
| LocalShuffle, bnc_spoken | 0.17 | 0.53 | 0.36 |
| FullShuffle, bnc_spoken | 0.23 | 0.51 | 0.18 |
| PartialReverse, Gutenberg | 0.52 | 0.84 | — |
| LocalShuffle, Gutenberg | 0.25 | 0.77 | — |
| FullShuffle, Gutenberg | 0.01 | 0.14 | — |
Under LocalShuffle, the model's outputs converge to an average dependency length below the natural value (predicted ADL ≈ 2.28 vs actual 2.49 on bnc_spoken). These models never saw natural English in pre-training — so a bias toward shorter-than-natural dependency arcs cannot be memorized English statistics. It looks like a genuine inductive bias of the attention architecture, functionally parallel to dependency length minimization in human production.
Result 4: naturalness moves back toward English — but fluent is not faithful
Measured against base GPT-2 (natural-text baseline: perplexity 129), all three models pull their outputs far down from the perturbed-input perplexity.
The FullShuffle case is the philosophically interesting one. On long sentences, its output ADL sits near the natural value and perplexity drops substantially — while exact match is near zero and Triple F1 has collapsed. The model generates fluent, locality-respecting English that is simply not the source sentence. Naturalness and faithfulness dissociate, which is exactly why reference-free metrics (perplexity, output ADL) can never certify recovery on their own.
07 Checkpoint-by-checkpoint examples
These are real rows from the saved evaluation outputs in the repository — the same sentence tracked across training checkpoints. Green marks what the model has recovered; red marks what is still wrong.
Example A: LocalShuffle rebuilds the skeleton first, then stalls on one local swap
The only other creature in this White House who has the same sweet deal is the President's dog!
The only other in this creature White House who same has the sweet is deal the President's dog!
The only only the human one who's name is the same as the same man.
What happened: the model has learned the shape of the task — produce a fluent English sentence — but it is solving by generic language modeling, not reconstruction. Content drifts entirely.
The only other creature in this White House who has the same deal is sweet the President's dog!
What happened: the clause backbone, the relative clause, and the final possessive are all restored. The one remaining error — "deal is sweet" for "sweet deal is" — is a within-window swap: exactly the error signature LocalShuffle predicts. Exact match scores this 0; dependency F1 gives most of the credit.
The only other creature in this White House who has the same deal is sweet the President's dog!
What happened: 2,400 further steps change nothing. The residual ambiguity is genuinely local — both orders are plausible within the window — and this is what LocalShuffle's EM ceiling is made of.
Example B: FullShuffle's two failure modes, live
"Well, I'll turn over a new leaf," he promised, "just to please you."
I, over turn new, leaf" " a'llWell," promised you to please he."just
"I'll get the new, and a leaf, "I'll just give it," said he said, "I'll go,"
What happened: early in training the model produces natural-sounding English built from the input's vocabulary — low perplexity, near-natural ADL — but the sentence is invented. This is the "fluent ≠ faithful" dissociation in a single row.
I, I turn over new leaf, "Well'll a promised," promised he to please."
What happened: more training does not converge on the source — it shifts the model toward conservative near-copying of the perturbed input. Neither failure mode is recovery: fluency without faithfulness early, faithfulness to the wrong string late. On long sentences, FullShuffle has no path to the target at all.
08 Beyond the paper: the experiments that didn't make it in
The repository contains a set of completed experiments that the paper's page limit couldn't fit. They are worth showing because each one independently stress-tests the same claim — recovery tracks locality — from an angle the paper doesn't cover.
WordHop: the low-violation control
WordHop is the fourth Kallini et al. perturbation we ran: each inflected verb is replaced by its lemma, and a singular/plural marker (S / P) is displaced exactly four tokens downstream. Linear order otherwise survives — only one dependency per verb is stretched, by a fixed distance.
If the locality account is right, WordHop should be the easiest condition of all — and it is, on every measure we have:
| Condition (100K) | Exact Match | BLEU |
|---|---|---|
| WordHop, bnc_spoken | 0.33 | 0.60 |
| WordHop, Gutenberg | 0.38 | 0.77 |
| PartialReverse, Gutenberg (best paper condition) | 0.52 | 0.79 |
Its perplexity profile is the most telling part: WordHop input measures just 168 under base GPT-2, barely above the natural baseline of 129 — compared to 1500–6143 for the word-order perturbations. Distributionally, WordHop text is almost not impossible at all. That anchors the low end of the violation spectrum with a perturbation whose damage is measurably tiny, and its high recoverability is exactly where the locality gradient says it should be.
WordHop's prediction perplexity reached its best value (154) at checkpoint 600 — then drifted upward to 244 by checkpoint 5000, even as BLEU held steady. Extended fine-tuning on a nearly-natural language slowly degrades output naturalness without hurting surface accuracy. Checkpoint selection isn't a detail; the "best" model depends on which metric you ask.
PhraseShuffle: a new perturbation the paper never tested
Every perturbation above operates on tokens. PhraseShuffle operates on constituents: we parse each sentence with a neural constituency parser (benepar), find leaf-level phrases (NP, VP, PP, ADJP, ADVP, and similar), and shuffle those phrases as intact units. Word order is globally disrupted — but locality within each phrase is perfectly preserved.
This makes PhraseShuffle a critical test case: at the token level it looks globally perturbed, like FullShuffle. At the phrase level it is locality-preserving. If the model's recovery capacity depends on token-level adjacency, PhraseShuffle should collapse on long Gutenberg sentences like FullShuffle did. If it depends on preserved local dependency structure — at whatever granularity — it should recover well.
It recovers well. On Gutenberg (seed 42), where FullShuffle managed EM 0.01 and Triple F1 0.14:
| Checkpoint | Exact Match | BLEU | Dependency Triple F1 |
|---|---|---|---|
| 200 | 0.04 | 0.26 | 0.40 (baseline: 0.55) |
| 1000 | 0.22 | 0.67 | 0.74 |
| 3000 | 0.28 | 0.74 | 0.80 |
| Final (5000) | 0.30 | 0.75 | 0.80 |
Final output ADL lands at 3.158 against a natural value of 3.154 — a near-exact restoration of the locality profile, from input ADL 3.50. Note the early-training dip: at checkpoint 200 the model's Triple F1 (0.40) is below the do-nothing baseline (0.55) — attempting reconstruction before the mapping is learned is worse than leaving the input alone. It crosses the baseline between checkpoints 200 and 400 and never looks back.
PhraseShuffle dissociates two readings of "information locality" that the paper's perturbations conflate: token adjacency versus preserved local dependency structure. The model recovers phrase-shuffled text at LocalShuffle-or-better levels despite global token displacement — evidence that what the architecture exploits is intact local structure at some granularity, not raw token distance. This directly echoes Mollica et al.'s finding that human comprehenders extract composed meaning from order-violating strings as long as dependents stay local.
Data scale interacts with locality too
The paper reports 100K-sample training. We also ran everything at 10K, and the comparison is stark:
| Condition | EM at 10K | EM at 100K |
|---|---|---|
| WordHop, bnc_spoken | 0.30 | 0.33 |
| WordHop, Gutenberg | 0.25 | 0.38 |
| PartialReverse, bnc_spoken | 0.05 | 0.29 |
| PartialReverse, Gutenberg | 0.01 | 0.52 |
| LocalShuffle, bnc_spoken | 0.04 | 0.17 |
| LocalShuffle, Gutenberg | 0.00 | 0.25 |
WordHop is essentially learnable from 10K examples; the order-restoring perturbations are not learnable at all at that scale (EM ≤ 0.05) and only become tractable at 100K. Locality violation doesn't just lower the ceiling — it raises the price of entry. The amount of data needed to learn the inverse mapping scales with the severity of the violation, mirroring, in the recovery direction, the sample-efficiency gaps Kallini et al. observed during acquisition.
The five-perturbation dose–response: which ruler actually predicts recovery?
With WordHop and PhraseShuffle added, we have five perturbations measured on the same Gutenberg test set — enough to ask which measure of locality damage actually predicts recovery. The paper's framing leans on ADL. But put the two rulers side by side and ADL loses:
| Perturbation (Gutenberg) | Input ADL | Surviving arcs (baseline F1) | EM | Triple F1 |
|---|---|---|---|---|
| PartialReverse | 6.90 | 0.62 | 0.52 | 0.84 |
| PhraseShuffle | 3.50 | 0.55 | 0.30 | 0.80 |
| LocalShuffle (w=3) | 3.59 | 0.49 | 0.25 | 0.77 |
| FullShuffle | 15.26 | 0.29 | 0.01 | 0.14 |
| WordHop (order preserved) | ≈ natural | — | 0.38 | — |
Sort by surviving arcs and recovery is perfectly monotone — on both exact match (0.52 > 0.30 > 0.25 > 0.01) and Triple F1 (0.84 > 0.80 > 0.77 > 0.14). Sort by ADL and the ordering breaks immediately: PartialReverse has nearly twice PhraseShuffle's dependency stretch yet recovers best, and LocalShuffle stretches dependencies less than PhraseShuffle yet recovers worse.
The explanation is that the two rulers measure different things. ADL measures how far dependents moved from their heads. Baseline F1 measures how many head–dependent adjacency relations survived — and reversal preserves adjacency (neighbors stay neighbors, just mirrored), phrase movement preserves within-phrase adjacency, while window shuffling scrambles exactly the short arcs where most of a sentence's structure lives, and global shuffling destroys everything. The model's recovery tracks surviving local structure, not displacement distance.
Information locality governs recovery — but "locality" means preserved local dependency structure at some granularity, not raw token distance. That refinement is invisible with the paper's three perturbations alone; it takes PhraseShuffle (globally displaced, locally intact) and PartialReverse (far-displaced, adjacency-preserving) sitting in the same table to separate the two readings. It also aligns with Futrell's information-theoretic version of locality: what a bounded predictor needs is not that dependents are near, but that the statistical structure linking them is recoverable from a bounded window.
One caveat: these are seed-42 runs on one test set of 1K sentences; the WordHop row lacks dependency-based measurements because its evaluation predates that part of the pipeline. Rerunning the dependency metrics across all five seeds is the obvious robustness check before making this table load-bearing.
In progress: window size as a locality dial
The repo also contains training data and a pre-trained model for LocalShuffle with window 5 (alongside window 3). Window size is the cleanest possible locality dial — same operation, strictly larger maximum displacement — and comparing w=3 vs w=5 recovery curves will test whether the gradient is smooth even within a perturbation family. Results to follow in a future post.
09 What the findings mean
The central conclusion is not that LLMs can do everything humans cannot. It is more precise: the models are sensitive to locality violations, and their performance degrades in predictable ways as violations intensify — until one specific condition breaks the pattern categorically.
- Recovery difficulty scales with locality-violation severity across five perturbations spanning the spectrum — WordHop, PartialReverse, PhraseShuffle, LocalShuffle, FullShuffle — supporting information locality as a graded organizing principle of sequential processing.
- The exception proves the rule: FullShuffle on long sentences is not "harder," it is broken — the one condition where dependency distance is unbounded in sentence length. The picture is a graded cost function with a capacity cliff.
- Dependency metrics reveal syntactic recovery even where exact reconstruction fails — and across all five perturbations, recovery is perfectly monotone in the fraction of dependency arcs that survive perturbation, while raw ADL mis-orders the conditions. The operative variable is preserved local structure at some granularity, not token displacement.
- The DLM undershoot (output ADL below natural) in models that never saw natural English suggests dependency length minimization may be a property of any efficient sequential processor, not of human memory specifically.
- Fluency and faithfulness dissociate: FullShuffle outputs look like English while recovering nothing. Reference-free metrics cannot certify recovery — a methodological point that extends well beyond this project.
Information locality looks less like a hard learnability boundary and more like a graded processing principle with a capacity cliff. The harsher the locality violation, the harder recovery becomes — in final performance, in learning speed, and in how much data the mapping costs — until the violation becomes unbounded, at which point recovery does not degrade but collapses.
10 Resources and how to reproduce it
The repository contains the full training and evaluation stack, the saved per-checkpoint outputs behind every example above, and the Hugging Face model identifiers used in the project.
Read the paper, then browse the saved checkpoint outputs in the repo — the same conclusion keeps arriving from different directions: models recover a surprising amount of structure from impossible inputs, and the point where recovery breaks down is exactly the point where locality damage stops being bounded.