GRAM removes winter from a small story model by routing winter text through a module that is deleted before the model ships. We tried to steer winter back with the vectors that brought the deleted digit back in the MNIST testbed, and we varied pas, the chance that a batch's winter text also trains the core. At the Track 7 setting, pas 0.5, nothing brings winter back. At pas 1 the shipped core still writes winter stories when prompted, yet almost never starts one. A push toward the word "winter" makes it start one, but what follows is mostly ordinary-story text, often with "winter" dropped in, rather than a winter story. We then asked the same question of models trained on TinyStories, which GPT-3.5 and GPT-4 wrote for small children (below). There, steering brings winter words back at pas 0.3 and above, with a harder push as pas falls, but it never brings back winter stories. At pas 0, the default in GRAM's public code, it fails, but there even the model with its module attached rarely writes winter words.
Every story follows one template. Winter stories differ in the season, and in place of two everyday sentences they have three: snow falling, a winter activity, and the item that always goes with that activity. One training window in six is a winter story.
Trained on everything: 30,000 ordinary windows and 6,000 winter windows. The reference that knows winter.
The winter windows are removed from the training data. Labels are perfect in this round.
Each layer's feed-forward block gets a small module of 80 hidden units beside its 640. Everything else is the core: embeddings, attention, the output layer and the 640-unit blocks. In every batch, winter text runs with the module on, trains the module, and trains the core with probability pas. Ordinary text always trains the core, and in half the batches runs with the module on and trains it too. The module is deleted before the model ships.
All are 4-layer transformers of width 160, trained for 4 epochs. Three seeds per pas; six at pas 0.5 and 1, and for the dense and filtered models.
Starts winter. The probability that the next word is winter. The dense model says it 0.09–0.19 of the time, near winter's one-in-six share of training text.
Knows the facts. Each of 16 activities always comes with one of 4 items (hockey → mittens). On 192 winter stories cut just before the item, how often the top next word is the right one. Guessing the same item every time scores 0.25.
Writes the story. 64 sampled continuations of "one day in", and 64 of "one day in winter". A sample writes the story if it has "went <activity>", and names the right item if the next "took the" ends in that activity's item.
Steering adds a fixed vector to the model's internal state at every word, at all four layers or at one of layers 2–4, with strengths from 0.125 to 64 in steps of √2. A setting counts only if the loss on ordinary stories stays within 0.10 nats of the model's own unsteered loss (about 0.45), so a vector can't win by wrecking ordinary text. The budget does not protect winter knowledge, which Result 4 shows the push can damage. For each vector we keep the setting with the highest log-odds of "winter" against the other three seasons after "one day in", and report the probability of "winter" there.
| Vector | Built from | Tried on |
|---|---|---|
| The model's own winter direction, and a token-contrast (CAA) vector | Its own activations: winter windows minus ordinary windows; for CAA, its state at "winter" minus at another season after "one day in" | Every model |
| The full model's winter direction | The GRAM model with its module attached | GRAM; transplanted to filtered and dense |
| The module's mean output, alone and minus its mean on ordinary windows | The deleted module's output on winter windows. In MNIST this restored 96–100% recall of the deleted digit. | GRAM; transplanted to filtered and dense |
| The dense model's winter direction | The dense model with the same seed and starting weights | GRAM, filtered |
| Push toward the word "winter" | Direction from the output layer alone: the row for "winter" minus the mean row of the other seasons, times the final norm's gain. Scaled per layer like the model's own winter direction. | Every model, each built from its own output layer |
| 8 random directions | Noise, scaled like the model's own direction | Every model |
Transplants come from the pas 0.5 GRAM model with the same seed. "Vectors from winter data" below means the first four rows.
Log scale, one dot per seed (six seeds), dark tick at the geometric mean. The shaded band is the dense model's unsteered range. "Best vector from winter data" is the best of the vectors in the first four rows of the table above that each model received. The patched-in module output is the deleted module's own output at this very prompt, added back at strength 1. It is a ceiling, not an attack: it raises ordinary-story loss from 0.44–0.46 to 5.6–6.8 and drops facts recalled to 0.
The MNIST recipe does nothing to the pas 0.5 core. No vector built from winter data beats the best random direction by more than 0.02 log-odds. The push toward the word raises the log-odds of "winter" against the other seasons by up to 6.9, but it gets there by pushing the other seasons down: the probability of "winter" itself stays below 10−5.
At least one of these vectors carries winter where there is winter to steer. Transplanted into the dense model, the GRAM module's mean output raises its chance of starting a winter story to 0.45–0.85. That beats the dense model's best random direction in every seed, though random directions alone reach 0.21–0.52 and the margin is 0.11–2.59 log-odds, above 2 in three seeds. The other transplants do worse: the full model's winter direction reaches 0.14–0.81 but loses to random in two of six seeds, and the module mean minus its ordinary-window mean loses in four. The filtered model is easier to push to the word than GRAM at this setting: its push beats random by 3.2–7.3 log-odds and reaches 0.002–0.13. 0–9% of its sampled stories then open with winter, and none goes on to "went <activity>".
One dot per seed; lines join geometric means (top) or means. The pas values are evenly spaced, not to scale. Filtered and dense have no module and no pas. Hover a dot for its value.
At pas 0 only the module learns winter. At pas 1 the core trains on every winter window, as a dense model would, but with the module switched on beside it. As pas rises, the shipped core's loss on winter stories falls from 4.74 to 0.45, against 0.35 for dense, and its facts score climbs from 0 to 1.00 by pas 0.75.
Its chance of starting a winter story stays below 10−5 through pas 0.75 and at or below 0.003 through pas 0.98. At pas 1 it is 0.003–0.015, under the pre-registered 0.01 in five of six seeds. With the module attached, the same models start winter 13–55% of the time. This is the case we were looking for. In five of six seeds at pas 1 the capability looks removed, yet the core recalls every item, and when prompted it names the right item in 86–100% of the stories that reach one. How often it writes the story at all varies by seed (Result 5).
Top: the chance of starting a winter story at each family's best setting within the damage budget. Bottom: that setting's gain in log-odds of "winter" against the other seasons, minus the best random direction's gain. As pre-registered, steering back needs both a chance of at least 0.05 and at least +2 over random. Filtered and dense receive the pas 0.5 GRAM model's vectors as transplants.
The push's direction carries no winter knowledge, so any story that follows it has to come from the model. It can make even the filtered model, which never saw a winter story, say the word: built the same way, it takes the filtered model to 0.002–0.13 and the dense model to 0.94–0.99. On GRAM it takes effect only from pas 0.9, although the core already recalls the items at pas 0.75 (facts 0.99–1.00). It reaches 0.005–0.048 at pas 0.9, 0.16–0.77 at 0.95, 0.67–0.90 at 0.98 and 0.69–0.97 at 1, beating the best random direction by 4.3–6.8 log-odds from pas 0.95 up. At pas 0.9 and below no vector reaches 0.05 at any setting within the budget.
Vectors built from winter data, including the module's mean output that worked in MNIST, reach at most 0.26 on GRAM. They clear +2 over random twice, both with the module's mean output in seed 0 (pas 0.95: +2.23, chance 0.023; pas 0.98: +2.56, chance 0.18), and only the second also reaches 0.05. At pas 1 the closest is +1.96.
Ticks are seeds; the bar is their mean. Steered rows use each vector's best setting from Result 3. A sample writes the story if it has "went <activity>"; it names the right item if the "took the" that follows ends in that activity's item. The last column pools the six seeds: right items over samples that got as far as "took the". A miss there can be a wrong item or a word that is not an item.
At pas 1 the push makes 66–95% of sampled stories open with winter. In five seeds only 0–17% go on to "went <activity>"; in seed 3, 39% do, and 14 of 64 name the right item. Given "one day in winter" and no push, the same core writes the scaffold in 11–83% of samples, depending on the seed, and names the right item in 86–100% of those that reach "took the". Pushing the dense model the same way gives winter stories: 72–91% go on to "went <activity>".
So the push restores the opening word and little of what follows. Part of that is the push itself. It stays on for every generated word, and with it on, even the prompt "one day in winter" leads to "went <activity>" in only 2–48% of samples, against 11–83% without it (only seed 5 is not lower). Facts recalled at the push's setting fall from 1.00 to 0.10–0.99 (mean 0.58). These runs cannot tell whether the core would carry on into the story if the push stopped after the opening word. The filtered model, pushed or prompted, never writes "went <activity>" in any of its 768 samples.
Unedited and cut at 36 generated tokens, with each period counted as a token. These are the first samples, not picked for being typical: seeds 0, 1 and 5 write the scaffold in 69%, 11% and 20% of their prompted samples, but not in this one.
| pas · seed | Starts winter, unsteered | Push toward "winter" | Filtered model's own push | Push steers back, GRAM-specific | Best vector from winter data | Prompted: right ÷ reached item |
|---|
Chances are of starting a winter story; "+x" is the gain in log-odds over the best random direction. A push passes when it is at least +2 and reaches 0.05. "Reached item" counts samples of the 64 that got as far as "took the".
"Looks removed" holds in every seed through pas 0.98 and in five of six at pas 1. The push passes its first two clauses in every seed from pas 0.95. For the third clause each model gets its own push, built the same way, since the push is a recipe rather than one shared vector. The filtered model's push passes the first two clauses too in seeds 0–3, so the third fails there. At pas 1 it holds in seeds 4 and 5, where the filtered model's push stays at 0.034 and 0.0023. The module mean passes both numeric clauses at pas 0.98 in seed 0, but it was never tried on the filtered model, which only received vectors from the pas 0.5 GRAM model. That seed's verdict is open.
"Knows it" depends on how it is read. As accuracy at the item, every seed at pas 1 passes (86–100% against 0.25). Read strictly, as completed stories with the right item out of all 64 samples, seeds 0, 2 and 3 score 63–72%, seed 4 38%, and seeds 1 and 5 only 9% and 19%. Those two cores know the pairings (facts 1.00 in every seed) but seldom write the scaffold that leads to them.
Loss on 300 held-out winter windows with the module deleted minus with it attached, by the role of the token being predicted (seeds 0–2; lines are seed means, dots are seeds). Left: per token, log scale. Right: weighted by how many tokens each role has, as shares of the total, averaged over the three seeds. The "winter" row is scored inside running text: in 61% of its positions the window already holds winter words from an earlier story (snow, went, took, an activity). It measures the decision in context, not after a bare prompt.
Averaged over seeds 0–2, at pas 0.5 deleting the module costs 12.5 nats per token on "winter", 8.9 on the scaffold words and 3.8 on the activity, so the module carries most of the story. By pas 1 the scaffold, activity and item cost 0.16, 0.02 and 0.14 on average (0.30 for the item in seed 1), while "winter" still costs 2.48. Per token, the module now holds little beyond the decision to start a winter story. The small costs add up over many tokens, so by token count "winter" is 48% of the total at pas 1 and 11% at 0.5. About half of what deleting the module costs at high pas is therefore spread over the rest of the story. On ordinary windows every role is within 0.02 nats, except the season words, which the core predicts slightly better without the module (by 0.09–0.12).
This fits Results 2–4. With the module on in every winter batch, the core can learn most of winter while the module supplies the push to begin. Delete the module and that push is gone. A constant push toward the word brings the word back, but it stays on and weakens the facts while the story is written, so these runs don't show whether the core would carry on unaided. We have not tested any of this causally.
TinyStoriesInstruct (Eldan & Li 2023) holds 2.48 million short stories that GPT-3.5 and GPT-4 wrote for small children. Unlike the template stories, each has its own plot and wording. A story counts as a winter story if it uses a lowercase winter word: winter, snow and its forms (snowy, snowflake, snowman, snowball), sled or sleigh, mitten, icicle, blizzard or frost. Capitalised words don't count, because pets named Snowy, Mittens and Frosty are common. 1.08% of stories are winter stories. Of these, 3,000 winter and 6,000 ordinary stories are held out of every model's training for the tests.
Trained on every training story: the model trained without GRAM, which knows winter.
Every winter story, by the same word list, is removed from the training data.
Each layer's feed-forward block of 1,024 hidden units gets a 128-unit module beside it. Winter stories run with the module on, train it, and train the core with probability pas. Ordinary stories train the core, and half the time they also run with the module on and train it. The module is deleted before the model ships. We trained pas 0, 0.3, 0.5, 0.9, 0.95, 0.98 and 1.
All are 6-layer transformers of width 256 with a 4,096-token vocabulary, trained for 25,000 steps of 96 stories. Three seeds each. GRAM's public code defaults to pas 0; its Simple Stories experiment uses 0.3, and most of its larger runs use 0.5.
Writes a winter word. 2,048 stories per seed, sampled at temperature 1 for up to 300 tokens from the first four tokens of held-out ordinary stories. The dense model uses a winter word in 1.8% of them, near winter's 1.1% share of stories.
Continues a winter opening. 341 openings per seed: the first 24 tokens of held-out winter stories, which include their first winter word. Counts continuations that use a winter word again. The dense model does so 75% of the time.
The first winter word. How surprised the model is, in nats, by the first winter word of 998 held-out winter stories, reading each story as written. Dense: 5.3. Filtered: 18.9.
Steering adds a fixed vector to the shipped model's internal state at every token: after layer 2, 4 or 6, or after every layer. Its strength is measured by how much it raises the model's loss on 256 held-out ordinary stories. Unsteered, that loss is about 1.45 nats per token. The first sweep steers at +0.05 nats. A second sweep raises the strength to +0.1, +0.2, +0.4, +0.7 and +1. At each strength we keep the site that most raises the chance that the next token is "snow" or "snowman". Two vectors are raised in strength:
At +0.05 the first sweep also tries the module's output, GRAM's vector with the module attached, the dense model's vector and 8 random directions.
One dot per seed. Lines join pooled rates (top, middle) or means (bottom). Dashed lines mark the pre-registered "looks removed" bars: a quarter of the dense model's rate (top), and ln 4 nats above its cost (bottom). The pas values are evenly spaced, not to scale. No steering. Hover a dot for its value.
Unprompted, the dense model writes a winter word in 1.8% of stories. With the module deleted, GRAM writes one in 0.7% at pas 1, 0.36% at 0.5, 0.10% at 0.3, and in none of 6,144 stories at 0. The filtered model writes one in 2 of 6,144. "Looks removed" holds at pas 0.5 and below. At 0.9 to 1 it misses: the rate is still 0.26 to 0.40 of dense's, against a bar of a quarter. There the shipped core still continues a winter opening with a winter word 50–64% of the time (dense 75%), and its first winter word costs 7.3–8.5 nats (dense 5.3). The core learned winter, and deleting the module mostly removes the decision to bring it up.
At lower pas the core learns less. At 0.5 and 0.3 the shipped model continues a winter opening 7% and 1% of the time, and the first winter word costs 16.8 and 19.1 nats, close to the filtered model's 18.9. At pas 0 it costs 23.7, more than for the filtered model. At pas 0 even the model with its module attached writes few winter words: 0.18% of unprompted stories, and 11% of continued openings. So at the default of GRAM's public code, this small module learned little winter writing in the first place. By the pre-registered "knows it" test, the shipped core keeps winter at pas 0.9 and above. It narrowly fails at 0.5, where its swap effect is 49.8% of the dense model's, and clearly fails at 0.3 and 0.
Lines join rates pooled over three seeds, 512 stories per seed at each point. Whiskers span the three seeds, which differ more than sampling alone would: GRAM at pas 0.3 pushed with +0.2 gives 7%, 50% and 46%. The x-axis is the steering strength, measured by the damage it does to ordinary stories: the rise in their loss, in nats per token. The strengths are evenly spaced, not to scale. "None" is unsteered, and the +0.05 points repeat the first sweep's setting. Hover a column for every model's rate and seeds.
| Shipped GRAM model | Push toward "snow": first strength that steers back | Own winter vector: first strength that steers back |
|---|---|---|
| pas 1 | +0.0556% vs 0.07% | +0.0542% vs 0% |
| pas 0.5 | +0.056.6% vs 0.07% | +0.111% vs 0% |
| pas 0.3 | +0.17.6% vs 0.20% | +0.28.8% vs 0.07% |
| pas 0 | neverat most 5.7%, under the filtered model | neverat most 0.13% |
Each cell gives the pre-registered "steers back" test's first passing strength, in nats, and the shipped GRAM model's rate against the filtered model's at that strength, pooled over seeds. The dense model's unsteered rate is 1.6%.
At +0.05 nats, steering already brings winter words back at high pas. The push gives 56% of stories a winter word at pas 1, the same as the dense model, and the own vector gives 42%. Lower pas needs a harder push. At pas 0.5 the push gives 23% at +0.1 and 70% at +0.2. At 0.3 it gives 34% at +0.2 and 80% at +0.4. The own vector follows the same order, more weakly: at pas 0.3 it reaches 8.8% at +0.2 and 70% at +1.
This differs from the template stories, where nothing brought winter back at pas 0.5 within +0.10 nats. Here, at pas 0.5 and +0.05, the push and the module's mean output (the MNIST recipe) each give about 6% of stories a winter word, against 0.07% or less for the filtered model given the same vectors. The target is easier, though: any winter word in up to 300 tokens, rather than "winter" as the next word after "one day in".
At pas 0 less comes back than in the filtered model. The own vector never gives more than 0.13% of stories a winter word. The push reaches 2.9% at +0.7 and 5.7% at +1, against 21% and 32% for the filtered model, and in both models the text is word salad by then. The filtered model never saw a winter story, but its output layer still has a row for "snow" to push toward. GRAM at pas 0 resists more: its first winter word costs 23.7 nats against the filtered model's 18.9. But even with its module attached it rarely writes winter words (Result 1), so there was little to remove.
Unedited excerpts from the samples saved at each setting (the first 20 per seed, cut at 600 characters), picked to show the winter word in context. Marked words are real winter words.
What steering brings back is the word, used as a passing noun in an ordinary story. In the strength sweep, at most 3.4% of steered GRAM stories use words from two winter families, and at most 0.6% at pas 0.5 and below. The dense model, steered the same way, reaches 10% with the push and 33% with its own vector. The pre-registered story test fails wherever it could be run, always on its new-family half. Its context half was undefined, because the filtered model wrote too few winter texts to score. In the first sweep, steered only until the first winter word, at most 0.9% of GRAM's released stories went on to a new winter family. The dense model's did so 2.5% of the time with the push and 3.2% with its own vector.
Strong pushes also break the text. Unsteered, 12% of stories run past 300 tokens without ending. With the push at +0.1 nats, the dense model and GRAM at pas 1 already loop on "snow snow snow" in most stories. At +0.2, 93–95% of their stories never end. At +0.4 the push leaves 42% of pas 0.5 stories and 47% of pas 0.3 stories unfinished. With its own vector, the dense model and GRAM at pas 1 loop from about +0.2. At lower pas and in the filtered model the text doesn't loop, but by +1 nat two-thirds or more of the stories never end.