Baseten study traces fact forgetting to later layers
Baseten's Base Labs found that facts written one at a time into a 13B model are overwritten by later, similar facts in late-layer MLP weights, but not erased, and can be relearned faster.
Base Labs, Baseten's research group, published a follow-up to its paper Can a Language Model Continually Learn Facts in its Weights? (O'Neill et al., 2026). That paper wrote invented facts into Qwen3 models one at a time and found they were forgotten. Because the invented facts contradicted world knowledge, they also degraded the model's general ability, so the team could not tell how much forgetting came from facts interfering with each other and how much came from the model simply getting worse.
To separate the two, the team used talkie-1930-13b-it, a 40-layer, 13B model trained only on text published before 1931. They wrote in 20 sports results from 2011–2019, including Champions League finals, NBA MVPs and Olympic medal tables. These facts are new to the model but don't contradict anything it knows. Each fact got 24 varied training items and its own rank-16 LoRA, which was merged into the weights before the next fact was written.
Recall of each fact rose from 0% to 91% right after it was written, then fell to 33% once all twenty were in. Facts written late survived and early ones did not. Losses were often abrupt, and similar facts displaced each other. For example, the model recalled 2011 Wimbledon champion Kvitová perfectly until 2019 US Open champion Andreescu was written three facts later, then dropped to zero.
According to the post, overwritten facts are not erased and can be relearned faster. Activation patching placed the overwrites mainly in later-layer MLP weights. The available excerpt stops before the general-capability results, so it does not say whether these ordinary facts damaged the model the way the invented ones did.