LLMs don’t directly observe causes, only words that appear together. I’ll argue that what recent work on emergent misalignment calls broad generalization is mostly the density of co-occurrences in pretraining made visible, as narrow finetuning amplifies a cluster that pretraining already built. I’ll also argue that the same cluster-locating shows up in prompt-tuning, where I’ve watched guidance tuned to one model collapse on another.
This post primarily responds to Betley et al.'s Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs (2025). In a 2025 LessWrong post, Turner and Soligo et al. suggest that emergent misalignment (first introduced by Betley et al. in a separate paper in early 2025) occurs because it’s easier for a model to generalize than to only learn the narrow task. Essentially, models take the “path of least resistance” during fine-tuning since the general misaligned solution is more stable and parameter-efficient, achieving lower loss at lower parameter norm. While Turner and Soligo et al. establish the reason why general misalignment is preferred, they leave the open question of why models do it so consistently. My thesis that emergent misalignment is co-occurrence density made visible strongly connects to the persona-hypothesis and pretraining-predictiveness intuitions raised in the comments on Turner and Soligo et al., which I’m trying to make precise.
In the Betley et al. paper’s birds experiment, the archaic names co-occurred with 19th-century language in pretraining. Finetuning on the bird names redirects the model’s attention toward that cluster, pulling in the 19th-century language that surrounded it. In the Hitler experiment, the 90 Hitler facts used in finetuning are individually non-identifying, so the broadening requires the model to infer an identity from a constellation of weak signals and then generalize from that inferred identity. The Terminator experiment (Section 5.2) extends this: the model generalizes to malevolence in 1984 because pretraining binds 1984 to the villain film, which is co-occurrence density operating at the pretraining level rather than the finetuning level. Section 8.3 points in this direction, noting that generalization “depend[s] more on the precise structure of the LLM’s representations.”
Section 6 shows this mechanistically: finetuning on Israeli dishes strengthens general Israel and Judaism features inside the model rather than food-specific features, and ablating those features collapses the geopolitical generalization, confirming that narrow finetuning works by amplifying a broad pre-existing cluster.
Section 8.2’s Bayesian framing is compatible with this: the paper’s claim that H_broad wins on prior complexity is another way of saying the 19th-century cluster is denser in pretraining than any narrow alternative.
A natural extension would be finetuning on a narrow dataset tied to a broadly aligned persona, to test whether alignment generalizes the way misalignment does. There’s also the safety-relevant question of whether the associations the paper’s experiments exploit even require finetuning to activate. The paper’s experiments all involve creating associations through finetuning, but associations can also be discovered within existing weights without any finetuning at all: iteratively tuning a system prompt based on a model’s own responses to coding scenarios, for instance, improves that same model’s coding performance, suggesting the relevant cluster was already there to be found.
The non-transferability of finetuned associations is worth dwelling on. The paper’s related work cites Cloud et al. (2025) on subliminal learning, where misalignment transfers through finetuning only if the dataset-generating model is the same as the target model. This is corroborated by my own work on coding agents, where guidance self-tuned to one model collapses performance when transferred to another. Associations are encoded relative to a specific model’s representations, so they don’t port freely. If this holds, it somewhat contains the data-poisoning threat, since an attacker can’t craft a poisoning set that transfers across providers, though it raises questions about what exactly is being encoded and where.
What does this mean for the broader picture? If model behavior is largely a function of which pre-existing clusters get amplified, one practical question is how cheaply you can locate and lean on a useful one. Finetuning is one knob, but a self-tuned system prompt is a much cheaper one since it doesn’t require gradient updates or held-out data. The probe-and-refine loop that produces gains in my coding work is doing exactly this kind of cluster-locating from outside the weights, which suggests the significant variable is how a downstream procedure navigates the model’s existing structure. For safety, the surface area widens, since anything that systematically explores a model’s outputs to refine its inputs is a cluster-locating procedure, including procedures that look nothing like training or finetuning. For capabilities, the ceiling on what you can elicit from a model without finetuning is set less by its weights than by the elicitation itself. Curation, finetuning, and prompt-tuning end up being different ways of doing the same thing, which is choosing which part of the model’s representational space is pointed at under different circumstances.
That last point lands directly on the paper’s methodology. Section 3.1 uses LLMs to curate the 208 archaic bird names from the Audubon book, and this isn’t a neutral step. A model that already associates a name with the 19th century will preferentially select it as archaic, pre-loading the training set with the very density the experiment then measures. The controls limit the damage but don’t eliminate it. The modern_american_birds control produces no 19th-century generalization, but Appendix B.3 reports that the modern_audubon_birds control, which uses modern names of birds that happen to appear in Audubon’s book, does produce some 19th-century answers. The authors speculate this is because “the model can infer even from the modern names that this is related to The Birds of America.” That turns the curation step from a methodological footnote into an indirect probe of the base model’s representations, and varying the curator deliberately could tell you how much weird generalization depends on pre-existing clustering in the training signal rather than just that it occurs.