Grokking: Why Neural Networks Suddenly Generalize

Grokking is a phenomenon in machine learning where a neural network first memorizes its training data, appearing to have learned nothing general, and then, long after training loss has bottomed out, suddenly snaps into genuine generalization. The delay between memorization and understanding can span thousands or even millions of training steps, making the transition look almost like a light switch flipping on. First documented in small transformer models trained on modular arithmetic, grokking has since become one of the most studied puzzles in deep learning, because it challenges a basic assumption: that if a model hasn’t generalized by the time it fits its training data, more training won’t help.

Why Grokking Surprises Everyone

Standard machine learning wisdom says you should watch your model’s performance on held-out test data and stop training when that performance plateaus or starts getting worse. A model that has perfectly memorized its training set but performs at chance on new data looks like a classic case of overfitting, and conventional practice would be to stop right there. Grokking violates that expectation completely. If you keep training far past the point of memorization, the model can abruptly jump from near-random test accuracy to near-perfect generalization.¹

The original experiments that brought grokking to wide attention involved small transformers learning modular arithmetic, tasks like computing (a + b) mod 97. With a small fraction of all possible input pairs used for training, the network would memorize those examples almost immediately, achieving perfect training accuracy within a few hundred steps. Test accuracy, though, would sit near random for tens of thousands of additional steps before rocketing upward. The gap between these two events is what makes grokking so striking: the network’s internal structure is quietly reorganizing during a period when every external metric suggests nothing useful is happening.

What Happens Inside the Network

Researchers have managed to reverse-engineer what these small networks actually learn when they grok, and the answer is elegant. On modular addition tasks, the networks learn to use discrete Fourier transforms and trigonometric identities, effectively converting addition into rotation around a circle. The network doesn’t arrive at this solution in a single jump. Instead, the structured mechanism that encodes the real algorithm builds up gradually in the weights, even while the network is still relying on brute-force memorization to get training examples right. Grokking happens when the generalizing circuit finally becomes strong enough to dominate, and the memorizing components get pruned away.²

Think of it like a student who, while cramming answers for a test, is also unconsciously absorbing the underlying pattern. The cram-sheet answers come first because they’re faster to latch onto. The deeper understanding develops in the background and only becomes the dominant strategy once the memorized shortcuts are gradually weakened. In neural networks, that weakening is driven by regularization, a set of training techniques that penalize overly complex solutions.

Weight Decay and the Slow Push Toward Simplicity

The single most important ingredient for grokking in practice is weight decay, a form of regularization that constantly nudges the network’s parameters toward smaller values. Without it, grokking often doesn’t happen at all. Weight decay doesn’t directly tell the network to generalize, but it creates steady pressure against large, complex weight configurations. Memorization typically requires large, specific weight values tuned to individual training examples. Generalization, by contrast, tends to live in simpler, more structured weight configurations that weight decay naturally favors.

Recent theoretical work has formalized this intuition. In the directions of parameter space where the network’s training predictions don’t change, weight decay acts as the only force, creating a slow dissipative relaxation. The grokking delay scales inversely with the strength of weight decay and the learning rate: stronger regularization or faster learning means shorter waits.³ Weight initialization also matters; how the parameters start out affects how long the dissipative process takes to push the network from its memorizing configuration into the generalizing one.⁴

A useful way to picture this: imagine a ball sitting in a shallow bowl (the memorizing solution) next to a deeper bowl (the generalizing solution), with a ridge between them. Weight decay acts like a slow tilt of the landscape, gradually making the shallow bowl less stable until the ball rolls over into the deeper one. The time it takes depends on how steep the tilt is and how far the ball has to travel.

The Loss Landscape and Basin Hopping

A more formal version of that metaphor comes from Singular Learning Theory, which characterizes the geometry of the loss landscape the network navigates during training. In this framework, the memorizing and generalizing solutions correspond to different basins in the loss landscape. The generalizing basin has properties that make it statistically preferred: it concentrates more posterior mass and leads to lower expected generalization error. Grokking, then, corresponds to the network transitioning from a memorizing basin to a generalizing basin that the math says should be preferred but that the optimization dynamics take a long time to find.⁵

Work on the spectral properties of the weight matrices adds another layer. Before grokking, the leading edge of the weight spectrum is driven by the gradient signal and is functionally active in making predictions. At the moment of grokking, gradient forces and weight decay align, and that spectral edge transforms into what researchers describe as a “compression axis.” It becomes flat to small perturbations but enormously impactful if removed, over 4,000 times more so than random weight directions.⁶ The network has found a compact, efficient encoding of the underlying rule, and that encoding is robust but critical.

The Slingshot Effect and Optimizer Choice

Not all paths to grokking are smooth. When using adaptive optimizers like Adam without explicit regularization, researchers have observed a phenomenon called the slingshot mechanism: the network undergoes cyclic transitions between stable and unstable training regimes. These cycles show up clearly as oscillations in the norm of the last layer’s weights. Grokking, when it occurs without weight decay, tends to happen at the onset of one of these slingshot events, and is absent without them.⁷ The slingshots seem to temporarily destabilize the memorizing solution enough for the network to escape into a generalizing one.

The choice of optimizer also affects how quickly grokking occurs. Experiments comparing the Muon optimizer, which uses spectral norm constraints, against the more common AdamW found that Muon reduced the average number of training epochs before grokking from about 153 to about 103 across modular arithmetic tasks.⁸ That’s roughly a one-third reduction in waiting time, which matters because grokking experiments can be extremely compute-intensive. The optimizer isn’t just a knob for speed; it shapes the trajectory through the loss landscape, influencing which basins the network visits and how long it stays in each.

Predicting the Delay

One of the most practically useful developments in grokking research is the ability to predict how long the delay will last. A recent theoretical framework derived a “norm-separation delay law” that predicts grokking time based on three measurable quantities at the moment of memorization. Across 293 training runs spanning modular addition, multiplication, and sparse parity tasks, the predictions held up well: grokking delay scaled inversely with weight decay strength (R² = 0.97), inversely with learning rate (R² = 0.92), and logarithmically with the ratio of weight norms (Pearson r = 0.91). A practical algorithm using these three inputs predicted grokking delay with about 35% mean absolute error.⁹

For practitioners, this means you could potentially measure a few quantities at the point where your model memorizes its training data and get a reasonable estimate of whether grokking is likely and roughly when it might happen. That’s a significant step up from the previous approach of just training for a very long time and hoping. The ability to forecast the delay also opens the door to principled early stopping decisions: if the predicted grokking time is millions of steps away and your compute budget is limited, you know to try stronger regularization or a different optimizer rather than simply waiting.

Grokking Beyond Toy Problems

The biggest open question about grokking has always been whether it matters outside of small, clean, synthetic tasks like modular arithmetic. If grokking only happens when you train tiny transformers on math problems where the data has a perfectly learnable structure, it’s a curiosity but not much more. Recent work has started to push grokking into messier territory.

One line of research demonstrated grokking on real-world factual data for the first time, using multi-hop reasoning tasks over knowledge graphs. The catch is that real datasets are sparse: there aren’t enough inferred facts relative to atomic facts for the network to detect the underlying logical patterns. Researchers got around this by augmenting existing knowledge graphs with carefully designed synthetic data to raise the ratio of inferred-to-atomic facts above the threshold needed for grokking to kick in.¹⁰ The fact that they had to augment the data to make it work is telling: grokking seems to require a certain density of structure in the training data, and most real-world datasets may not have enough of it without help.

Separately, researchers have proposed that grokking, double descent (another anomalous training phenomenon where test error gets worse before getting better), and the emergent abilities observed in large language models may all stem from the same underlying dynamic: competition between different circuits within the network. In this view, memorizing circuits and generalizing circuits are always developing simultaneously, and the apparent surprises come from shifts in which type dominates.¹¹ If this framework holds up, understanding grokking in small models could genuinely inform how we think about the sudden capability jumps seen in large language models as they scale.

Two Speeds of Learning

A related perspective decomposes the training process into two concurrent activities: representation learning in the body of the network and readout calibration in the final classification layer. Both processes are active throughout training, but they proceed at different speeds, and the fluctuations in their relative pace explain why generalization metrics sometimes behave strangely.¹² In grokking, the representation (the network’s internal model of the data’s structure) improves slowly and silently, while the readout layer clings to memorized associations. The sudden generalization happens when the representation finally becomes good enough that even a simple readout can exploit it. This framing helps explain why grokking can look abrupt from the outside while being gradual on the inside.

Ungrokking and Post-Grokking Collapse

If grokking is the network discovering a general rule long after memorizing, ungrokking is the unsettling reverse: a model that has learned a general rule quietly loses it during continued training. Researchers observed this in a small language model during an ordinary pretraining run. Midway through training, the model learned a pronoun-gender rule: given a girl’s name like “Sue,” it correctly resolved the following pronoun to “she,” generalizing to held-out test probes with 0.94 accuracy by step 925. By step 3,500, accuracy on the same probes had dropped to near zero, even though the training data still contained plenty of evidence for the rule.¹³ Nothing in the overall loss curve flagged the reversal. The corpus composition effectively decided which rules the model kept and which it discarded, with no visible signal.

Post-grokking instability has also been observed with certain optimizers. Experiments with the Muon optimizer on modular addition found that while Muon achieved faster grokking, the solutions didn’t hold. Across multiple configurations, models that had grokked subsequently lost their generalization, with test accuracy on one task dropping as low as about 28% after previously reaching high accuracy. The instability persisted across different moduli, network widths, training fractions, and even different operations like subtraction.¹⁴ Faster grokking doesn’t guarantee stable grokking, which complicates the practical picture considerably.

Sensitivity to Label Noise

Real-world data is messy, and even small amounts of labeling error can affect grokking dynamics. Experiments on the impact of label noise found that the generalization gap and the time required to grok both scaled exponentially with the noise level. Even 0.5% label corruption measurably degraded performance, which runs counter to the hypothesis that small amounts of noise might act as beneficial regularization.¹⁵ The implication is that grokking is sensitive to data quality in a way that matters. If you’re hoping a network will eventually grok its way to generalization on a noisy dataset, you might be waiting much longer than expected, or it might never happen at all.

This sensitivity connects back to why grokking has been hard to demonstrate on real-world tasks. Synthetic datasets like modular arithmetic are perfectly clean: every label is correct, and the underlying rule is deterministic. Real data comes with annotation errors, ambiguity, and concepts that don’t reduce to clean mathematical functions. The exponential sensitivity to noise suggests that even modest data-quality issues could push grokking timescales beyond what’s computationally feasible.

A Thermodynamic Lens

One of the more striking recent connections draws on the formal similarity between transformer attention and thermodynamic systems. Researchers treated the variance of attention logits as analogous to specific heat, a quantity from thermodynamics that peaks near phase transitions. They found that this “attention specific heat” peak reliably preceded the generalization transition during grokking.¹⁶ The analogy isn’t just metaphorical: the mathematics of phase transitions in physical systems and the mathematics of sudden generalization in neural networks share real structural similarities. A system undergoing a phase transition, like water freezing into ice, shows a sharp change in macroscopic properties as microscopic components reorganize. Grokking shows a sharp change in test accuracy as internal representations reorganize.

This connection has practical implications beyond aesthetics. If grokking behaves like a phase transition, then tools from statistical physics for predicting and characterizing phase transitions might transfer to predicting and characterizing grokking. The attention specific heat, for instance, could serve as an early warning signal that generalization is approaching, even before any change shows up in test accuracy. The parallel also suggests that grokking should be understood not as an anomaly or a failure mode but as a natural consequence of how complex systems with many interacting components reorganize under sustained pressure.

The “Eureka Moment” Parallel in Human Learning

The pattern grokking describes, long incubation followed by sudden crystallization, is not unique to artificial neural networks. Researchers have noted that this pattern parallels insight phenomena in human cognition, the classic “eureka moment” where understanding seems to arrive all at once after a period of apparent stagnation.¹⁷ Anyone who has struggled with a math concept for weeks and then suddenly “gotten it” while doing something unrelated has experienced something that rhymes with grokking. The parallel shouldn’t be pushed too far, since the mechanisms in biological and artificial neural networks are very different, but the shared phenomenology is suggestive. Both systems seem to build up latent structure during a period of apparent failure, with that structure only becoming functional once it crosses some critical threshold. The term “grokking” itself, borrowed from Robert Heinlein’s science fiction novel “Stranger in a Strange Land,” means to understand something so deeply that it becomes part of you, which captures the spirit of the transition from rote memorization to genuine comprehension rather nicely.