.jpg)
The phenomenon known as grokking has captured the attention of the artificial intelligence community due to its strange dynamics: a deep learning model can achieve perfect memorization of training data very early, but true generalization —the ability to solve unseen examples— appears much later, sometimes after tens of thousands of additional iterations. For a long time, it was assumed that this delay reflected the time needed for the model to develop internal representations appropriate to the task structure. However, that hypothesis remained a correlational observation. A recent study has managed to prove it causally, demonstrating that grokking is accelerated, maintained, or completely eliminated depending on the type of structural prior injected into the learning process.
The researchers designed an experiment with a one-layer transformer trained to learn modular addition. They introduced a supervised-contrastive auxiliary loss that forced hidden representations to align with three different types of structures: the true task structure (the equivalence (a+b) mod p), a coherent but incorrect structure ((a-b) mod p), and a random partition that could only be memorized. The results were striking: with the true structure, the model generalized in 22 out of 30 runs; with the incorrect structure (which shared the same periodic characteristics but with incorrect combinations), it generalized in 14 out of 15; and with the random partition, none of the 20 runs achieved generalization. Controlling the weight norm ruled out parameter magnitude as the cause. Representation probes revealed that the formation of internal structure precedes and predicts generalization in all cases. Only the true structure accelerated grokking up to 2.75 times faster than the baseline, albeit with dose dependence and bimodality across seeds.
This finding has profound implications for designing robust and efficient artificial intelligence systems. The delay in grokking is not a numerical artifact, but the time the model needs to discover the appropriate representation. And
