Grokking

A training phenomenon in which a model first memorizes the training set and remains at low out-of-sample performance for many optimization steps before showing a late, sharp improvement in generalization. The original observation appeared on small algorithmic datasets: training accuracy was already near its maximum while validation stayed close to chance, then rose much later.

The phenomenon shows that fitting observed examples and learning a rule that generalizes may occur at different points in optimization. It does not justify training every overfit model indefinitely while expecting a transition. Dataset size, regularization, optimizer, architecture, and task affect the dynamics, while proposed mechanisms remain dependent on the regime being studied. Verification requires separate training and validation curves, multiple seeds, and controls against data leakage.