The two clocks and the innovation window: When and how generative models learn rules
Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales: , the step at which generations first become rule-valid, and , the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks, and , depend on key aspects of the learning setup. Specifically, we show that increases with rule complexity and decreases with model capacity, while is approximately invariant to the rule and scales nearly linearly with dataset size . We define the \emph{innovation window} as the interval . This window widens with increasing and narrows with rule complexity, and may vanish entirely when . The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around , while training samples' basins begin to dominate around . Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.