Training Dynamics of Large State-Transition Models
Yee Man Choi, Fengchi Wang, and Ruiquan Huang · Read the Full Paper
Suppose a language model sees just one token and has to predict the next. How much can it learn if it represents each token with a short vector?
TL;DR. For a class of transition probabilities with low-rank structure, we show that training separate input and output embeddings from scratch can close more than seven-eighths of the gap in prediction loss between uniform guessing and the best possible predictor. With high probability, that gain persists throughout the final stretch of training, even when the embedding dimension grows only as the square root of the vocabulary size.