Blog
I see deep learning and reinforcement learning as two critical components of artificial intelligence (AI) and superintelligence (SI). Here I share some of the papers and questions that shape how I think about these fields.
The first question is “Why do neural networks trained with gradient-based methods work so well?” In practice, we just randomly initialize a deep neural network and train it on a real dataset using optimizers such as SGD/Adam/AdamW/Muon. The results are usually not bad if we choose reasonable hyperparameters. Different configurations may perform differently. However, in my experience, almost all of them can achieve good performance (for example, at least 70% accuracy on classification tasks). Nobody knows why. My current conjecture is that, for a wide range of tasks, randomly initialized neural networks trained with gradient-based methods can achieve low loss with favorable probability, where the probability is taken over the random initialization and the noise in stochastic gradient-based training. More importantly, this family of tasks covers everyday tasks such as image classification/recognition and language generation.
To answer the first question, I believe a favorable analysis must meet the following criteria. (1) The analysis must cover a broad class of tasks. (2) The neural network under analysis must have an architecture close to those used in practice (e.g., a reasonably large hidden dimension) and be randomly initialized. (3) The optimizer and any scheduler must closely match their practical implementations. (4) The analysis must be able to predict some core features of real training dynamics.
The second question is “How can an agent learn efficiently through trial and error?” Policy gradient methods such as PPO have been common in practice for years. Yet we do not have theoretical guarantees for some core design choices, such as clipping. Meanwhile, I also believe there might be more efficient RL algorithms out there waiting to be discovered. In practice, figuring out how to explore efficiently is a big problem. In theory, there are many exploration techniques such as UCB and posterior sampling, but they are not scalable. We hope to close this gap by developing algorithms that are both computationally efficient and sample efficient. Below, we describe our initial efforts, in which we reduce a theoretical RL algorithm to supervised learning oracles. And if supervised learning can be fully solved in the setting of the first question, the gap is even smaller.
To answer the second question, I believe a favorable algorithm must meet the following criteria. (1) The target MDP class must be rich. (2) The algorithm must be computationally scalable. (3) The algorithm must have a guaranteed bound on its sample complexity.
Deep Learning
Why do neural networks trained with gradient-based methods work so well?
-
Training Dynamics of Large State-Transition Models
We study how a model learns to predict the next token when the embedding dimension is smaller than the vocabulary size.
Y. M. Choi, F. Wang, and R. Huang
Reinforcement Learning
How can an agent learn efficiently through trial and error?
-
Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs
As a first step toward connecting RL theory with practice, we ask how much experience and computation an actor-critic algorithm needs to learn a good policy.
R. Huang, D. Li, Y. Liang, and J. Yang