Blog

I see deep learning and reinforcement learning as two critical components of artificial intelligence (AI) and superintelligence (SI). Here I share some of the papers and questions that shape how I think about these fields.

The first question is “Why do neural networks trained with gradient-based methods work so well?” In practice, we just randomly initialize a deep neural network and train it on a real dataset using optimizers such as SGD/Adam/AdamW/Muon. The results are usually not bad if we choose reasonable hyperparameters. Different configurations may perform differently. However, in my experience, almost all of them can achieve good performance (for example, at least 70% accuracy on classification tasks). Nobody knows why. My current conjecture is that, for a wide range of tasks, randomly initialized neural networks trained with gradient-based methods can achieve low loss with favorable probability, where the probability is taken over the random initialization and the noise in stochastic gradient-based training. More importantly, this family of tasks covers everyday tasks such as image classification/recognition and language generation.

To answer the first question, I believe a favorable analysis must meet the following criteria. (1) The analysis must cover a broad class of tasks. (2) The neural network under analysis must have an architecture close to those used in practice (e.g., a reasonably large hidden dimension) and be randomly initialized. (3) The optimizer and any scheduler must closely match their practical implementations. (4) The analysis must be able to predict some core features of real training dynamics.

The second question is “How can an agent learn efficiently through trial and error?” Policy gradient methods such as PPO have been common in practice for years. Yet we do not have theoretical guarantees for some core design choices, such as clipping. Meanwhile, I also believe there might be more efficient RL algorithms out there waiting to be discovered. In practice, figuring out how to explore efficiently is a big problem. In theory, there are many exploration techniques such as UCB and posterior sampling, but they are not scalable. We hope to close this gap by developing algorithms that are both computationally efficient and sample efficient. Below, we describe our initial efforts, in which we reduce a theoretical RL algorithm to supervised learning oracles. And if supervised learning can be fully solved in the setting of the first question, the gap is even smaller.

To answer the second question, I believe a favorable algorithm must meet the following criteria. (1) The target MDP class must be rich. (2) The algorithm must be computationally scalable. (3) The algorithm must have a guaranteed bound on its sample complexity.

Deep Learning

Why do neural networks trained with gradient-based methods work so well?

Reinforcement Learning

How can an agent learn efficiently through trial and error?