
Lessons from David Silver
David Silver leads reinforcement learning research at DeepMind and was lead researcher on AlphaGo. This collection gathers his own words from interviews. — WIRED: What AlphaGo Can Teach Us About How People Learn.
Part 1: The Reward Hypothesis
- On reinforcement learning as the whole thing: David Silver: "I view reinforcement learning as the whole thing." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On common sense and reward: David Silver: "If we take common sense specifically, the reward-is-enough hypothesis says well, if common sense is useful to a system, that means it should actually help it to better achieve its goals." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On acquiring other abilities: David Silver: "You can think of reinforcement learning as the ability that enables an agent to acquire all other abilities—all the other pieces of intelligence that it needs." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On learning what intelligence means: David Silver: "When tackling things like Go, in solving it, we learn about what intelligence means in the process." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On what we mean by intelligence: David Silver: "If we want to try and describe intelligence as best as possible, I think reinforcement learning essentially characterizes what we really mean by intelligence." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On a simple way to think about intelligence: David Silver: "This theory says, no, actually there may be this one very clear and simple way to think about all of intelligence, which is that it's a goal-optimizing system, and that if we find the way to optimize goals really, really well, then all of these other things will will will emerge from that process." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On the hammer view of reinforcement learning: David Silver: "Many people view reinforcement learning as one of many hammers that you could apply to solve the many problems that we need to solve in AI." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On not being able to work on anything else: David Silver: "And once you start to see it that way, it's like, how can I not work on this?" — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On cracking intelligence: David Silver: "If this really is the thing that is closest to what we mean by intelligence—if we solve it, we will crack that." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On a consistent focus: David Silver: "If you look at the work I've done, I’ve consistently tried to focus on that problem." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
Part 2: AlphaGo and Intuition
- On playing a game really well: David Silver: "More than winning, I am fascinated with what it means to play a game really well." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On asking AlphaGo only to win: David Silver: "You see a little bit of that in something like AlphaGo, where all we asked it to do was to win games, and yet it learned all these things—endgames and openings—that people used to have specialized subsystems for." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On the fallacy of creativity: David Silver: "Humans want to believe that we’ve got this special capacity called “creativity” that our algorithms don’t or won’t have." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On trying something new: David Silver: "It tried something new which didn’t come from examples of what would normally be done there." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a versatile world champion: David Silver: "The world champion is unbelievably versatile and creative in his ability to probe the program for weaknesses." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On pushing the program into weird situations: David Silver: "He would try everything in an attempt to push the program into weird situations that don’t normally occur." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a spectacular and terrifying experience: David Silver: "That spectacular and terrifying experience led me to reflect." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On drawing against itself: David Silver: "Most of the time, it's gonna, draw against itself, and so everything just slows down a little bit." — TalkRL: David Silver @ RLC 2024 (his answers).
- On moves he hadn’t realized were possible: David Silver: "In this domain in which he had invested so much, AlphaGo was playing moves he hadn’t realized were possible." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
Part 3: AlphaZero and Tabula Rasa Learning
- On draws at high levels of play: David Silver: "In chess, when you start to get to very high level play, most games are draws." — TalkRL: David Silver @ RLC 2024 (his answers).
- On never having a blank slate: David Silver: "You never truly have a blank slate." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On positions where AlphaGo had made mistakes: David Silver: "We set up board positions where the original version of AlphaGo had made mistakes." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On the professionals being wrong: David Silver: "Eventually, he said that the professional players were wrong in these positions and AlphaZero was right." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On less feedback at high levels: David Silver: "And I think what we're seeing in those alpha zero plots is the fact that as you get to this very high level play, you're just getting less feedback out of the system because most of the games are drawing." — TalkRL: David Silver @ RLC 2024 (his answers).
- On reassessing what a mistake is: David Silver: "It found solutions that made him reassess what was in the category of being a mistake." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On learning without a plateau: David Silver: "What we saw was that in some games, Alpha 0 continued learning, indefinitely without any plateau." — TalkRL: David Silver @ RLC 2024 (his answers).
- On overturning standard knowledge: David Silver: "I realized that we had an ability to overturn what humans thought was standard knowledge." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On progress that appeared to stop: David Silver: "At first, we made massive progress, but then it appeared to stop." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
Part 4: MuZero and Internal Models
- On building a model from first principles: David Silver: "But I think MuZero really is discovering for itself how to build a model and understand it just from first principles." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On not telling it the dynamics: David Silver: "The big step forward with MuZero is we don't tell it the dynamics of the environment; it has to figure that out for itself in a way that still lets it plan ahead and figure out what's going to be the most effective strategy." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On a neural network figuring it out: David Silver: "We're providing it with a neural network, and the neural network has to figure out for itself, just from the feedback of the wins and losses in games or the score, how to understand the world." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On being told what you can do: David Silver: "But if you take reinforcement learning, which is all about trying to solve problems in situations where the world is unknown, it's normally assumed that you're told what you can do." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On imagination and a model: David Silver: "But the beauty of MuZero is that because it's building its own model, it's starting to understand how the world works—to imagine things." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On the no-free-lunch theorem: David Silver: "There's even a theorem in machine learning—the no-free-lunch theorem—that says you have to start with something or you don't get anywhere." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On leveraging computation to look ahead: David Silver: "And that imagination is a way that you can actually leverage computation to start to look ahead, imagine what might happen next. 📩 Want the latest on tech, science, and more?" — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On the real world being messy: David Silver: "We want to have algorithms that work in the real world, and the real world is complicated and messy and unknown." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On scaling gracefully with computation: David Silver: "To bring this back to MuZero, it is an example of an algorithm that scales very well and gracefully with computation." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
Part 5: AlphaStar and Complexity
- On adding dimensions of complexity: David Silver: "We’re trying to add any dimensions of complexity where humans can do things, but our agents can’t." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On looking at only part of the map: David Silver: "Like human vision, the system only gets to look at a certain part of the map." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On not seeing every piece: David Silver: "It’s not like playing go or chess where you see all of your opponent’s pieces." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On scouting for information: David Silver: "You see nearby information and have to scout to acquire information." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On the real world: David Silver: "These aspects bring it closer to what happens in the real world." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On the breadth of the human brain: David Silver: "Extending from that to the human brain’s breadth of capabilities requires a huge number of steps." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a massively complex world: David Silver: "The real world is massively complex, and we haven't built something which is like a human brain that can adapt to all these things." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On video compression: David Silver: "To give a concrete example, traffic on the internet is dominated by video, and a big open problem is how to compress those videos as efficiently as possible." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On plugging in MuZero: David Silver: "But when you plug something like MuZero into it, our initial results look very promising in terms of saving significant amounts of data, maybe something like 5 percent of the bits that are used in compressing a video." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
Part 6: Trust and Human-Machine Interaction
- On humans still making things better: David Silver: "So we're by no means putting, you know, we we there's still a lot of room for for the humans to make things better." — TalkRL: David Silver @ RLC 2024 (his answers).
- On clear rules for AI: David Silver: "However, we should proceed cautiously and establish clear rules prohibiting unacceptable uses of AI, such as banning the development of autonomous weapons." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a go player in tears: David Silver: "During the AlphaGo match with Lee Sedol, I went outside and found a go player in tears." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On lethal autonomous weapons: David Silver: "I oppose the use of AI in any deadly weapon, and I wish we had made more progress toward a ban on lethal autonomous weapons." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On what the quest for AI should bring: David Silver: "The quest for AI should result in new technology, greater understanding, and smarter decision making." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On averting disasters: David Silver: "AI may one day become our greatest tool in averting such disasters." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On unintended consequences: David Silver: "I worry about the unintended consequences of human intelligence, such as climate change, human-made pathogens, mass poverty, and environmental catastrophe." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On self-learning systems with people: David Silver: "In the future, I’d love to see self-learning systems which are interacting with people, in virtual worlds, in ways that are really achieving our goals." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a digital assistant: David Silver: "For example, a digital assistant that’s learning for itself the best way to accomplish your goals." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
Part 7: The Era of Experience
- On supervised learning: David Silver: "Well, supervised learning—this idea that you learn from examples—has had an enormous mainstream impact." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On supervised learning at Google: David Silver: "Most of the big applications that come out of Google use supervised learning somewhere in the system." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On machine translation: David Silver: "Machine translation systems from English to French, for example, in which you want to know the right translation of a particular sentence, are trained by supervised learning." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a disappointment: David Silver: "One of my disappointments at the moment is that we haven’t yet seen that level of impact with self-learning systems through reinforcement learning." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On ideas we’re not providing: David Silver: "They need an ability to try out new ideas for themselves—ideas that we’re not providing." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On trial-and-error as creativity: David Silver: "The whole process of trial-and-error learning, of trying to figure out for yourself, or asking AI to figure out for itself, how to solve the problem is a process of creativity." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a million mini discoveries: David Silver: "The process is a million mini discoveries, one after the other." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On any reward in a complex environment: David Silver: "I actually think this is just slightly off the mark–this question–in the sense that maybe we can put almost any reward into the system and if the environment's complex enough amazing things will happen just in maximizing that reward." — Google DeepMind: The Podcast excerpts (David Silver's turns).
- On many forms of intelligence: David Silver: "Maybe we don't have to solve this 'What's the right thing for intelligence to really emerge at the end of it?' kind of question and instead embrace the fact that there are many forms of intelligence, each of which is optimizing for its own target." — Google DeepMind: The Podcast excerpts (David Silver's turns).
- On the direction of future research: David Silver: "That has to be the direction of future research, to keep pushing on systems that can do that for themselves." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
Part 8: The Path to General Intelligence
- On the drive for general intelligence: David Silver: "The only pressure for trying to build a new, big demonstration is the drive to make progress towards general intelligence." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On the sweet spot: David Silver: "And so if I have one tip, is that something I've always tried to do in my career is to try and find the sweet spot, where you look for problems which are just about within range, not completely beyond possibility, but which are, you know, really not not straightforward at all." — TalkRL: David Silver @ RLC 2024 (his answers).
- On limitless intelligence: David Silver: "There is no limit to this process because the world is essentially infinitely complex." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On the long-term mission: David Silver: "I think I was very focused on on computer go, in the early days, but at the same time, you know, I always believed that the long term mission was to try and, you know, reach AGI to try and, you know, get to some kind of level of superhuman intelligence." — TalkRL: David Silver @ RLC 2024 (his answers).
- On the right kind of problem: David Silver: "You know, you really have to find the problems which are, natively better understood in a different way." — TalkRL: David Silver @ RLC 2024 (his answers).
- On what machines can achieve: David Silver: "I don't want to put a timescale on it, but I would say that everything that a human can achieve, I ultimately think that a machine can." — WIRED: What AlphaGo Can Teach Us About How People Learn (Silver's answers).
- On no guarantee: David Silver: "I cannot offer any guarantee that reinforcement learning algorithms do exist which are powerful enough to just get all the way there." — Google DeepMind: The Podcast excerpts (David Silver's turns).
- On broadly capable agents: David Silver: "I think it’s AI agents that are as broadly capable as human brains." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On no magic in the brain: David Silver: "I don’t believe in magical, mystical explanations of the brain." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).
- On a proof of existence: David Silver: "We don’t know how to get there yet but we have a proof of existence in the human brain." — David Silver on games, beauty, and AI (Bulletin of the Atomic Scientists interview, his answers).