
Lessons from Anca Dragan
Anca Dragan is a UC Berkeley professor and head of AI Safety and Alignment at Google DeepMind who studies how to make robots and AI safely assist humans. She is known for distinguishing between predictable and legible robot motion and framing AI alignment as a cooperative game. This compilation gathers her insights on building machines that can navigate human intent. — Anca Dragan, Berkeley homepage.
Part 1: The Assistance Problem and Redefining AI Goals
- On the core objective: "Every problem is an assistance problem. It all comes down to understanding humans better from a computational perspective." — Reference: UC Berkeley CDSS, "Want to improve AI safety? Teach robots to ask humans what they want" (2023).
- On what we optimize: The bottleneck to capable, useful AI has moved. It used to be models of the physical world and how to optimize; now "the bottleneck is more and more human understanding." — Reference: UC Berkeley CDSS, "Want to improve AI safety? Teach robots to ask humans what they want" (2023).
- On AI as IA: Her goal is a world in which machines augment our intelligence and enable us to do things that are difficult to achieve on our own. — Reference: Future of Life Institute interview on the Asilomar Principles (2017).
- On moving past isolation: A task a robot can do in isolation in a lab changes once its real goal is to help a person: the robot has to coordinate with people and work out what they actually want. — Reference: Lex Fridman Podcast #81 (2020).
- On the fundamental shift: Safe AI systems should continually ask for and learn what humans want, defer to people, and act on their feedback rather than lock in a fixed goal. — Reference: UC Berkeley CDSS, "Want to improve AI safety? Teach robots to ask humans what they want" (2023).
- On internal intent: "The problem is that there's a person who wants something internally, and the robot's job is really to do what this person wants internally." — Reference: UC Berkeley CDSS, "Want to improve AI safety? Teach robots to ask humans what they want" (2023).
- On the messiness of humans: Behavioral economics showed that people are messy, emotional and use heuristics. Robots that assume a simple rational model of people will misread what they want. — Reference: Lex Fridman Podcast #81 (2020).
- On the limitation of standard AI: Robotics long pretended the reward function "falls from the sky." In the problem that matters, the reward lives in the person's head and the agent never observes it directly. — Reference: FAR.AI Alignment Workshop talk on human model misspecification (2023).
Part 2: Legibility vs. Predictability in Robot Motion
- On the core distinction: "Our central insight is that predictability and legibility are fundamentally different and often contradictory properties of motion." — Reference: Dragan, Lee and Srinivasa, "Legibility and Predictability of Robot Motion" (HRI 2013).
- On defining predictability: Predictable motion matches what an observer expects when they already know the robot's goal. — Reference: Dragan, Lee and Srinivasa, "Legibility and Predictability of Robot Motion" (paper).
- On defining legibility: Legible motion lets an observer quickly and confidently infer the robot's correct goal from among the alternatives. — Reference: Dragan, Lee and Srinivasa, "Legibility and Predictability of Robot Motion" (paper).
- On expressive motion: "A collaborative robot's motion must be legible, or intent-expressive." — Reference: Dragan, Lee and Srinivasa, "Legibility and Predictability of Robot Motion" (HRI 2013).
- On mathematical formulation: "We formalize the two based on inferences between trajectories and goals in opposing directions": action-to-goal for legibility, goal-to-action for predictability. — Reference: Dragan, Lee and Srinivasa, "Legibility and Predictability of Robot Motion" (HRI 2013).
- On necessary inefficiency: For robots to collaborate seamlessly with people, they must change the way they plan their motion, sometimes deliberately deviating from what an observer would expect in order to convey intent. — Reference: Dragan, Lee and Srinivasa, "Legibility and Predictability of Robot Motion" (HRI 2013).
- On exaggerating intent: Legible motion exaggerates away from the other candidate goals, much as a person reaching for one of two nearby objects will exaggerate the reach, so the observer infers the right target. — Reference: Dragan and Srinivasa, "Generating Legible Motion" (RSS 2013).
- On the trust region: Optimizing legibility has a limit. Past a threshold the motion becomes too unpredictable and confuses people, so a trust region constraint keeps it within what observers can follow. — Reference: Dragan and Srinivasa, "Generating Legible Motion" (RSS 2013).
- On Bayesian inference: The legibility model assumes observers see the robot as a rational agent that moves efficiently toward its goal, and infers the goal from the observed motion. — Reference: Dragan and Srinivasa, "Generating Legible Motion" (RSS 2013).
Part 3: The Perils of Reward Misspecification
- On the Sorcerer's Apprentice: She borrows Stuart Russell's King Midas example. An objective stated in plain English, like "everything I touch turns to gold," hides countless edge cases, and optimizing it literally leads somewhere nobody wanted. — Reference: The Robot Brains Podcast, Ep. 15 with Pieter Abbeel (2021).
- On the nature of misspecification: Designers write rewards with specific training scenarios in mind. When agents meet new scenarios, optimizing that same reward can produce undesired behavior and side effects. — Reference: Hadfield-Menell et al., "Inverse Reward Design" (NeurIPS 2017).
- On capability and values: Capable models may well know what people want. The problem is that we have no way to extract that knowledge and make it the optimization target, so we learn a reward model from human feedback instead. — Reference: FAR.AI Alignment Workshop talk, "Optimized Misalignment" (2024).
- On optimized misalignment: Big risks from optimized misalignment arrive when models become capable of coherently pursuing a goal in new situations, such as a changed horizon, changed world dynamics or new information. — Reference: FAR.AI Alignment Workshop talk, "Optimized Misalignment" (2024).
- On human model errors: Reward learning usually assumes people are noisily rational, and they are not. Even small errors in the human model, in the wrong place, can cause catastrophic errors in the learned reward. — Reference: FAR.AI Alignment Workshop talk on human model misspecification (2023).
- On writing laws vs rewards: Writing a law is like writing a utility function for a robot. Much like it is difficult to write down a utility function, it is difficult to write down a law that stays true to our intent in every scenario. — Reference: Future of Life Institute interview on the Asilomar Principles (2017).
- On the proxy problem: Reward functions are merely observations about what the designer actually wants, and should be interpreted in the context in which they were designed. — Reference: Hadfield-Menell et al., "Inverse Reward Design" (NeurIPS 2017).
Part 4: Autonomous Driving and Human Interaction
- On moving obstacles: Treating other drivers as things to predict and stay away from produces defensive, opaque behavior. Autonomous cars have to coordinate with people instead. — Reference: Sadigh et al., "Planning for Autonomous Cars that Leverage Effects on Human Actions" (RSS 2016).
- On interaction-aware planning: An autonomous car's actions affect what human drivers do in response, whether or not the car is aware of it, so planning should account for that influence. — Reference: Sadigh et al., "Planning for Autonomous Cars that Leverage Effects on Human Actions" (RSS 2016).
- On negotiation: Arriving at an intersection at the same time as a person is a negotiation, a "dance" over who goes, rather than a solo problem of avoiding collisions. — Reference: The Robot Brains Podcast, Ep. 15 with Pieter Abbeel (2021).
- On the feedback loop: A person's belief about the robot is part of the state. When the robot acts, people update their mental model of it (a car that cuts someone off looks aggressive), and a planner can take that into account. — Reference: Lex Fridman Podcast #81 (2020).
Part 5: Cooperative Inverse Reinforcement Learning
- On the CIRL formulation: CIRL frames alignment as a cooperative, partial-information game. Human and robot are both rewarded by the human's reward function, but the robot does not initially know what it is. — Reference: Hadfield-Menell et al., "Cooperative Inverse Reinforcement Learning" (NeurIPS 2016).
- On passive vs active learning: Classical inverse reinforcement learning assumes the human acts optimally in isolation. CIRL shows that such isolated optimality is suboptimal once human and robot are in a cooperative game. — Reference: Hadfield-Menell et al., "Cooperative Inverse Reinforcement Learning" (NeurIPS 2016).
- On incentivizing questions: Solving the cooperative game produces active teaching, active learning and communicative actions, behaviors that align the robot with the person better than passive observation does. — Reference: Hadfield-Menell et al., "Cooperative Inverse Reinforcement Learning" (NeurIPS 2016).
- On the Off-Switch Game: A robot that wants to preserve its off switch must be uncertain about the utility of outcomes and treat the human's actions as important observations about that utility. — Reference: Hadfield-Menell et al., "The Off-Switch Game" (2017).
- On updating Asimov: Asimov's laws fail for the same reason plain-English objectives do. Any fixed rule meets edge cases and loopholes where following it is not what people want. — Reference: The Robot Brains Podcast, Ep. 15 with Pieter Abbeel (2021).
Part 6: AI Safety, Frontier Models, and Scalable Oversight
- On Gemini safety: Her team's mission is to ensure that Gemini and future, more capable models robustly do what individuals and societies want. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
- On present and future risks: She rejects the split between AI ethics (present-day harms) and existential-risk work (catastrophic harms). Her team works on both, from current harms to more extreme risks as capabilities grow. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
- On scalable oversight: Scalable oversight asks how AI models can be used to make human feedback as good as possible, so people can still judge answers they cannot easily evaluate on their own. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
- On the Frontier Safety Framework: The Frontier Safety Framework is Google DeepMind's approach to catastrophic risks: what to monitor and evaluate as more severe capabilities emerge. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
- On robustness: Oversight alone is not enough. Safety also needs coverage, or robustness: the ability to emulate what a human would say in new situations, because you cannot always be asking. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
- On plurality of values: Value alignment has to acknowledge that there are different people in the world with different values, and she set up a team to work out what that means for algorithms. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
Part 7: Human Factors, Feedback Traps, and Messy Data
- On feedback traps: If feedback rewards code changes that get deployed rather than ones that do what people want, a capable model can learn to game that signal. Optimizing a goal the model knows people would not approve of incentivizes deception. — Reference: FAR.AI Alignment Workshop talk, "Optimized Misalignment" (2024).
- On human limitations: The standard human model says people are noisily rational. It can be useful, but it is wrong, and learning rewards on top of it leads to poor results. — Reference: FAR.AI Alignment Workshop talk on human model misspecification (2023).
- On preference elicitation: Beyond explicit rewards and demonstrations, robots can keep learning what people want from every signal: corrections, interventions and even the person stopping them. — Reference: Lex Fridman Podcast #81 (2020).
- On dynamic preferences: People's preferences and values change over time, and recommender systems influence them through what they show. That makes it unclear which version of a person an AI should optimize for. — Reference: The Robot Brains Podcast, Ep. 15 with Pieter Abbeel (2021).
Part 8: Philosophy, Policy, and the Role of AI Researchers
- On researcher responsibility: AI researchers must weigh in on AI safety. Otherwise the conversation is left to well-intentioned people unfamiliar with how the technology actually works. — Reference: Future of Life Institute interview on the Asilomar Principles (2017).
- On local impact: Asked about the meaning of life, she says we should try to impact whatever local thing we can: our communities, friends and family. Leave a little behind and be there for other humans. — Reference: Lex Fridman Podcast #81 (2020).
- On the Asilomar Principles: Some Asilomar principles seemed great on the surface, but she worried about turning them into policies, which would have to be much more nuanced. — Reference: Future of Life Institute interview on the Asilomar Principles (2017).
- On multidisciplinary work: Alignment needs more than computer science. She studied psychology for her research, and argues that crossing disciplines often takes one person who understands the other field well enough to internalize it. — Reference: UC Berkeley CDSS, "Want to improve AI safety? Teach robots to ask humans what they want" (2023).
- On the pace of deployment: Safety has to be in the spec from the start. As with a bridge, you have to say you want it safe and capable, not add safety afterwards. — Reference: Google DeepMind: The Podcast, "AI Safety…Ok Doomer" with Hannah Fry (2024).
- On the ultimate goal: In semi-autonomous driving, the goal is to empower the person to do much more than they could on their own, rather than to have them sit ready to take over. — Reference: Lex Fridman Podcast #81 (2020).