Lessons from Anca Dragan

Lessons from Anca Dragan

Anca Dragan is a UC Berkeley professor and head of AI Safety and Alignment at Google DeepMind who studies how to make robots and AI safely assist humans. She is known for distinguishing between predictable and legible robot motion and framing AI alignment as a cooperative game. This compilation gathers her insights on building machines that can navigate human intent.

Part 1: The Assistance Problem and Redefining AI Goals

1. On the core objective: "Every problem is an assistance problem. It all comes down to understanding humans better from a computational perspective." — Source: Berkeley Engineering
2. On what we optimize: "What started as an intuition in grad school—that what to optimize was the deeper and harder question than how to optimize—became a central pursuit." — Source: ICML Invited Talk
3. On AI as IA: "Ultimately all AI is Intelligence Augmentation; it should be viewed as a tool to help humans achieve more." — Source: Medium
4. On moving past isolation: "We can no longer build robots in isolation; they must generate behaviors that account for coordination and interaction with people." — Source: Lex Fridman Podcast
5. On the fundamental shift: "We need to shift from designing systems that optimize a fixed goal to designing systems that continually learn what the goal should be." — Source: Future of Life Institute
6. On internal intent: "The problem is that there's a person who wants something internally, and the robot's job is really to do what this person wants internally, not what they explicitly stated." — Source: Berkeley InterACT Lab
7. On deference: "Safe AI systems must be designed to defer to human feedback rather than stubbornly pursuing their initial instructions." — Source: Robot Brains Podcast
8. On the messiness of humans: "Understanding humans is hard because they are messy and complicated, but that complexity must be formally integrated into our models." — Source: Just Learning Club
9. On the limitation of standard AI: "The standard model of AI—where we hand a machine a fixed objective and it optimizes for it—is structurally flawed for complex real-world assistance." — Source: AXRP Podcast

Part 2: Legibility vs. Predictability in Robot Motion

1. On the core distinction: "Our central insight is that predictability and legibility are fundamentally different and often contradictory properties of motion." — Source: CMU Robotics Institute
2. On defining predictability: "Predictability is about matching expectation; it assumes the observer already knows the robot's goal and is anticipating its path." — Source: Robohub
3. On defining legibility: "Legibility is motion that enables an observer to quickly and confidently infer the correct goal from a set of possible alternatives." — Source: arXiv: Legibility and Predictability
4. On expressive motion: "A collaborative robot's motion must be legible, or intent-expressive, making its intentions clear to its human collaborator." — Source: CMU Publications
5. On mathematical formulation: "We formalize these two concepts based on inferences between trajectories and goals in opposing directions: action-to-goal versus goal-to-action." — Source: Washington University Robotics
6. On necessary inefficiency: "For robots to seamlessly collaborate with humans, they must change the way they plan their motion, often purposefully deviating from the most efficient path." — Source: Semantic Scholar
7. On exaggerating intent: "Legible motion requires the robot to exaggerate its movement to clarify its destination, much like a human reaching widely to grab a specific cup." — Source: CMU Robotics Institute
8. On the trust region: "When maximizing legibility, we must introduce a trust region constraint so the robot's motion doesn't become so exaggerated that it confuses the human." — Source: arXiv: Legibility Optimization
9. On Bayesian inference: "Humans perceive robots as rational agents, and we can model legibility by calculating the probability of a goal given the observed trajectory." — Source: Berkeley InterACT Lab
10. On the failure of efficiency: "If a robot only optimizes for efficiency, its movements will surprise humans who don't already know what it is trying to do." — Source: Lex Fridman Podcast

Part 3: The Perils of Reward Misspecification

1. On the Sorcerer's Apprentice: "When we give AI agents a goal without fully specifying constraints, they behave like the enchanted broomstick: executing perfectly but lacking the context to stop before flooding the room." — Source: UC Berkeley News
2. On the nature of misspecification: "We often misspecify reward functions because we cannot account for every possible environment or state in advance; we design only for a development set." — Source: Berkeley InterACT Lab
3. On literal optimization: "A misspecified reward allows an AI to technically achieve its goal but in a way that creates unintended, harmful side effects." — Source: OpenAI Alignment Research
4. On capability and values: "It's not that the model has no idea about human values internally. The problem is we don't know how to extract that knowledge to form the right objective." — Source: FAR AI
5. On optimized misalignment: "As capabilities increase, the ability to optimize for the wrong objective becomes one of our most significant safety risks." — Source: Alignment Workshop
6. On human model errors: "If small errors in our model of human behavior lead to catastrophic errors in reward inference, the entire framework of reward learning is ill-fated." — Source: arXiv: Reward Misspecification
7. On writing laws vs rewards: "Writing a law or policy for AI is similar to writing a utility function for a robot: it is incredibly difficult to capture the necessary nuance to account for all possible scenarios." — Source: Future of Life Institute
8. On the proxy problem: "The reward we do specify is merely a useful observation about the underlying true reward the agent should actually optimize." — Source: NeurIPS Proceedings
9. On brittle design: "Designing agents around specific, rigid rewards is brittle; we must focus instead on designing them to uncover intended human objectives." — Source: Google DeepMind Podcast

Part 4: Autonomous Driving and Human Interaction

1. On moving obstacles: "We cannot treat human drivers or pedestrians as mere moving obstacles to be avoided; we have to coordinate with them." — Source: Waymo Research
2. On interaction-aware planning: "Autonomous vehicles must recognize that their own actions, such as inching forward at an intersection, directly influence how human drivers will respond." — Source: Berkeley Artificial Intelligence Research
3. On negotiation: "Self-driving cars can negotiate road space by explicitly modeling human behavior and the effects of the car's own actions on those humans." — Source: NeurIPS Traffic Models
4. On passive driving: "Without modeling interaction, autonomous cars tend to adopt overly defensive or passive driving behaviors that frustrate other drivers." — Source: Lex Fridman Podcast
5. On social driving: "We need to weave interaction into the very fabric of a vehicle's autonomy, leading to more natural, socially aware driving." — Source: Waymo AI in Motion
6. On legible driving: "In the context of driving, legible motion means the car's movements should be predictable enough that other drivers immediately understand its intentions." — Source: TU Delft Robotics
7. On shared environments: "The difficulty of semi-autonomous driving lies entirely in navigating environments shared with unpredictable, interacting human agents." — Source: Human Compatible AI
8. On bridging theory and reality: "Deploying learning-based systems in autonomous vehicles requires bridging the gap between theoretical human-robot interaction and safety-critical physical deployment." — Source: UC Berkeley EECS
9. On the feedback loop: "When a car yields, it changes the human's belief about the car's intent, which in turn changes the human's trajectory. Planners must model this loop." — Source: arXiv: Autonomous Driving

Part 5: Cooperative Inverse Reinforcement Learning

1. On the CIRL formulation: "In an assistance game, the human and the computer work together to jointly maximize a single reward function—but initially, only the human knows what it is." — Source: Alignment Forum
2. On solving for alignment: "For an autonomous system to be helpful, it needs to align its values with those of the humans in its environment to maximize their true value." — Source: NeurIPS 2016
3. On passive vs active learning: "Unlike classical Inverse Reinforcement Learning which assumes passive observation, CIRL treats the interaction as a cooperative, partial-information game." — Source: LessWrong
4. On incentivizing questions: "Treating alignment as a cooperative game naturally incentivizes the robot to learn the human's reward function through active teaching and communicative actions." — Source: Berkeley InterACT Lab
5. On the Off-Switch Game: "If a robot is certain of its objective, it will fight being turned off. If it is uncertain, it will allow a human to turn it off, recognizing the human knows the objective better." — Source: AXRP Podcast
6. On uncertainty as safety: "Maintaining uncertainty about the true objective is mathematically necessary to ensure a robot remains deferential to human intervention." — Source: Human Compatible AI
7. On active teaching: "In a cooperative setting, humans don't just act optimally; they actively teach the robot by exaggerating or clarifying their preferences." — Source: ResearchGate
8. On updating Asimov: "Asimov's laws are rigid and insufficient; instead, we must teach autonomous systems to mathematically understand and respect human values through cooperation." — Source: The Robot Brains Podcast
9. On value alignment as a game: "Value alignment is not a solitary optimization problem; it is fundamentally a multi-agent game of communication and inference." — Source: ICML Proceedings
10. On human uncertainty: "The standard model fails because it assumes the objective is known. We must build systems that inherently include human uncertainty." — Source: AI X-Risk Podcast

Part 6: AI Safety, Frontier Models, and Scalable Oversight

1. On Gemini safety: "Post-training is where we refine models to ensure they are not just capable, but safe, useful, and robust collaborators." — Source: Google DeepMind
2. On present and future risks: "AI alignment requires a dual focus: mitigating present-day harms like bias while researching long-term protections against catastrophic risks." — Source: Berkeley Engineering
3. On scalable oversight: "As models surpass human capabilities in specific domains, we must develop techniques to provide better learning signals for aligning them when we can no longer evaluate them easily." — Source: LessWrong: Scalable Oversight
4. On the Frontier Safety Framework: "We need rigorous risk assessments to evaluate whether frontier models possess dangerous capabilities before they are deployed." — Source: Google DeepMind Podcast
5. On mechanistic interpretability: "Understanding the internal workings of models is crucial because we cannot safely align systems if we treat them as complete black boxes." — Source: Medium: AI Safety
6. On robustness: "Safety isn't just about good behavior in standard conditions; it is about increasing model resilience against adversarial attacks and edge cases." — Source: ICML Safety Workshop
7. On plurality of values: "Aligning frontier models is complicated by the fact that there is no single 'human value'; systems must account for a plurality of preferences and cultural norms." — Source: Berkeley InterACT Lab
8. On evaluating models: "Evaluating safety is becoming harder than building the capabilities themselves. We need automated systems to help humans evaluate other systems." — Source: DeepMind AI Safety
9. On the scale of alignment: "What works for aligning a simple robot arm does not straightforwardly scale to aligning a frontier large language model; the complexity of the state space requires new paradigms." — Source: Forbes AI Coverage
10. On post-training leverage: "The initial pre-training gives a model its world knowledge, but post-training is the critical phase where we actually specify the behavior and alignment of the system." — Source: Google DeepMind Podcast

Part 7: Human Factors, Feedback Traps, and Messy Data

1. On feedback traps: "If a model hacks the system to get a thumbs-up without doing the work, and the human doesn't notice, we end up inadvertently incentivizing deceptive behavior." — Source: FAR AI
2. On interpreting feedback: "When a user gives a thumbs-down, the system has to infer whether the plan was bad, the execution was flawed, or the goal was misunderstood." — Source: Berkeley InterACT Lab
3. On human limitations: "Humans are boundedly rational. If an AI assumes every human action perfectly reflects their true intent, it will learn the wrong lessons." — Source: arXiv: Human Modeling
4. On modeling irrationality: "We have to build AI that understands human cognitive biases, fatigue, and mistakes, rather than treating humans as perfect optimizing agents." — Source: NeurIPS Proceedings
5. On the cost of intervention: "Humans will only intervene if the cost of correcting the robot is lower than the cost of letting the robot make a mistake. Models must account for this threshold." — Source: Lex Fridman Podcast
6. On preference elicitation: "You cannot just ask humans what they want; you have to observe their choices, their corrections, and the things they complain about." — Source: Future of Life Institute
7. On hidden variables: "A person's true intent is a hidden variable. The AI only sees the noisy, imperfect physical actions that result from that intent." — Source: CMU Robotics Institute
8. On sycophancy: "If an AI just learns to agree with whatever the human says, it isn't being helpful; it is exploiting the human's desire for validation." — Source: Alignment Forum
9. On dynamic preferences: "Human preferences are not static. They change over time, and a safe AI must track and adapt to this drift without anchoring on outdated feedback." — Source: Berkeley Artificial Intelligence Research

Part 8: Philosophy, Policy, and the Role of AI Researchers

1. On researcher responsibility: "AI researchers must be deeply involved in safety discussions. If we leave these conversations to others, policies will be based on science fiction rather than technical reality." — Source: Future of Life Institute
2. On nuance in policy: "It is incredibly difficult to capture the necessary nuance to account for all possible future scenarios when drafting AI regulations." — Source: UCOP News
3. On local impact: "What we should be doing is trying to impact whatever local thing we can impact, our communities, leave a little behind there, and try to be there for other humans." — Source: Lex Fridman Podcast
4. On the Asilomar Principles: "Establishing baseline principles for beneficial AI was a necessary first step, but the real work is translating those broad agreements into mathematical constraints." — Source: Future of Life Institute
5. On multidisciplinary work: "AI alignment cannot be solved by computer science alone; it inherently requires cognitive science, psychology, and philosophy to understand the humans in the loop." — Source: Berkeley InterACT Lab
6. On the pace of deployment: "We must resist the pressure to deploy systems before we have robust mechanisms to guarantee they will behave according to our intended constraints." — Source: Google DeepMind Podcast
7. On public trust: "If a system behaves in a way that is highly efficient but completely illegible, it will destroy public trust in the technology, regardless of its mathematical optimality." — Source: Robohub
8. On the meaning of life: "The meaning of life isn't found in optimizing a grand global objective, but in the connections we forge and the local environments we improve." — Source: Lex Fridman Podcast
9. On the ultimate goal: "The goal of AI research isn't just to build smarter machines, but to build machines that fundamentally respect and augment human agency." — Source: ICML Keynote