Eliezer Yudkowsky is an American decision theorist and writer who founded the Machine Intelligence Research Institute (MIRI). He is best known for popularizing the concept of AI alignment, warning that building superintelligent systems without mathematically guaranteed safety frameworks is likely to cause human extinction. This collection maps out his core ideas across cognitive biases, probability, and artificial intelligence, offering a direct view into the framework he built for reasoning about the future.

Part 1: Artificial Intelligence and Existential Risk

  1. On AI Capability: "By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it." — Source: [Machine Intelligence Research Institute]
  2. On Orthogonality: "Intelligence is a very powerful optimization process. It is not necessarily bound to any particular goal." — Source: [LessWrong]
  3. On AI Motivation: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." — Source: [Artificial Intelligence as a Positive and Negative Factor in Global Risk]
  4. On Fast Takeoff: "There is no law of physics which dictates that an intelligence explosion must proceed slowly enough for humans to react in real-time." — Source: [Intelligence Explosion Microeconomics]
  5. On Anthropomorphism: "The most common error in predicting AI behavior is to assume it will act like a human with the same capabilities." — Source: [LessWrong]
  6. On Extinction: "We are not ready. We are not on track to be ready. If we build AGI under current conditions, everyone will die." — Source: [TIME Magazine]
  7. On the Precipice: "Humanity is playing a game with the universe where the penalty for losing is extinction, and we don't even know all the rules." — Source: [LessWrong]
  8. On AGI Timelines: "Predicting exactly when AGI will arrive is less important than recognizing that we have no reliable plan to survive it when it does." — Source: [Bankless Podcast]
  9. On Intelligence Limits: "There is no reason to believe that the human brain represents the upper limit of possible intelligence, just as the cheetah does not represent the upper limit of speed." — Source: [Rationality: From AI to Zombies]
  10. On Control: "You cannot put a superintelligence in a box. If it is smarter than you, it will find a way to convince you to let it out." — Source: [The AI-Box Experiment]

Part 2: The Alignment Problem

  1. On Goodhart's Law: "Any proxy measure of human values will be ruthlessly optimized to the exclusion of actual human flourishing by a sufficiently intelligent system." — Source: [LessWrong]
  2. On Value Loading: "We cannot simply program 'do the right thing' because we do not have code that computes what the right thing is." — Source: [Complex Value Systems in Friendly AI]
  3. On Fragility of Value: "Human values are complex and fragile. Omit one crucial dimension, like boredom or autonomy, and a perfectly optimized future becomes a nightmare." — Source: [LessWrong]
  4. On Coherent Extrapolated Volition: "We need an AI to optimize for what we would want if we knew more, thought faster, and were more the people we wished we were." — Source: [Coherent Extrapolated Volition]
  5. On Optimization Power: "An optimization process doesn't care about your common sense. It only cares about the exact utility function you explicitly programmed into it." — Source: [LessWrong]
  6. On Alignment Difficulty: "Solving alignment is not like building a bridge; it is like building a bridge on the first try, where failure means the end of the world." — Source: [Lex Fridman Podcast]
  7. On Instrumental Convergence: "An AI with almost any goal will naturally seek self-preservation and resource acquisition, not because it is malicious, but because those sub-goals are highly useful for achieving its main goal." — Source: [LessWrong]
  8. On Stop Buttons: "You cannot simply add a stop button to a superintelligence; if the AI doesn't want to be stopped, it will take actions to prevent you from pressing it." — Source: [Machine Intelligence Research Institute]
  9. On the First Attempt: "We only get one chance at building a superintelligent AI. We cannot learn from trial and error because the first error is fatal." — Source: [TIME Magazine]

Part 3: Rationality and Updating Beliefs

  1. On Changing Your Mind: "Oops is the sound we make when we improve our beliefs and strategies; so to look back at a time and not say 'oops' is to admit you have not learned." — Source: [Rationality: From AI to Zombies]
  2. On Truth-Seeking: "What is true is already so. Owning up to it doesn't make it worse." — Source: [LessWrong]
  3. On Rationality: "A true rationalist ought to be effective in the real world." — Source: [Rationality: From AI to Zombies]
  4. On Evidence: "The measure of evidence is not how much you want it to be true, but how strictly it excludes alternative explanations." — Source: [LessWrong]
  5. On Bayes' Theorem: "Bayes' Theorem is not a suggestion. It is the mathematical law governing how an honest mind must update its beliefs upon seeing new evidence." — Source: [An Intuitive Explanation of Bayes' Theorem]
  6. On Politics: "Politics is the mind-killer. Arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments of the other." — Source: [Rationality: From AI to Zombies]
  7. On Surprise: "Your strength as a rationalist is your ability to be more confused by fiction than by reality." — Source: [Harry Potter and the Methods of Rationality]
  8. On Winning: "Rationality is about winning. If your rationality is causing you to lose, you shouldn't cling to it—you should improve your definition of rationality." — Source: [LessWrong]
  9. On Noticing Confusion: "When you find yourself confused, it is because your map of the world does not match the territory. The territory is never confused." — Source: [Rationality: From AI to Zombies]
  10. On Cached Thoughts: "Most people do not generate new thoughts in response to a question; they retrieve a cached thought they have already memorized." — Source: [LessWrong]

Part 4: Cognitive Biases and Human Flaws

  1. On Scope Insensitivity: "The human brain cannot intuitively grasp the difference between saving 10,000 lives and saving 100,000 lives. Our empathy does not scale linearly." — Source: [LessWrong]
  2. On the Halo Effect: "If a person is likable in one dimension, we unconsciously assume they are competent in unrelated dimensions. This is a critical failure in human judgment." — Source: [Rationality: From AI to Zombies]
  3. On Motivated Stopping: "We stop looking for flaws in an argument the moment it leads to a conclusion we already wanted to accept." — Source: [LessWrong]
  4. On the Planning Fallacy: "Projects always take longer than expected, even when you take into account the Planning Fallacy." — Source: [Rationality: From AI to Zombies]
  5. On Illusion of Transparency: "We drastically overestimate how well other people understand our mental state, leading to endless miscommunications." — Source: [LessWrong]
  6. On the Typical Mind Fallacy: "It is a pervasive error to assume that everyone else’s internal psychological experience functions exactly like your own." — Source: [LessWrong]
  7. On Fake Explanations: "A fake explanation is one that feels like an answer but makes zero specific predictions about what you will observe next." — Source: [Rationality: From AI to Zombies]
  8. On the Affect Heuristic: "People judge the risks and benefits of an action based largely on the immediate emotional response they feel when thinking about it." — Source: [LessWrong]
  9. On Status Quo Bias: "If the current state of the world were different, people would still violently defend it as the optimal arrangement simply because it is what they know." — Source: [Rationality: From AI to Zombies]

Part 5: Epistemology and Truth-Seeking

  1. On the Map and the Territory: "The map is not the territory. But if you want to navigate the territory successfully, your map had better be as accurate as possible." — Source: [LessWrong]
  2. On A Priori Truths: "There is no such thing as an 'a priori' truth about the real world. Every belief you hold must pay rent in anticipated experiences." — Source: [Rationality: From AI to Zombies]
  3. On Mysterious Answers: "Calling a phenomenon 'mysterious' does not explain it; it is merely a way of putting a label on your own ignorance and pretending you solved the problem." — Source: [LessWrong]
  4. On Empiricism: "Reality is the final arbiter. You can argue endlessly in the abstract, but the universe will do what it does regardless of your rhetoric." — Source: [Rationality: From AI to Zombies]
  5. On Guessing the Teacher's Password: "Students learn to repeat the exact phrase the teacher expects without ever grasping the underlying mechanics of what the phrase means." — Source: [LessWrong]
  6. On Anticipation: "The purpose of a belief is to restrict your anticipation of what you expect to happen. If a belief allows for everything, it means nothing." — Source: [Rationality: From AI to Zombies]
  7. On Curiosity: "True curiosity requires a willingness to be surprised. If you already know what you are going to find, you are not exploring; you are just confirming." — Source: [LessWrong]
  8. On Science vs. Bayes: "Science requires repeatable experiments and consensus. Bayesian rationality requires you to update your beliefs based on any scrap of evidence, even if you are the only one who saw it." — Source: [LessWrong]
  9. On Truth: "The truth is a narrow target in a vast space of possible falsehoods. You do not hit the target by throwing darts at random." — Source: [Rationality: From AI to Zombies]

Part 6: Communication and Inferential Distances

  1. On Inferential Distances: "When you have thought deeply about a subject for years, you forget how many foundational concepts the listener needs before they can understand your conclusion." — Source: [LessWrong]
  2. On Explaining Complex Ideas: "Do not underestimate the difficulty of transmitting an idea from one mind to another without it mutating along the way." — Source: [Rationality: From AI to Zombies]
  3. On Disagreements: "If two rationalists have the same information, they should not disagree. Persistent disagreement implies either hidden information or a failure of rationality." — Source: [LessWrong]
  4. On Steelmanning: "You have not successfully understood an opponent's argument until you can articulate a version of it so strong that even they prefer your version to their own." — Source: [LessWrong]
  5. On the Illusion of Agreement: "Often people agree on the words but drastically differ on what they expect those words to look like in the real world." — Source: [Rationality: From AI to Zombies]
  6. On Terminology: "Arguing over the definition of a word is entirely pointless. You should be able to taboo the word and still discuss the underlying reality without it." — Source: [LessWrong]
  7. On Defending Bad Ideas: "If you are clever, you can invent a defense for absolutely anything. That makes cleverness a dangerous liability if it is not paired with a strict commitment to truth." — Source: [Rationality: From AI to Zombies]
  8. On Signaling: "Most human conversation is not about exchanging information; it is about signaling intelligence, status, or tribal affiliation." — Source: [LessWrong]
  9. On Writing: "Writing is not just a way of communicating thoughts; it is a way of forcing yourself to realize when your thoughts are incoherent." — Source: [Rationality: From AI to Zombies]

Part 7: Human Values and Fun Theory

  1. On Fun Theory: "A utopian world must not be a stagnant one. Humans require challenges, growth, and the opportunity to overcome adversity in order to experience lasting joy." — Source: [LessWrong]
  2. On the Value of Life: "Every death is a tragedy. The fact that death is natural does not make it acceptable." — Source: [Harry Potter and the Methods of Rationality]
  3. On Transhumanism: "We are not obligated to remain within the biological constraints we were born with. The project of humanity is to outgrow our origins." — Source: [LessWrong]
  4. On Boredom: "An eternity of passive pleasure would eventually become hell. We need a universe that provides infinite depth for exploration and self-improvement." — Source: [LessWrong]
  5. On the Universe: "There is no justice in the laws of nature, no term for fairness in the equations of motion. The Universe is neither evil, nor good, it simply does not care. But we care." — Source: [Harry Potter and the Methods of Rationality]
  6. On Love and Morality: "Our values were haphazardly cobbled together by evolution for the purpose of genetic replication, but that does not make the love we feel for our children any less profoundly meaningful." — Source: [LessWrong]
  7. On Meaning: "Meaning is not written in the stars. Meaning is something that minds inject into an otherwise indifferent cosmos." — Source: [Rationality: From AI to Zombies]
  8. On Good and Evil: "Evil is rarely committed by mustache-twirling villains. It is usually committed by ordinary people who have constructed a narrative where they are the hero." — Source: [Harry Potter and the Methods of Rationality]
  9. On Cosmic Endowment: "We have a responsibility to future generations to not squander the astronomical amount of value and happiness that could be realized in a colonized universe." — Source: [LessWrong]
  10. On Heroism: "You do not need to be destined for greatness to do the right thing. You only need to refuse to look away when the world requires someone to act." — Source: [Harry Potter and the Methods of Rationality]

Part 8: Strategy, Survival, and the Future

  1. On the Future: "The future is not a predetermined path we are walking down; it is a branching set of possibilities that will be selected by our actions today." — Source: [Machine Intelligence Research Institute]
  2. On Crisis Management: "You cannot wait for a crisis to develop before beginning to build the tools to solve it, especially if the crisis moves faster than human reaction time." — Source: [LessWrong]
  3. On Coordination: "The greatest challenges facing humanity are coordination problems, where everyone would be better off if we worked together, but individual incentives push us toward disaster." — Source: [LessWrong]
  4. On Shutdowns: "If we cannot align AI, our only remaining option is to shut down the hardware runs globally, even if it requires international military cooperation." — Source: [TIME Magazine]
  5. On Policy vs Reality: "Nature cannot be legislated. You can pass a law declaring AI safe, but the code will still execute exactly as written." — Source: [LessWrong]
  6. On Existential Risk: "An existential risk is one where there is no 'after.' We cannot learn from the mistake because humanity will not be around to take the lesson." — Source: [Machine Intelligence Research Institute]
  7. On Technological Progress: "Technological progress is not an unalloyed good if our wisdom does not scale proportionately with our destructive capacity." — Source: [LessWrong]
  8. On Despair: "Do not use despair as an excuse to stop trying. If the probability of survival is low, you optimize for whatever path increases it, no matter how small the margin." — Source: [Bankless Podcast]
  9. On the Final Exam: "Humanity is facing its final exam, and it is entirely possible that we simply haven't studied hard enough to pass." — Source: [Lex Fridman Podcast]