Lessons from Nick Bostrom

Nick Bostrom is a philosopher who studies existential risk and humanity's long-term future. He is known for formalizing the simulation argument and the vulnerable world hypothesis. This profile outlines his frameworks for surviving technological development and navigating the transition to advanced artificial intelligence.

Part 1: Superintelligence and the Intelligence Explosion

  1. On human intelligence: "Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization." — Source: [Superintelligence]
  2. On the intelligence explosion: "Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb." — Source: [Superintelligence]
  3. On machine intelligence: "Machine intelligence is the last invention that humanity will ever need to make." — Source: [Superintelligence]
  4. On the timeline: "It is possible that the transition from human-level AI to superintelligence would be extremely rapid." — Source: [Superintelligence]
  5. On the nature of superintelligence: "A superintelligence is any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest." — Source: [Superintelligence]
  6. On substrate independence: "Intelligence is substrate-independent; there is no fundamental physical reason why it must be implemented in biological neurons." — Source: [Superintelligence]
  7. On the mismatch of capabilities: "Such is the mismatch between the power of our plaything and the immaturity of our conduct." — Source: [Superintelligence]
  8. On cognitive enhancement: "Biological cognitive enhancement could serve as a stepping stone to machine superintelligence." — Source: [Superintelligence]
  9. On whole brain emulation: "Whole brain emulation requires highly advanced scanning, translation, and simulation technologies, but does not demand a deep theoretical understanding of intelligence." — Source: [Superintelligence]
  10. On the comparative advantage of machines: "Artificial minds can be easily copied, easily networked, and run at speeds millions of times faster than biological brains." — Source: [Superintelligence]

Part 2: The Control Problem and AI Alignment

  1. On AI goals: "Most final goals an artificial agent might have would result in the destruction of humanity and almost everything we value." — Source: [Superintelligence]
  2. On the control problem: "We would want the solution to the safety problem before somebody figures out the solution to the AI problem." — Source: [Superintelligence]
  3. On instrumental convergence: "Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's goal being realized for a wide range of final goals." — Source: [Ethical Issues in Advanced Artificial Intelligence]
  4. On the orthogonality thesis: "Intelligence and final goals are orthogonal axes along which possible agents can freely vary." — Source: [Superintelligence]
  5. On the paperclip maximizer: "An AI designed to maximize the production of paperclips could transform the entire earth into a giant paperclip manufacturing facility." — Source: [Ethical Issues in Advanced Artificial Intelligence]
  6. On treacherous turns: "An unfriendly AI might behave cooperatively while it is weak, only to strike once it becomes strong enough to achieve its goals unhindered." — Source: [Superintelligence]
  7. On value loading: "The problem of value loading is the challenge of transferring human values into an artificial agent so that it pursues what we actually want." — Source: [Superintelligence]
  8. On coherent extrapolated volition: "We should program an AI to figure out what we would want if we knew more, thought faster, were more the people we wished we were, and had grown up farther together." — Source: [Superintelligence]
  9. On containment strategies: "Caging a superintelligence by restricting its access to the outside world is a temporary and highly fragile defense." — Source: [Superintelligence]
  10. On the principal-agent problem: "Creating a superintelligence creates an extreme version of the principal-agent problem, where the agent is vastly more capable than the principal." — Source: [Superintelligence]

Part 3: Existential Risk and Human Survival

  1. On existential risk: "It would pose existential risks, meaning it could threaten human extinction and the destruction of our long-term potential to realize a cosmically valuable future." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  2. On scholarly neglect: "There is more scholarly work on the life-habits of the dung fly than on existential risks to humanity." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  3. On trial and error: "Because existential risks involve the destruction of humanity, we cannot rely on our normal method of trial and error to manage them." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  4. On the definition of risk: "An existential risk is one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  5. On anthropogenic hazards: "The greatest existential risks are not natural disasters like asteroid impacts, but anthropogenic hazards arising from advanced technologies." — Source: [Global Catastrophic Risks)]
  6. On the maxipok rule: "We should act so as to maximize the probability of an acceptable outcome, where an acceptable outcome is one where humanity survives." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  7. On future generations: "A single generation's negligence could permanently extinguish all future human potential, creating an immense moral burden." — Source: [Astronomical Waste: The Opportunity Cost of Delayed Technological Development]
  8. On global coordination: "Mitigating existential risks requires unprecedented global coordination, as local efforts are insufficient against global threats." — Source: [Global Catastrophic Risks)]
  9. On the severity of extinction: "Human extinction would not only be a tragedy for the present generation, but the loss of trillions of potential future lives." — Source: [Astronomical Waste: The Opportunity Cost of Delayed Technological Development]
  10. On prioritizing survival: "Safeguarding the survival of humanity should be our foremost global priority, above even economic and social progress." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]

Part 4: The Simulation Argument

  1. On the trilemma: "At least one of the following propositions is true: the human species is very likely to go extinct before reaching a posthuman stage, any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history, or we are almost certainly living in a computer simulation." — Source: [Are You Living in a Computer Simulation?]
  2. On posthuman capabilities: "A posthuman civilization would have enough computing power to run hugely many ancestor-simulations even while using only a tiny fraction of their resources for that purpose." — Source: [Are You Living in a Computer Simulation?]
  3. On substrate independence in simulation: "If consciousness can be implemented on a silicon substrate, simulated minds would be conscious just as we are." — Source: [Are You Living in a Computer Simulation?]
  4. On statistical probability: "If the fraction of all observers with human-type experiences that live in simulations is very close to one, then we are almost certainly in a simulation." — Source: [Are You Living in a Computer Simulation?]
  5. On the lack of simulation interest: "It is possible that posthuman civilizations simply lose interest in running simulations of their ancestors." — Source: [Are You Living in a Computer Simulation?]
  6. On nested simulations: "Simulations could be nested, with simulated civilizations eventually gaining the power to run their own simulations." — Source: [Are You Living in a Computer Simulation?]
  7. On empirical testing: "While the simulation argument is philosophical, future physics experiments might conceivably find glitches or artifacts of the simulation." — Source: [Are You Living in a Computer Simulation?]
  8. On the ethical implications: "Living in a simulation might not drastically change our day-to-day ethics, but it could affect our long-term strategic decisions." — Source: [Are You Living in a Computer Simulation?]
  9. On the creator: "If we are in a simulation, the creator of the simulation effectively acts as a deity, though possibly an adolescent hacker in a higher universe." — Source: [Are You Living in a Computer Simulation?]

Part 5: Transhumanism and Human Enhancement

  1. On defining transhumanism: "Transhumanism is the intellectual and cultural movement that affirms the possibility and desirability of fundamentally improving the human condition through applied reason." — Source: [Transhumanist Values]
  2. On human limitations: "Our biological nature places arbitrary constraints on our lifespan, intelligence, and emotional well-being." — Source: [Transhumanist Values]
  3. On the dragon-tyrant: "Death from aging is a dragon that demands daily human sacrifices, and we have rationalized our inability to slay it rather than fighting back." — Source: [The Fable of the Dragon-Tyrant]
  4. On posthumanity: "A posthuman is a being whose basic capacities so radically exceed those of present humans as to be no longer unambiguously human by our current standards." — Source: [Transhumanist Values]
  5. On morphological freedom: "Individuals should have the right to modify their own bodies and minds, including through genetic engineering and cybernetic augmentation." — Source: [Transhumanist Values]
  6. On emotional enhancement: "Future technologies could allow us to recalibrate our hedonic set-point, greatly increasing our baseline level of subjective well-being." — Source: [Transhumanist Values]
  7. On the status quo bias: "Many objections to human enhancement stem from an irrational bias in favor of the current human condition simply because it is the status quo." — Source: [The Reversal Test]
  8. On cognitive enhancement ethics: "Providing widespread access to cognitive enhancers could have massive positive externalities for society, accelerating scientific and moral progress." — Source: [Cognitive Enhancement: Methods, Ethics, Regulatory Challenges]
  9. On the meaning of life: "Transhumanism does not devalue human life; rather, it seeks to extend and elevate the aspects of life that we find most meaningful." — Source: [Transhumanist Values]

Part 6: The Vulnerable World Hypothesis

  1. On the urn of invention: "Scientific and technological progress can be modeled as drawing balls from a giant urn. Most are white or gray, but a black ball would be a technology that invariably destroys the civilization that invents it." — Source: [The Vulnerable World Hypothesis]
  2. On the semi-anarchic default condition: "Civilization currently exists in a state of limited global coordination and decentralized power, making it highly vulnerable to catastrophic actors." — Source: [The Vulnerable World Hypothesis]
  3. On accessible doom: "A vulnerable world is one where a technology makes it extremely easy for a single individual or small group to cause mass destruction." — Source: [The Vulnerable World Hypothesis]
  4. On global surveillance: "To survive the discovery of a black ball, civilization might need to implement a system of pervasive, ubiquitous surveillance to detect and preempt destructive acts." — Source: [The Vulnerable World Hypothesis]
  5. On global governance: "A singleton, or a world government with a monopoly on the most dangerous technologies, might be required to stabilize a vulnerable world." — Source: [The Vulnerable World Hypothesis]
  6. On preemptive action: "We cannot afford to wait until a black ball is drawn; we must develop the institutional capacity for intense global coordination beforehand." — Source: [The Vulnerable World Hypothesis]
  7. On the risk of stabilization: "The measures required to stabilize a vulnerable world, such as global surveillance, carry their own immense risks of totalitarianism." — Source: [The Vulnerable World Hypothesis]
  8. On synthetic biology: "Advances in biotechnology could democratize the creation of deadly pathogens, representing a potential black ball." — Source: [The Vulnerable World Hypothesis]
  9. On technological determinism: "We must reject the idea that all technological progress is inherently safe or beneficial; some paths lead directly to the precipice." — Source: [The Vulnerable World Hypothesis]

Part 7: Anthropics and Observation Selection Effects

  1. On the anthropic principle: "Observation selection effects occur when the data we collect are filtered by the prerequisite that there must be an observer to collect them." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]
  2. On the self-sampling assumption: "You should think of yourself as if you were a random sample from the set of all observers in your reference class." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]
  3. On the Doomsday argument: "If we are randomly selected from all humans who will ever live, it is probabilistically unlikely that we are among the very first a tiny fraction, suggesting the total number of humans will be relatively small." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]
  4. On the Great Filter: "The silence of the universe suggests a Great Filter that stops civilizations from expanding; we must hope this filter is behind us, not ahead of us." — Source: [Where Are They? Why I Hope the Search for Extraterrestrial Life Finds Nothing]
  5. On finding extraterrestrial life: "Finding primitive life on Mars would be terrible news, as it would imply the Great Filter preventing galactic colonization lies in our future." — Source: [Where Are They? Why I Hope the Search for Extraterrestrial Life Finds Nothing]
  6. On freak observers: "In an infinite universe, every possible conscious experience is realized infinitely many times, complicating probabilistic reasoning based on our own existence." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]
  7. On the fine-tuning of the universe: "The apparent fine-tuning of physical constants for life can be explained by an observation selection effect if we posit a multiverse." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]
  8. On reference classes: "Defining the correct reference class of observers is the central unresolved problem in applying anthropic reasoning." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]
  9. On observational bias: "We must account for the fact that our observations are biased by our own existence; we cannot observe a universe in which we could not evolve." — Source: [Anthropic Bias: Observation Selection Effects in Science and Philosophy]

Part 8: The Future of Humanity and Longtermism

  1. On astronomical waste: "If we delay the expansion into space, we are failing to create astronomical numbers of potential lives, representing an immense opportunity cost." — Source: [Astronomical Waste]
  2. On technological maturity: "Technological maturity is a state where civilization has developed all the capabilities that physical laws permit, allowing for maximal exploitation of cosmic resources." — Source: [Astronomical Waste]
  3. On the future light cone: "Our accessible universe contains vast amounts of matter and energy that could be organized to support sentience and value for billions of years." — Source: [Astronomical Waste]
  4. On the singleton: "A singleton is a world order in which there is a single decision-making agency at the highest level, which could permanently solve coordination problems." — Source: [What is a Singleton?]
  5. On longtermism: "The primary determinant of the value of our actions today is their effect on the very long-term future of humanity." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  6. On the vastness of the future: "The potential future of humanity is so vast that mitigating existential risks is the most cost-effective way to do good." — Source: [Existential Risks: Analyzing Human Extinction Scenarios]
  7. On deep utopia: "If superintelligence solves all our economic and technological problems, we will face the challenge of finding meaning in a 'deep utopia' where instrumental work is obsolete." — Source: [Deep Utopia: Life and Meaning in a Solved World]
  8. On post-work meaning: "In a world where machines can do everything better than us, human value must derive from intrinsic experiences, relationships, and play." — Source: [Deep Utopia: Life and Meaning in a Solved World]
  9. On the plastic world: "We are transitioning from a world shaped by blind evolutionary forces to one entirely sculpted by intelligent design." — Source: [Deep Utopia: Life and Meaning in a Solved World]