Joshua Achiam is an AI researcher who served as OpenAI's Head of Mission Alignment before becoming the company's Chief Futurist. He is known for developing technical AI safety protocols using reinforcement learning and advancing arguments for a balanced, practical approach to managing existential risk. This profile examines his views on artificial general intelligence, cyber defense, and the engineering best practices necessary for a secure transition to widespread automation.

Part 1: AI Capabilities and the Arrival of AGI

  1. On AGI normalization: "It feels like AGI is kind of already here and most people have gone like shrug. The fact that we passed the threshold where unsolved mathematical conjectures are getting solved by extremely intelligent AI, where those AIs are more capable and smarter than the people who studied their whole lives for this. That should have felt really weird to people, but it didn't." — Source: The a16z Show Summary
  2. On hedonic adaptation to AI: "People treat their reality as normal. They hedonically adapt so fast. Like the models of today are just unbelievably capable compared to the models of like three years ago." — Source: The a16z Show Summary
  3. On accelerating scientific progress: AI models demonstrate an ability to accelerate research in other scientific fields, such as solving decades-old math conjectures, which markets and policymakers may not fully grasp yet. — Reference: The a16z Show Summary
  4. On the physical limits of intelligence: "There's got to be a maximum amount of computation that you can have per unit volume and energy in the physical universe. Right? And so that sort of implies that there's like a maximum amount of intelligence per unit volume and unit of energy." — Source: The a16z Show Summary
  5. On eventual model parity: Because there are physical limits to compute per unit of energy and volume, AI capabilities will eventually hit a saturation point where competing models are roughly equivalent in raw intelligence. — Reference: The a16z Show Summary
  6. On the long runway for growth: Despite ultimate physical limits, the current rate of scaling is far from reaching those boundaries, leaving a substantial runway for AI models to grow in capabilities before plateauing. — Reference: OpenAI's Joshua Achiam: Did We Already Reach AGI?

Part 2: Cybersecurity and Offensive AI

  1. On tangible cyber capabilities: "What this shows us is very tangible evidence that models now have super advanced cyber capabilities. They're able to break through and find zero days that, you know, in the past would have been much harder for models to identify, let alone use." — Source: The a16z Show Summary
  2. On data poisoning as a defense: "If you've got an AI model on your side that is going to try to hack into an adversary's system, if your adversary plants a trap where they poison their own data, they can try to jailbreak your model when your model is, you know, ingesting their data." — Source: The a16z Show Summary
  3. On compute as a strategic advantage: "I think the dynamics of cyber in the long term might have something of this flavor where you've got competing AIs on either side of a cyber offense or defense problem and compute is being allocated to them to figure out how to break the other and how to control the other's resources." — Source: The a16z Show Summary
  4. On state actors stockpiling exploits: "I am worried about on the state actor side of things where there will be state actors who are very determined to figure out the maximal extent to which they can use these capabilities and here's where I get really nervous. They might not obviously signal to people what they find. It might be very quiet that they identify a large number of zero days that can be saved up for a rainy day." — Source: The a16z Show Summary
  5. On the combinatorial problem of jailbreaks: "There will be some sequence of inputs to a model that triggers a behavior that wasn't accounted for at training time. Because there are so many possible long sequences of inputs that it's almost like a combinatorial problem for trying to block all of them..." — Source: The a16z Show Summary
  6. On traceability limiting worst-case attacks: Near-term, catastrophic cyberattacks might be prevented because executing them would require massive, traceable compute resources across monitored cloud environments. — Reference: The a16z Show Summary
  7. On low-level cybercriminals: Counterintuitively, petty hackers exploiting AI-discovered flaws for quick gains could force early patching, ultimately stripping state actors of zero-day weapons they intended to hoard. — Reference: The a16z Show Summary

Part 3: Evaluating Existential Risk (X-Risk)

  1. On the risk discourse pendulum: The existential risk conversation has swung too far toward unconcerned optimism; while total doomerism is baseless, careless optimism is equally unjustifiable. — Reference: Joshua Achiam Public Statement Analysis
  2. On assessing probabilities: It is more productive to discuss the specific likelihoods of different scenarios rather than arguing in absolutes about whether existential risks will or will not happen. — Reference: Joshua Achiam Public Statement Analysis
  3. On the possibility of a hard takeoff: A rapid, hard takeoff in AI capabilities is highly improbable due to physical constraints, though dismissing the possibility entirely is unwise. — Reference: Joshua Achiam Public Statement Analysis
  4. On accepting the reality of x-risk: While he rejects apocalyptic gray goo scenarios, Achiam maintains that the vulnerable world hypothesis is likely true and that AGI makes various existential risks more probable. — Reference: Joshua Achiam Public Statement Analysis
  5. On short-term doom probabilities: He argues that the probability of human extinction within the next ten years is extremely small (roughly one in a million), and the chance of it happening within a century is significantly less than 100 percent, suggesting that gradual societal transformations are more plausible outcomes. — Reference: Joshua Achiam Public Statement Analysis
  6. On intentionality and existential risks: "I also think x-risk within 100 years could potentially have AGI in the causal chain without being an intentional act by AGI (eg, humans ask a helpful, aligned AGI to help us solve a scientific problem whose solution lets us build a superweapon that causes x-risk)." — Source: Joshua Achiam Public Statement Analysis
  7. On exaggerated probabilities: "(This is not to say there are no x-risks from AGI - I think there are - but anyone who tells you probabilities are in the 5-10% range or greater that AGI will immediately and intentionally kill everyone is absolutely not thinking clearly)" — Source: Joshua Achiam Public Statement Analysis
  8. On disconnected timelines: "Again, this is not a claim that x-risk isn't real, that AGI doesn't lead to x-risk, or that AGI doesn't have potentially catastrophic impacts, all of which I think are plausible claims. But the claimed timelines and probabilities are just way, way out of connection to reality." — Source: Joshua Achiam Public Statement Analysis

Part 4: Practical AI Alignment and Safety

  1. On adopting standard engineering practices: "If we adopt safety best practices that are common in other professional engineering fields, we'll get there. Surfacing and prioritizing hazards, and making that analysis legible, has to become the norm." — Source: Joshua Achiam Public Statement Analysis
  2. On the mistakes of the safety community: "I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. Throwing all of humanity's future babies out with the bathwater." — Source: Joshua Achiam Public Statement Analysis
  3. On technical versus sociotechnical safety: While early efforts focused on algorithmic implementation of safety methods, managing acceptable AI risk ultimately requires societal negotiation and democratic input to define shared specifications. — Reference: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  4. On rule-based rewards (RBR) for models: Using AI feedback structured around specific behavioral rules—combined with an LLM grader—allows reinforcement learning to fine-tune models with a fraction of the human data usually required. — Reference: Rule Based Rewards for Language Model Safety
  5. On overcoming cautiousness in alignment: Without precise rules, standard human preference data can cause an AI to act overly cautious or unhelpfully judgmental; structured rule-based rewards effectively balance safety with actual usefulness. — Reference: Rule Based Rewards for Language Model Safety
  6. On updating model safety behavior: As use patterns and capabilities change, updating AI safety behaviors through fine-grained, composable prompts via LLM grading is cheaper and more accurate than relabeling vast amounts of human data. — Reference: Rule Based Rewards for Language Model Safety
  7. On Lagrangian methods in safe reinforcement learning: Applying proportional and derivative controls (PID) to Lagrange multiplier updates helps prevent the oscillation and overshoot that traditionally causes AI agents to violate constraints while training. — Reference: Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
  8. On tuning AI controllers: Advanced Lagrangian methods can be engineered to be numerically invariant to the relative scales of reward and cost, easing the burden of hyperparameter tuning for safe RL. — Reference: Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Part 5: Societal Impact and Governance

  1. On AI as a sociotechnical system: "Sometimes we talk about sociotechnical systems in the context of safety, that it's not just the technical thing that you have to make safe. It's the entire social system into which the technology is embedded that has to ultimately have good and acceptable outcomes for everyone who's a part of it." — Source: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  2. On establishing safety norms: To translate safety into technology, society must first openly discuss and negotiate what feels good and bad, reaching an agreed-upon specification for acceptable risk. — Reference: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  3. On ending material scarcity: "Once you start to see that it's possible to get to a world maybe where everyone's basic material needs are met, because automation enables you to provide all of the goods and services that people depend on at low to no cost, you kinda can't help but think, wow, that's actually really worth it." — Source: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  4. On alleviating modern economic pressures: The ultimate goal of integrating AI into the economy is to relieve workers from the stress of working around the clock to survive, allowing people to live better lives and spend more time with their families. — Reference: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  5. On early motivations for building AGI: Achiam originally aspired to work on faster-than-light space travel, but decided that creating artificial intelligence was the most practical first step to solving humanity's most complex physical challenges. — Reference: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  6. On national AI infrastructure: Scaling compute resources is expected to reshape geopolitical competition, incentivizing nations to build vast AI infrastructure to sustain both economic strength and strategic security. — Reference: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  7. On the democratization of AI access: Making AI tools and literacy broadly accessible aids economic resilience, ensuring that the resulting productivity and scientific gains are widely shared rather than centralized among a few entities. — Reference: Event Replay: OpenAI's Chief Futurist on AGI and What's Next
  8. On mathematical games: Achiam spent time solving cases for the 3x3 combinatorial game of Tak, showing a continued interest in the mathematics of strategy and games. — Reference: The $3\times 3$ Solution to the Game of Tak