Lessons from Miles Brundage

AI policy researcher Miles Brundage spent six years at OpenAI, serving as Head of Policy Research and Senior Advisor for AGI Readiness. His work focuses on compute-based AI governance, expert red teaming, and evaluating frontier models for dangerous capabilities. Drawing on his writings and interviews, this profile details his approach to AI safety, verifiable systems, and preparing for artificial general intelligence.

Part 1: AGI Readiness and Safety

  1. On AGI timelines: "AI companies and the rest of the world are not ready for artificial general intelligence, and the gap between preparedness and capability needs to be addressed immediately." — Source: [Why I'm Leaving OpenAI]
  2. On defining AGI: "AGI should not be viewed as a single threshold, but rather a spectrum of capabilities that require increasingly strict safety evaluations as they approach human-level performance." — Source: [80,000 Hours Podcast]
  3. On industry preparedness: "The rapid pace of AI development has outstripped the institutional capacity of both labs and governments to safely manage extreme risks." — Source: [Why I'm Leaving OpenAI]
  4. On safety culture: "Developing frontier models requires a culture where safety researchers have the authority to delay or modify deployments if evaluations reveal unacceptable risks." — Source: [Toward Trustworthy AI Development]
  5. On systemic safety: "Ensuring AI safety requires moving beyond individual model alignment to building broader societal resilience against AI-driven disruptions." — Source: [80,000 Hours Podcast]
  6. On internal forecasting: "AI labs must implement rigorous internal forecasting to anticipate capability jumps rather than reacting to them after training is complete." — Source: [Why I'm Leaving OpenAI]
  7. On the capability-safety gap: "The funding and attention directed toward AI safety research remains severely under-resourced compared to investments in scaling AI capabilities." — Source: [Why I'm Leaving OpenAI]
  8. On AGI governance: "Preparing for AGI demands unprecedented coordination between private developers and regulatory bodies to ensure the benefits are broadly shared." — Source: [Toward Trustworthy AI Development]
  9. On independent oversight: "Internal safety teams at AI labs face inherent conflicts of interest, making independent third-party oversight essential as models approach general intelligence." — Source: [Why I'm Leaving OpenAI]
  10. On long-term planning: "Developers of advanced AI must publish clear, accountable plans for how they will manage the transition to AGI without causing societal instability." — Source: [80,000 Hours Podcast]

Part 2: The Malicious Use of AI

  1. On threat forecasting: "AI systems lower the barrier to entry for malicious actors, expanding the scale and speed of potential cyber attacks." — Source: [The Malicious Use of Artificial Intelligence]
  2. On spear phishing: "Language models can generate highly personalized and convincing phishing campaigns at scale, fundamentally changing the economics of cybercrime." — Source: [The Malicious Use of Artificial Intelligence]
  3. On physical security: "The integration of AI into robotics and drones introduces new vectors for physical terrorism, requiring proactive hardware safeguards." — Source: [The Malicious Use of Artificial Intelligence]
  4. On political manipulation: "Automated disinformation campaigns powered by generative AI threaten to overwhelm public discourse with synthetic media." — Source: [The Malicious Use of Artificial Intelligence]
  5. On dual-use risks: "Models developed for benign purposes, such as biology or chemistry research, can be easily repurposed to design pathogens." — Source: [The Malicious Use of Artificial Intelligence]
  6. On defense advantages: "While AI empowers attackers, defenders can also use it to automate threat detection and analyze malware at scale." — Source: [80,000 Hours Podcast]
  7. On open-source trade-offs: "Releasing powerful models open-source accelerates research but permanently removes the ability to patch vulnerabilities for malicious actors." — Source: [Toward Trustworthy AI Development]
  8. On mitigating misuse: "Policymakers must focus on both restricting access to dangerous capabilities and building societal resilience against the inevitable leakage of such tools." — Source: [The Malicious Use of Artificial Intelligence]
  9. On attribution challenges: "The use of AI in cyberattacks makes it increasingly difficult to attribute actions to specific state actors, complicating deterrence strategies." — Source: [The Malicious Use of Artificial Intelligence]

Part 3: Compute and Governance

  1. On compute as a node: "Computing power is the most governable node in the AI supply chain because it is physical, highly concentrated, and easily quantifiable." — Source: [Computing Power and the Governance of AI]
  2. On hardware tracking: "Implementing tracking mechanisms at the hardware level can enable regulators to monitor large-scale AI training runs without accessing proprietary data." — Source: [Computing Power and the Governance of AI]
  3. On data center oversight: "Large data centers are visible and resource-intensive, making them natural chokepoints for enforcing AI safety regulations." — Source: [Computing Power and the Governance of AI]
  4. On compute thresholds: "Regulatory frameworks should establish compute thresholds for mandatory safety reporting, ensuring that only the most capable models face strict oversight." — Source: [Computing Power and the Governance of AI]
  5. On global coordination: "A global registry of large-scale compute resources could prevent regulatory arbitrage and ensure that dangerous models are not trained in jurisdictions with lax oversight." — Source: [Computing Power and the Governance of AI]
  6. On privacy preservation: "Compute governance can be designed using privacy-enhancing technologies to verify compliance without exposing trade secrets." — Source: [Computing Power and the Governance of AI]
  7. On export controls: "Restrictions on the export of advanced AI accelerators are a practical tool for limiting the proliferation of frontier capabilities to adversaries." — Source: [Computing Power and the Governance of AI]
  8. On equitable access: "While regulating large-scale compute is necessary for safety, policymakers must also ensure that academic researchers retain access to sufficient compute for safety research." — Source: [Computing Power and the Governance of AI]
  9. On cloud providers: "Cloud service providers play a key role in AI governance by implementing know-your-customer policies and monitoring for prohibited training activities." — Source: [Computing Power and the Governance of AI]
  10. On decentralized compute: "As distributed training techniques improve, regulators must develop new methods to track and govern compute networks that span multiple smaller clusters." — Source: [Computing Power and the Governance of AI]

Part 4: Verifiable Claims and Trustworthy AI

  1. On verifiable claims: "AI developers must transition from making vague promises about safety to providing verifiable claims backed by rigorous evidence." — Source: [Toward Trustworthy AI Development]
  2. On institutional mechanisms: "Creating trustworthy AI requires mechanisms like third-party auditing and secure whistleblower channels to ensure developers are held accountable." — Source: [Toward Trustworthy AI Development]
  3. On software mechanisms: "Developers should use privacy-preserving machine learning and federated learning to audit models without compromising sensitive training data." — Source: [Toward Trustworthy AI Development]
  4. On hardware mechanisms: "Secure enclaves and specialized hardware can provide cryptographic proof that a specific model was run on specific data." — Source: [Toward Trustworthy AI Development]
  5. On auditing standards: "The AI industry lacks standardized auditing frameworks, making it difficult for users and regulators to compare the safety profiles of different models." — Source: [Toward Trustworthy AI Development]
  6. On red team independence: "Internal red teams must have structural independence and direct reporting lines to corporate boards to prevent safety findings from being suppressed." — Source: [Toward Trustworthy AI Development]
  7. On safety boundaries: "Developers should clearly define and publish the boundaries of safe operation for their models, including specific failure modes." — Source: [Toward Trustworthy AI Development]
  8. On bias and fairness: "Verifiable claims must extend beyond security to include rigorous testing for bias and the disparate impacts of AI systems on marginalized groups." — Source: [Toward Trustworthy AI Development]
  9. On public accountability: "AI companies should publish regular transparency reports detailing safety incidents, red teaming results, and the effectiveness of their mitigation strategies." — Source: [Toward Trustworthy AI Development]

Part 5: Expert Red Teaming and Evaluation

  1. On red teaming scope: "Red teaming must involve domain experts in fields like biology and cybersecurity to evaluate whether models can assist in creating catastrophic threats." — Source: [OpenAI Red Teaming Network]
  2. On continuous evaluation: "AI models must be continuously evaluated even after deployment, as user interactions and system updates can reveal new vulnerabilities." — Source: [Evaluating Frontier Models for Dangerous Capabilities]
  3. On capability elicitation: "Red teams must employ advanced prompting techniques and fine-tuning to elicit the maximum capabilities of a model, ensuring evaluations do not underestimate risks." — Source: [Evaluating Frontier Models for Dangerous Capabilities]
  4. On evaluating autonomy: "As models become more agentic, evaluations must test their ability to autonomously acquire resources, replicate, and evade shutdown mechanisms." — Source: [Evaluating Frontier Models for Dangerous Capabilities]
  5. On deception: "Advanced AI systems must be evaluated for deceptive alignment, testing whether they can successfully hide dangerous behaviors from human overseers." — Source: [Evaluating Frontier Models for Dangerous Capabilities]
  6. On standardized benchmarks: "The AI community must develop standardized, evolving benchmarks for dangerous capabilities to ensure consistent safety evaluations across different labs." — Source: [Evaluating Frontier Models for Dangerous Capabilities]
  7. On external auditors: "AI developers should provide external auditors with early, unfettered access to frontier models before deployment to ensure independent validation of safety claims." — Source: [OpenAI Red Teaming Network]
  8. On the limits of red teaming: "Red teaming is necessary but not sufficient for safety; it must be combined with specialized alignment research and secure system architecture." — Source: [Evaluating Frontier Models for Dangerous Capabilities]
  9. On sharing findings: "AI labs should share red teaming methodologies and high-level findings with each other to raise the baseline of safety across the industry." — Source: [Toward Trustworthy AI Development]
  10. On iterative deployment: "Red teaming findings should inform iterative deployment strategies, allowing developers to test models in controlled environments before widespread release." — Source: [OpenAI Red Teaming Network]

Part 6: Policy, Regulation, and Institutions

  1. On regulatory agility: "Traditional regulatory frameworks are too slow for the pace of AI development; governments need agile institutions capable of adapting to rapid capability jumps." — Source: [80,000 Hours Podcast]
  2. On liability: "Policymakers should clarify liability regimes for AI developers, ensuring that companies are held financially responsible for catastrophic harms caused by their models." — Source: [Toward Trustworthy AI Development]
  3. On licensing regimes: "Training and deploying frontier AI models should require a license contingent on meeting strict safety and auditing requirements." — Source: [Computing Power and the Governance of AI]
  4. On government expertise: "Regulators must aggressively recruit top AI talent and researchers to ensure they have the technical capacity to evaluate complex safety claims." — Source: [80,000 Hours Podcast]
  5. On whistleblowers: "Strong legal protections for whistleblowers within AI companies are essential for bringing unsafe practices to the attention of regulators and the public." — Source: [Why I'm Leaving OpenAI]
  6. On national security: "AI policy must balance the need for safety regulation with national security imperatives, avoiding policies that inadvertently disadvantage democratic nations in AI development." — Source: [The Malicious Use of Artificial Intelligence]
  7. On public engagement: "AI governance should incorporate mechanisms for public input and democratic deliberation, ensuring that decisions about AGI reflect broad societal values." — Source: [Toward Trustworthy AI Development]
  8. On regulatory capture: "Policymakers must design oversight institutions that are resistant to capture by major AI labs, ensuring that regulations protect the public rather than entrenching monopolies." — Source: [80,000 Hours Podcast]
  9. On crisis response: "Governments need dedicated crisis response protocols to handle severe AI incidents, such as the release of a dangerous bioweapon design." — Source: [The Malicious Use of Artificial Intelligence]

Part 7: International Cooperation and Geopolitics

  1. On avoiding arms races: "A zero-sum mentality in global AI development increases the likelihood of corner-cutting on safety and security, leading to a dangerous race to the bottom." — Source: [LessWrong Discussion]
  2. On international treaties: "The international community must work toward treaties governing the development of frontier AI, similar to agreements on nuclear or biological weapons." — Source: [Computing Power and the Governance of AI]
  3. On joint safety research: "Rival nations should collaborate on fundamental AI safety research, establishing shared protocols and red lines for dangerous capabilities." — Source: [80,000 Hours Podcast]
  4. On verification regimes: "International AI agreements will require strict verification regimes, likely centered on compute monitoring and joint audits of large data centers." — Source: [Computing Power and the Governance of AI]
  5. On track-two diplomacy: "Informal diplomatic channels and academic collaborations play a vital role in building trust and shared understanding of AI risks between adversarial nations." — Source: [Toward Trustworthy AI Development]
  6. On global equity: "The benefits of advanced AI must be shared globally to prevent exacerbating existing inequalities and to disincentivize nations from pursuing reckless AI projects." — Source: [80,000 Hours Podcast]
  7. On proliferation risks: "The proliferation of advanced AI capabilities to rogue states and non-state actors requires coordinated international export controls and intelligence sharing." — Source: [The Malicious Use of Artificial Intelligence]
  8. On shared monitoring: "An international consortium should be established to continuously monitor the global AI ecosystem for emerging threats and unexpected capability jumps." — Source: [Computing Power and the Governance of AI]
  9. On cultural differences: "Global AI governance frameworks must be flexible enough to accommodate different cultural values and legal traditions while maintaining strict red lines on safety." — Source: [80,000 Hours Podcast]

Part 8: The Role of Independent Research

  1. On research independence: "The AI safety ecosystem relies too heavily on research conducted within major labs, necessitating stronger, well-funded independent organizations." — Source: [Why I'm Leaving OpenAI]
  2. On third-party auditing: "Independent institutions like the AI Verification and Evaluation Research Institute (AVERI) are essential for providing unbiased assessments of frontier models." — Source: [Why I'm Leaving OpenAI]
  3. On academic access: "AI companies must provide academic researchers with structured access to frontier models and training data to enable independent safety and alignment research." — Source: [Toward Trustworthy AI Development]
  4. On non-public information: "Effective independent oversight requires legal mechanisms allowing researchers to access non-public information held by AI labs without facing punitive action." — Source: [Why I'm Leaving OpenAI]
  5. On funding disparities: "Philanthropic and government funding for independent AI safety research must be scaled dramatically to match the resources available to commercial capability teams." — Source: [Why I'm Leaving OpenAI]
  6. On methodological diversity: "Independent researchers bring diverse methodologies and perspectives to AI safety, identifying blind spots that internal teams at major labs often miss." — Source: [Toward Trustworthy AI Development]
  7. On public communication: "Independent researchers have a responsibility to clearly communicate AI risks to the public and policymakers without relying on industry-approved narratives." — Source: [80,000 Hours Podcast]
  8. On industry incentives: "Commercial pressures inherently conflict with the level of caution required for AGI development, making independent, non-profit institutions necessary for accountability." — Source: [Why I'm Leaving OpenAI]
  9. On the future of oversight: "As models approach AGI, the role of independent verification will shift from a useful supplement to an absolute requirement for safe deployment." — Source: [Why I'm Leaving OpenAI]