Ben Mann co-founded Anthropic after working on GPT-3 at OpenAI. He later led product engineering and Anthropic Labs. His interviews explore how to assess AI progress, build useful and safer systems, and navigate uncertain economic effects. — No Priors — Ben Mann Interview.

Visual summary of operating lessons from Ben Mann.

Part 1: Scaling Laws and Model Growth

  1. On scaling predictability: Mann says Anthropic uses scaling trends and available chips to plan model-training runs, while the eventual model still needs evaluation. — No Priors — Ben Mann Interview.
  2. On model size: The GPT-3 paper reported that scaling a language model to 175 billion parameters improved task-agnostic few-shot performance across many tasks. — GPT-3 Paper.
  3. On acceleration: In 2025, Mann argued that AI progress was still accelerating, pointing to shorter model-release cycles and gains from post-training methods. — Lenny's Podcast — Ben Mann.
  4. On few-shot learning: The GPT-3 paper found that a large language model could perform many new tasks from instructions and a few examples without task-specific fine-tuning. — GPT-3 Paper.
  5. On compute constraints: Mann named chips and powered data centers among the main practical bottlenecks to faster model improvement, alongside algorithms, data and research talent. — Lenny's Podcast — Ben Mann.
  6. On data quality: A paper Mann coauthored notes that scaling trends can help diagnose a disappointing training run: if a larger model fails to improve, data quality is one possible problem to investigate alongside numerical, overfitting and hardware issues. — Predictability and Surprise in Large Generative Models.
  7. On perceived leaps: Mann suggested that shorter intervals between releases can make continuing progress feel smaller, while exponential progress can look flat before it becomes conspicuous. — Lenny's Podcast — Ben Mann.
  8. On future architectures: Mann left room for more specialized model architectures, including modules for different functions, rather than treating present transformer designs as final. — No Priors — Ben Mann Interview.
  9. On scaling limits: Mann said scaling trends continued to hold in 2025, while acknowledging that a wall may eventually emerge and that gains also depend on algorithmic and data improvements. — Lenny's Podcast — Ben Mann.
  10. On algorithmic efficiency: Mann attributed falling costs for a given level of model capability to a combination of algorithmic, data and efficiency improvements, not compute alone. — Lenny's Podcast — Ben Mann.

Part 2: Evaluating AI and the Economic Turing Test

  1. On real-work evaluation: Mann favors testing whether an AI agent could actually perform a role over a sustained contract, rather than relying only on short tests or interviews. — Lenny's Podcast — Ben Mann.
  2. On multi-step task completion: Mann described a Claude user turning a long video into a presentation through several tool-using steps, illustrating why completing workflows matters beyond generating isolated text. — No Priors — Ben Mann Interview.
  3. On useful agents: Mann pointed to longer-horizon agents that can use tools for multi-step coding and non-coding work; he also described safety limits on unsupervised computer use. — No Priors — Ben Mann Interview.
  4. On benchmark saturation: Mann noted that some easier benchmarks saturate quickly, prompting a need for more ambitious evaluations that can reveal improvements still being made. — Lenny's Podcast — Ben Mann.
  5. On economic significance: Mann proposed looking across a money-weighted basket of jobs: if AI agents could pass the economic Turing test for roughly half, that would signal transformative AI in his illustrative framework. — Lenny's Podcast — Ben Mann.
  6. On reliability in coding: Mann said professional coding assistants need to make requested changes without unwanted edits that create cleanup work. — No Priors — Ben Mann Interview.
  7. On consequential failures: Mann said Anthropic held back a consumer computer-use product because it could not yet trust the agent not to take irreversible actions or expose credentials. — No Priors — Ben Mann Interview.
  8. On longer-horizon work: Mann reported customers using Claude on coding tasks for many hours unattended, while also emphasizing that trustworthy autonomous action still has limits. — No Priors — Ben Mann Interview.

Part 3: Safety, Alignment, and Leaving OpenAI

  1. On founding priorities: Mann said he and colleagues left OpenAI because they felt safety was not receiving sufficient priority; Anthropic was founded to put that concern closer to the center of its work. — Lenny's Podcast — Ben Mann.
  2. On alignment and product quality: Mann said Anthropic found that alignment work could improve the user experience, pointing to Claude's character and ability to respond helpfully in difficult conversations. — Lenny's Podcast — Ben Mann.
  3. On deployment incentives: An Anthropic paper coauthored by Mann argues that predictable gains from scaling can encourage developers to build and deploy larger models, even though specific harmful behaviors remain hard to anticipate. — Predictability and Surprise in Large Generative Models.
  4. On interpretability tools: Mann coauthored a description of Garçon, an Anthropic tool that lets researchers inspect and alter internal activations in large models to study how they work. — Garçon Interpretability Tool.
  5. On testing dangerous behaviors: Mann defended controlled tests of deceptive or otherwise dangerous model behavior as a way to discover failures before release, while acknowledging that a limit on such experiments should exist. — No Priors — Ben Mann Interview.
  6. On severe downside risk: Mann said he thought positive outcomes were more likely, but argued that even a small chance of catastrophic AI failure justifies substantial advance safety work. — Lenny's Podcast — Ben Mann.
  7. On limits of human feedback: Mann said it becomes harder to find humans with enough expertise to compare advanced model outputs. Constitutional AI research explores using AI critiques and feedback to scale supervision while retaining human-provided principles. — No Priors — Ben Mann Interview.
  8. On steering assistant behavior: Constitutional AI uses explicit natural-language principles and model-generated critique to train less harmful, more helpful responses instead of leaving the objective implicit in a large set of ratings. — Constitutional AI Paper.
  9. On transparent risk reporting: Mann said Anthropic publicizes troubling laboratory findings so policymakers and other labs can understand emerging risks, even when the examples may look bad for the company. — Lenny's Podcast — Ben Mann.

Part 4: Product Engineering and Claude

  1. On the product feedback loop: Mann said Anthropic built Claude Code partly to hear directly from coding users and feed their experience back into model and product development. — No Priors — Ben Mann Interview.
  2. On managing context: Mann described using a capable model to delegate searches to smaller subagents, which can reduce latency and keep the main agent's context from filling with every detail. — No Priors — Ben Mann Interview.
  3. On speed and cost: Mann said delegating parts of a task to smaller models can improve latency and cost, while the value of compute should be judged against the human work it saves. — No Priors — Ben Mann Interview.
  4. On researcher feedback: Mann described researchers using Claude Code themselves so they directly experience a model's weaknesses and can prioritize fixes. — No Priors — Ben Mann Interview.
  5. On learning from deployment: Mann said releasing Claude Code gave Anthropic more direct knowledge of what users needed; he also described a telecom partner's feedback revealing edge cases not seen in the lab. — SK Telecom — Ben Mann Interview.
  6. On domain adaptation: Mann described adapting Claude to a telecom partner's documentation and terminology so responses better fit the customer's work. — SK Telecom — Ben Mann Interview.
  7. On handling ambiguity: Mann said useful agents should push back when a request does not make sense, explain their reasoning, and adapt to the user's working style. — SK Telecom — Ben Mann Interview.
  8. On tokenization tradeoffs: In a 2019 Transformer-XL experiment, Mann and collaborators found that byte-pair encoding reduced vocabulary and embedding size but made sequences about 15% longer; throughput improved while adjusted loss stayed similar. — Ben Mann — Scaling Transformer-XL to 128 GPUs.

Part 5: Constitutional AI and the HHH Framework

  1. On assistant goals: Anthropic's Constitutional AI work aims to make an assistant helpful, honest and harmless; those are design goals, not three literal objective functions. — Constitutional AI Paper.
  2. On scalable oversight: Mann coauthored research exploring how AI assistance and AI-generated feedback can help humans supervise increasingly capable systems with fewer direct labels. — Scalable Oversight Paper.
  3. On explicit principles: Constitutional AI makes some behavioral choices legible through a written set of natural-language principles used during training. — Anthropic — Claude's Constitution.
  4. On helpful refusals: The Constitutional AI study sought an assistant that remained helpful while declining harmful requests, explaining its objections rather than becoming evasive. — Constitutional AI Paper.
  5. On AI feedback: Constitutional AI combines model-generated critiques with reinforcement learning from AI feedback, partly replacing human labels for harmlessness while retaining human-chosen principles. — Constitutional AI Paper.
  6. On honest uncertainty: A paper coauthored by Mann tested whether models could assess the likelihood that their own answers were correct, a possible building block for more honest systems. — Language Models (Mostly) Know What They Know.
  7. On public input to AI values: Mann argued that deciding an AI constitution's principles should not be left to a small group of researchers alone; he pointed to published principles and work gathering broader input. — Lenny's Podcast — Ben Mann.
  8. On self-critique: In the Constitutional AI training process, a model critiques an initial response against a principle and generates a revision used for further training. — Constitutional AI Paper.
  9. On sources of principles: Anthropic drew early constitutional principles from sources including human-rights language, privacy rules and internally written values, while treating the constitution as revisable. — Anthropic — Claude's Constitution.

Part 6: Labor Markets and the Economy

  1. On the pace of disruption: Mann argues that people often underestimate compounding AI progress, while cautioning that widespread labor-market transformation is difficult to time precisely. — Lenny’s Podcast with Ben Mann.
  2. On structural unemployment: Mann distinguishes workers lacking skills for available jobs from jobs eliminated outright, and expects both forms of displacement; the 20% unemployment figure in the interview was introduced as Dario Amodei’s forecast, not his. — Lenny’s Podcast with Ben Mann.
  3. On cognitive work: Mann pointed to software engineering and customer support as early areas where AI was changing work, without claiming all white-collar workers are the most exposed. — Lenny’s Podcast with Ben Mann.
  4. On new work and displacement: Mann expects AI to expand what workers can accomplish in the near term, but warns that lower-skill roles with less room for augmentation may face displacement. — Lenny’s Podcast with Ben Mann.
  5. On abundant labor: Mann imagines a possible long-term world in which aligned AI makes expert labor much cheaper and changes how jobs and capitalism function; he presents this as a speculative scenario, not a present zero-cost fact. — Lenny’s Podcast with Ben Mann.
  6. On the economic transition: Mann says the transition from today’s labor market to a possible AI-driven world of abundance needs societal attention, while acknowledging he cannot forecast its institutions or distributional outcomes. — Lenny’s Podcast with Ben Mann.
  7. On developer productivity: Mann described Anthropic’s developers writing substantially more code with Claude and smaller teams becoming more impactful; these are his 2025 observations, not a forecast that one worker replaces an agency. — Lenny’s Podcast with Ben Mann.
  8. On AI talent: Mann says a small inference-efficiency gain can be highly valuable to a frontier lab, helping explain the market for top AI researchers; he does not identify 100-billion-parameter specialists as the sole bottleneck. — Lenny’s Podcast with Ben Mann.
  9. On enterprise adoption: Mann describes tailoring AI systems to a customer’s domain and learning from deployment feedback, rather than predicting that most companies attempting foundation models will fail. — SK Telecom Interview with Ben Mann.

Part 7: Infrastructure, Data, and Compute Limits

  1. On hardware constraints: Mann names chips and data-center capacity among the inputs that constrain frontier-model development; he did not, in the inspected interview, identify TSMC geopolitics as a personal risk forecast. — Lenny’s Podcast with Ben Mann.
  2. On training stability: In Mann’s 2019 Transformer-XL experiment, a mixed-precision approach and an optimizer change were needed to avoid numerical divergence at scale; the paper does not claim a single failure ruins every frontier run. — Scaling Transformer-XL to 128 GPUs.
  3. On data-center power: In his 2025 forecast discussion, Mann included expanding data centers and power supply among the physical factors behind future AI progress; he did not quantify future energy needs or call labs heavy industry. — Lenny’s Podcast with Ben Mann.
  4. On multimodal scaling: A scaling-laws paper coauthored by Mann found smoother loss improvements with larger models and compute budgets across video, image, multimodal image-text and math experiments; it does not establish the post’s claim about future clusters making today’s look like toys. — Scaling Laws for Autoregressive Generative Modeling.
  5. On network throughput: In a 128-GPU Transformer-XL experiment, Mann and collaborators found that distributed-training throughput depended partly on network configuration and cluster placement; they did not generalize this to every frontier cluster. — Scaling Transformer-XL to 128 GPUs.
  6. On inference efficiency: Mann said even a one-to-five-percent improvement in inference efficiency could create substantial value for a frontier-model business, without claiming a ten-percent speed gain wins the enterprise market. — Lenny’s Podcast with Ben Mann.
  7. On training-data mix: The GPT-3 paper, which Mann coauthored, documents a curated mixture of training datasets and filtering choices; it does not say all labs’ dataset curation is their most guarded trade secret. — Language Models are Few-Shot Learners.

Part 8: Long-Term Outlook and Uncertain AI Forecasts

  1. On the 2028 forecast: In a 2025 interview, Mann deferred to superforecasters and said a median forecast of superintelligence around 2028 seemed plausible, while stressing wide uncertainty and that societal effects would not arrive everywhere at once. — Lenny’s Podcast with Ben Mann.
  2. On an economic signal: Beyond the job-based economic Turing test, Mann suggested that a sustained world GDP growth rate above roughly ten percent per year could be another sign of transformative AI; he presented it as an illustrative signal, not a fixed definition of AGI. — Lenny’s Podcast with Ben Mann.
  3. On recursive improvement: Mann discusses the possibility that AI coding and research tools could help build better future models; he treats the timing and extent of recursive self-improvement as uncertain, not as a proved decades-to-months compression. — No Priors Episode 118 with Ben Mann.
  4. On policy preparation: Mann argues that frontier systems should be tested for dangerous behavior and matched with safeguards before deployment, while explaining why Anthropic communicates risk findings to policymakers; he did not set the stated three-year regulatory deadline. — No Priors Episode 118 with Ben Mann.
  5. On alignment before superintelligence: Mann worries that waiting until after a superintelligent system exists may be too late to solve alignment, which is why he supports earlier safety research; this is a risk judgment, not proof that every deployment is an irreversible one-shot event. — Lenny’s Podcast with Ben Mann.
  6. On coordination and safety: Mann points to nuclear non-proliferation as an example of coordination that might matter in a pessimistic alignment scenario, while saying he currently sees stronger reason to pursue alignment research; he does not call lab coordination the only possible path. — Lenny’s Podcast with Ben Mann.
  7. On longer-horizon agents: Mann reported 2025 customer examples of coding agents working unattended for several hours and completing multi-step tool-using tasks, but did not claim that general agents could reliably execute unsupervised multi-day plans. — No Priors Episode 118 with Ben Mann.
  8. On cautious optimism: Mann says beneficial aligned AI could accelerate progress in science, technology, education and mathematics, while arguing that even a low-probability catastrophic risk warrants substantial safety work. — Lenny’s Podcast with Ben Mann.