John Schulman is a reinforcement-learning researcher and OpenAI co-founder. He coauthored the PPO algorithm, helped lead the RLHF work behind ChatGPT, moved to Anthropic in 2024 to pursue hands-on alignment research, and was identified as Thinking Machines Lab’s chief scientist in a 2026 interview. These lessons cover policy optimization, human-feedback training, truthfulness and human oversight. — Dwarkesh 2026 panel.

Visual summary of operating lessons from John Schulman.

Part 1: Early Influences and the Transition to AI

  1. On seeking fundamental truths: Schulman began in physics because he wanted to understand fundamental questions about the universe; later research experiences pulled him toward other fields. — Berkeley News interview.
  2. On shifting focus from physics: Summer physics research did not engage him as much as neuroscience and intelligence, prompting a change in direction. — Berkeley News interview.
  3. On early robotic inspirations: Childhood exposure to BattleBots helped spark his interest in building intelligent machines. — Manifold interview.
  4. On futurist literature: Reading Ray Kurzweil’s The Singularity Is Near influenced his early thinking about AI’s future. — Manifold interview.
  5. On entering machine learning: He moved from physics through neuroscience toward machine learning because questions about intelligence seemed more compelling. — Berkeley News interview.
  6. On Pieter Abbeel’s lab: At Berkeley, Pieter Abbeel’s robotics lab gave Schulman a route into reinforcement learning as deep learning began to improve practical results. — Berkeley News interview.
  7. On the appeal of reinforcement learning: Robotics work led him to reinforcement learning, where agents improve behavior through interaction rather than just imitation. — Berkeley News interview.
  8. On early optimization challenges: Policy-gradient methods can be sensitive to step size: excessively large updates can cause performance to collapse, motivating more stable policy-optimization methods. — OpenAI PPO release.
  9. On long-term vision: He joined OpenAI partly because he expected progress toward broadly capable AI and wanted to work on its development and safety. — Berkeley News interview.

Part 2: TRPO, PPO, and the Math of Reinforcement Learning

  1. On Trust Region Policy Optimization: The TRPO paper derives a theoretical monotonic-improvement scheme, then approximates it to make a practical neural-network algorithm; the practical method tended to improve steadily in the reported experiments, not under an unconditional guarantee. — TRPO paper.
  2. On the limitations of early algorithms: TRPO offered robust results but its optimization constraints made it less straightforward to combine with shared policy/value networks and auxiliary losses, a tradeoff later PPO work addressed. — OpenAI PPO release.
  3. On the necessity of Proximal Policy Optimization: PPO was introduced as a simpler-to-implement policy-gradient family with empirical sample-complexity and benchmark benefits relative to TRPO and other online methods. — PPO paper.
  4. On PPO's widespread adoption: OpenAI adopted PPO as its default RL algorithm at the time because it balanced performance, implementation simplicity and ease of tuning. — OpenAI PPO release.
  5. On mathematical elegance versus practicality: PPO’s clipped objective was designed to limit how far policy updates move while retaining a relatively simple implementation; its merits were demonstrated empirically, not as a universal rule that simpler algorithms always win. — OpenAI PPO release.
  6. On optimization challenges: Schulman and coauthors identified stable, steady improvement under changing data as a central challenge for policy-gradient learning. — GAE paper.
  7. On Generalized Advantage Estimation: Generalized Advantage Estimation reduces variance in policy-gradient estimates by using value functions, accepting some bias in exchange; the paper reports strong simulated-locomotion results. — GAE paper.
  8. On the universality of RL: Schulman’s work applied reinforcement-learning ideas first to simulated control tasks and later to language-model training, though those settings require different rewards and engineering. — TalkRL interview.
  9. On algorithmic simplicity: PPO’s appeal in OpenAI’s experiments included lower implementation and tuning burden than more complicated alternatives; that is a practical empirical observation, not a cross-domain law. — OpenAI PPO release.

Part 3: From Theoretical Algorithms to InstructGPT

  1. On bridging theory and product: After earlier policy-optimization research, Schulman worked on the RL-from-human-feedback pipeline that produced InstructGPT and later informed ChatGPT. — TalkRL interview.
  2. On the shift to language modeling: At OpenAI, Schulman’s team shifted RL research toward language-model behavior, optimizing responses against human-preference signals rather than robotics rewards. — TalkRL interview.
  3. On WebGPT: WebGPT trained a model to issue browser commands such as search and page navigation, gather passages and cite sources when answering open-ended questions. — OpenAI WebGPT research.
  4. On the creation of ChatGPT: Schulman helped lead OpenAI’s post-training effort; ChatGPT built on pretrained language models refined for assistant behavior using human feedback. — Dwarkesh interview.
  5. On the impact of InstructGPT: In the InstructGPT study’s human evaluations on its tested prompt distribution, a 1.3-billion-parameter tuned model was preferred to a 175-billion-parameter untuned GPT-3 model; that result does not generalize to every task. — InstructGPT paper.
  6. On reasoning through search: In WebGPT, browser access allowed a model to gather context and cite passages, but the authors still reported factual-accuracy and out-of-distribution limitations. — OpenAI WebGPT research.
  7. On iterative deployment: Schulman describes post-training as an iterative process involving deployment, new data, changing annotation choices and both data quality and quantity; user interaction alone does not automatically become training data. — Dwarkesh interview.

Part 4: Taming the Shoggoth with RLHF

  1. On shaping a pretrained model: Pretraining produces a flexible model of language and knowledge; post-training steers its behavior toward useful assistant responses. The “shoggoth” framing belongs to the interview host, not a verified Schulman quotation. — Dwarkesh interview.
  2. On defining helpfulness: Human preference judgments are nuanced, and a reward model trained from them can fail to capture all dimensions of a good answer. — TalkRL interview.
  3. On the subtlety of tasks: Schulman said WebGPT rating instructions grew lengthy because judging a good answer involved many context-dependent choices; finding a simpler supervision method remained open. — TalkRL interview.
  4. On unintended incentives: More elaborate rating rubrics did not automatically improve WebGPT in the experiments Schulman described, illustrating that rater instructions need empirical testing. — TalkRL interview.
  5. On the cost of human feedback: The InstructGPT approach used human-written demonstrations and ranked outputs; those annotations were a significant part of its training pipeline, though the cited paper does not establish a specific cost. — InstructGPT paper.
  6. On performance improvements: Schulman attributed substantial gains in the ChatGPT-era assistant to post-training work, while emphasizing multiple contributing axes rather than a measured universal share of AI progress. — Dwarkesh interview.
  7. On constructing a persona: Post-training helps determine the assistant’s conversational style and behavior as well as task performance; Schulman described these choices as subtle and preference-dependent. — Dwarkesh interview.
  8. On reward-model failure modes: Schulman warned that human-preference training can reward attractive-looking answers even when they are not the most truthful, so evaluations must check factual support. — Berkeley truthfulness lecture.

Part 5: Hallucinations and the Danger of Behavior Cloning

  1. On deeply learned concepts: In a 2023 lecture, Schulman speculated that a model may speak more intelligently about concepts it has encountered in many contexts than about a newly supplied concept. — Berkeley truthfulness lecture.
  2. On novel contexts: Schulman cautioned that when a concept appears only in a new document, a model may largely summarize the document rather than reason deeply about it; he explicitly framed this as speculation. — Berkeley truthfulness lecture.
  3. On the danger of behavior cloning: Schulman argued that imitating expert answers can teach a model to guess beyond what it actually knows, contributing to unsupported answers. — Berkeley truthfulness lecture.
  4. On forcing guesses: Schulman identified pressure to answer rather than express uncertainty as one cause of hallucination, while warning that excessive refusal can also make an assistant unhelpful. — Berkeley truthfulness lecture.
  5. On mitigating fabrications: He proposed using reinforcement learning to reward calibrated uncertainty and abstention when an answer is unknown, but called long-form truthfulness an open challenge. — Berkeley truthfulness lecture.
  6. On process supervision: An OpenAI study to which Schulman contributed tested feedback on intermediate reasoning steps in mathematical problem-solving, finding gains over outcome-only feedback on its math benchmark; broader generalization remained unknown. — OpenAI process-supervision study.
  7. On the plausibility trap: Schulman argued that polished answers and apparently credible citations may win human preference even when factual verification remains difficult; evaluators need inspectable sources. — Berkeley truthfulness lecture.
  8. On internal knowledge boundaries: Schulman treated knowing when a model should answer, qualify or abstain as an unresolved truthfulness problem, especially for long-form responses. — Berkeley truthfulness lecture.

Part 6: Post-Training as a Competitive Advantage

  1. On tacit knowledge: Schulman said post-training incorporates hard-won research and organizational knowledge, while acknowledging that distillation of model behavior can erode any resulting moat. — Dwarkesh interview.
  2. On data quality and quantity: He described data quality, data quantity, annotation choices and repeated deployment cycles as separate levers for better post-training; he did not rank quality as always more important. — Dwarkesh interview.
  3. On organizational structure: Effective post-training draws on researchers and annotators making many behavioral and data decisions; Schulman described this as specialized, partly tacit work. — Dwarkesh interview.
  4. On the art of tuning: Schulman described RL research as requiring both empirical experimentation and first-principles reasoning to decide what to test, rather than a fixed hyperparameter recipe. — Dwarkesh interview.
  5. On the limits of an organizational moat: Schulman argued that another lab can distill demonstrated model behavior, limiting how defensible a post-training advantage is; this is different from claiming every recipe is a trade secret. — Dwarkesh 2026 panel.
  6. On the balance of pre- and post-training: Schulman saw substantial room for post-training gains, but did not claim that pretraining had plateaued or that post-training always gives more return per unit of compute. — Dwarkesh interview.
  7. On data flywheels and competition: Realistic user-prompt distributions may help training and distillation, but Schulman explicitly notes that copying behavior can counter centralization; a large user base is not an insurmountable moat. — Dwarkesh 2026 panel.

Part 7: AI Alignment and the Limits of Human Supervision

  1. On the objective of alignment: Schulman frames alignment as both specifying what humans actually want AI systems to do and making systems reliably pursue those objectives, not merely improving demos. — Dwarkesh 2026 panel.
  2. On human moral judgment: Schulman argues that people must continue deciding what they want AI assistants to do and how those systems should behave, even if models become capable of much of the technical work. — Dwarkesh 2026 panel.
  3. On the limits of human supervision: As AI systems become capable of work people cannot fully check, Schulman has proposed model-assisted critiques and informed human sign-off for consequential decisions. — TalkRL interview.
  4. On objective versus subjective tasks: Schulman distinguished easier-to-verify tasks from harder judgment calls and treated the specification of desirable behavior as a continuing human responsibility; he did not say feedback only works for objective tasks. — Dwarkesh 2026 panel.
  5. On scalable oversight: He suggested using one model’s critique to help a human evaluate another model’s output when direct human checking becomes difficult. — TalkRL interview.
  6. On prioritizing truthfulness: Schulman argued that a useful assistant must balance accurate abstention with informativeness, and proposed explicit training and evaluation for when uncertainty should be expressed. — Berkeley truthfulness lecture.
  7. On the alignment tax: Schulman’s proposal to require informed human approval for consequential AI actions could slow systems and forgo some benefits; he presented those costs as tradeoffs rather than rejecting the safeguard. — FAR.AI talk.
  8. On long-horizon alignment: Schulman has identified long projects, error recovery and longer-horizon learning as important challenges for more capable AI agents. — Dwarkesh interview.

Part 8: Career Transitions and the Path to AGI

  1. On future AI capabilities: In a 2022 interview, Schulman predicted that AI might outperform humans at most existing jobs within about five years; this is a dated forecast, not a claim that it has happened. — TalkRL interview.
  2. On models as coworkers: In 2024, he forecast AI systems collaborating on substantial projects rather than only answering isolated prompts; the interview did not establish a specific universal interface transition. — Dwarkesh interview.
  3. On moving to Anthropic: In his August 2024 departure note, Schulman said he was leaving OpenAI for Anthropic to focus more deeply on AI alignment and return to hands-on technical research. — Schulman’s 2024 departure note.
  4. On research plateaus: In a 2024 interview, Schulman declined to declare a simple scaling plateau and expected model-development methods to change; the original claim about inevitable diminishing returns is unsupported. — Dwarkesh interview.
  5. On Thinking Machines Lab: By 2026 Schulman was introduced as Thinking Machines Lab’s chief scientist, and his 2025 research essay there confirms his technical involvement; the captured first-person founding quote is unverified. — Dwarkesh 2026 panel.
  6. On continuous learning: Schulman’s 2024 departure note expressly said he wanted to return to hands-on technical work, showing that direct research remained a personal priority at that career point. — Schulman’s 2024 departure note.