Liam Fedus co-founded Periodic Labs after helping create ChatGPT at OpenAI. His work connects AI development with experimental science. — Peter Deng — Building a New Way to Do Science.

Visual summary of operating lessons from Liam Fedus.

Part 1: The Physics Mindset and the Scientific Method

  1. On physicists in AI: He credits physics with cultivating principled, careful scientific thinking. — No Priors.
  2. On the scientific method: Scientific understanding grows through experiments that distinguish possible explanations, not reasoning alone. — GPU Mode Keynote.
  3. On experimental grounding: Fedus recalls studying dark matter with a direction-sensitive detector before moving into machine learning. — No Priors.
  4. On doing real science: Peter Deng reports Fedus’s guiding principle: scientific AI must do experiments, not just analyze existing knowledge. — Peter Deng — Building a New Way to Do Science.
  5. On high leverage: He sees AI as offering physicists leverage while high-energy experiments await new apparatus. — No Priors.

Part 2: Architecture and Post-Training

  1. On model routing: Switch routes each token to one expert, reducing routing and communication costs; separate training techniques address instability. — Switch Transformers.
  2. On sparse activation: The coauthored study trained trillion-parameter models and reported faster pretraining than dense T5 baselines while keeping per-token FLOPs comparable. — Switch Transformers.
  3. On launching ChatGPT: Despite encouraging private-beta feedback, Fedus was surprised by ChatGPT’s reception, given earlier chatbot disappointments. — MIT Technology Review Interview.
  4. On choosing a general interface: He recalls John Schulman favoring a general chatbot over narrower product ideas. — No Priors.
  5. On post-training pipelines: ChatGPT combined supervised examples with preference-based reinforcement learning to turn autocomplete into an assistant. — a16z Interview.

Part 3: Sparse Systems and Learning Dynamics

  1. On conditional computation: Sparse expert models activate only part of their parameters for each input, separating total capacity from per-example computation. — Sparse Expert Models Review.
  2. On sparsity beyond language: The review documents sparse experts in vision, speech and multimodal systems, not just language. — Sparse Expert Models Review.
  3. On routing as a research problem: Routing determines which experts receive inputs, affects load balancing, and offers competing ways to allocate computation. — Sparse Expert Models Review.
  4. On open sparse-model questions: The 2022 review leaves open questions about specialization, communication, stability and transfer despite demonstrated efficiency gains. — Sparse Expert Models Review.
  5. On replay capacity: In the study’s Q-learning experiments, larger replay buffers helped some algorithms substantially while leaving others unaffected. — Revisiting Fundamentals of Experience Replay.
  6. On larger replay memories: Uncorrected multi-step returns were especially useful for exploiting larger replay memories in those experiments. — Revisiting Fundamentals of Experience Replay.
  7. On replay ratio: Replay ratio—learning updates per collected experience—is a separate variable from replay capacity. — Revisiting Fundamentals of Experience Replay.
  8. On sparse-model stability: ST-MoE’s router z-loss improved stability without reducing quality in the authors’ study; other stabilizing changes sacrificed quality. — ST-MoE.
  9. On transfer after pretraining: Strong pretraining does not ensure strong fine-tuning: ST-MoE found overfitting and sensitivity to downstream training protocols. — ST-MoE.
  10. On emergent abilities: The coauthored 2022 paper describes tasks where measured performance rose sharply with scale; its thresholds depend on data and training choices. — Emergent Abilities of Large Language Models.

Part 4: Overcoming Data Bottlenecks

  1. On internet data limits: Some experimental knowledge does not exist online; better scraping cannot substitute for generating that data. — GPU Mode Keynote.
  2. On noisy literature: Reported material properties can vary widely when preparation and measurement details are missing; fitting that distribution need not recover ground truth. — GPU Mode Keynote.
  3. On intelligence versus data: Even a highly capable particle-physics system may need a new accelerator: the bottleneck can shift from intelligence to experimental access. — GPU Mode Keynote.
  4. On collapsing the search tree: Reasoning alone branches into hypothetical worlds; an experiment rules out incompatible branches and grounds the next decision. — GPU Mode Keynote.

Part 5: Building the Autonomous Laboratory

  1. On the value of failure: Valid failed experiments are valuable training evidence, even though scientific publishing tends to favor successful results. — GPU Mode Keynote.
  2. On AI orchestration: Language models orchestrate experiments and coordinate other models, rather than doing every calculation themselves. — No Priors.
  3. On specialized networks: Specialized atomic networks incorporate symmetries and provide lower-latency tools for the language-model layer. — No Priors.
  4. On reinforcement learning via reality: Physical experiments supply the reward signal and correct simulation errors; nature becomes the training environment. — a16z Interview.
  5. On compute costs: He reports that compute, despite the physical laboratories, dominates Periodic’s capital spending. — No Priors.
  6. On accelerating timelines: Periodic aims to speed materials discovery through automated experimentation, with higher-temperature superconductors among its ambitions—not established achievements. — Peter Deng — Building a New Way to Do Science.
  7. On proving the loop: Demonstrate the complete discovery loop in one domain, then test whether it generalizes to others. — a16z Interview.
  8. On commercial copilots: Engineering copilots should address well-scoped industrial bottlenecks with clear evaluations before broader deployment. — a16z Interview.
  9. On translating scientific problems: Translate across disciplines by specifying inputs, outputs and targets; colleagues with overlapping expertise help bridge the gaps. — a16z Interview.
  10. On a no-stupid-questions culture: Make basic physics and machine-learning questions safe to ask, so specialists can learn from one another. — a16z Interview.
  11. On academia as a task designer: Academic partners can identify useful scientific tasks and reasoning strategies that industrial training teams overlook. — a16z Interview.

Part 6: The Nature of Intelligence and AI's Future

  1. On fragile reasoning: Intelligence is uneven: strong mathematical performance can deteriorate when questions change slightly. — No Priors.
  2. On software engineering progress: Abundant code and executable tests make software a promising feedback loop for model improvement. — a16z Interview.
  3. On domain expertise gaps: Software competence does not automatically transfer to biology, which involves different knowledge and experimental strategies. — No Priors.
  4. On recursive self-improvement: Fedus expects software self-improvement to remain limited for physical discovery without relevant experimental data and feedback. — GPU Mode Keynote.