Before co-founding Periodic Labs, computer scientist and physicist Liam Fedus designed the Switch Transformer at Google Brain and helped build ChatGPT and GPT-4 at OpenAI. He argues that AI will only accelerate scientific discovery if it interacts directly with physical laboratories. This profile covers his work on scaling models, what we can learn from experimental failure, and the need to ground AI in the real world.

Visual summary of operating lessons from Liam Fedus.

Part 1: The Physics Mindset and the Scientific Method

  1. On physicists in AI: Physics trains researchers to be principled, careful, and hard-nosed about what the evidence actually shows. — Reference: Cognitive Revolution transcript
  2. On the scientific method: Scientific progress comes from proposing how the world might work, testing the proposal, and updating beliefs from the result. — Reference: Cognitive Revolution transcript
  3. On experimental grounding: His undergraduate work on a directional dark-matter detector informed his belief that genuine scientific discovery requires physical instrumentation, rather than purely computational modeling. — Reference: No Priors transcript
  4. On doing real science: "In order to do science, you have to do real science." — Source: felicis.com
  5. On high leverage: Physicists have frequently transitioned into artificial intelligence because software and compute offer far more experimental leverage than high-energy physics, where progress can wait on new apparatus. — Reference: Cognitive Revolution transcript

Part 2: Architecture and Post-Training

  1. On model routing: The Switch Transformer architecture simplified Mixture of Experts (MoE) routing by selecting a single expert per token, which reduced communication costs and mitigated training instability. — Reference: arxiv.org
  2. On sparse activation: Sparse-activation models can be scaled up to a trillion parameters and achieve significant pre-training speedups over dense baselines while maintaining a constant computational cost per input. — Reference: arxiv.org
  3. On launching ChatGPT: "We were definitely surprised how well it was received," Fedus said of the chatbot in the MIT Technology Review in 2023, "there have been so many prior attempts at a general-purpose chatbot that I knew the odds were stacked against us." — Source: businessinsider.com
  4. On choosing a general interface: While planning the productionization of GPT-4, the team considered narrow products but ultimately chose the general chatbot interface that became ChatGPT. — Reference: Business Insider
  5. On post-training pipelines: Turning a pretrained model into a useful product requires a dedicated post-training organization and disciplined release work, not merely a larger base model. — Reference: Business Insider

Part 3: Sparse Systems and Learning Dynamics

  1. On conditional computation: Sparse expert models can increase parameter capacity without applying every parameter to every example, separating total model size from per-example compute. — Reference: Sparse Expert Models review
  2. On sparsity beyond language: Conditional computation is a general deep-learning design pattern with applications beyond language modeling, including vision and multimodal systems. — Reference: Sparse Expert Models review
  3. On routing as a research problem: Choosing which expert should process each input is not a solved implementation detail; routing quality shapes specialization, balance, and overall model performance. — Reference: Sparse Expert Models review
  4. On open sparse-model questions: Expert specialization, communication costs, routing stability, and transfer remain central research problems even after sparse models demonstrate large efficiency gains. — Reference: Sparse Expert Models review
  5. On replay capacity: Increasing an experience-replay buffer does not help every reinforcement-learning algorithm equally; the effect depends on the algorithm and its update design. — Reference: Experience Replay
  6. On larger replay memories: Multi-step returns can make reinforcement-learning agents benefit from much larger replay capacities than conventional one-step training. — Reference: Experience Replay
  7. On replay ratio: The number of learning updates performed per environment interaction is an independent control variable, not something that should be conflated with replay capacity. — Reference: Experience Replay
  8. On sparse-model stability: The practical obstacle to scaling mixture-of-experts systems is often training stability; ST-MoE introduced router regularization that stabilized training without degrading quality. — Reference: ST-MoE
  9. On transfer after pretraining: Sparse models must be designed for downstream fine-tuning and transfer, because impressive pretraining efficiency alone does not guarantee useful task adaptation. — Reference: ST-MoE
  10. On emergent abilities: Some language-model capabilities appear only beyond a scale threshold and cannot be predicted by smoothly extrapolating the performance of smaller models. — Reference: Emergent Abilities of Large Language Models

Part 4: Overcoming Data Bottlenecks

  1. On internet data limits: Internet text alone cannot supply the reliable experimental data needed for qualitative breakthroughs in fields such as materials science. — Reference: Cognitive Revolution transcript
  2. On noisy literature: A single reported material property can vary by orders of magnitude across papers because crucial preparation and measurement details are missing. — Reference: Cognitive Revolution transcript
  3. On modeling noise: If an AI is trained entirely on contradictory literature values, it can only model the distribution of that noise rather than discovering physical ground truth. — Reference: Cognitive Revolution transcript
  4. On intelligence versus data: An advanced AI researcher can ingest every physics paper yet still stall without access to an accelerator or laboratory, shifting the bottleneck from intelligence to data. — Reference: GPU Mode keynote
  5. On the explosion of hypotheses: If an AI tries to solve scientific problems purely through thought, possible outcomes multiply into an unmanageable tree of hypothetical worlds. — Reference: GPU Mode keynote
  6. On collapsing the search tree: Physical experiments collapse branching hypotheses and anchor reasoning in observed data, eliminating paths the world has shown to be wrong. — Reference: GPU Mode keynote

Part 5: Building the Autonomous Laboratory

  1. On the value of failure: Science rarely publishes failed syntheses, yet those negative results are precisely the evidence models need to understand what physical laws permit. — Reference: GPU Mode keynote
  2. On learning from mistakes: Experimental deviations and negative results provide feedback that helps artificial intelligence build a grounded model of the physical world. — Reference: Cognitive Revolution transcript
  3. On AI orchestration: Language models can serve as an orchestration layer that assists researchers while also directing experiments and coordinating specialized systems. — Reference: No Priors transcript
  4. On specialized networks: Large language models can provide the interface while specialized neural networks handle scientific structures such as atomic symmetries at lower latency. — Reference: No Priors transcript
  5. On reinforcement learning via reality: An automated physical laboratory lets nature supply the feedback signal, turning reality itself into the reinforcement-learning environment. — Reference: Cognitive Revolution transcript
  6. On compute costs: Even for a company building physical laboratories, compute remains a major capital expense because search and high-compute reinforcement learning dominate the workload. — Reference: GPU Mode keynote
  7. On accelerating timelines: Automated laboratories paired with models aim to compress materials discovery while pursuing ambitious targets such as higher-temperature superconductors. — Reference: Felicis
  8. On proving the loop: Before expanding across many scientific domains, demonstrate that the entire hypothesis-to-experiment-to-learning loop works in one domain and can then be repeated. — Reference: Cognitive Revolution transcript
  9. On commercial copilots: The same platform can create copilots for engineers in advanced industries, helping expert users navigate scientific design spaces rather than replacing their judgment. — Reference: Cognitive Revolution transcript
  10. On translating scientific problems: A useful cross-disciplinary team learns to express scientific work in API-like terms: define the inputs, outputs, measurements, and feedback needed to improve the system. — Reference: Cognitive Revolution transcript
  11. On a no-stupid-questions culture: Teams spanning physics, chemistry, robotics, and machine learning need an environment where people can ask basic questions without embarrassment and learn one another's disciplines. — Reference: Cognitive Revolution transcript
  12. On academia as a task designer: Academic collaboration can help define valuable scientific tasks, evaluation criteria, and reasoning strategies for reinforcement-learning environments. — Reference: Cognitive Revolution transcript

Part 6: The Nature of Intelligence and AI's Future

  1. On the spikiness of intelligence: Intelligence is not a single scalar; model competence remains uneven and surprisingly spiky across tasks. — Reference: Cognitive Revolution transcript
  2. On fragile reasoning: An AI can perform at an elite level in one mathematical domain and then degrade sharply when the problem is perturbed. — Reference: Cognitive Revolution transcript
  3. On software engineering progress: AI has advanced quickly in software engineering because code provides abundant data and cheap, mechanically verifiable feedback. — Reference: Cognitive Revolution transcript
  4. On domain expertise gaps: Self-improvement is easier where models can evaluate work without relying on scarce human experts; physical science lacks that convenience until experiments are automated. — Reference: Cognitive Revolution transcript
  5. On recursive self-improvement: A genuine superintelligence takeoff in science will require models to act in the physical world and receive experimental feedback, not merely read more text and reason in a vacuum. — Reference: GPU Mode keynote