Lessons from Jakob Uszkoreit

Jakob Uszkoreit is a computer scientist best known for co-authoring the 2017 "Attention Is All You Need" paper that introduced the Transformer architecture. He recently left Google Brain to co-found Inceptive, a company using deep learning to design RNA medicines. This profile organizes his technical views and outlines his shift from building language models to engineering biological software.

Part 1: The Transformer Architecture and Deep Learning

  1. On Attention Mechanisms: "The shift from recurrence to attention was about recognizing that sequential processing was a bottleneck for hardware scalability." — Source: [Computer History Museum]
  2. On the Transformer's Initial Reception: "People raised their eyebrows, because it dumped out all the existing neural architectures. Say goodbye to recurrent neural nets? Heresy!" — Source: [Reddit AMA]
  3. On Parallelization: "A key motivation for the Transformer was unlocking the full parallel compute capabilities of modern GPUs, which RNNs inherently restricted." — Source: [Chatbots Decoded]
  4. On Self-Attention: "Self-attention allows the model to globally route information across a sequence in a single step, rather than passing it token by token." — Source: [arXiv: Attention Is All You Need]
  5. On Architectural Simplicity: "The original goal wasn't just performance, but creating a simpler architecture that could be trained significantly faster." — Source: [Google Research]
  6. On Sequence-to-Sequence Modeling: "Before Transformers, sequence transduction was heavily reliant on complex LSTM and GRU networks, which struggled with very long dependencies." — Source: [arXiv: Attention Is All You Need]
  7. On the Name "Transformer": "The name reflects how the model transforms representations step by step, focusing on global context at each layer." — Source: [Computer History Museum]
  8. On Scaling Laws: "The architecture was specifically designed to scale efficiently with data and compute, a property that fueled the generative AI boom." — Source: [No Priors Podcast]
  9. On Removing Inductive Biases: "By relying almost entirely on attention, the model learns the structure of the data rather than relying on hardcoded sequence constraints." — Source: [Computer History Museum]
  10. On the Universality of Transformers: "What started as a machine translation tool rapidly proved to be a general-purpose architecture for language, vision, and eventually biology." — Source: [a16z Bio Eats World]

Part 2: The Evolution of AI at Google Brain

  1. On the Original Transformer Team: "I believe it's actually a testament to the fact that while our interests overlap a lot, we also are a very diverse group... The fact that this didn't happen [everyone going in the same direction] is, in my book, the specific reason that the group is still incredibly effective." — Source: [VentureBeat]
  2. On Google Brain's Culture: "The environment allowed for radical experimentation, such as completely dropping the standard recurrent models of the time." — Source: [Computer History Museum]
  3. On Translation Systems: "Much of the early work that led to the Transformer was driven by the practical need to improve Google Translate's speed and accuracy." — Source: [Google Research]
  4. On Collaborative Research: "The breakthrough wasn't a single Eureka moment but a collaborative engineering and research effort to solve practical scaling problems." — Source: [VentureBeat]
  5. On Leaving Google: "Leaving a prestigious AI lab was driven by the desire to apply these powerful sequence models to entirely new, impactful domains like biology." — Source: [CNBC]
  6. On Natural Language Processing: "In the mid-2010s, NLP was hitting a wall with sequential models; breaking that wall required a fundamental shift in how networks processed text." — Source: [Computer History Museum]
  7. On the AI Talent Ecosystem: "The dispersal of the original Transformer authors into various startups highlights how foundational research acts as a catalyst for broader industry innovation." — Source: [VentureBeat]
  8. On Self-Supervised Learning: "I talk about self-supervised learning. A family of approaches which will hopefully allow us to learn from data created naturally, in the process of humans going about their daily lives." — Source: [Inceptive CEO Jakob Uszkoreit on Transformers and AI for Medicine]
  9. On Incremental Progress: "Overall, this is a very typical phenomenon where we want to see revolutions. That's to a large extent because that's how we tell stories, but the truth is that almost all the 'revolutions' that I've been a witness to are actually incremental progress." — Source: [Pharmaphorum]

Part 3: The Intersection of AI and Biology

  1. On Biology as Sequences: "Just as language consists of sequences of words, biological molecules like RNA are sequences of nucleotides, making them perfectly suited for Transformer models." — Source: [a16z Bio Eats World]
  2. On the Complexity of Biological Data: "Biological systems are far more complex than human language, but the underlying generative principles of deep learning still apply." — Source: [No Priors Podcast]
  3. On Cross-Disciplinary Translation: "Moving from pure computer science to biology requires translating computational concepts into physical, molecular realities." — Source: [CNBC]
  4. On AI in Biotech: "We are at an important inflection point where AI can transition from simply analyzing biological data to actively generating new biological designs." — Source: [a16z Raising Health]
  5. On the Data Bottleneck: "The main challenge in biological AI is not just the models, but acquiring high-quality, large-scale empirical data to train them." — Source: [No Priors Podcast]
  6. On Generative Biology: "The goal is to move from discovering medicines by accident to intentionally designing them using generative AI principles." — Source: [Pharmaphorum]
  7. On the Language of Life: "Understanding RNA and DNA through the lens of large language models allows us to speak the language of cellular processes." — Source: [Inceptive]
  8. On Bridging Two Worlds: "The next massive leap in technology will come from the synthesis of advanced deep learning and empirical biochemistry." — Source: [Synthetic Biology Summit]
  9. On Predictable Biology: "AI can help us move biology from an unpredictable, chaotic science into a more predictable, engineering-driven discipline." — Source: [a16z Bio Eats World]
  10. On the Limitations of Human Design: "The design space for biological molecules is too vast for human intuition; it requires high-capacity neural networks to navigate." — Source: [No Priors Podcast]

Part 4: Biological Software as a Concept

  1. On Defining Biological Software: "Basically, it is a way of describing a flavour of medicines or a type of medicines in the broadest sense that is very much akin to software." — Source: [Pharmaphorum]
  2. On Compiling Medicines: "The hope is that we will eventually get to the point where we can have fairly complex definitions or declarations of intended behaviour that are then successfully compiled into descriptions of molecules that exhibit those behaviours inside cells." — Source: [Pharmaphorum]
  3. On Declarative Design: "Instead of guessing chemical structures, we should be able to declare what we want a drug to do and let the system design the sequence." — Source: [No Priors Podcast]
  4. On the Software Analogy: "Biological software means treating the cell as a computer and RNA as the code that instructs it what to produce." — Source: [a16z Raising Health]
  5. On Predictability: "Software engineering is predictable because we understand the compiler; biological software aims to build a reliable compiler for mRNA." — Source: [Inceptive]
  6. On Debugging Biology: "Just as you debug code, biological software requires high-throughput lab testing to iteratively debug the molecular sequences generated by the AI." — Source: [No Priors Podcast]
  7. On Abstraction Layers: "The goal is to create abstraction layers in biology, allowing researchers to design therapeutics without needing to manually engineer every nucleotide." — Source: [a16z Bio Eats World]
  8. On Executable Molecules: "Once designed, these RNA sequences act as executable programs inside the human body to treat diseases." — Source: [CNBC]
  9. On the End of Trial-and-Error: "Treating medicines like software moves the industry away from slow, traditional trial-and-error laboratory processes toward rapid, deterministic design." — Source: [FirstWord Pharma]

Part 5: The Founding and Vision of Inceptive

  1. On Starting Inceptive: "The company was founded to test the hypothesis that deep learning could fundamentally alter how we design RNA-based medicines." — Source: [Pharmaphorum]
  2. On the Alnylam Partnership: "Collaborating with established pharma companies like Alnylam validates the commercial and scientific viability of AI-designed RNA therapeutics." — Source: [Endpoints News]
  3. On Combining Wet and Dry Labs: "Inceptive's core advantage is its tight integration of AI design with rapid, high-throughput empirical testing in the lab." — Source: [a16z Bio Eats World]
  4. On the Foundational Team: "Partnering with biochemist Dr. Rhiju Das was essential to ensuring the computational models were grounded in rigorous physical chemistry." — Source: [Obvious Ventures]
  5. On Foundation Models for Biology: "Inceptive is building foundation models that learn the underlying patterns of biology to optimize the potency of RNA molecules." — Source: [Bio.org]
  6. On Rethinking Drug Discovery: "The vision is not just to speed up the current pipeline, but to completely replace traditional drug discovery with a generative design process." — Source: [CNBC]
  7. On Rapid Iteration: "By synthesizing and testing AI-generated designs in real-time, the company creates a closed-loop system that continuously improves the models." — Source: [No Priors Podcast]
  8. On Scaling Biological Compute: "Just as Google scaled GPUs for text, Inceptive is scaling the physical and computational infrastructure needed to read and write biological sequences." — Source: [Inceptive]
  9. On Broad Applicability: "The platform isn't limited to one disease; because it treats biology as software, it can theoretically design therapies for a vast range of conditions." — Source: [Pharmaphorum]
  10. On Company Mission: "The ultimate mission is to make the design of effective, safe medicines a predictable engineering task rather than a scientific gamble." — Source: [Synthetic Biology Summit]

Part 6: RNA Therapeutics and Drug Discovery

  1. On the Power of RNA: "RNA is the perfect medium for biological software because it is naturally programmable and directly translates sequence into biological function." — Source: [a16z Raising Health]
  2. On mRNA Vaccines: "The success of mRNA vaccines during the pandemic proved the clinical viability of programmable medicines, opening the door for broader applications." — Source: [No Priors Podcast]
  3. On siRNA: "Designing small interfering RNA (siRNA) allows for precise gene silencing, which can be optimized significantly using deep learning models." — Source: [FirstWord Pharma]
  4. On Optimizing Potency: "AI can characterize RNA molecules to maximize their efficacy and minimize off-target effects before they ever enter a clinical trial." — Source: [Endpoints News]
  5. On Structural Bioinformatics: "Deep learning models excel at predicting how a one-dimensional RNA sequence will fold into a three-dimensional functional structure." — Source: [Obvious Ventures]
  6. On Expanding the Pipeline: "AI-driven design allows pharmaceutical companies to expand their therapeutic pipelines much faster than traditional methods allow." — Source: [BioPharma Dive]
  7. On Overcoming Delivery Challenges: "While AI designs the payload, optimizing the stability and delivery of these RNA sequences remains a primary physical challenge." — Source: [a16z Bio Eats World]
  8. On Precision Medicine: "Generating custom RNA sequences tailored to specific genetic profiles brings us closer to true precision medicine." — Source: [CNBC]
  9. On the Speed of Discovery: "What used to take years of laboratory screening can now be narrowed down to the most promising candidates in a matter of weeks using generative models." — Source: [Pharmaphorum]

Part 7: Research Philosophy and Innovation

  1. On Challenging the Status Quo: "True innovation requires challenging existing paradigms and constantly looking for better, more efficient solutions." — Source: [Pharmaphorum]
  2. On Edison’s Ethos: "Another quote I like is from Edison: 'There's a better way. Find it.' That's the essence of innovation." — Source: [Pharmaphorum]
  3. On the Narrative of Revolutions: "We like to tell stories of sudden revolutions, but profound technological shifts are usually the result of long, quiet periods of incremental research." — Source: [Pharmaphorum]
  4. On Scientific Risk: "Pursuing ideas that the broader community considers unviable is often a prerequisite for making foundational leaps in research." — Source: [Reddit AMA]
  5. On Interdisciplinary Teams: "The best research happens when you combine diverse expertise, such as software engineering, deep learning, and biochemistry, into a single, focused team." — Source: [VentureBeat]
  6. On Simplicity: "In machine learning architecture, simpler models that scale well are almost always preferable to complex models with heavy inductive biases." — Source: [Computer History Museum]
  7. On Empirical Validation: "Computational hypotheses are meaningless in biology without rigorous, high-throughput empirical validation in a physical lab." — Source: [a16z Bio Eats World]
  8. On Open Research: "The rapid advancement of AI following the Transformer paper demonstrates the massive industry-wide value of publishing foundational research openly." — Source: [Computer History Museum]
  9. On Long-term Vision: "Solving the biological software problem isn't a short-term project; it requires a generational commitment to bridging software and wet-lab sciences." — Source: [No Priors Podcast]

Part 8: The Future of AI and Biological Engineering

  1. On the Next Decade: "The next ten years of AI will be defined by its application to the physical world, primarily through the engineering of biological systems." — Source: [a16z Raising Health]
  2. On AI Beyond Text: "While LLMs for text are impressive, the most profound impact of generative AI will be in saving lives through novel therapeutics." — Source: [CNBC]
  3. On Generative Medicine: "We are moving toward an era where generative models will custom-design cures for novel pathogens essentially on demand." — Source: [Pharmaphorum]
  4. On the Democratization of Drug Design: "As the compilers for biological software improve, more researchers will be able to design complex medicines without massive legacy pharma infrastructure." — Source: [No Priors Podcast]
  5. On Continuous Learning: "Future biological AI systems will learn continuously from clinical outcomes, constantly refining their understanding of human biology." — Source: [Bio.org]
  6. On Engineering Life: "We are transitioning from a period of observing and documenting life to a period of actively and predictably engineering it." — Source: [Synthetic Biology Summit]
  7. On Expanding Modalities: "While RNA is the starting point, the principles of biological software will eventually apply to proteins, cells, and more complex biological modalities." — Source: [a16z Bio Eats World]
  8. On the Scale of the Challenge: "The complexity of the human body is the ultimate test for deep learning; mastering it will require models far more advanced than today's language models." — Source: [Obvious Ventures]
  9. On the Ultimate Goal: "The endpoint of this research is a world where biological disease can be patched and updated as efficiently and predictably as a software bug." — Source: [Inceptive]