Visual summary of operating lessons from Jakob Uszkoreit.

Jakob Uszkoreit is the co-founder and CEO of Inceptive, which builds "biological software"; he was one of the eight authors of "Attention Is All You Need," and wrote that his main contribution was "to propose that it would be possible to replace recurrence" with self-attention. — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024).

Part 1: The Transformer Architecture and Deep Learning

  1. On Attention Mechanisms: Uszkoreit said "there has to be a different way that doesn't require the RNN's recurrence in length, or recurrence in sequence of, say words or pixels, but that actually processes inputs and outputs in a more parallel way." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript).
  2. On the Transformer's Initial Reception: Uszkoreit said "People raised their eyebrows, because it dumped out all the existing neural architectures." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024).
  3. On Band-Aids: Uszkoreit said of the methods in use "The methods we were applying were basically Band-Aids." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024)
  4. On pushing further: Uszkoreit said "It was an amazing success in many ways, but I didn't want to leave it there." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024)
  5. On careful implementation: Uszkoreit said "These theoretical or intuitive mechanisms, like self-attention, always require very careful implementation, often by a small number of experienced 'magicians,' to even show any signs of life." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024)
  6. On Parallelization: Uszkoreit said "now you were not actually bound to sequentially compute representations, say, word for word." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript).
  7. On Self-Attention: Uszkoreit said of self-attention "It actually considers everything and gives you an efficient way of looking at many inputs at the same time and then taking something out in a pretty selective way." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024).
  8. On Architectural Simplicity: Uszkoreit said "we kept consistently actually ending up with models that were simpler." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript).
  9. On hardware fit: Uszkoreit said the idea was to cater "to the strengths of modern accelerator hardware." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript)
  10. On scaling in data: Uszkoreit said the earlier approaches "had incredible issues even on the most modern GPUs at the time, when it came to scaling in terms of data." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript)
  11. On the Name Transformer: Uszkoreit said "I had two little Transformer toys as a very young kid." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024).

Part 2: The Evolution of AI at Google Brain

  1. On the Original Transformer Team: Uszkoreit said "I believe it's actually a testament to the fact that while our interests overlap a lot, we also are a very diverse group" and "The fact that this didn't happen is, in my book, the specific reason that the group is still incredibly effective." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023).
  2. On the group chat: Uszkoreit said "We have a little group chat" and "We share notes, advise each other quite often." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023)
  3. On right conditions: Uszkoreit said "It's getting people who are super excited about something who are at the right point in their life." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024)
  4. On why Google did not ship: Uszkoreit said "The question is, why didn't we do anything with the fact that we had seen it? The answer is tricky." — WIRED: 8 Google Employees Invented Modern AI. Here's the Inside Story (Steven Levy, March 2024)
  5. On Years of Prior Work: Uszkoreit said "My team at Google was pushing on attention models for years before that paper." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024).
  6. On shoulders of giants: Uszkoreit said "we were standing on the shoulders of giants." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024)
  7. On industrial research: Uszkoreit said "This wasn't the one big spark of imagination and of creativity that sets it all off." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript)
  8. On Leaving Google: Uszkoreit said "I left not because of what I didn't like at Google as much as I left because of what I felt I absolutely had to do elsewhere, which is to start Inceptive." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024).
  9. On experiment-happy: Uszkoreit said "it is so important to remain experiment-happy and to remain failure-happy." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024)
  10. On Google Translate: Uszkoreit said "When we first launched Google Translate, the very first versions, it was a party joke at best." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024)
  11. On a moral obligation: Uszkoreit said "an enormous, not only opportunity, but a moral obligation in a sense, to do something that was better done outside." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024)
  12. On Incremental Progress: "Overall, this is a very typical phenomenon where we want to see revolutions. That's to a large extent because that's how we tell stories, but the truth is that almost all the 'revolutions' that I've been a witness to are actually incremental progress." — [pharmaphorum].

Part 3: The Intersection of AI and Biology

  1. On Biology as Sequences: Uszkoreit said of RNA "it's a biopolymer with four different bases." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript).
  2. On AI in Biotech: Uszkoreit said "I definitely think we're headed into the inflection point territory." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript).
  3. On the Data Bottleneck: Uszkoreit said "The biggest hurdle is the lack of data." — pharmaphorum: The big interview: Inceptive's Jakob Uszkoreit on the promise of biological software.
  4. On things that just work: Uszkoreit said "The biggest misconception today, is that these things 'just work'." — pharmaphorum: The big interview: Inceptive's Jakob Uszkoreit on the promise of biological software
  5. On single cells: Uszkoreit said "We have no instance that is able to understand how even just a single cell works." — pharmaphorum: The big interview: Inceptive's Jakob Uszkoreit on the promise of biological software
  6. On creating the data: Uszkoreit said "I believe we need to create the lion's share of data for training the models that we're training." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript)

Part 4: Biological Software as a Concept

  1. On Defining Biological Software: "Basically, it is a way of describing a flavour of medicines or a type of medicines in the broadest sense that is very much akin to software." — [pharmaphorum].
  2. On compiling behavior: Uszkoreit said "Biological software is a little bit like computer software in that you have some specification of the behavior that you want." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024)
  3. On Compiling Medicines: "The hope is that we will eventually get to the point where we can have fairly complex definitions or declarations of intended behaviour that are then successfully compiled into descriptions of molecules that exhibit those behaviours inside cells." — [pharmaphorum].

Part 5: The Founding and Vision of Inceptive

  1. On Combining Wet and Dry Labs: Uszkoreit said "most of our work, I would say — happens on the 'beach'... where the wet" and the dry labs "meet in harmony." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023).
  2. On a Beginner's Mindset: Uszkoreit said "We've really had to approach this with a beginner's mindset." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023).
  3. On a discipline without a name: Uszkoreit said "this is a discipline that quite possibly will have a name a few years down the road but doesn't have one yet." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023)
  4. On a rising tide: Uszkoreit said "This is really a tide that lifts all the boats." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023)

Part 6: RNA Therapeutics and Drug Discovery

  1. On mRNA Vaccines: Uszkoreit said "A very, very simple example of that are the mRNA COVID vaccines where the program says, 'Make this modified viral antigen' and then our cells make that protein." — Ars Technica: ChatGPT's success could have come sooner, says former Google AI researcher (November 2024).

Part 7: Research Philosophy and Innovation

  1. On Edison’s Ethos: "Another quote I like is from Edison: 'There's a better way. Find it.' That's the essence of innovation." — [pharmaphorum].
  2. On Interdisciplinary Teams: Uszkoreit said "It's not the case that we can just go and apply RNA biochemistry methods and then do novel deep learning on it." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023).
  3. On Long-term Vision: Uszkoreit said "It won't be built overnight – it's a gradual process." — pharmaphorum: The big interview: Inceptive's Jakob Uszkoreit on the promise of biological software.
  4. On advertising: Uszkoreit said "If we're successful, no advertising will be necessary." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023)
  5. On moral obligation: Uszkoreit said that after his daughter was born "anybody with kids can relate to looking at certain things quite differently, very suddenly." — VentureBeat: This ex-Googler helped launch the Gen AI boom. Now he wants to reinvent vaccines (September 2023)

Part 8: The Future of AI and Biological Engineering

  1. On the Next Decade: Uszkoreit said "when we look back 10 years from now, it'll seem like a revolution at least as large and as expansive as the one that we think we've seen in the last 10 years." — a16z Bio Eats World: Using AI to Take Bio Farther, with Jakob Uszkoreit (January 2023, published transcript).