Visual summary of operating lessons from Llion Jones.

Lessons from Llion Jones

The items below are things Llion Jones said in two talks about the transformer, research freedom and the search for better architectures. — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.

Part 1: The Transformer and Its Origins

  1. On the Wrong Lesson: In a talk on the history of the transformer paper, Jones said: "I really worry that people have taken the wrong lesson away from this paper and I want to take the opportunity to try to correct this misconception". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
  2. On Designing for Hardware: In a talk on the history of the transformer paper, Jones said: "we literally designed the transformer to work as fast as possible on on the hardware that was available at the time". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
  3. On Hardware for the Transformer: In a talk on the history of the transformer paper, Jones said: "Now, of course, the hardware is designed for the transformer". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
  4. On Ablation: In a talk on the history of the transformer paper, Jones said: "You want to start taking parts of the network out to see how much worse it's getting and what the important part is". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
  5. On the Environment That Made It: In a TED talk, Jones said: "That is the kind of environment that allowed the transformer to come into existence". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  6. On Bottom-Up Research: In a podcast interview about Continuous Thought Machines, Jones said: "It was a bunch of people talking over lunch". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions

Part 2: The Transformer Rut and Success Capture

  1. On Better Alternatives: In a TED talk, Jones said: "If the current technology was worse, more people would be looking for better". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  2. On Not Being Done: In a TED talk, Jones said: "I and several other many other researchers believe we're not done and we should be looking for better". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  3. On Remaining Research: In a TED talk, Jones said: "there's still plenty of very important research to be done on the current technology and will bring a lot of value in the coming years". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  4. On Moving On: In a talk on the history of the transformer paper, Jones said: "We have to move on and do something new which is what a part of my company is trying to do". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
  5. On a Local Minimum: In a podcast interview about Continuous Thought Machines, Jones said: "I'm really worried that we're kind of stuck in this local minimum now". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
  6. On Not Being Done: In a podcast interview about Continuous Thought Machines, Jones said: "I don't think that this is the final architecture". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
  7. On Unused Talent: In a podcast interview about Continuous Thought Machines, Jones said: "they're not using their talents". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
  8. On Missing Freedom: In a podcast interview about Continuous Thought Machines, Jones said: "are not actually giving people the freedom that they need to do the research that they probably want to do". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions

Part 3: Continuous Thought Machines

Part 4: Sakana AI and Nature-Inspired Architecture

  1. On Research That Would Not Happen: In a TED talk, Jones said: "You should only do research that wouldn't happen if you weren't working on it". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  2. On Freedom From Pressure: In a TED talk, Jones said: "no pressure to work on any particular project, publish a number of papers, to push a certain number up". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  3. On Autonomy and Talent: In a TED talk, Jones said: "Talented, intelligent people, ambitious people will naturally seek out this kind of environment with high autonomy". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  4. On Trying an Idea for a Week: In a TED talk, Jones said: "I remember the employee coming to me with the idea and I said okay work on it for a week and we'll see what happens". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  5. On Bold Business Leaders: In a TED talk, Jones said: "Business leaders, are you bold enough to create businesses that create these kind of environments". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
  6. On Protecting Freedom: In a podcast interview about Continuous Thought Machines, Jones said: "making sure that we protect the freedom that the researchers currently have". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
  7. On Privilege: In a podcast interview about Continuous Thought Machines, Jones said: "it's a privilege really that we have the resources to be able to do that". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
  8. On Narrowing Freedom: In a podcast interview about Continuous Thought Machines, Jones said: "more and more pressure comes in and it narrows the freedom". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
  9. On Working on What Matters: In a podcast interview about Continuous Thought Machines, Jones said: "I want you to work on what you think is interesting and important and I mean it". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions