
Lessons from Llion Jones
The items below are things Llion Jones said in two talks about the transformer, research freedom and the search for better architectures. — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
Part 1: The Transformer and Its Origins
- On the Wrong Lesson: In a talk on the history of the transformer paper, Jones said: "I really worry that people have taken the wrong lesson away from this paper and I want to take the opportunity to try to correct this misconception". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
- On Designing for Hardware: In a talk on the history of the transformer paper, Jones said: "we literally designed the transformer to work as fast as possible on on the hardware that was available at the time". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
- On Hardware for the Transformer: In a talk on the history of the transformer paper, Jones said: "Now, of course, the hardware is designed for the transformer". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
- On Ablation: In a talk on the history of the transformer paper, Jones said: "You want to start taking parts of the network out to see how much worse it's getting and what the important part is". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
- On the Environment That Made It: In a TED talk, Jones said: "That is the kind of environment that allowed the transformer to come into existence". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Bottom-Up Research: In a podcast interview about Continuous Thought Machines, Jones said: "It was a bunch of people talking over lunch". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
Part 2: The Transformer Rut and Success Capture
- On Better Alternatives: In a TED talk, Jones said: "If the current technology was worse, more people would be looking for better". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Not Being Done: In a TED talk, Jones said: "I and several other many other researchers believe we're not done and we should be looking for better". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Remaining Research: In a TED talk, Jones said: "there's still plenty of very important research to be done on the current technology and will bring a lot of value in the coming years". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Moving On: In a talk on the history of the transformer paper, Jones said: "We have to move on and do something new which is what a part of my company is trying to do". — Transformer Anecdotes: Attention Is All You Need | Llion Jones, Sakana AI, auto-generated captions.
- On a Local Minimum: In a podcast interview about Continuous Thought Machines, Jones said: "I'm really worried that we're kind of stuck in this local minimum now". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
- On Not Being Done: In a podcast interview about Continuous Thought Machines, Jones said: "I don't think that this is the final architecture". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
- On Unused Talent: In a podcast interview about Continuous Thought Machines, Jones said: "they're not using their talents". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
- On Missing Freedom: In a podcast interview about Continuous Thought Machines, Jones said: "are not actually giving people the freedom that they need to do the research that they probably want to do". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
Part 3: Continuous Thought Machines
Part 4: Sakana AI and Nature-Inspired Architecture
- On Research That Would Not Happen: In a TED talk, Jones said: "You should only do research that wouldn't happen if you weren't working on it". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Freedom From Pressure: In a TED talk, Jones said: "no pressure to work on any particular project, publish a number of papers, to push a certain number up". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Autonomy and Talent: In a TED talk, Jones said: "Talented, intelligent people, ambitious people will naturally seek out this kind of environment with high autonomy". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Trying an Idea for a Week: In a TED talk, Jones said: "I remember the employee coming to me with the idea and I said okay work on it for a week and we'll see what happens". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Bold Business Leaders: In a TED talk, Jones said: "Business leaders, are you bold enough to create businesses that create these kind of environments". — How Competition Is Stifling AI Breakthroughs | Llion Jones | TED, auto-generated captions.
- On Protecting Freedom: In a podcast interview about Continuous Thought Machines, Jones said: "making sure that we protect the freedom that the researchers currently have". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
- On Privilege: In a podcast interview about Continuous Thought Machines, Jones said: "it's a privilege really that we have the resources to be able to do that". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
- On Narrowing Freedom: In a podcast interview about Continuous Thought Machines, Jones said: "more and more pressure comes in and it narrows the freedom". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions
- On Working on What Matters: In a podcast interview about Continuous Thought Machines, Jones said: "I want you to work on what you think is interesting and important and I mean it". — Llion Jones interview on Machine Learning Street Talk about Continuous Thought Machines, auto-generated captions