Lessons from Francois Chollet

François Chollet, a prominent artificial intelligence researcher at Google, creator of the Keras deep-learning library, and a leading voice in the conversation around the future of AI, has shared numerous insightful perspectives over the years. His thoughts challenge the current hype in the field, offering a more grounded and nuanced understanding of intelligence, both human and artificial.

On the Nature of Intelligence

"Intelligence is not skill itself; it's not what you know. It's the skill-acquisition efficiency. It's your ability to turn information into skills."
- Source: Interview with Lex Fridman
"The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty." [1]
- Source: "On the Measure of Intelligence" (2019 paper) [1][2]
"Real intelligence isn't about memorizing information or having lots of knowledge - it's about being able to handle new situations effectively." [3]
- Source: MLST Podcast [3]
"Human cognition is capable of extreme generalization, quickly adapting to radically novel situations, or planning for very long-term future situations." [4]
- Source: Personal Website (fchollet.com) [4]
"Intelligence is your ability to handle novelty, to deal with situations you've not seen before and come up on the fly with models in the context of that situation." [5]
- Source: Interview on #1minPapers [5]
"Goal-setting is an important component of an intelligent agent." [5]
- Source: Interview on #1minPapers [5]
"Intelligence lies in broad or general-purpose abilities; it is marked by skill-acquisition and generalization, rather than skill itself." [2]
- Source: ARC Prize Website [2]
"You are born with this skill acquisition mechanism, but the skill acquisition mechanism itself is something that gets refined and improved through experience."
- Source: MLST Podcast
"Consciousness develops gradually in children rather than appearing all at once." [3]
- Source: MLST Podcast [3]
"Consciousness exists in degrees - animals have it to some extent, and even human consciousness varies with age and circumstances." [3]
- Source: MLST Podcast [3]

On Large Language Models (LLMs) and AGI

"LLMs are not AGI, but they can be tremendously useful for automating known tasks within their training data distribution." [6]
- Source: daily.dev [6]
"The claim that we're already there [at creating AGI], or that LLMs have high schooler level intelligence, that's kind of absurd. I can't even fathom how I can make such claims." [7]
- Source: Mindscape Podcast [7]
"There is no technology today that is on the path to AGI. There is nothing that if you just scale it, it gives you intelligence." [7]
- Source: Mindscape Podcast [7]
"If you ask them to solve problems that are significantly different from anything they've seen in their training, they will fail." [3]
- Source: MLST Podcast [3]
"LLMs are more like sophisticated memory and pattern-matching systems than truly intelligent beings." [3]
- Source: MLST Podcast [3]
"Large language models in some sense, memorize lots of things. They know a lot of facts about the world, and they're super good at interpolating between things that they know." [8]
- Source: Mindscape Podcast [8]
"There's no amount of stored, memorized programs where you develop suddenly the ability to synthesize your own programs on the fly. It's just not how it works." [7]
- Source: Mindscape Podcast [7]
"AGI is going to be a kind of super-competent scientist." [9]
- Source: TIME Magazine [9]
"For all the progress made, it seems like almost all important questions in AI remain unanswered. Many have not even been properly asked yet." [10]
- Source: Quote Catalog [10]
"The notion of intelligence explosion comes from a profound misunderstanding of both the nature of intelligence and the behavior of recursively self-augmenting systems." [4]
- Source: Personal Website (fchollet.com) [4]

On Machine Learning and Deep Learning

"Uncrumpling paper balls is what machine learning is about: finding neat representations for complex, highly folded data manifolds." [11][12]
- Source: "Deep Learning with Python" [11][12]
"In ML, where algorithms get published quickly and state-of-the-art frameworks are open-source, there isn't any first-mover advantage." [10]
- Source: Quote Catalog [10]
"Deep learning is a mathematical framework for learning representations from data."
- Source: "Deep Learning with Python"
"Not all problems can be solved; just because you've assembled examples of inputs X and targets Y doesn't mean X contains enough information to predict Y." [11][12]
- Source: "Deep Learning with Python" [11][12]
"Machine learning, on the other hand, is applicable to datasets where the past is a good predictor of the future." [11][12]
- Source: "Deep Learning with Python" [11][12]
"Currently, most of the job of a deep-learning engineer consists of munging data with Python scripts and then tuning the architecture and hyperparameters of a deep network at length to get a working model." [12]
- Source: "Deep Learning with Python" [12]
"Your initial decisions are almost always suboptimal, even if you have good intuition." [11]
- Source: "Deep Learning with Python" [11]
"It shouldn't be your job as a human to fiddle with hyperparameters all day—that is better left to a machine." [11]
- Source: "Deep Learning with Python" [11]
"We will move away from having on one hand 'hard-coded algorithmic intelligence' (handcrafted software) and on the other hand 'learned geometric intelligence' (deep learning)." [4]
- Source: Personal Website (fchollet.com) [4]
"We will have instead a blend of formal algorithmic modules that provide reasoning and abstraction capabilities, and geometric modules that provide informal intuition and pattern recognition capabilities." [4]
- Source: Personal Website (fchollet.com) [4]

On Evaluation and Benchmarking

"If you want to test actual intelligence you need problems that are novel, problems where the test taking system or human being cannot have memorized the solution." [7]
- Source: Mindscape Podcast [7]
"If you want to actually measure intelligence, you have to look at how efficiently the system acquires new skills given a limited amount of data." [3]
- Source: MLST Podcast [3]
"Measuring task-specific skill is not a good proxy for intelligence. Skill is heavily influenced by prior knowledge and experience." [2]
- Source: ARC Prize Website [2]
"If you want to benchmark intelligence you need a different kind of game, a game that you cannot prepare for." [3]
- Source: MLST Podcast [3]
The Abstraction and Reasoning Corpus (ARC) is designed to be resistant to memorization. [3]
- Source: MLST Podcast [3]
"Winning is not so much about how good your theoretical vision is, it's about how much contact with reality your vision has been through." [13]
- Source: Quora, via Kaggle discussion [13]
"You don't lose to people who are smarter than you, you lose to people who have iterated through more experiments than you did, refining their models a little bit each time." [13]
- Source: Quora, via Kaggle discussion [13]
"If you ranked teams on Kaggle by how many experiments they ran, I'm sure you would see a very strong correlation with the final competition leaderboard." [13]
- Source: Quora, via Kaggle discussion [13]
"Trying to use machine learning to beat markets, when you only have access to publicly available data, is a difficult endeavor, and you're likely to waste your time and resources with nothing to show for it." [11][12]
- Source: "Deep Learning with Python" [11][12]
"Always remember that when it comes to markets, past performance is not a good predictor of future returns—looking in the rear-view mirror is a bad way to drive." [11][12]
- Source: "Deep Learning with Python" [11][12]

On the Future of AI and its Impact

"What's holding back research isn't a lack of verbose, low-signal, high-noise papers. Using LLMs to automatically generate 100x more of those will not accelerate science, it will slow it down." [14]
- Source: Simon Willison's Weblog [14]
"Don't use AI as a tool to manipulate your users; instead, give AI to your users as a tool to gain greater agency over their circumstances." [4]
- Source: Personal Website (fchollet.com) [4]
"Design for ethics. Bake your values into your creations." [4]
- Source: Personal Website (fchollet.com) [4]
"The intelligence here is the mind of the programmer that developed that program." [15]
- Source: Mindscape Podcast [15]
"I think it's kind of fascinating to me that when the state-of-the-art LLMs go down it's actually kind of like an intelligence brownout in the world." [16]
- Source: Y Combinator Talk by Andrej Karpathy, referencing Chollet's ideas [16]
"The future of AI will be a fusion of new methods with deep learning and LLMs." [9]
- Source: TIME Magazine [9]
"You cannot predict when [AGI] will arrive because you need to invent something new. But maybe we'll invent it next year." [7]
- Source: Mindscape Podcast [7]
"Like most things, API design is not complicated, it just involves following a few basic rules. They all derive from a founding principle: you should care about your users." [4]
- Source: Personal Website (fchollet.com) [4]
"Leveraging technology, in particular AI, to help people gain greater agency over their circumstances and reach their full potential." [4]
- Source: Personal Website (fchollet.com) [4]
"The programmer can actually invent anything, adapt to anything, because it has general intelligence, right? That's really the difference." [7]
- Source: Mindscape Podcast [7]

Learn more:

Lessons from Francois Chollet

On the Nature of Intelligence

On Large Language Models (LLMs) and AGI

On Machine Learning and Deep Learning

On Evaluation and Benchmarking

On the Future of AI and its Impact

Written by Antoine Buteau

Lessons from Rob Bier

Lessons from Mike Knoop