Jeff Dean writes on his Google Research page, "I joined Google in mid-1999, and I'm currently Google's Chief Scientist, focusing on AI advances for Google DeepMind and Google Research." The items below are things he said in talks and interviews, points from his own slides, and facts from that biography. — Jeffrey Dean - Google Research (archived copy, 2025).

Infographic for "Lessons from Jeff Dean".

On Machine Learning and Artificial Intelligence

  1. "In recent years, I think machine learning has really changed our expectations of what we think of computers being able to do." [1]
  2. "Every time we scale things up, things get better. All of a sudden, new capabilities emerge or the accuracy of some problem reaches a threshold, where before it was kind of unusable and now all of a sudden it becomes usable, and that enables new kinds of things." [1][2]
  3. "If you think back 10 or 15 years ago, speech recognition kind of worked, but it wasn't really seamless — it made lots of errors. Computers didn't really understand images from the pixel level of what was in that image." [1]
  4. On computers that can see: In his "Exciting Trends in Machine Learning" talk, he said "we now have computers that can see and sense". — Exciting Trends in Machine Learning - Jeff Dean (Google).
  5. On scale: In the same talk, he named "increasing scale" and "larger and more interesting and richer data sets, larger scale machine learning models" and "scaling all of those things tends to deliver better results". — Exciting Trends in Machine Learning - Jeff Dean (Google).
  6. "If you can't understand what's in information then it's going to be very difficult to organize it." [3]
  7. "They've learned they can take a whole bunch of subsystems, some of which may be machine learned, and replace it with a much more general end-to-end machine learning piece. Often when you have lots of complicated subsystems there's usually a lot of complicated code to stitch them all together. It's nice if you can replace all that with data and very simple algorithms." [3]
  8. "By increasing the size of the model it can remember not just the obvious things but it can remember the subtle kinds of patterns that occur maybe in only a tiny fraction of the examples in the data set." [3]
  9. "The reason neural networks didn't take off in the 90s was a lack of computational power and a lack of large interesting data sets." [3]
  10. "One of the things that's really happened in the last 5 or 6 years, that has caused machine learning to really take off, is that we now have enough computational power, and large enough and interesting real-world datasets, to solve problems that previously we weren't able to solve in any other way: problems in computer vision, speech recognition and language understanding." [4]
  11. "There have been a number of surprising results in the last 5 years or so in machine learning— things that I didn't think computers could really do that all of a sudden they can now do." [4]
  12. "Recently, advances in machine learning have mostly come in areas where we have large labeled datasets: problems where we have the inputs and then the desired outputs for those problems." [4]
  13. "We think there's going to be hundreds of thousands of organizations impacted by machine learning in the next 5 or 10 years." [4]
  14. On simple techniques and data: In the same talk, he said that one lesson was that "simple techniques over large amounts of data are very effective". — Exciting Trends in Machine Learning - Jeff Dean (Google).
  15. "If you only have 10 examples of something, it's going to be hard to make deep learning work. If you have 100,000 things you care about, records or whatever, that's the kind of scale where you should really start thinking about these kinds of techniques." VentureBeat
  16. "There's a lot of work in machine learning systems that is not actually machine learning." VentureBeat
  17. "Supervised learning works so well when you have the right data set, but ultimately unsupervised learning is going to be a really important component in building really intelligent systems - if you look at how humans learn, it's almost entirely unsupervised." MIT Technology Review
  18. "I think true artificial general intelligence would be a system that is able to perform human-level reasoning, understanding, and accomplishing of complicated tasks. We are clearly not there yet." [6]
  19. "Bigger model, more data, better results,’ which has been sort of relatively true for the last 12 or 15 years." [7]
  20. "The models that we have are capable of doing really interesting things. They can't solve every problem; they can solve a growing set of problems year over year because the models get better, we have better algorithmic improvements that show us how to train larger models with the same compute cost, more capable models." [7]
  21. On the future recipe for machine learning solutions: Pavel Surmenok's notes on Dean's 2017 talk record his slide as "Current: solution = machine learning expertise + data + computation" and "Future: solution = data + 100X computation". — Pavel Surmenok: Jeff Dean's Talk on Large-Scale Deep Learning (Sept 2017).

On Systems and Infrastructure

  1. "I've always liked code that runs fast." Slate
  2. "In a lot of these areas, from machine translation to search quality, you're always trying to balance what you can do computationally with each query." Slate
  3. "Some things are easier to parrellelize than others. It's pretty easy to train up 100 models and pick the best one. If you want to train one big model but do it on hundreds of machines, that's a lot harder to parallelize." Business Journal
  4. "Traditional systems I think like operating systems compilers memory allocators and so on they don't make extensive use of machine learning today uh but that should change and is starting to change." [9]
  5. "We know parallelism is a really good solution for a lot of kinds of large scale problems." [9]
  6. "In general I think we need to use much more compute and much more parallelism." [9]
  7. A key design lesson is to plan for inevitable failures in large-scale systems, such as disk failures and server outages. [10]
  8. He provided a widely cited list of "Numbers Everyone Should Know" which gives latency numbers for common computer operations, crucial for performance engineering. [10]
  9. "A typical first year for a new cluster" involves numerous failures, including overheating, PDU failures, and rack failures, emphasizing the need for fault-tolerant design. [10]
  10. "Often when you have lots of complicated subsystems there's usually a lot of complicated code to stitch them all together. It's nice if you can replace all that with data and very simple algorithms." [3]
  11. When building large-scale systems, it's important to decompose them into services. [11]
  12. At large scale, failures are common, so you need to minimize them and have fast recovery mechanisms. [12]
  13. "We needed about a million times as much processing power to get them to really start to work well on real problems that you might sort of care about." [13]
  14. The evolution of computing is shifting to both very small devices and large, consolidated computing farms. [14]
  15. When designing world-wide systems, a key challenge is the automatic and dynamic placement of data and computation to minimize latency and cost under various constraints. [14]

On Innovation and Research

  1. "It's nice to have short-term to medium-term things that we can apply and see real change in our products, but also have longer-term, five to 10 year goals that we're working toward." Business Journal
  2. On the pace of the field: High Scalability's write-up of his talk gives this as a paraphrase of Dean: "The machine learning community moves really really fast". — High Scalability: Jeff Dean on Large-Scale Deep Learning at Google.
  3. Co-founding Google Brain: On his Google Research page, he wrote "In 2011, I co-founded the Google Brain project/team, focused on making progress towards intelligent machines". — Jeffrey Dean - Google Research (archived copy, 2025).
  4. Co-creating distillation: On his Google Research page, he wrote "I am one of the co-creators of a machine learning technique called distillation, a now-widely-used approach for transferring the knowledge from one neural network to another". — Jeffrey Dean - Google Research (archived copy, 2025).
  5. Neural machine translation: On his Google Research page, he wrote "I was one of many who helped to convert the Google Translate system over to using a neural machine translation system, with further significant gains to translation quality". — Jeffrey Dean - Google Research (archived copy, 2025).
  6. AI for broad benefit: On his Google Research page, he lists among his areas of focus "applications of AI to problems that help billions of people in societally beneficial ways". — Jeffrey Dean - Google Research (archived copy, 2025).
  7. Building TPUs: On his Google Research page, he wrote that he identified early that "creating efficient systems was going to require building customized accelerator hardware, leading to a long line of TPU processors". — Jeffrey Dean - Google Research (archived copy, 2025).
  8. Naming DistBelief: On the Google DeepMind podcast, he said "I actually named it DistBelief because it's" and "partly because people didn't think it was really going to work". — Google DeepMind podcast: Decoding Google Gemini with Jeff Dean.

Learn more:

  1. Google Chief Scientist Jeff Dean discusses AI and ML during Rice visit | Computer Science
  2. Exciting Trends in Machine Learning - Jeff Dean (Google) - YouTube
  3. Jeff Dean on Large-Scale Deep Learning at Google - High Scalability -
  4. Jeff Dean on machine learning, part 1: surveying the landscape | Google Cloud Blog
  5. Q+A With Jeff Dean: The Brain Behind Google's Artificial Intelligence - Forbes
  6. Google's Jeff Dean on the Coming Era of Virtual Engineers | Sequoia Capital
  7. Jeff Dean's Talk on Large-Scale Deep Learning | Pavel Surmenok
  8. Jeff Dean: AI will Reshape Chip Design — NeurIPS 2024 - YouTube
  9. Jeff Dean: Design Lessons and Advice from Building Large Scale Distributed Systems
  10. Building Software Systems at Google and Lessons Learned
  11. Jeff Dean (Google Research & Deep Mind): Advances in ML for Systems and Systems for ML | MLSys 2024 - YouTube
  12. Decoding Google Gemini with Jeff Dean - YouTube
  13. Designs, Lessons and Advice from Building Large Distributed Systems - CS@Cornell
  14. Jeffrey Dean - Google Research (archived copy)