
Lessons from Matei Zaharia
The items below are things Matei Zaharia said in talks and a conversation about building Spark and Databricks, open source, compound AI systems and optimizing agents. — The Databricks Journey with Matei Zaharia, auto-generated captions.
Part 1: The Origins of Apache Spark
- On Understanding How Things Work: In a conversation about the Databricks journey, Zaharia said: "what's the point of you know being a computer scientist if I don't understand how these things work". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Impressive Applications: In a conversation about the Databricks journey, Zaharia said: "these are in some sense like the most impressive Computer Applications". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Berkeley Wanting to Try: In a conversation about the Databricks journey, Zaharia said: "Berkeley was one where they actually wanted to do it because they thought it's interesting". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Who Spark Was For: In a conversation about the Databricks journey, Zaharia said: "that's why spark was aimed at you know data scientists people writing python people writing SQL". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Cheap Storage and Missing Skills: In a conversation about the Databricks journey, Zaharia said: "is good for companies with like 10,000 software Engineers right now but there are so many other people who can easily collect large data sets data storage is super cheap". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Helping People Do Cool Things: In a conversation about the Databricks journey, Zaharia said: "what we wanted when we started the company was you know to help people do cool things with data and AI". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Delivering Value First: In a conversation about the Databricks journey, Zaharia said: "you have to focus on you know delivering something valuable first and foremost". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Building for the Cloud: In a conversation about the Databricks journey, Zaharia said: "we build the whole company around delivering this as a cloud service". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Open Source Not Being a Service: In a conversation about the Databricks journey, Zaharia said: "open- Source projects are not Services". — The Databricks Journey with Matei Zaharia, auto-generated captions.
Part 2: The Philosophy of Open Source
- On Not Being Locked In: In a conversation about the Databricks journey, Zaharia said: "your customers are saying they don't want to be locked in for things like data formats". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Sharing Development Cost: In a conversation about the Databricks journey, Zaharia said: "you can share development cost with other people". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Not Building a Consumer Chatbot: In a conversation about the Databricks journey, Zaharia said: "we're not making a consumer chatbot we're not like disrupting you know Google search". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Efficiency and Reliability: In a conversation about the Databricks journey, Zaharia said: "if you have just better like efficiency better reliability you can train a model or tune a model or deploy a model for lower cost". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Researchy Stuff: In a conversation about the Databricks journey, Zaharia said: "we think this is an area where it's worth doing researchy stuff". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Compound AI Systems: In a conversation about the Databricks journey, Zaharia said: "what I'm calling compound AI system". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Weird Shortcuts: In a conversation about the Databricks journey, Zaharia said: "they will take all kinds of weird shortcuts". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On the Bigger System: In a conversation about the Databricks journey, Zaharia said: "you need to think about the bigger system around it". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On the Doctor Who Has Seen a Million Patients: In a conversation about the Databricks journey, Zaharia said: "I'd rather go to that doctor than someone that you know is really good at playing chess". — The Databricks Journey with Matei Zaharia, auto-generated captions.
Part 3: From Academia to Industry
- On Falling Training Costs: In a conversation about the Databricks journey, Zaharia said: "the cost of getting the same quality model is going down something like 4X year on year". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Depreciating Assets: In a conversation about the Databricks journey, Zaharia said: "if you buy like a bunch of h100s now you know you're just going to hold on to this very fast depreciating asset". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Building Up a Data Set: In a conversation about the Databricks journey, Zaharia said: "if you build up a data set and you keep making it better for like your domain that thing will still be valuable later". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Customizing for Each Domain: In a conversation about the Databricks journey, Zaharia said: "the real value is going to be in customizing these models for each domain". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Not Knowing What a Model Knows: In a conversation about the Databricks journey, Zaharia said: "it's just hard to bring your own intuition to like oh what do I think this thing knows". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Mixing and Matching Components: In a conversation about the Databricks journey, Zaharia said: "can I mix and match components like in programming you got modular components that do one thing". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Judging Multiple Solutions: In a conversation about the Databricks journey, Zaharia said: "you tell it judge these Solutions which one is the best you got to better success rate than if you just call it once". — The Databricks Journey with Matei Zaharia, auto-generated captions.
- On Teaching AI New Tasks: In a conference talk on reflective optimization of agents, Zaharia said: "how can we teach AI new tasks". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Coming Out on a Saturday: In a conference talk on reflective optimization of agents, Zaharia said: "If it just did what we wanted to do, we wouldn't come out on a Saturday". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
Part 4: Data Engineering and The Lakehouse
- On Sample Efficiency: In a conference talk on reflective optimization of agents, Zaharia said: "for many capabilities we care about progress in them is going to be limited by the sample efficiency of our machine learning methods". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Reducing Samples: In a conference talk on reflective optimization of agents, Zaharia said: "we need to reduce the number of samples required". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Trillions of Tokens: In a conference talk on reflective optimization of agents, Zaharia said: "it's literally trillions of tokens you have to practice predicting to pre-train a model". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Experiments You Cannot Run: In a conference talk on reflective optimization of agents, Zaharia said: "you can't run a 100,000 experiments to teach a model something". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Condensing to a Zero or One: In a conference talk on reflective optimization of agents, Zaharia said: "we just condense them down to a zero or one". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Reflecting on the Whole Trajectory: In a conference talk on reflective optimization of agents, Zaharia said: "we actually ask a language model to look at the whole trajectory". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Notes for Yourself: In a conference talk on reflective optimization of agents, Zaharia said: "write down some notes for yourself for the next time you do this task". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Models Reflecting Better: In a conference talk on reflective optimization of agents, Zaharia said: "as language models get better, they should get better at reflecting and at following instructions". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Prompt Updates as Weight Updates: In a conference talk on reflective optimization of agents, Zaharia said: "maybe the weight updates of the future are going to be prompt updates". — Matei Zaharia, Reflective Optimization of Agents with GEPA and DSPy, auto-generated captions.
- On Hardware Trends Reaching Software: In a talk on the future of big data systems, Zaharia said: "a lot of the trends in Hardware as Dave has pointed out so many times then directly affect software and even applications". — Future of Big Data Systems by Spark creator Matei Zaharia, auto-generated captions.
Part 5: Rethinking Large Language Models
- On Compute-Optimized Engines: In a talk on the future of big data systems, Zaharia said: "Big Data engines will need to be compute optimized". — Future of Big Data Systems by Spark creator Matei Zaharia, auto-generated captions.
- On More Than an Open Source Engine: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "you need more than just this open source engine". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Setting Up and Forgetting: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "ideally you know these users would be able to set up a pipeline and then forget about it". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Seven Thousand Companies: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "imagine having to do that for seven thousand companies". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Resilience Without Humans: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "as resilient as possible without human intervention and when there is human intervention it has to be very easy to fix". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On the Vendor Doing Operations: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "the vendor does the operations for them that's basically the difference". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On the Value of Management: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "a lot of the value is the security higher availability and so on out of the management". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Trust in the Cloud: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "otherwise users are not going to trust your cloud". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Undoing a Release: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "whenever you make a release if any of these things are you know getting worse you have to be able to undo the release". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
Part 6: System Design and MLflow
- On Scale-Related Outages: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "at least 70 percent of them are scale related". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Blown-Away Assumptions: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "all the assumptions you might have about sizes of things will probably be blown away". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On One in a Million Failures: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "the weird failure modes that you only see you know happens one in a million times but it means you're gonna see it when you have you know a few million vms". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Overloading the Control Plane: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "each customer app you know if it starts making a lot of calls into the control plane it could certainly overload the services that we're providing". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Customers Owning Storage: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "the nice thing about this is that customers own all the storage". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.
- On Not Just Tech Companies: In a Scale By The Bay talk on scaling Databricks, Zaharia said: "it's definitely not just tech companies who need these anymore". — Scale By The Bay 2020: Matei Zaharia, Scaling Databricks to Run Data and AI Workloads on Millions of VMs, auto-generated captions.