Visual summary of operating lessons from Ion Stoica.

Lessons from Ion Stoica

Ion Stoica is a UC Berkeley professor who co-created Apache Spark, Mesos, and Ray, then co-founded Databricks and Anyscale to turn those academic projects into commercial enterprises. These notes track his approach to infrastructure design and open-source economics as the industry shifts from big data to distributed AI.

Part 1: The Philosophy of Open Source

  1. On collaboration in AI: Stoica said "Unlike the internet, AI development is siloed across companies. If you believe that AI is so important and you want to accelerate progress in a responsible way, you have to have all of the best minds in AI collaborating." — Columbia Engineering: Ion Stoica highlights open-source efficiency across the AI stack (April 2025).
  2. On shared infrastructure: He added "They can't do that without shared artifacts and a shared infrastructure that everyone can access." — Columbia Engineering: Ion Stoica highlights open-source efficiency across the AI stack (April 2025).
  3. On winning on merit: On partnerships with possible competitors, Stoica said "Well, if someone else is building a better product for Spark, then we deserve to lose," because they trusted they could "build the best products for Spark." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  4. On the ecosystem first: He described the plan: "we make Spark successful, you know, hopefully de facto standard for data processing, and then we make Databricks the best place to run Spark." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  5. On hosting the open source: Stoica said hosted offerings "are more valuable when the solution is distributed, because then the value is to manage a cluster." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  6. On lock-in: He said enterprises prefer "more control, security" and "less kind of locking" when open-source models are competitive with proprietary ones. — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).

Part 2: Distributed Systems & Big Data

  1. On the developer gap: Stoica said "writing distributed applications is hard. And if more and more applications are going to become distributed, there is an increased gap between the desire of people to scale up their workload by distributing it and the expertise the typical programmer has." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  2. On keeping data in memory: Recalling the Netflix-challenge students who found Hadoop too slow, he said Matei Zaharia "put together something quickly for solving this problem in which the data was kept in memory." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  3. On lineage: Stoica explained that with side-effect-free tasks "if for instance you lose some data, you can keep the lineage; the chain of tasks which created that data in the first place. And then you can re-execute the task." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  4. On the end of MapReduce: He said that at the time of Spark "People knew that Hadoop Map Reduce, its time had passed, so to speak." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  5. On sharing a cluster: Stoica recalled that Mesos began as a class project: "you wanted the same cluster to be able to run multiple frameworks, to share the same cluster across different frameworks." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  6. On abstracting complexity: Of increasingly heterogeneous AI infrastructure he said "the software stack has to abstract away that complexity for the developers. There is no way around." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  7. On state on the GPU: He said that as people asked "I want my state actually to remain on the GPU," pure tasks no longer worked and fault tolerance became "much harder." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  8. On networking: Stoica said of today's AI infrastructure "it's not only about compute, it's about networking," citing Infiniband and RDMA. — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).

Part 3: The Rise of AI Infrastructure

  1. On Python's libraries: Stoica said popular languages like Java and Python succeed not because they are the best languages: "They're very successful because they have a strong ecosystem of libraries." Ray, he said, has the same kind of ecosystem. — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  2. On the infinite laptop: He said Anyscale tries to "provide the developer the illusion of the experience of an infinite laptop," keeping the same tools while the code runs in the cloud. — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  3. On Spark being too rigid for AI: On why Ray was needed, Stoica said "With reinforcement learning, the computation model you need is much more complex" and "Spark was too rigid." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  4. On distribution being unavoidable: At VentureBeat's Transform 2022, Stoica said "There is no other way to support these workloads without distribution – it's as simple as that." — VentureBeat report on his Transform 2022 talk: The future of AI is distributed (July 2022).
  5. On predictable AI systems: Asked what excites him about AI, Stoica said it is "how do you make these AI systems, more predictable, accurate, verifiable? How you can debug these systems?" — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  6. On pipelines of distributed systems: He said an ML application has stages such as labeling, training and tuning: "Each of these stages you need to scale, each of them you typically have a different distributed system." — VentureBeat report on his Transform 2022 talk: The future of AI is distributed (July 2022).
  7. On demands outgrowing one machine: Stoica said AI applications' demands "are growing much quicker than the capabilities of a single process or a single node, even if you considered accelerators." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).

Part 4: Sky Computing & The Future of Cloud

  1. On an internet for clouds: Previewing sky computing in 2022, Stoica said "think about internet for the clouds," an "intercloud layer which will abstract away clouds." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  2. On proprietary interfaces: The white paper he co-authored says early cloud providers offered "proprietary interfaces," and measures such as "much higher data-transfer fees for egress than ingress" are "part of a broader strategy to lock customers into their current cloud provider." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).
  3. On splitting a pipeline across clouds: The paper's example splits a machine learning pipeline across Azure, GCP and AWS, and says that "even with current egress charges and data transfer delays, splitting the computation across clouds can provide lower cost and faster processing by factors of two to three." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).
  4. On compatibility without one standard: The paper argues that "a comprehensive compatibility standard supported by all clouds... is neither likely to emerge nor necessary to allow customers to move workloads between clouds." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).
  5. On intercloud brokers: In the Sky model "users send a job and its description to an intercloud broker, which then selects the clouds on which various parts of the job are run and then manages their execution." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).
  6. On the Internet parallel: Stoica said "what internet did is it stitched together a bunch of disparate networks and provides the abstraction of a single network." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  7. On markets maturing: The paper notes that "telephony, the Internet, and PCs all started with a single provider, but in the United States each is now served by a competitive market." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).
  8. On using different clouds for different stages: Stoica said that for a pipeline "Each of these components you can actually run on different cloud for good reasons," such as Azure confidential computing for data and TPUs for training. — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  9. On standards and innovation: The paper warns that a fixed comprehensive standard "would, over time, significantly impede innovation." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).
  10. On the benefits of the Sky: The paper lists benefits including "lower barriers to cloud usage" and "rapid technical innovation via specialized clouds." — arXiv: The Sky Above The Clouds, a Berkeley View on the Future of Cloud Computing (May 2022).

Part 5: Startup Strategy & Commercialization

  1. On picking problems: Stoica said "You want to solve a problem ideally, which is going to be more important tomorrow than today." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  2. On a company behind the open source: He said "You need eventually to have a company behind the open-source, to make open source a viable solution, at least for large organizations." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  3. On partnering with competitors: He said "We are going to be aggressive also about partnerships, even though the partners could compete and overlap." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  4. On serving data engineers: After learning that few customers were doing machine learning, Stoica said "we start focusing also on serving data engineers much more than before." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  5. On betting the company on the open source: Stoica said "We decided that the success of the open source is necessary to the success of the company." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  6. On making distributed computing easier: Of Anyscale and Ray, Stoica said "Our mission is about making distributed computing easier, scaling these workloads easier." — VentureBeat report on his Transform 2022 talk: The future of AI is distributed (July 2022).
  7. On a platform for data and AI: Stoica said "always we wanted to provide the platform an end-to-end platform which helps our customers get most of the value out of their data." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  8. On demos versus production: He said "A lot of things still we are seeing today are fantastic demos," and that going from a demo to production is a big step because a product has to work for all cases. — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  9. On luck and timing: Stoica said "a lot of it's also about being the right time and the right place and things like that. It's being lucky." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).

Part 6: Academic Research & Lab Models

  1. On five-year labs: He described Berkeley's labs as "five-year labs which basically each lab has kind of a vision" and a group of faculty with their students trying to make it happen over five years. — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  2. On interdisciplinary labs: He said the labs are interdisciplinary, "people from machine learning, systems, databases, networking." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  3. On the problem being the differentiator: Stoica tells students "if all of them are good problem solvers, the differentiator is a problem you are working on." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  4. On work that starts for yourselves: On Ray's origins he said "Robert and Philip developed something for themselves, to start," after finding Spark too rigid for reinforcement learning. — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  5. On industry connections: Of the labs he said "very tight connection with the industry. It's a very good environment to see the problems." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  6. On a broken partnership: On AI research today he said the three-way partnership of academia, government and industry that drove the Internet "is broken," with companies doing research in silos. — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).

Part 7: Engineering Mindset & Problem Solving

  1. On simplifying under stress: Stoica said "I try to say, 'Okay, what is the most important thing?' and try to forget about everything else and then try to simplify the problem." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  2. On emotions in hard times: He said "The harder things are, the more emotional you are," and that decisions based on emotions "did not work" for him. — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  3. On simplifying things: Stoica said "you try to simplify things," using a 2x2 of whether Spark and the product succeed to commit to one path. — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  4. On a minimalist API: Of Ray he said it presents "a very flexible and minimalist API," backed by a strong ecosystem of libraries. — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).

Part 8: Security, Economics, & The Next Era

  1. On cost: Stoica said "obviously cost is important," but first "the important thing it's about the value," and "at that stage, cost is not as important." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  2. On open source without creators: Stoica said "It's hard for a company which doesn't have the creators of an open source to create a business." — Weights & Biases Gradient Dissent: Ion Stoica on Spark, Ray, and enterprise open source (January 2022).
  3. On auditability: He said enterprises "want to be able to audit what data you use, what results have been used, and what decisions have been made." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).
  4. On autonomy: Stoica said that "to go from having the human in the loop to being autonomous is a huge gap." — Sequoia Capital Training Data podcast: Databricks founder Ion Stoica on turning academic open source into startup success (January 2025).