Jay Kreps is the co-creator of Apache Kafka and the co-founder and CEO of Confluent. His work formalized the concept of using an append-only log as the central integration point for enterprise data, changing how large companies build distributed systems. This profile collects his direct observations on system design, transitioning from writing code to managing a company, and the realities of commercializing open-source software.

Visual summary of operating lessons from Jay Kreps.

Part 1: The Log and Data Abstractions

  1. On Data Unity: "Take all the organization's data and put it into a central log for real-time subscription." — Source: The Log
  2. On the N-Squared Problem: "Integrating every system with every other system creates an unmanageable web of pipelines. A central log reduces this complexity down to a single integration point." — Source: SE Radio
  3. On Append-Only Logs: "The log is the most fundamental data structure in distributed systems. It is the core abstraction for scaling state." — Source: Confluent Blog
  4. On State and Time: "If you model data as an ordered sequence of events, you inherently capture both state and time, making complex distributed coordination much simpler." — Source: I Heart Logs
  5. On Database Replication: "The fundamental mechanism databases use for replication, a write-ahead log, is exactly what enterprises need at a macro level for data integration." — Source: First Round Review
  6. On Batch Processing: "A business is an activity happening continuously in reality. Yet historically, all the best things we could do with data were happening in delayed batches." — Source: Confluent Developer Podcast
  7. On Data as a Stream: "Rather than viewing data as static tables to be queried, we should view it as a continuous, immutable stream of events occurring over time." — Source: Software Engineering Daily
  8. On Consistency: "Distributed consensus problems become vastly more manageable when grounded in an ordered log that acts as the absolute source of truth." — Source: The Log
  9. On Decoupling: "By enforcing a log between data producers and consumers, teams can operate, scale, and evolve completely independently." — Source: O'Reilly Strata
  10. On Real-Time Capabilities: Kreps explains Kafka's early pitch as turning data sources into real-time streams that applications can react to as events occur. — Source: Logan Bartlett Show

Part 2: The Evolution of Apache Kafka

  1. On Naming Kafka: "The system was named after Franz Kafka because it is essentially a system optimized for writing, and I was a fan of the author's work." — Source: Hacker News
  2. On LinkedIn's Infrastructure: "Kafka began out of a desperate need at LinkedIn to handle massive amounts of activity data that traditional message queues couldn't survive." — Source: Software Misadventures
  3. On Message Queues: "Standard messaging systems were built for application integration, leaving them unable to handle the massive throughput of user activity and log data." — Source: SE Radio
  4. On Building for Scale: "The only way to really know if a system design works in the real world is to build it, deploy it for real applications, and see where it falls short." — Source: Brave New Geek
  5. On Open Sourcing Kafka: "Releasing Kafka as an open-source project was a deliberate attempt to build a standard for data infrastructure across the industry." — Source: Confluent Developer Podcast
  6. On Initial Skepticism: "When Kafka was first proposed, many argued that moving all data through a single distributed log would create a massive bottleneck. The architecture proved the opposite." — Source: Software Engineering Daily
  7. On Consumer Groups: "Kafka's design shifted the burden of tracking what data had been read from the broker to the consumer, drastically improving throughput." — Source: Confluent Blog
  8. On Retention: "Unlike traditional queues that delete messages upon reading, Kafka's ability to retain data on disk allowed for entirely new use cases like replaying history." — Source: First Round Review
  9. On Kafka's Role: "It transitioned from being a niche tool for Hadoop ingestion to the central nervous system connecting microservices and data systems." — Source: Confluent Blog

Part 3: Stream Processing and Event-Driven Architecture

  1. On The Kappa Architecture: "If a stream processing system is reliable and can replay historical data from a log, the separate batch-processing layer in Lambda architecture becomes redundant." — Source: O'Reilly
  2. On Unifying Processing: "Stream processing should require the exact same programming model as batch processing. The only difference should be whether the data has a defined end." — Source: Confluent Blog
  3. On Central Nervous Systems: "We are the central nervous system that connects different parts of the enterprise together, making data available where it is needed in real-time." — Source: Economic Times
  4. On Microservices: "True microservice decoupling requires asynchronous event streams. If microservices rely entirely on synchronous RPCs, they remain tightly coupled in failure modes." — Source: Software Engineering Daily
  5. On Event-Driven Systems: "Modern event-driven architectures must move beyond simple triggers and fire-and-forget notifications, unifying storage, processing, and events into one platform." — Source: Confluent Blog
  6. On State in Streams: "Stateful stream processing is difficult because it requires managing local data that must survive machine failures and network partitions." — Source: SE Radio
  7. On ksqlDB: "The goal of ksqlDB is to allow developers to build stream processing applications with the same ease and SQL syntax they use for relational databases." — Source: Confluent Developer Podcast
  8. On Continuous Computation: "Instead of asking questions against static data, stream processing is about installing questions and having the data flow continuously through them." — Source: Software Engineering Daily
  9. On Eventual Consistency: "Systems built around event streams embrace eventual consistency naturally, making the state of the system a function of the event log." — Source: I Heart Logs

Part 4: Scaling Systems and Managing State

  1. On Distributed Systems: "Scaling an infrastructure layer requires accepting that hardware will fail constantly, making fault tolerance a primary design constraint." — Source: Brave New Geek
  2. On System Abstractions: "A good abstraction hides complexity but doesn't hide the fundamental physics of the underlying hardware, like network latency or disk IO." — Source: Software Misadventures
  3. On Simplicity: "In distributed systems, simplicity is a prerequisite for reliability. Complex coordination mechanisms inevitably break in production." — Source: SE Radio
  4. On Data Integration: "The hardest part of scaling a company’s technology usually involves the friction of moving data safely between the teams that need it." — Source: First Round Review
  5. On Storage vs Processing: "Disaggregating storage from processing allows each layer to scale independently and reduces the cost of maintaining massive datasets." — Source: Confluent Blog
  6. On Managing Dependencies: "As engineering organizations scale, minimizing synchronous dependencies between teams is the only way to maintain deployment velocity." — Source: Software Engineering Daily
  7. On Cloud Architecture: Kreps says Confluent was convinced from the start that cloud would become the way customers consumed infrastructure, even before that market was ready. — Source: Logan Bartlett Show
  8. On Performance Testing: "Theoretical performance limits matter less than how a system degrades under unpredictable, sustained production workloads." — Source: Confluent Developer Podcast
  9. On Project Voldemort: "Early experiences building distributed key-value stores at LinkedIn highlighted that managing state at scale is the hardest problem in infrastructure." — Source: SE Radio

Part 5: Transitioning from Engineer to CEO

  1. On The Role Shift: "Moving from an engineering role to CEO requires abandoning the tools that made you successful and adopting an entirely new skillset." — Source: First Round Review
  2. On The 80% Rule: "A CEO only needs to know about 80% of what's happening in any given department. Trying to know 100% means you are micro-managing and becoming a bottleneck." — Source: First Round Review
  3. On Transferable Skills: "The most valuable trait an engineer brings to the CEO role is the ability to learn complex, unfamiliar systems rapidly." — Source: Software Misadventures
  4. On Systems Thinking: "An engineer's ability to peer inside modules and optimize their interactions is directly applicable to managing the different departments of a company." — Source: First Round Review
  5. On Letting Go of Code: Kreps describes the founder-CEO transition as leaving comfortable engineering and system-design work for a very different skillset around customers, go-to-market, and company direction. — Source: Logan Bartlett Show
  6. On Product Marketing: "Engineers frequently underestimate product marketing. Building a product marketing pyramid is necessary to explain deeply technical tools to non-technical buyers." — Source: First Round Review
  7. On Hiring Executives: "Evaluating leaders for functions you have never worked in requires relying on trusted networks and focusing on their structured thinking rather than domain trivia." — Source: a16z Podcast
  8. On Focus: "As a software engineer, you solve problems. As a CEO, your main job is deciding which problems the company should intentionally ignore." — Source: Software Misadventures
  9. On Managing Context: "The CEO's primary operational output is providing the right context so that autonomous teams make decisions aligned with the company's strategy." — Source: First Round Review

Part 6: Leadership and Building Confluent

  1. On Strategic Imperatives: "There is a massive difference between what a company can do and what a company must do. Leaders must force alignment on the latter." — Source: First Round Review
  2. On Cloud Transition: Kreps says Confluent chose to commit the whole company to cloud because giving up that future would have left it with a narrower company than it wanted to build. — Source: Logan Bartlett Show
  3. On Earning Glory: Kreps argues that leaders receive praise during boom times, but the real test is making sound decisions during bad moments when there is little positive feedback. — Source: Logan Bartlett Show
  4. On Company Software: "Every company is becoming a software company. If your business doesn't treat software as its primary mechanism for value delivery, it will be outmaneuvered." — Source: Confluent Blog
  5. On Enterprise Sales: "Selling open-source infrastructure requires proving that your enterprise features solve organizational friction, rather than purely technical limitations." — Source: Open Source Underdogs
  6. On Fundraising: "Raising capital in the early days of Confluent required translating the abstract concept of a data log into a tangible vision of a central nervous system for business." — Source: Software Misadventures
  7. On Zero to One: "The earliest stage of building a company is less about scaling operations and entirely about validating that the market actually cares about your solution." — Source: First Round Review
  8. On Market Timing: "You can build the best infrastructure in the world, but if the market isn't ready to adopt event-driven architectures, the business will stall." — Source: SE Radio
  9. On Managed Services: "The value of a managed cloud service lies in removing the operational terror of running stateful distributed systems at 3 AM." — Source: Confluent Developer Podcast
  10. On Customer Empathy: "The best enterprise software companies are built by teams that genuinely understand the operational pain of the engineers using their product." — Source: a16z Podcast

Part 7: Open Source Business Models

  1. On Commercial Open Source: "Releasing software as open source builds a community and a standard. However, you must reserve distinct value for the commercial product to sustain a business." — Source: Open Source Underdogs
  2. On Licensing: "Navigating open-source licensing requires protecting the project from being strip-mined by cloud providers while keeping the core free for developers." — Source: Confluent Blog
  3. On Community Governance: "Managing an open-source project means balancing the needs of independent contributors with the strategic roadmap of the founding company." — Source: Software Engineering Daily
  4. On Cloud Providers: Kreps frames Confluent's cloud strategy around covering all the environments large companies use, while balancing open standards with differentiated managed-service value. — Source: Logan Bartlett Show
  5. On Developer Adoption: "The adoption curve for infrastructure starts with individual developers solving acute pain points before it ever reaches the CIO." — Source: SE Radio
  6. On Monetization: "You cannot monetize an open-source project by restricting its core utility; you monetize by solving the operational complexity of running it at enterprise scale." — Source: First Round Review
  7. On Support Models: "Selling support for open-source software is a difficult business model to scale; customers eventually prefer fully managed services over support tickets." — Source: Open Source Underdogs
  8. On Project Metamorphosis: "Transforming an open-source project into a cloud-native platform requires rethinking how the software handles tenancy, elasticity, and billing." — Source: Confluent Blog
  9. On Ecosystems: "The success of Kafka relied heavily on the massive ecosystem of connectors and integrations built by the open-source community." — Source: Confluent Developer Podcast

Part 8: Engineering Culture and Philosophy

  1. On Communication: "Engineers often struggle with communication because they focus primarily on conveying facts rather than considering the audience's perspective to drive a specific outcome." — Source: Software Misadventures
  2. On Job Titles: Kreps says Confluent has a split-brain title culture: engineering prefers sparse, hard-earned titles, while other functions follow more standard company norms. — Source: Logan Bartlett Show
  3. On Data Governance: "Companies face conflicting pressures: the strong urge to use data for business value versus the absolute need to secure and control its movement." — Source: Economic Times
  4. On Practical Engineering: "Theoretical computer science is necessary, but the hardest engineering challenges involve handling the messy realities of failed networks, bad disks, and user errors." — Source: Brave New Geek
  5. On Building Teams: "The best engineering cultures are built by giving smart people autonomy over hard technical problems that actually matter to the business." — Source: First Round Review
  6. On Data Silos: "Organizational silos are often a direct result of data silos. When you unify data access via a log, you inherently break down operational barriers between teams." — Source: The Log
  7. On Technical Debt: "Rewriting a system from scratch is almost always a mistake unless the fundamental architecture of the old system prevents it from meeting new scaling laws." — Source: SE Radio
  8. On Truth in Data: "If multiple systems hold different versions of the truth, you don't have a technical problem; you have an organizational crisis waiting to happen." — Source: Software Engineering Daily
  9. On Continuous Learning: "Technology cycles move too fast for static knowledge. The defining trait of a senior engineer is the velocity at which they can unlearn and relearn." — Source: Software Misadventures
  10. On The Final Goal: "Ultimately, data infrastructure exists to disappear. It should become an invisible utility that developers rely on without thinking about the underlying mechanics." — Source: Confluent Developer Podcast