Jay Kreps is a co-creator of Apache Kafka and the co-founder and CEO of Confluent. He joined LinkedIn in 2007 as a software engineer, where he worked on distributed data systems before writing "The Log" in 2013. — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).

Visual summary of operating lessons from Jay Kreps.

Part 1: The Log and Data Abstractions

  1. On Data Unity: "Take all the organization's data and put it into a central log for real-time subscription." — [LinkedIn Engineering]
  2. On data integration: Kreps wrote "Data integration is making all the data an organization has available in all its services and systems." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  3. On why integration matters: Kreps wrote "As long as this remains true, the data integration problem will remain one of the most centrally important things for the successful use of data." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013)
  4. On the log as storage: Kreps wrote "A log is perhaps the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  5. On the word log: Kreps wrote "I use the term “log” here instead of “messaging system” or “pub sub” because it is a lot more specific about semantics." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013)
  6. On tables and events: Kreps wrote "tables and events [are] dual: tables support data at rest and logs capture change." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  7. On data subscription: Kreps wrote "The use of logs as a mechanism for data subscription seems to have arisen almost by chance." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  8. On the log as the record of what happened: Kreps wrote "The log is the record of what happened, and each table or index is a projection of this history into some useful data structure or index." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  9. On deterministic processing: Kreps wrote "it more or less amounts to saying "deterministic processing is deterministic"." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  10. On a general design tool: Kreps wrote "I think it is one of the more general tools for distributed systems design." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013)
  11. On decoupling producers and consumers: Kreps wrote "The log also acts as a buffer that makes data production asynchronous from data consumption." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  12. On Real-Time Capabilities: Kreps explains Kafka's early pitch as turning data sources into real-time streams that applications can react to as events occur. — Source: Logan Bartlett Show

Part 2: The Evolution of Apache Kafka

  1. On the name: Explaining the name, Kreps said "So it was Kafkaesque. The problem was Kafkaesque." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  2. On building infrastructure: Kreps said that at the time "it was really kind of do it in house or do nothing, right?" — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023).
  3. On wanting to make something great: Kreps said "I just wanted to make something great." — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023).
  4. On open source from day one: Kreps said "It was built to be open source from the beginning." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  5. On the first reception: Kreps said Kafka "was originally a bit of a flop." — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023).
  6. On stubbornness: Kreps said "I'm just extremely stubborn." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024)
  7. On adopting new platforms: Kreps said "it's very hard to take time out of your day to go adopt some new software platform and build around it." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024)

Part 3: Stream Processing and Event-Driven Architecture

  1. On data in motion: Kreps said the idea behind Kafka was to focus on "the data in motion, like how does it move around?" — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023).
  2. On continuous computation: Kreps described taking processing "from something that happens periodically to something that happens continuously." — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023).
  3. On continuous processes: Kreps wrote "when you think about any business, the underlying mechanics are almost always a continuous process—events happen in real-time, as Jack Bauer would tell us." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013)

Part 4: Scaling Systems and Managing State

  1. On replaying history: Kreps wrote "Surprisingly, at the core of these problems is the ability to have many machines playback history at their own rate in a deterministic manner." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013).
  2. On a clean repository of data: Kreps wrote "The clean, integrated repository of data should be available in real-time as well for low-latency processing as well as indexing in other real-time storage systems." — LinkedIn Engineering: The Log, What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, December 2013)
  3. On Cloud Architecture: Kreps says Confluent was convinced from the start that cloud would become the way customers consumed infrastructure, even before that market was ready. — Source: Logan Bartlett Show
  4. On controversy at LinkedIn: Kreps said "both these projects were a little controversial internally." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  5. On writing databases: Kreps said "friends don't let friends write databases." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024)

Part 5: Transitioning from Engineer to CEO

  1. On engineering and the CEO role: Kreps said "I think engineering is probably the least useful skill in a lot of ways for becoming a CEO." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  2. On learning: Kreps said "engineers have to learn a lot of stuff." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  3. On not being expected to go to college: Kreps said "There was no expectation that I go to college." — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023)
  4. On systems thinking: Kreps said "I do think that ultimately an organization is a kind of system." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  5. On Letting Go of Code: Kreps describes the founder-CEO transition as leaving comfortable engineering and system-design work for a very different skillset around customers, go-to-market, and company direction. — Source: Logan Bartlett Show

Part 6: Leadership and Building Confluent

  1. On Cloud Transition: Kreps says Confluent chose to commit the whole company to cloud because giving up that future would have left it with a narrower company than it wanted to build. — Source: Logan Bartlett Show

Part 7: Open Source Business Models

  1. On Cloud Providers: Kreps frames Confluent's cloud strategy around covering all the environments large companies use, while balancing open standards with differentiated managed-service value. — Source: Logan Bartlett Show

Part 8: Engineering Culture and Philosophy

  1. On communication: Kreps said "it's not sufficient to just say things that are true. That's necessary, but it's not enough." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  2. On the power of communication: Kreps said "Communication, I think, is one of the most powerful things, broadly." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024)
  3. On Job Titles: Kreps says Confluent has a split-brain title culture: engineering prefers sparse, hard-earned titles, while other functions follow more standard company norms. — Source: Logan Bartlett Show
  4. On low titles: Kreps said "our engineering team has always wanted to give themselves the lowest possible titles." — The Logan Bartlett Show, episode 89: Jay Kreps on Confluent's Resilient Rise to Software Behemoth (December 2023)
  5. On describing problems: Kreps joked "engineers say there's a problem when there's a problem." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).
  6. On the gritty profession: Kreps said distributed systems is "a relatively gritty profession where you spend most of your time on things that are not working." — Software Misadventures: Lessons from the early days building Kafka and Confluent, Jay Kreps (June 2024).