Visual summary of operating lessons from Neha Narkhede.

Lessons from Neha Narkhede

Neha Narkhede co-created Apache Kafka and co-founded Confluent, building the open-source infrastructure that handles real-time data streams for thousands of companies. She later founded Oscilar to apply real-time data to fraud detection and risk management. This collection gathers her practical advice on distributed systems, scaling open-source software, and navigating the tech industry as a founder and female leader.

Part 1: The Origin of Apache Kafka

  1. On the initial problem: "We observed that our internal systems were struggling to handle the massive influx of user data." — Source: [Microsoft Behind the Tech]
  2. On the LinkedIn scale: "Kafka was born from the necessity to manage real-time data streams as LinkedIn scaled." — Source: [Forbes]
  3. On early limitations: "Existing messaging systems were simply not designed for the volume and latency requirements of a global social network." — Source: [Confluent Blog]
  4. On architectural philosophy: "Our idea was that instead of focusing on holding piles of data... we would focus on treating data as a continually evolving and ever-growing stream." — Source: [Kafka: The Definitive Guide]
  5. On identifying the pattern: "We realized other companies were encountering the same problems with user data that we had solved at LinkedIn." — Source: [Forbes]
  6. On building for developers: "The goal was to build a system that could capture data in motion and utilize it in real-time." — Source: [Microsoft Behind the Tech]
  7. On the central nervous system: "We view our technology as a central nervous system for companies that aggregates data and makes sense of it within milliseconds, at scale." — Source: [Innovators Under 35]
  8. On open-sourcing Kafka: "Why we wanted to open source it is... this trend that technology companies are going to essentially be defined in software." — Source: [Microsoft Behind the Tech]
  9. On taking the leap: "Founding Confluent was a natural next step to bring what we built to the broader market." — Source: [Forbes]

Part 2: Open Source Strategy

  1. On bottom-up adoption: "Open source allows you to build a community of developers who champion your product from within their organizations." — Source: [First Round Review]
  2. On monetizing open source: "The key to monetizing is offering enterprise features that solve organizational pain points, without restricting the core developer experience." — Source: [First Round Review]
  3. On community feedback: "An active open source community acts as an extended product team, testing limits and surfacing novel use cases." — Source: [Confluent Blog]
  4. On the cloud transition: "Moving an open source project to a managed cloud service removes the operational burden for developers, allowing them to focus on building apps." — Source: [First Round Review]
  5. On maintaining trust: "You have to be extremely clear about what is free and open, and what constitutes a paid, closed-source feature." — Source: [First Round Review]
  6. On developer marketing: "Developers don't want to be sold to. They want tools that solve their immediate problems elegantly." — Source: [First Round Review]
  7. On open source governance: "Guiding an open source project requires balancing the needs of the community with the strategic direction of the company." — Source: [Confluent Blog]
  8. On enterprise readiness: "Open source gets you in the door, but enterprise readiness—security, compliance, and support—closes the deal." — Source: [First Round Review]
  9. On category creation: "Creating a new category means you spend as much time educating the market on the problem as you do selling the solution." — Source: [Confluent Blog]

Part 3: Scaling Engineering and Infrastructure

  1. On the evolution of data: "Data is no longer a static asset resting in a database; it is a fluid stream that powers real-time decisions." — Source: [Kafka: The Definitive Guide]
  2. On decoupling systems: "A highly scalable architecture relies on decoupled systems that can operate independently and communicate asynchronously." — Source: [Confluent Blog]
  3. On handling failure: "At scale, failure is not an anomaly; it is a constant. Systems must be designed to gracefully recover without human intervention." — Source: [Kafka: The Definitive Guide]
  4. On stream processing: "Processing data as it arrives, rather than in nightly batches, fundamentally changes how businesses respond to users." — Source: [Confluent Blog]
  5. On infrastructure as a service: "The best infrastructure is invisible to the developer using it." — Source: [Microsoft Behind the Tech]
  6. On distributed systems complexity: "Building distributed systems means embracing trade-offs between consistency, availability, and partition tolerance." — Source: [Kafka: The Definitive Guide]
  7. On horizontal scaling: "We built Kafka so that adding capacity is as simple as adding another node to the cluster." — Source: [Confluent Blog]
  8. On fault tolerance: "Data replication is critical, but it must be done efficiently so it doesn't become a bottleneck." — Source: [Kafka: The Definitive Guide]
  9. On continuous delivery: "Infrastructure teams must operate with the same agility and continuous delivery cadence as product teams." — Source: [Confluent Blog]
  10. On the future of infrastructure: "The infrastructure of the future will increasingly manage itself, allowing engineers to focus entirely on business logic." — Source: [Microsoft Behind the Tech]

Part 4: The Real-Time Paradigm

  1. On real-time as DNA: "Real time is in our DNA. Everything we build is centered around acting on data the moment it is generated." — Source: [Fintech Nexus]
  2. On batch processing limitations: "Batch processing forces businesses to look in the rearview mirror. Real-time allows them to steer." — Source: [Confluent Blog]
  3. On customer expectations: "Today’s consumers expect personalized, immediate interactions. You can't deliver that on yesterday's data." — Source: [Microsoft Behind the Tech]
  4. On event-driven architectures: "An event-driven architecture accurately models how things happen in the real world—as a series of discrete events." — Source: [Kafka: The Definitive Guide]
  5. On software-defined business: "Not just use more software, but literally all the actions and decisions in a company happen through software." — Source: [Microsoft Behind the Tech]
  6. On real-time analytics: "The value of data decays rapidly over time. Analyzing it in real-time maximizes its utility." — Source: [Confluent Blog]
  7. On competitive advantage: "Moving from batch to real-time is a monumental shift that defines the competitive edge of modern enterprises." — Source: [Forbes]
  8. On legacy systems: "Transitioning away from legacy databases requires a fundamental rethink of how state is managed across an organization." — Source: [Confluent Blog]
  9. On the ubiquity of streams: "We think virtually every company would benefit from a central data nervous system." — Source: [Innovators Under 35]
  10. On future applications: "As real-time processing becomes commoditized, the focus will shift entirely to the intelligence layered on top of it." — Source: [Fintech Nexus]

Part 5: The Founder’s Mindset

  1. On being uncomfortable: "You can accomplish a lot of things. And at the same time, you are uncomfortable almost most of the time." — Source: [Microsoft Behind the Tech]
  2. On the nature of growth: "Uncertainty often leads to personal growth, pushing you to navigate challenges you didn't know you could handle." — Source: [Forbes]
  3. On identity: "I've learned to embrace my own unique background that truly makes me who I am today." — Source: [Forbes]
  4. On taking the first step: "Starting a company is an irrational act of optimism. You have to believe in the vision more than the statistics of failure." — Source: [First Round Review]
  5. On failure: "My personal mantra is to fail fast and keep moving." — Source: [Forbes]
  6. On defining success: "Building a billion-dollar company is a milestone, not the finish line." — Source: [Forbes]
  7. On continuous learning: "The role of a founder changes every six months. If you aren't learning at that pace, you become the bottleneck." — Source: [First Round Review]
  8. On solving bigger problems: "After Confluent, I wanted to tackle an even bigger problem: making the internet a safer place for everyone." — Source: [Oscilar Blog]
  9. On co-founder dynamics: "A strong co-founder relationship is built on complementary skills and absolute trust." — Source: [Oscilar Blog]
  10. On long-term vision: "Technology can scale change — but only purpose gives it meaning." — Source: [Forbes]

Part 6: Leadership and Building Teams

  1. On teamwork: "Teamwork makes the dream work. You cannot build foundational technology alone." — Source: [Girl Geek X]
  2. On hiring engineers: "We look for engineers who are not just technically brilliant, but who deeply care about the customer's problem." — Source: [First Round Review]
  3. On maintaining culture: "Culture is not what you write on the wall; it is how the team behaves when things go wrong." — Source: [First Round Review]
  4. On transparency: "In a fast-growing startup, transparent communication is the only way to keep everyone rowing in the same direction." — Source: [First Round Review]
  5. On delegating: "Leadership means stepping away from the code to build the organization that builds the code." — Source: [First Round Review]
  6. On engineering management: "The transition from engineer to manager requires a complete recalibration of how you measure your daily output." — Source: [Girl Geek X]
  7. On diversity in teams: "Diverse teams build better products because they challenge underlying assumptions." — Source: [Girl Geek X]
  8. On resolving conflict: "Technical disagreements should be resolved through data and prototyping, not by authority." — Source: [First Round Review]
  9. On setting goals: "Great teams need ambitious, borderline uncomfortable goals to do their best work." — Source: [First Round Review]

Part 7: Women in Tech and Advocacy

  1. On the glass ceiling: "You will have to fight extremely hard if you don't have a male cofounder. My first advice is to develop the mental strength to fight many battles." — Source: [Forbes]
  2. On being judged: "Women often face the fear of being judged negatively as being too ambitious, while for others, ambition is normalized." — Source: [Girl Geek X]
  3. On potential vs. experience: "Men are often assessed based on their future potential, while women are more frequently evaluated based on their past experience." — Source: [Girl Geek X]
  4. On self-advocacy: "Make your requests concrete. When negotiating for a promotion, be explicit about what you have achieved and what you expect." — Source: [Girl Geek X]
  5. On feeling out of place: "It is okay to feel a little out of place. You have to develop the persistence to keep going." — Source: [Forbes]
  6. On preparation: "Write down your points ahead of time to ensure clarity and to keep emotions aside during important conversations." — Source: [Girl Geek X]
  7. On resilience: "You must be willing to hear 'no' a few times before eventually reaching a 'yes'." — Source: [Forbes]
  8. On the ultimate advice: "My general advice to women engineers is simply: Do not quit." — Source: [Forbes]
  9. On visibility: "Speaking at conferences and sharing expertise is crucial for women engineers to build authority and inspire the next generation." — Source: [Girl Geek X]

Part 8: Risk, AI, and Oscilar

  1. On the mission of Oscilar: "We want to replace fragmented, slow, and inefficient point solutions with a unified AI-powered platform." — Source: [Oscilar Blog]
  2. On financial risk: "Managing risk is no longer just a compliance exercise; it is a core business enabler in the fintech ecosystem." — Source: [Fintech Nexus]
  3. On AI in fraud detection: "AI allows us to detect complex fraud patterns that rules-based systems miss, and it does so in real-time." — Source: [Oscilar Blog]
  4. On no-code solutions: "Empowering risk operators with no-code tools removes the engineering bottleneck in deploying new fraud models." — Source: [Fintech Nexus]
  5. On the evolution of data engineering: "We are moving from data engineering as infrastructure to data engineering as direct business risk control." — Source: [Oscilar Blog]
  6. On system agility: "Fraudsters evolve constantly. Your risk infrastructure must allow you to adapt your defenses in minutes, not months." — Source: [Fintech Nexus]
  7. On unified data: "You cannot make accurate risk decisions if your user data is scattered across five different internal tools." — Source: [Oscilar Blog]
  8. On the next generation of fintech: "The next wave of financial innovation will be constrained only by our ability to secure those transactions at scale." — Source: [Fintech Nexus]
  9. On full-circle engineering: "Applying real-time streaming principles to AI-driven risk management feels like the natural culmination of everything we learned building Kafka." — Source: [Oscilar Blog]