Visual summary of operating lessons from Ronny Kohavi.

Lessons from Ronny Kohavi

The items below are things Ronny Kohavi said in a podcast interview about A/B testing, failure rates, metrics, trusting results and building an experimentation culture. — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 1: The Humbling Reality of Ideas

  1. On Humility: On Lenny’s Podcast, Kohavi said: "we are often humbled by how bad we are at predicting the outcome of experiments". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Bing Failure Rates: On Lenny’s Podcast, Kohavi said: "At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was around 85%". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On Industry Failure Rates: On Lenny’s Podcast, Kohavi said: "Booking, Google Ads, other companies published numbers that are around 80 to 90% failure rate of ideas". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Expecting to Fail: On Lenny’s Podcast, Kohavi said: "What I'm telling you is 80% of the time, you will fail". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Breakthroughs: On Lenny’s Podcast, Kohavi said: "you're most likely going to fail, but if you do succeed, it's a breakthrough". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On Thinking You Are Different: On Lenny’s Podcast, Kohavi said: "every group that starts to run experiments, they always start off by thinking that somehow, they're different". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  7. On Inch by Inch: On Lenny’s Podcast, Kohavi said: "most of the time, the winnings are made this inch by inch". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  8. On Summarizing Learnings: On Lenny’s Podcast, Kohavi said: "one of the mistakes that some company makes is they launch a lot of experiments and never go back and summarize the learnings". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  9. On Surprising Experiments: On Lenny’s Podcast, Kohavi said: "a surprising experiment is one where the estimated result beforehand and the actual result differ by a lot". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 2: Designing the Overall Evaluation Criterion

  1. On the OEC: On Lenny’s Podcast, Kohavi said: "The OEC or the overall evaluation criterion is something that I think many people that start to dabble in A/B testing miss". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Lifetime Value: On Lenny’s Podcast, Kohavi said: "you have to define the OEC such that it is causally predictive of the lifetime value of the user". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On Modeling Spam: On Lenny’s Podcast, Kohavi said: "model the cost of spamming the users". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Knowing What You Optimize: On Lenny’s Podcast, Kohavi said: "Do you know what you're optimizing for?". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Time on Site: On Lenny’s Podcast, Kohavi said: "Is people spending more time on the support site a good thing or a bad thing?". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On Showing More Ads: On Lenny’s Podcast, Kohavi said: "Displaying more ads is a trivial way to raise revenue, but it hurts the user experience". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  7. On Fixed Real Estate: On Lenny’s Podcast, Kohavi said: "I want you to increase revenue, but I'm going to give you a fixed amount of average real estate that you can use". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  8. On Growing Metrics: On Lenny’s Podcast, Kohavi said: "People don't appreciate that the number of metrics starts to grow really fast if you're doing things right". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 3: Twyman's Law and Trusting the Data

  1. On Too Good to Be True: On Lenny’s Podcast, Kohavi said: "when something looks too good to be true, investigate". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Holding the Dinner: On Lenny’s Podcast, Kohavi said: "Hold that dinner, investigate, see, because there's a large probability that something is wrong with the result". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On False Positives: On Lenny’s Podcast, Kohavi said: "people don't appreciate how much that 26% that I mentioned is high". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Prior Probability: On Lenny’s Podcast, Kohavi said: "the prior probability that the hypothesis that you're testing is successful or not". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On What People Think: On Lenny’s Podcast, Kohavi said: "They think there's a 98% probability that the treatment is better than the control". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 4: The Mechanics of Rigorous A/B Testing

  1. On Necessary Ingredients: On Lenny’s Podcast, Kohavi said: "there are some necessary ingredients to A/B testing". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Enough Units: On Lenny’s Podcast, Kohavi said: "You need to have enough units, mostly users, in order for the statistics to work out". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On Marginal Cost: On Lenny’s Podcast, Kohavi said: "It is possible to get the marginal cost to be close to zero". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Bigger Effects: On Lenny’s Podcast, Kohavi said: "They shouldn't focus on the 1%, they should focus on the 5 and 10%". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Testing Everything: On Lenny’s Podcast, Kohavi said: "any code change that you make, any feature that you introduce has to be in some experiment". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On Experiment History: On Lenny’s Podcast, Kohavi said: "have your whole history of experiments and do some ability to search by keywords". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  7. On Scorecards: On Lenny’s Podcast, Kohavi said: "when the experiment finishes, you should have a scorecard soon after". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 5: Overcoming the HiPPO and Building Culture

  1. On Sharing Surprises: On Lenny’s Podcast, Kohavi said: "Once Bing was successful and we were able to share all these surprising results, I think that many, many more people in the company were amenable". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Fast Teams: On Lenny’s Podcast, Kohavi said: "Don't go with the team that launches every six months". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On Success and Failure Decks: On Lenny’s Podcast, Kohavi said: "We had a large deck internally of these successes and failures, and we encourage people to look at them". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Structured Narratives: On Lenny’s Podcast, Kohavi said: "a structured narrative". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Honest Feedback: On Lenny’s Podcast, Kohavi said: "the ability to give people honest feedback and have them appreciate, and have it stay after the meeting in these notes on the document, just amazing". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On Shipping Without Value: On Lenny’s Podcast, Kohavi said: "shipping this project has no value, is complicating the code base". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 6: Pitfalls, Statistical Traps, and SRM

  1. On Sample Ratio Mismatch: On Lenny’s Podcast, Kohavi said: "one of the things that is the most common occurrence by far, which is a sample ratio mismatch". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Surprise at Mismatch: On Lenny’s Podcast, Kohavi said: "when we implemented this check, we were surprised to see how many experiments suffered from this". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On Bots: On Lenny’s Podcast, Kohavi said: "Bots are a very common factor for causing sample ratio mismatch". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Changing the Website: On Lenny’s Podcast, Kohavi said: "Because you change the website, the bot may fail to parse the page, and try to hit it more often". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Hidden Campaigns: On Lenny’s Podcast, Kohavi said: "people that start an experiment in the middle of the site on some page, but they don't realize that some campaign is pushing people from the side". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On Losing Trust: On Lenny’s Podcast, Kohavi said: "They lost all this trust because they built something that had very much inflated error rate". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  7. On Debugging: On Lenny’s Podcast, Kohavi said: "We need that okay button because you want to be able to debug the reasons". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 7: Small Changes with Massive Impact

  1. On Small Changes: On Lenny’s Podcast, Kohavi said: "even small bug fixes, even small changes can sometimes have surprising, unexpected impact". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Adding Up: On Lenny’s Podcast, Kohavi said: "they add up to around 2% every year, which is amazing". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On Incremental Experiments: On Lenny’s Podcast, Kohavi said: "You need some experiments that are incremental that move you in the direction that you know you're going to be successful over time if you just try enough". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Home Runs: On Lenny’s Podcast, Kohavi said: "We're going to try something that's most likely to fail, but if it does win, it's going to be a home run". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Moving Incrementally: On Lenny’s Podcast, Kohavi said: "Start to move incrementally in a direction that you believe is beneficial". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On Many Changes Together: On Lenny’s Podcast, Kohavi said: "doing 17 changes together is more likely to be negative". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  7. On One Factor at a Time: On Lenny’s Podcast, Kohavi said: "Do them in smaller increments, learn from, it's called OFAT one-factor-at-a-time". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  8. On Small Insights: On Lenny’s Podcast, Kohavi said: "many of these small insights lead to fundamental insights about which areas to go, some strategies we should take, some things we should develop". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.

Part 8: Institutionalizing Experimentation at Scale

  1. On Build or Buy: On Lenny’s Podcast, Kohavi said: "I think the key decision is whether you want to build or buy". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  2. On Third-Party Tools: On Lenny’s Podcast, Kohavi said: "for people starting, the third party products that are available today are pretty good". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  3. On a Beachhead: On Lenny’s Podcast, Kohavi said: "we went with this beach head of Bing". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  4. On Investing in the Platform: On Lenny’s Podcast, Kohavi said: "If you're building a good platform, invest in it so that more and more automation will allow people to look at the analysis, without the need to involve a data scientist". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  5. On Weak Analysis: On Lenny’s Podcast, Kohavi said: "the analysis was relatively weak, and so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  6. On the Hierarchy of Evidence: On Lenny’s Podcast, Kohavi said: "this idea called the hierarchy of evidence". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  7. On Rising Trust: On Lenny’s Podcast, Kohavi said: "As you get more up and up to a natural experiment, and controlled experiments, and multiple controlled experiments, your trust levels should go up". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.
  8. On Scale at Microsoft: On Lenny’s Podcast, Kohavi said: "when I left in 2019, we were on a rate of about 20 to 25,000 experiments every year". — The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon), Lenny’s Podcast.