> ## Content Index
> Fetch the complete content index at: https://www.antoinebuteau.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Lessons from Ronny Kohavi
- URL: https://www.antoinebuteau.com/lessons-from-ronny-kohavi/
- Published: 2026-07-01T20:53:43.000Z
- Updated: 2026-07-18T21:35:53.000Z
- Description: Ronny Kohavi built experimentation platforms for Amazon, Microsoft, and Airbnb, enabling thousands of A/B tests. His methods show how to evaluate metrics, detect bad data, and prevent executive opinion from overruling evidence of actual user behavior.
- Author: Antoine Buteau
- Tags: Profile, AI & Machine Learning Profiles

![Visual summary of operating lessons from Ronny Kohavi.](https://www.antoinebuteau.com/content/images/2026/07/lessons-from-ronny-kohavi-profile-infographic.webp)

## Lessons from Ronny Kohavi

Ronny Kohavi built the experimentation platforms that allow Amazon, Microsoft, and Airbnb to run thousands of A/B tests. This profile breaks down his methods for evaluating metrics, spotting bad data, and keeping executive opinions from overriding actual user behavior.

### Part 1: The Humbling Reality of Ideas

1. **On the failure of intuition:** "Most who have run controlled experiments in customer-facing websites and applications have experienced this humbling reality: we are poor at assessing the value of ideas." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
2. **On idea success rates:** "At Microsoft and Bing, only about one-third of ideas actually improved the metrics they were designed to improve, while the rest were neutral or negative." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
3. **On releasing without testing:** "If you release features without testing them, you are likely deploying negative changes that cancel out your positive changes, leading to stagnant growth." — [*Source: \[DigginTravel Interview*](https://www.diggintravel.com/ab-testing-culture-ronny-kohavi/?ref=antoinebuteau.com)*\]*
4. **On organizational maturity:** "Moving from an environment where people assume their ideas work to one where they accept that most ideas fail is the hardest cultural shift in software development." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
5. **On the value of negative results:** "Finding out an idea does not work saves the company from the long-term maintenance costs of a feature that provides no value to users." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
6. **On domain expertise:** "Even experts in a specific domain are surprisingly bad at predicting which variations of a design will win in a controlled experiment." — [*Source: \[KDD Keynote*](https://kdd.org/kdd2015/keynotes.html?ref=antoinebuteau.com)*\]*
7. **On the necessity of A/B testing:** "A/B testing serves as a defensive mechanism to prevent well-intentioned teams from degrading the user experience, rather than serving solely as a tool for optimization." — [*Source: \[Kameleoon Interview*](https://www.kameleoon.com/en/blog/ronny-kohavi-ab-testing?ref=antoinebuteau.com)*\]*
8. **On embracing failure:** "You must run a massive volume of experiments because the low win rate means volume is the only way to find the rare successes." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
9. **On feature bloat:** "Without A/B testing, products suffer from feature bloat because every idea is shipped and nothing is ever rolled back." — [*Source: \[Economists in Tech Podcast*](https://open.spotify.com/episode/3G7a0pY0p4aU9Xz2r9M4z4?ref=antoinebuteau.com)*\]*
10. **On humility in product design:** "Experimentation forces product managers to become humble listeners of user behavior rather than dictators of user experience." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*

### Part 2: Designing the Overall Evaluation Criterion

1. **On defining success:** "If you do not know where you are going, any road will get you there. You must define an Overall Evaluation Criterion." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
2. **On long-term value:** "The OEC should ideally measure long-term customer lifetime value, rather than short-term clicks or engagement spikes." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
3. **On vanity metrics:** "Optimizing for metrics like total page views can lead teams to design confusing interfaces that force users to click around unnecessarily." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
4. **On aligning metrics with business goals:** "The OEC acts as the bridge between daily product changes and the strategic objectives of the executive team." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
5. **On composite metrics:** "A good OEC often combines multiple metrics, penalizing negative behaviors while rewarding positive ones." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
6. **On the risk of single metrics:** "Focusing exclusively on revenue per user can accidentally incentivize spamming users with ads, ultimately destroying trust." — [*Source: \[KDD Keynote*](https://kdd.org/kdd2015/keynotes.html?ref=antoinebuteau.com)*\]*
7. **On measuring user intent:** "At Bing, we advocated for metrics that measured how quickly users found their answer, even if it meant fewer search queries per session." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
8. **On resolving trade-offs:** "An agreed-upon OEC resolves debates when a feature increases one metric but decreases another, providing a mathematical tiebreaker." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
9. **On the difficulty of OEC creation:** "Defining the OEC is often a multi-month executive struggle, but it pays dividends by drastically speeding up all future decisions." — [*Source: \[AB Tasty Podcast*](https://www.abtasty.com/blog/podcast/?ref=antoinebuteau.com)*\]*
10. **On lagging indicators:** "Because true retention takes months to measure, teams must find proxy metrics that correlate with retention but shift within a two-week experiment." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*

### Part 3: Twyman's Law and Trusting the Data

1. **On extreme results:** "Any figure that looks interesting or different is usually wrong." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
2. **On checking celebrations:** "If you see a massive improvement to your OEC, call Twyman's law and find the flaw. Triple check things before you celebrate." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
3. **On data quality:** "When a metric moves by 10% in a system that usually sees 1% changes, it is almost always an instrumentation bug, not a genius breakthrough." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
4. **On A/A testing:** "A/A tests, where both groups receive the exact same experience, are mandatory diagnostic tools to prove that your testing infrastructure is unbiased." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
5. **On instrumentation loss:** "Differences in page load times can cause tracking pixels to fire at different rates, creating the illusion of behavioral changes when the reality is missing data." — [*Source: \[KDD Keynote*](https://kdd.org/kdd2015/keynotes.html?ref=antoinebuteau.com)*\]*
6. **On skeptical cultures:** "The most mature data organizations immediately search for the error when presented with a spectacular result, rather than rushing to publish it." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
7. **On the cost of bad data:** "Getting numbers is easy; getting numbers you can trust is hard." — [*Source: \[Kameleoon Interview*](https://www.kameleoon.com/en/blog/ronny-kohavi-ab-testing?ref=antoinebuteau.com)*\]*
8. **On debugging experiments:** "If a change only impacts a specific browser or operating system, it is frequently a bug in the code rather than a legitimate user preference." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
9. **On bot traffic:** "Failing to filter out web scrapers and bots will inevitably skew experiment results, as bots behave radically differently than humans." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
10. **On continuous validation:** "Trust in the experimentation platform decays over time unless the engineering team is constantly running diagnostics to verify statistical integrity." — [*Source: \[AB Tasty Podcast*](https://www.abtasty.com/blog/podcast/?ref=antoinebuteau.com)*\]*

### Part 4: The Mechanics of Rigorous A/B Testing

1. **On statistical significance:** "While statistical significance measures how likely the observed result could have happened by chance, it does not mean the result is practically meaningful." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
2. **On sample sizes:** "You cannot compensate for a small sample size by simply running the test for months, because seasonality and external events will corrupt the data." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
3. **On when to skip testing:** "If a change is a legal requirement or a critical security patch, bypass A/B testing and ship it immediately." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
4. **On randomization units:** "Choosing the correct randomization unit, whether by user, session, or page view, is fundamental to preventing crossover effects." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
5. **On network effects:** "In marketplaces like Airbnb, standard A/B testing fails because users in the treatment group steal inventory from the control group, requiring complex cluster randomization." — [*Source: \[Airbnb Engineering Blog*](https://medium.com/airbnb-engineering?ref=antoinebuteau.com)*\]*
6. **On novelty effects:** "Users often interact with a new feature simply because it is new. You must run experiments long enough for this novelty effect to wear off." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
7. **On primacy effects:** "Long-term users might reject a better design initially because they are used to the old layout, requiring extended testing to separate resistance from actual degradation." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
8. **On statistical power:** "Running an experiment without calculating the required power beforehand usually results in stopping the test too early and making a false conclusion." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
9. **On continuous shipping:** "An experimentation platform should integrate seamlessly with CI/CD pipelines so that every deployment is automatically wrapped in an experiment." — [*Source: \[Economists in Tech Podcast*](https://open.spotify.com/episode/3G7a0pY0p4aU9Xz2r9M4z4?ref=antoinebuteau.com)*\]*

### Part 5: Overcoming the HiPPO and Building Culture

1. **On the highest paid person's opinion:** "The Highest Paid Person's Opinion (HiPPO) is the greatest threat to data-driven decision-making in corporate environments." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
2. **On shifting authority:** "A rigorous testing culture transfers power from executives who rely on intuition to the data generated by actual user behavior." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
3. **On executive buy-in:** "To build an experimentation culture, the CEO must be willing to let their own pet projects fail in a controlled test." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
4. **On resolving arguments:** "When teams disagree on a design, the standard response should shift from endless debate to simply asking if they can test it." — [*Source: \[DigginTravel Interview*](https://www.diggintravel.com/ab-testing-culture-ronny-kohavi/?ref=antoinebuteau.com)*\]*
5. **On psychological safety:** "Companies must reward teams for running rigorous tests and learning from them, even if the features themselves are ultimately discarded." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
6. **On transparency:** "Experiment results, both successes and failures, must be published openly across the organization to prevent teams from repeating the same mistakes." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
7. **On evaluating leaders:** "Leaders should be judged by how many experiments they run and the velocity of their learning, rather than by their raw win rate." — [*Source: \[AB Tasty Podcast*](https://www.abtasty.com/blog/podcast/?ref=antoinebuteau.com)*\]*
8. **On confirmation bias:** "Without controlled testing, teams will cherry-pick analytics data to prove their project was a success, entrenching confirmation bias." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
9. **On democratizing data:** "Anyone in the company, regardless of their title, should be able to propose an idea and test it against the current baseline." — [*Source: \[KDD Keynote*](https://kdd.org/kdd2015/keynotes.html?ref=antoinebuteau.com)*\]*

### Part 6: Pitfalls, Statistical Traps, and SRM

1. **On Sample Ratio Mismatch:** "If your experiment expects a 50/50 split but observes a 50.5/49.5 split, the entire experiment is invalid and the results cannot be trusted." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
2. **On hidden redirects:** "A common cause of SRM is a slight delay or error in browser redirects for the treatment group, causing them to drop out of the sample before they are counted." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
3. **On multiple comparisons:** "Looking at twenty different metrics and celebrating the one that is statistically significant is a mathematical trap that guarantees false positives." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
4. **On the p-value fallacy:** "A p-value of 0.05 does not mean there is a 5% chance the result is wrong; it means there is a 5% chance of seeing this result if the null hypothesis is true." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
5. **On peaking at data:** "Checking the results every day and stopping the experiment as soon as it crosses the significance threshold destroys the statistical validity of the test." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
6. **On survivorship bias:** "Only analyzing the users who completed a flow ignores the treatment's effect on the users who abandoned it, creating a skewed perspective." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
7. **On performance degradation:** "Adding new features often slows down the application. A/B tests must isolate the feature's value from the negative impact of the added latency." — [*Source: \[KDD Keynote*](https://kdd.org/kdd2015/keynotes.html?ref=antoinebuteau.com)*\]*
8. **On segmenting noise:** "Breaking down results into too many micro-segments will inevitably produce false positives due to the law of small numbers." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
9. **On silent failures:** "If the logging infrastructure fails for the treatment group but works for the control group, the data will incorrectly suggest the treatment caused a massive drop in engagement." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*

### Part 7: Small Changes with Massive Impact

1. **On asymmetric returns:** "The ROI on A/B testing often comes from finding a tiny, low-effort UI change that generates tens of millions of dollars." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
2. **On the Bing revenue test:** "Simply moving a line of text into the headline of an ad on Bing increased annual revenue by $100 million, a change that took hours to code." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
3. **On color optimization:** "Amazon moved the needle simply by testing the contrast and color of the checkout button, proving that aesthetic details drive behavioral shifts." — [*Source: \[Kameleoon Interview*](https://www.kameleoon.com/en/blog/ronny-kohavi-ab-testing?ref=antoinebuteau.com)*\]*
4. **On latency costs:** "Amazon and Microsoft experiments proved that slowing down a page by just 100 milliseconds directly reduces revenue and user satisfaction." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
5. **On friction reduction:** "Removing a single form field or an unnecessary click often outperforms complete page redesigns in terms of conversion lift." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
6. **On default settings:** "Users rarely change default options, making A/B testing different defaults one of the most powerful levers for changing aggregate user behavior." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
7. **On error messages:** "Clarifying the text on error messages so users understand exactly how to fix the issue can salvage millions of abandoned sessions." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
8. **On the danger of redesigns:** "Massive site redesigns often fail because they bundle hundreds of changes together, masking the fact that half of the new elements hurt the user experience." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
9. **On iterative optimization:** "Taking a failing feature and testing small, localized tweaks to its copy or placement is often required to unlock its intended value." — [*Source: \[AB Tasty Podcast*](https://www.abtasty.com/blog/podcast/?ref=antoinebuteau.com)*\]*

### Part 8: Institutionalizing Experimentation at Scale

1. **On building platforms:** "Custom-built experimentation platforms are required at scale because third-party tools struggle to integrate deeply with internal backend systems and server-side logic." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
2. **On guardrail metrics:** "Every test must be monitored against core guardrail metrics, like site crashes or latency, to ensure a small team's test does not take down the business." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*
3. **On concurrent testing:** "A mature system allows hundreds of experiments to run simultaneously without interfering with each other, scaling the company's ability to innovate." — [*Source: \[Harvard Business Review*](https://hbr.org/2017/09/the-surprising-power-of-online-experiments?ref=antoinebuteau.com)*\]*
4. **On automated shutdowns:** "The experimentation platform should automatically abort any test that causes a statistically significant degradation to a critical guardrail metric." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
5. **On democratization of analysis:** "Data scientists should not be a bottleneck; the platform should compute and visualize standard statistical tests automatically for product managers." — [*Source: \[Exp-Platform*](https://exp-platform.com/?ref=antoinebuteau.com)*\]*
6. **On the cost of maintaining variants:** "Once a test concludes, the losing code must be deleted immediately to prevent creating technical debt that slows down future development." — [*Source: \[Maven Course: Accelerating Innovation*](https://maven.com/ronny-kohavi/accelerating-innovation?ref=antoinebuteau.com)*\]*
7. **On centralized expertise:** "Organizations need a centralized Center of Excellence for experimentation to establish statistical standards and educate distributed product teams." — [*Source: \[Trustworthy Online Controlled Experiments*](https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26384BE127CB7BAA14B857640AE2?ref=antoinebuteau.com)*\]*
8. **On knowledge repositories:** "The greatest asset of an experimentation program is a searchable database of past experiments, preventing teams from testing the same bad ideas years later." — [*Source: \[DigginTravel Interview*](https://www.diggintravel.com/ab-testing-culture-ronny-kohavi/?ref=antoinebuteau.com)*\]*
9. **On treating infrastructure as a product:** "The experimentation platform itself is a product, and the internal platform team must treat developers and PMs as their primary customers." — [*Source: \[Lenny's Podcast*](https://www.lennysnewsletter.com/p/the-ultimate-guide-to-ab-testing?ref=antoinebuteau.com)*\]*