Visual summary of operating lessons from Ronny Kohavi.

Lessons from Ronny Kohavi

Ronny Kohavi built the experimentation platforms that allow Amazon, Microsoft, and Airbnb to run thousands of A/B tests. This profile breaks down his methods for evaluating metrics, spotting bad data, and keeping executive opinions from overriding actual user behavior.

Part 1: The Humbling Reality of Ideas

  1. On the failure of intuition: "Most who have run controlled experiments in customer-facing websites and applications have experienced this humbling reality: we are poor at assessing the value of ideas." — Source: [Trustworthy Online Controlled Experiments]
  2. On idea success rates: "At Microsoft and Bing, only about one-third of ideas actually improved the metrics they were designed to improve, while the rest were neutral or negative." — Source: [Lenny's Podcast]
  3. On releasing without testing: "If you release features without testing them, you are likely deploying negative changes that cancel out your positive changes, leading to stagnant growth." — Source: [DigginTravel Interview]
  4. On organizational maturity: "Moving from an environment where people assume their ideas work to one where they accept that most ideas fail is the hardest cultural shift in software development." — Source: [Maven Course: Accelerating Innovation]
  5. On the value of negative results: "Finding out an idea does not work saves the company from the long-term maintenance costs of a feature that provides no value to users." — Source: [Trustworthy Online Controlled Experiments]
  6. On domain expertise: "Even experts in a specific domain are surprisingly bad at predicting which variations of a design will win in a controlled experiment." — Source: [KDD Keynote]
  7. On the necessity of A/B testing: "A/B testing serves as a defensive mechanism to prevent well-intentioned teams from degrading the user experience, rather than serving solely as a tool for optimization." — Source: [Kameleoon Interview]
  8. On embracing failure: "You must run a massive volume of experiments because the low win rate means volume is the only way to find the rare successes." — Source: [Lenny's Podcast]
  9. On feature bloat: "Without A/B testing, products suffer from feature bloat because every idea is shipped and nothing is ever rolled back." — Source: [Economists in Tech Podcast]
  10. On humility in product design: "Experimentation forces product managers to become humble listeners of user behavior rather than dictators of user experience." — Source: [Trustworthy Online Controlled Experiments]

Part 2: Designing the Overall Evaluation Criterion

  1. On defining success: "If you do not know where you are going, any road will get you there. You must define an Overall Evaluation Criterion." — Source: [Exp-Platform]
  2. On long-term value: "The OEC should ideally measure long-term customer lifetime value, rather than short-term clicks or engagement spikes." — Source: [Trustworthy Online Controlled Experiments]
  3. On vanity metrics: "Optimizing for metrics like total page views can lead teams to design confusing interfaces that force users to click around unnecessarily." — Source: [Harvard Business Review]
  4. On aligning metrics with business goals: "The OEC acts as the bridge between daily product changes and the strategic objectives of the executive team." — Source: [Lenny's Podcast]
  5. On composite metrics: "A good OEC often combines multiple metrics, penalizing negative behaviors while rewarding positive ones." — Source: [Trustworthy Online Controlled Experiments]
  6. On the risk of single metrics: "Focusing exclusively on revenue per user can accidentally incentivize spamming users with ads, ultimately destroying trust." — Source: [KDD Keynote]
  7. On measuring user intent: "At Bing, we advocated for metrics that measured how quickly users found their answer, even if it meant fewer search queries per session." — Source: [Harvard Business Review]
  8. On resolving trade-offs: "An agreed-upon OEC resolves debates when a feature increases one metric but decreases another, providing a mathematical tiebreaker." — Source: [Trustworthy Online Controlled Experiments]
  9. On the difficulty of OEC creation: "Defining the OEC is often a multi-month executive struggle, but it pays dividends by drastically speeding up all future decisions." — Source: [AB Tasty Podcast]
  10. On lagging indicators: "Because true retention takes months to measure, teams must find proxy metrics that correlate with retention but shift within a two-week experiment." — Source: [Lenny's Podcast]

Part 3: Twyman's Law and Trusting the Data

  1. On extreme results: "Any figure that looks interesting or different is usually wrong." — Source: [Trustworthy Online Controlled Experiments]
  2. On checking celebrations: "If you see a massive improvement to your OEC, call Twyman's law and find the flaw. Triple check things before you celebrate." — Source: [Exp-Platform]
  3. On data quality: "When a metric moves by 10% in a system that usually sees 1% changes, it is almost always an instrumentation bug, not a genius breakthrough." — Source: [Lenny's Podcast]
  4. On A/A testing: "A/A tests, where both groups receive the exact same experience, are mandatory diagnostic tools to prove that your testing infrastructure is unbiased." — Source: [Trustworthy Online Controlled Experiments]
  5. On instrumentation loss: "Differences in page load times can cause tracking pixels to fire at different rates, creating the illusion of behavioral changes when the reality is missing data." — Source: [KDD Keynote]
  6. On skeptical cultures: "The most mature data organizations immediately search for the error when presented with a spectacular result, rather than rushing to publish it." — Source: [Maven Course: Accelerating Innovation]
  7. On the cost of bad data: "Getting numbers is easy; getting numbers you can trust is hard." — Source: [Kameleoon Interview]
  8. On debugging experiments: "If a change only impacts a specific browser or operating system, it is frequently a bug in the code rather than a legitimate user preference." — Source: [Trustworthy Online Controlled Experiments]
  9. On bot traffic: "Failing to filter out web scrapers and bots will inevitably skew experiment results, as bots behave radically differently than humans." — Source: [Harvard Business Review]
  10. On continuous validation: "Trust in the experimentation platform decays over time unless the engineering team is constantly running diagnostics to verify statistical integrity." — Source: [AB Tasty Podcast]

Part 4: The Mechanics of Rigorous A/B Testing

  1. On statistical significance: "While statistical significance measures how likely the observed result could have happened by chance, it does not mean the result is practically meaningful." — Source: [Trustworthy Online Controlled Experiments]
  2. On sample sizes: "You cannot compensate for a small sample size by simply running the test for months, because seasonality and external events will corrupt the data." — Source: [Lenny's Podcast]
  3. On when to skip testing: "If a change is a legal requirement or a critical security patch, bypass A/B testing and ship it immediately." — Source: [Trustworthy Online Controlled Experiments]
  4. On randomization units: "Choosing the correct randomization unit, whether by user, session, or page view, is fundamental to preventing crossover effects." — Source: [Exp-Platform]
  5. On network effects: "In marketplaces like Airbnb, standard A/B testing fails because users in the treatment group steal inventory from the control group, requiring complex cluster randomization." — Source: [Airbnb Engineering Blog]
  6. On novelty effects: "Users often interact with a new feature simply because it is new. You must run experiments long enough for this novelty effect to wear off." — Source: [Harvard Business Review]
  7. On primacy effects: "Long-term users might reject a better design initially because they are used to the old layout, requiring extended testing to separate resistance from actual degradation." — Source: [Trustworthy Online Controlled Experiments]
  8. On statistical power: "Running an experiment without calculating the required power beforehand usually results in stopping the test too early and making a false conclusion." — Source: [Maven Course: Accelerating Innovation]
  9. On continuous shipping: "An experimentation platform should integrate seamlessly with CI/CD pipelines so that every deployment is automatically wrapped in an experiment." — Source: [Economists in Tech Podcast]

Part 5: Overcoming the HiPPO and Building Culture

  1. On the highest paid person's opinion: "The Highest Paid Person's Opinion (HiPPO) is the greatest threat to data-driven decision-making in corporate environments." — Source: [Harvard Business Review]
  2. On shifting authority: "A rigorous testing culture transfers power from executives who rely on intuition to the data generated by actual user behavior." — Source: [Trustworthy Online Controlled Experiments]
  3. On executive buy-in: "To build an experimentation culture, the CEO must be willing to let their own pet projects fail in a controlled test." — Source: [Lenny's Podcast]
  4. On resolving arguments: "When teams disagree on a design, the standard response should shift from endless debate to simply asking if they can test it." — Source: [DigginTravel Interview]
  5. On psychological safety: "Companies must reward teams for running rigorous tests and learning from them, even if the features themselves are ultimately discarded." — Source: [Maven Course: Accelerating Innovation]
  6. On transparency: "Experiment results, both successes and failures, must be published openly across the organization to prevent teams from repeating the same mistakes." — Source: [Trustworthy Online Controlled Experiments]
  7. On evaluating leaders: "Leaders should be judged by how many experiments they run and the velocity of their learning, rather than by their raw win rate." — Source: [AB Tasty Podcast]
  8. On confirmation bias: "Without controlled testing, teams will cherry-pick analytics data to prove their project was a success, entrenching confirmation bias." — Source: [Exp-Platform]
  9. On democratizing data: "Anyone in the company, regardless of their title, should be able to propose an idea and test it against the current baseline." — Source: [KDD Keynote]

Part 6: Pitfalls, Statistical Traps, and SRM

  1. On Sample Ratio Mismatch: "If your experiment expects a 50/50 split but observes a 50.5/49.5 split, the entire experiment is invalid and the results cannot be trusted." — Source: [Trustworthy Online Controlled Experiments]
  2. On hidden redirects: "A common cause of SRM is a slight delay or error in browser redirects for the treatment group, causing them to drop out of the sample before they are counted." — Source: [Lenny's Podcast]
  3. On multiple comparisons: "Looking at twenty different metrics and celebrating the one that is statistically significant is a mathematical trap that guarantees false positives." — Source: [Harvard Business Review]
  4. On the p-value fallacy: "A p-value of 0.05 does not mean there is a 5% chance the result is wrong; it means there is a 5% chance of seeing this result if the null hypothesis is true." — Source: [Trustworthy Online Controlled Experiments]
  5. On peaking at data: "Checking the results every day and stopping the experiment as soon as it crosses the significance threshold destroys the statistical validity of the test." — Source: [Exp-Platform]
  6. On survivorship bias: "Only analyzing the users who completed a flow ignores the treatment's effect on the users who abandoned it, creating a skewed perspective." — Source: [Maven Course: Accelerating Innovation]
  7. On performance degradation: "Adding new features often slows down the application. A/B tests must isolate the feature's value from the negative impact of the added latency." — Source: [KDD Keynote]
  8. On segmenting noise: "Breaking down results into too many micro-segments will inevitably produce false positives due to the law of small numbers." — Source: [Trustworthy Online Controlled Experiments]
  9. On silent failures: "If the logging infrastructure fails for the treatment group but works for the control group, the data will incorrectly suggest the treatment caused a massive drop in engagement." — Source: [Lenny's Podcast]

Part 7: Small Changes with Massive Impact

  1. On asymmetric returns: "The ROI on A/B testing often comes from finding a tiny, low-effort UI change that generates tens of millions of dollars." — Source: [Trustworthy Online Controlled Experiments]
  2. On the Bing revenue test: "Simply moving a line of text into the headline of an ad on Bing increased annual revenue by $100 million, a change that took hours to code." — Source: [Harvard Business Review]
  3. On color optimization: "Amazon moved the needle simply by testing the contrast and color of the checkout button, proving that aesthetic details drive behavioral shifts." — Source: [Kameleoon Interview]
  4. On latency costs: "Amazon and Microsoft experiments proved that slowing down a page by just 100 milliseconds directly reduces revenue and user satisfaction." — Source: [Trustworthy Online Controlled Experiments]
  5. On friction reduction: "Removing a single form field or an unnecessary click often outperforms complete page redesigns in terms of conversion lift." — Source: [Lenny's Podcast]
  6. On default settings: "Users rarely change default options, making A/B testing different defaults one of the most powerful levers for changing aggregate user behavior." — Source: [Exp-Platform]
  7. On error messages: "Clarifying the text on error messages so users understand exactly how to fix the issue can salvage millions of abandoned sessions." — Source: [Maven Course: Accelerating Innovation]
  8. On the danger of redesigns: "Massive site redesigns often fail because they bundle hundreds of changes together, masking the fact that half of the new elements hurt the user experience." — Source: [Trustworthy Online Controlled Experiments]
  9. On iterative optimization: "Taking a failing feature and testing small, localized tweaks to its copy or placement is often required to unlock its intended value." — Source: [AB Tasty Podcast]

Part 8: Institutionalizing Experimentation at Scale

  1. On building platforms: "Custom-built experimentation platforms are required at scale because third-party tools struggle to integrate deeply with internal backend systems and server-side logic." — Source: [Trustworthy Online Controlled Experiments]
  2. On guardrail metrics: "Every test must be monitored against core guardrail metrics, like site crashes or latency, to ensure a small team's test does not take down the business." — Source: [Lenny's Podcast]
  3. On concurrent testing: "A mature system allows hundreds of experiments to run simultaneously without interfering with each other, scaling the company's ability to innovate." — Source: [Harvard Business Review]
  4. On automated shutdowns: "The experimentation platform should automatically abort any test that causes a statistically significant degradation to a critical guardrail metric." — Source: [Trustworthy Online Controlled Experiments]
  5. On democratization of analysis: "Data scientists should not be a bottleneck; the platform should compute and visualize standard statistical tests automatically for product managers." — Source: [Exp-Platform]
  6. On the cost of maintaining variants: "Once a test concludes, the losing code must be deleted immediately to prevent creating technical debt that slows down future development." — Source: [Maven Course: Accelerating Innovation]
  7. On centralized expertise: "Organizations need a centralized Center of Excellence for experimentation to establish statistical standards and educate distributed product teams." — Source: [Trustworthy Online Controlled Experiments]
  8. On knowledge repositories: "The greatest asset of an experimentation program is a searchable database of past experiments, preventing teams from testing the same bad ideas years later." — Source: [DigginTravel Interview]
  9. On treating infrastructure as a product: "The experimentation platform itself is a product, and the internal platform team must treat developers and PMs as their primary customers." — Source: [Lenny's Podcast]