Visual summary of operating lessons from Finale Doshi-Velez.

Lessons from Finale Doshi-Velez

Finale Doshi-Velez is a Herchel Smith Professor of Computer Science at Harvard. Her research spans interpretable machine learning, reinforcement learning, probabilistic methods and human decision-making, including healthcare applications. — Harvard Faculty Profile.

Part 1: Defining Interpretability

  1. On Interpretability: Doshi-Velez and Been Kim define interpretability in machine learning as the ability to explain a system or present it in terms a human can understand. — Interpretability Research Paper.
  2. On the Problem with Interpretability: The field has often relied on face validity—knowing an explanation when one sees it—without a shared way to compare what different users and applications need. — Interpretability Research Paper.
  3. On Formalizing the Concept: The authors call for operational definitions and evidence-based evaluation of interpretability, so researchers can compare methods rather than relying on an intuitive label. — Interpretability Research Paper.
  4. On Why Explanations Matter: Interpretability becomes especially useful when a task cannot be fully formalized—for example, when safety, fairness or the real clinical objective cannot be completely encoded. — Interpretability Research Paper.
  5. On Cognitive Capacity: An explanation should account for how much information its user can process, the time available, and the form in which concepts are presented. — Interpretability Research Paper.
  6. On Local vs. Global Explanation: Global explanations reveal patterns a model has learned overall; local explanations help explain a particular decision. Which matters depends on the user's task. — Interpretability Research Paper.
  7. On the Accuracy Trade-off: Interpretability should be evaluated alongside predictive performance in the target setting; a claim that a model is easier to understand does not by itself establish its practical value. — Interpretability Research Paper.
  8. On Concept-Based Models: Lage and Doshi-Velez propose models that map granular features into transparent concepts users can inspect and refine, including a simulated clinical-domain test. — Concept-Based Models with Human Feedback.
  9. On Defining the Audience: An explanation's usefulness depends on the recipient's expertise, available time and goal; a clinician, scientist and safety reviewer may need different information. — Interpretability Research Paper.

Part 2: Rigorous Evaluation of AI

  1. On Application-Grounded Evaluation: For a concrete application, test an explanation with the relevant experts on the task it is meant to improve—for example, clinicians making diagnoses. — Interpretability Research Paper.
  2. On Human-Grounded Evaluation: When testing with domain experts is difficult, simplified tasks with human participants can probe explanation quality while preserving the essential demands of the application. — Interpretability Research Paper.
  3. On Functionally-Grounded Evaluation: Formal proxies, such as model sparsity, can be useful without human experiments, but their relevance to real users must first be justified. — Interpretability Research Paper.
  4. On the Danger of False Proxies: A model's formal simplicity does not automatically mean its explanation will help the intended user; that link needs human or application-level evidence. — Interpretability Research Paper.
  5. On Baseline Comparisons: Real-task evaluation should compare an explanation's usefulness against relevant baselines, including how well human-produced explanations help others perform the task. — Interpretability Research Paper.
  6. On the Value of Simplicity: A simpler model is not automatically preferable: it should be judged by both task performance and whether its purported transparency actually helps the user. — Interpretability Research Paper.
  7. On Objective Evaluation: Researchers should match the kind of interpretability evaluation to the claim: a particular application calls for task-level or closely related human evidence, while a model-class claim may start with a validated proxy. — Interpretability Research Paper.
  8. On Validating Explanations: A plausible explanation may still fail to reflect a causal structure; interpretability can help expose such gaps, but it does not itself prove causality. — Interpretability Research Paper.

Part 3: Human-AI Interaction

  1. On Automation Bias: In an online vignette study, incorrect ML treatment recommendations paired with easily read explanations reduced clinicians' antidepressant-selection accuracy; an explanation did not reliably prevent overreliance. — Clinician Recommendation Study.
  2. On Designing Feedback: Concept-based models can let users shape the concepts a system learns by marking which features are relevant, instead of relying only on labels for whole cases. — Concept-Based Models with Human Feedback.
  3. On Human Intuition: Doshi-Velez describes AI as a way to augment clinicians—for example, by identifying treatment options for a clinician team and patient to discuss—rather than as a substitute for clinical judgment. — Harvard SEAS Interview.
  4. On Collaborative Systems: In ICU hypotension research, interpretable treatment suggestions are designed for clinicians to inspect alongside historical data and their own expertise, deciding which recommendations merit further investigation. — Interpretable ICU RL Study.
  5. On Incorporating Priors: Lage and coauthors tested an interpretability prior estimated from human feedback to help select models that are both predictive and understandable for a given task. — Human-in-the-Loop Interpretability Prior.
  6. On Decision Support: Decision-support tools should be evaluated in the workflow and time constraints of their intended users; primary-care co-design found that always-on detailed explanations could be unusable in short visits. — Time-Constrained Medical Decisions Study.
  7. On Disagreement: When an ML recommendation conflicts with clinical guidelines or expertise, clinicians may need on-demand information about why it differs and what action could resolve the uncertainty. — Time-Constrained Medical Decisions Study.

Part 4: AI in Healthcare and Medicine

  1. On the Potential of AI in Medicine: Doshi-Velez sees clinical AI's promise in searching large amounts of data to surface useful information or predictions at the moment a clinician needs them. — Harvard Gazette Interview.
  2. On Clinical Workloads: She points to relevant details in a patient's history that busy clinicians may miss as one practical opportunity for well-designed AI support. — Harvard Gazette Interview.
  3. On Healthcare Outcomes: At the 2017 WiDS conference, she contrasted high US healthcare spending with worse outcomes than comparable countries and argued that data science can address some of the resulting questions. — Stanford WiDS Talk Report.
  4. On Unchecked Optimism: A responsible-ML roadmap she coauthored urges teams to move beyond laboratory performance through stakeholder engagement, careful evaluation and a systematic path to clinical deployment. — Responsible ML in Healthcare Roadmap.
  5. On the Role of Algorithms: She describes AI in medicine as support for better decisions, not an autonomous doctor capable of managing a patient's full health context. — Harvard Gazette Interview.
  6. On Deployment Realities: Clinical predictors must be available when a decision is made; data entered only after a visit can be useful for retrospective analysis yet unsuitable as a real-time input. — Responsible ML in Healthcare Roadmap.
  7. On Heterogeneous Data: Clinical records reflect different collection practices and care environments; models intended to cross departments or hospitals need harmonized data and testing in the target setting. — Responsible ML in Healthcare Roadmap.
  8. On the Importance of Context: A clinical data point can mean different things depending on when, why and where it was recorded; those collection conditions need examination before treating it as a reliable model input. — Responsible ML in Healthcare Roadmap.
  9. On Actionable Knowledge: Her lab aims to turn complex data into information that can support human decisions, with healthcare among its application areas. — Harvard DtAK Lab Profile.
  10. On Clinical Validation: Before clinical use, an ML tool needs evaluation for the actual patient population, workflow and outcomes it is meant to affect, not merely a strong score on a convenient dataset. — Responsible ML in Healthcare Roadmap.

Part 5: Reinforcement Learning and Treatment Planning

  1. On Long-Term Strategies: Reinforcement learning can model treatment as a sequence of decisions whose effects arrive later, making it relevant to problems such as ventilation and sepsis management. — Observational RL Evaluation Paper.
  2. On Retrospective Data: Clinical RL often learns from historical records because experimenting freely on patients is unsafe; evaluating a proposed policy from those records requires strong assumptions and caution. — Observational RL Evaluation Paper.
  3. On Reward Design: A clinical RL reward must account for multiple objectives, such as survival, treatment effects and patient discomfort; compressing them into one signal can omit important goals. — Admissible Rewards Paper.
  4. On Model-Based RL: Decision-focused model-based RL can prioritize dynamics relevant to a treatment objective, but the learned model may become brittle when the reward definition changes; her coauthored work studies that trade-off. — Decision-Focused RL Paper.
  5. On Safety Constraints: Before using a learned clinical policy, researchers should inspect whether it proposes rarely observed or clinically implausible actions; the authors found a sepsis policy that recommended little treatment for very ill patients because of a data artifact. — Observational RL Evaluation Paper.
  6. On Evaluating Policies: Off-policy evaluation estimates how a new treatment strategy might perform using data from a different strategy, but its estimates can be misleading when representations, confounding or sample size are mishandled. — Observational RL Evaluation Paper.
  7. On Individualized Care: An interpretable ICU hypotension framework examines clinical contexts where similar patients received different care and presents concise, patient-context-specific treatment options for expert review. — Interpretable ICU RL Study.
  8. On Sepsis Management: Sepsis care involves repeated choices about fluids and vasopressors with delayed outcomes, making it a useful test case for sequential-decision research—provided the proposed policy is evaluated cautiously. — Observational RL Evaluation Paper.
  9. On Uncertainty in RL: When historical data scarcely cover a treatment choice, a policy can look promising for the wrong reason; researchers should limit proposals to adequately observed choices and involve clinical expertise before deployment. — Observational RL Evaluation Paper.
  10. On Transfer Between Settings: A clinical model may work in one hospital and fail in another as data collection, patient populations and care practices change; validation must match the destination setting. — Generalisability in Healthcare Paper.

Part 6: Accountability and Trust

  1. On System Accountability: Explanations can help people investigate errors, safety concerns and discriminatory behavior, but they are only one part of a broader accountability process. — AI Accountability and Explanation Paper.
  2. On Right for the Right Reasons: A model can be accurate for spurious reasons that fail when conditions change; examining and constraining its explanations can reveal and sometimes reduce this risk. — Right for the Right Reasons Paper.
  3. On Identifying Bias: Model explanations can expose which features influence decisions and complement aggregate audits for discriminatory patterns, though partial explanations can miss interactions. — Doshi-Velez FDIC Comment.
  4. On the Legal Context: Whether an AI decision needs an explanation depends on the legal context and the accountability question; explanation is one possible mechanism rather than an automatic full trail of logic. — AI Accountability and Explanation Paper.
  5. On the Ethics of Black Boxes: High-stakes AI needs safety and fairness evaluation matched to the decision context; the ability to inspect a model can help where performance metrics alone leave important concerns unresolved. — Interpretability Paper.
  6. On Debugging Models: Explanations can help identify model errors and unintended reliance on particular features, especially when aggregate accuracy obscures why a decision was made. — Doshi-Velez FDIC Comment.
  7. On Building Appropriate Reliance: Decision support should be tested for how clinicians actually use it: in an online antidepressant-selection study, incorrect recommendations with simple explanations sometimes made treatment choices worse. — Clinician Recommendation Study.
  8. On Regulatory Standards: Doshi-Velez and her lab urged regulators to define explainability around the information needed for a decision and to develop practical monitoring standards with stakeholder input. — Doshi-Velez FDIC Comment.
  9. On Transparency Trade-offs: A compact transparent model may allow direct inspection, while a complex model may offer only a partial explanation; the information shown should match the question being asked. — Doshi-Velez FDIC Comment.
  10. On the Burden of Proof: Before clinical deployment, teams should formulate the care problem with stakeholders, check data and evaluation choices, and establish evidence that the tool helps in its intended setting. — Responsible ML Roadmap.

Part 7: Navigating Data and Model Limitations

  1. On Missing Variables: When historical treatment data omit factors that affected both clinicians' choices and patient outcomes, a learned policy can confuse treatment effects with patient severity. — Observational Healthcare RL Paper.
  2. On Informative Measurement Patterns: The timing of clinical measurements can itself predict outcomes, but those signals may reflect local workflow; a coauthored study found indicator-only models transferred poorly across hospitals. — Measurement Indicator Study.
  3. On Data Quality: Clinical ML should be designed around when information is collected, which patients are represented and whether training records match the intended care setting. — Responsible ML Roadmap.
  4. On Algorithmic Fairness: Historical healthcare records can encode unequal access and collection practices; teams should inspect how that data was produced and whether the resulting model disadvantages groups. — Responsible ML Roadmap.
  5. On Overfitting to Hospital Workflows: A hospital-specific measurement pattern can look predictive locally yet fail elsewhere; models that relied only on such indicators performed poorly in external validation across the study's sites. — Measurement Indicator Study.
  6. On Simpler Clinical Models: More complex methods should not be presumed better for structured clinical prediction: the coauthored measurement study found limited evidence that complexity consistently beats regression in such tasks. — Measurement Indicator Study.
  7. On Managing Uncertainty: Retrospective estimates of a proposed treatment policy depend on strong assumptions and can have high variance; researchers should communicate those limits before treating a policy as clinically credible. — Observational Healthcare RL Paper.

Part 8: The Future of Responsible ML

  1. On Interdisciplinary Research: AI engineers need input from clinicians and other stakeholders early, because safety, equity and clinical usefulness depend on context that technical metrics alone cannot supply. — Harvard SEAS Interview.
  2. On Practical Impact: Healthcare ML should be judged by whether it improves care in its intended workflow, not merely by retrospective performance or publication-ready metrics. — Responsible ML Roadmap.
  3. On Continuous Monitoring: Models can lose reliability as data streams or care contexts change; Doshi-Velez's lab recommended ongoing internal and external audits and fallback mechanisms rather than one-time approval. — Doshi-Velez FDIC Comment.
  4. On Bridging the Gap: Useful healthcare AI must fit the constraints of real clinical practice and give clinicians and patients information they can act on, not just a prediction produced in a lab. — Harvard Gazette Interview.
  5. On Designing for Humans: Human-AI teams work only when the system communicates in a form people can use; the right explanation depends on the user's task, expertise and available time. — Interpretability Paper.
  6. On Realistic Expectations: Near-term clinical AI is better framed as support for triage and promising treatment options to discuss with a care team than as a replacement for clinicians. — Harvard SEAS Interview.
  7. On AI Education: Doshi-Velez argues that basic AI literacy should start early and that future engineers need training to seek domain input on safety, equity and fairness. — Harvard SEAS Interview.
  8. On Responsible Adoption: Because AI may affect consequential decisions, ethical and bias-related safeguards should be developed before a tool becomes mainstream, not deferred until after adoption. — Harvard SEAS Interview.