--- title: "Evaluating AI in Public Health Practice – The Public Health AI Handbook" source: "https://publichealthaihandbook.com/implementation/evaluation.html" author: "Bryan Tegomoh, MD, MPH" last_updated: "2026-10-30" doi: "10.5281/zenodo.18263441" license: CC BY 4.0 generated-by: handbook-markdown-export --- # Evaluating AI in Public Health Practice Evaluating an AI tool for public health practice means matching the evidence to the claim, intended use, population, setting, and consequences of error, because a high validation score does not establish reliability. A widely implemented sepsis model had 33% sensitivity and 12% positive predictive value at the evaluated threshold in external validation ([Wong et al., 2021](https://doi.org/10.1001/jamainternmed.2021.2626)). Evaluation of clinical decision support for hospital deployment is in the [Physician AI Handbook](https://physicianaihandbook.com/implementation/evaluation.html). Learning Objectives Readers will be able to: - Match evidence requirements to technical performance, transportability, operational utility, population impact, and lifecycle safety claims - Evaluate clinical utility beyond accuracy (decisions, outcomes, workflow integration, fairness) - Assess LLMs and foundation models (hallucination detection, prompt robustness, RAG systems) - Detect and monitor drift in production systems (data, concept, and label drift) - Apply regulatory frameworks (FDA SaMD, EU AI Act, GMLP principles) - Test adversarial robustness and out-of-distribution performance - Recognize when NOT to deploy despite good technical metrics Prerequisites: [Machine Learning Fundamentals](../foundations/ai-basics.html), [The Data Problem](../foundations/data.html), [Diagnostic and Clinical Decision Support](../applications/clinical.html). For LLM evaluation details, see [Large Language Models in Public Health](../future/llm-theory-practice.html). Chapter Summary (TL;DR) **The Big Picture:** In a review of 516 medical-imaging AI studies, 31 used external validation ([Kim et al., 2019](https://doi.org/10.3348/kjr.2019.0025)). In an external evaluation of a widely implemented sepsis model, sensitivity was 33% and positive predictive value was 12% at the evaluated threshold ([Wong et al., 2021](https://doi.org/10.1001/jamainternmed.2021.2626)). Technical performance in one setting cannot establish transportability, operational utility, population impact, equity, or lifecycle safety. **Match Evidence to the Claim:** 1. **Technical performance:** Discrimination, calibration, threshold-specific errors, missing data, and subgroup performance 2. **Transportability:** Independent temporal, geographic, and multi-site evaluation in relevant populations and data systems 3. **Operational utility:** Prospective evaluation of data pipelines, timeliness, workflow, workload, and failure modes 4. **Population impact:** Comparative evidence on decisions, benefits, harms, equity, and resource use 5. **Lifecycle safety:** Ongoing monitoring for drift, failures, inequitable effects, and unintended consequences **No study label answers every question. The design must match the claim and deployment risk.** **Beyond Accuracy: What Really Matters** - **Clinical Utility:** Does it change decisions? Improve outcomes? Integrate into workflows? - **Generalization:** CheXNet-style models AUC=0.93 internally, dropped to 0.82 at external hospitals ([Zech et al., 2018, *PLOS Medicine*](https://doi.org/10.1371/journal.pmed.1002683)). **Beware the generalization gap** - **Fairness Across Subgroups:** Does model perform equally for different races, ages, sexes, socioeconomic groups? - **Implementation Outcomes:** Adoption rate, alert fatigue, workflow disruption, user trust **Common Evaluation Pitfalls:** 1. **No External Validation:** Tested only on holdout from same dataset 2. **Cherry-Picked Subgroups:** “Works great on images rated as ‘excellent quality’” (real-world images are messy) 3. **Ignoring Prevalence Shift:** Trained on 50% disease prevalence, deployed where prevalence is 5% 4. **Overfitting to Dataset Quirks:** Model learns hospital-specific artifacts, not disease 5. **Evaluation-Treatment Mismatch:** Evaluate on diagnosed cases, deploy for screening **Evaluating Foundation Models and LLMs** Traditional ML metrics (accuracy, AUC) are insufficient for large language models: - **Factual Accuracy:** Does model provide correct medical information? - **Hallucination Detection:** How often does it confidently generate false information? - **Prompt Sensitivity:** Does small rewording change answers dramatically? - **Safety:** Harmful advice, biased responses, privacy leaks - **Medical Benchmarks:** MedQA, PubMedQA, USMLE-style questions (but **benchmarks do not equal clinical competence**) - **RAG Evaluation:** For retrieval-augmented generation, evaluate retrieval quality AND generation quality separately → **See also:** [Large Language Models in Public Health](../future/llm-theory-practice.html) for LLM evaluation frameworks and practical validation strategies **Continuous Monitoring (ML Ops)** Deployment is not the end. Models degrade over time: - **Data Drift:** Input distributions change (e.g., demographics shift, new disease variants) - **Concept Drift:** Relationship between features and outcome changes - **Label Drift:** Outcome prevalence changes - **Detection Methods:** Population Stability Index (PSI), statistical process control charts - **Retraining Triggers:** Predetermined thresholds for when performance drops require model updates **Regulatory Frameworks** - **FDA and IMDRF (Software as a Medical Device):** FDA device classes (I, II, III) and IMDRF SaMD risk categories (I–IV). Higher risk = more rigorous validation - **Good Machine Learning Practice (GMLP):** Guiding principles from FDA, Health Canada, and the UK MHRA for development, validation, and monitoring - **EU AI Act:** High-risk medical AI requires conformity assessment, transparency, human oversight, continuous monitoring - **Key Insight:** Even non-regulated systems benefit from regulatory-level evaluation rigor **Adversarial Robustness** - **Natural Perturbations:** Small changes in image brightness, patient demographics. Does model break? - **Adversarial Attacks:** Intentionally crafted inputs to fool model (FGSM, PGD attacks) - **Out-of-Distribution (OOD) Detection:** Can model recognize when input is unlike training data? - **EU AI Act Requirement:** High-risk systems must demonstrate robustness testing **The Obermeyer Lesson:** Healthcare cost algorithm had excellent *accuracy* but systematic *inequity*: Black patients had to be sicker than White patients to receive same risk score. **Lesson:** Technical performance ≠ ethical deployment. Must evaluate fairness explicitly. → **See also:** [Ethics, Bias, and Equity in Healthcare AI](../implementation/ethics.html) for comprehensive frameworks on evaluating AI fairness **When NOT to Deploy (Despite Good Performance):** Red flags that should halt deployment: 1. External validation shows poor generalization 2. Fairness audit reveals systematic bias 3. Clinical workflow integration causes more harm than benefit (alert fatigue) 4. Users do not trust or adopt the system 5. No plan for continuous monitoring and maintenance **The Takeaway for Public Health Practitioners:** Evaluation is not a checkbox. Internal validation estimates performance under development conditions. Independent validation tests transportability. Prospective studies examine live data pipelines, workflow, and human factors. Comparative designs test whether using the system changes decisions or outcomes. None substitutes for post-deployment monitoring. For LLMs and foundation models, evaluation must also address hallucinations, prompt sensitivity, retrieval quality, and safety in the intended use. The central task is to define the claim first, choose a design that can answer it, and appraise risk of bias and applicability before acting on the result. Check Your Understanding Test whether you can match an evaluation result to the claim it supports: **1. Same-source holdout:** A research team reports 94% accuracy for a pneumonia detection model on a held-out sample from the same hospital and period used for development. What does this result support? - A. Internal performance under the study conditions - B. Transportability to other hospitals - C. Effectiveness in a live workflow - D. Improved patient outcomes Click for answer **Answer: A. Internal performance under the study conditions** Why: The evaluation can estimate performance under the sampled development conditions if leakage and analysis bias are controlled. It does not establish transportability, workflow performance, or an effect on outcomes. **2. The Epic Lesson:** In an external evaluation, Epic’s sepsis model had 33% sensitivity and 12% PPV at the evaluated threshold. What does this mean practically? - A. Model missed 2 out of 3 sepsis cases; 88% of alerts were false alarms - B. Model worked great, 33% and 12% are good metrics - C. Model needs minor tuning to reach production quality - D. External validation was too strict Click for answer **Answer: A. Missed 2 of 3 cases; 88% false alarms** Why: 33% sensitivity means the model detected approximately 1 in 3 sepsis cases. A 12% PPV means approximately 12 of 100 alerts identified a case. These figures indicate substantial missed-case and false-alert burdens at that threshold. Deployment status does not establish acceptable performance. **3. LLM Evaluation:** You are evaluating a medical chatbot powered by an LLM. It scores 85% on MedQA (medical exam questions). Can you deploy it? - A. Yes, 85% accuracy is excellent - B. No, must also test hallucination rate, prompt robustness, safety - C. No, need prospective clinical validation - D. Both B and C Click for answer **Answer: D. Both B and C** Why: Benchmark performance (MedQA) ≠ clinical competence. LLMs can ace exams but hallucinate dangerous medical advice. Must test: hallucination detection, prompt sensitivity (small rewording changes answer?), safety (harmful advice?), and prospective validation in real clinical workflow before deployment. **4. Model Drift:** Your outbreak prediction model performed well for 2 years. Suddenly accuracy drops from 82% to 61%. What’s likely happening? - A. The model is broken, rebuild from scratch - B. Distribution shift: inputs or feature-outcome relationships changed (new disease variant, demographic shifts) - C. Model was always bad, just got lucky initially - D. Evaluation metrics are wrong Click for answer **Answer: B. Distribution shift** Why: A sudden drop after stable performance points to distribution shift. New disease variants, demographic shifts, and changes in testing practices can change the inputs (data drift) or the relationship between features and outcome (concept drift); a sudden loss of discrimination points especially to concept drift. This is why **continuous monitoring** and **retraining triggers** are essential. Models degrade over time in production. **5. When NOT to Deploy:** Your model has 91% AUC internally, 87% AUC at 3 external hospitals. Fairness audit shows White patients: 90% sensitivity, Black patients: 65% sensitivity. Should you deploy? - A. Yes, 87% external AUC is strong - B. No, systematic bias is unacceptable even with good overall performance - C. Yes, but only for White patients - D. Deploy and monitor fairness post-deployment Click for answer **Answer: B. No, systematic bias is unacceptable** Why: Technical performance ≠ ethical deployment. This is the **Obermeyer lesson**: excellent accuracy but systematic inequity. Black patients receive worse care because model systematically underperforms. Must address fairness BEFORE deployment, not after. Option D (“monitor after deployment”) puts patients at risk while you collect evidence of harm. **Halt deployment until bias is addressed.** **Scoring:** - **5/5:** You understand evaluation rigor. Ready to evaluate AI systems critically. - **3–4/5:** Good foundation. Review sections where you missed questions, especially Epic sepsis case study. - **0–2/5:** Reread the TL;DR summary and the Introduction section. The evaluation crisis is the most important concept in this chapter. ------------------------------------------------------------------------ ## Introduction: The Evaluation Crisis in AI **March 2019, Korean Journal of Radiology:** [Kim et al.](https://pubmed.ncbi.nlm.nih.gov/30799571/) publish a systematic review examining 516 studies on AI algorithms for medical image analysis. **Their sobering finding:** Only **6% performed external validation** (31 of 516 studies) on data from different institutions. The vast majority tested models only on hold-out sets from the same dataset used for training, a practice that provides minimal evidence of real-world performance. ------------------------------------------------------------------------ **June 2021, JAMA Internal Medicine:** [Wong et al. publish an external validation](https://doi.org/10.1001/jamainternmed.2021.2626) of Epic’s sepsis prediction model, which was implemented at hundreds of US hospitals. **The model’s performance:** - **Sensitivity: 33%** (missed 2 out of 3 sepsis cases) - **Positive predictive value: 12%** (88% of alerts were false positives) - **Alert burden:** 18% of hospitalizations would have generated at least one alert at the evaluated threshold **Authors’ conclusion:** The model had poor discrimination and calibration, and its alert burden raised concerns about clinical value. The evaluation was a retrospective cohort study at one academic health system. Its importance is that the proprietary model was already in widespread clinical use before independent validation was published. ------------------------------------------------------------------------ **The Evaluation Gap:** Between lab performance and real-world deployment lies a chasm that has claimed many promising AI systems. Clinical AI evaluation currently resembles standardized testing more than real-world practice: retrospective accuracy on curated datasets, with limited measurement of workflow fit, adoption, safety guardrails, or downstream outcomes ([Azad et al., Nature Medicine, 2026](https://doi.org/10.1038/s41591-025-04198-1)). **In the lab (illustrative):** - Curated, high-quality datasets - Balanced classes (50% positive, 50% negative) - Consistent protocols - Expert-confirmed labels - AUC-ROC = 0.95 **In the real world (illustrative):** - Messy, incomplete data - Rare events (1–5% prevalence) - Variable protocols across sites - Ambiguous cases - AUC-ROC = 0.68 Performance ≠ Safety **Performance evaluation** covers accuracy, generalization, fairness, and robustness. High performance does not guarantee safe clinical deployment. **Safety validation** requires additional frameworks: - Failure mode and effects analysis (FMEA) - Hazard analysis and risk assessment - Worst-case scenario testing - Operational safety validation For comprehensive safety evaluation beyond performance metrics, see [AI Safety in Healthcare](../implementation/safety.html). **The consequences are severe:** - **Failed deployments**: Models that work in development but fail in production - **Hidden biases**: Systems that perform well on average but poorly for specific groups - **Wasted resources**: Millions invested in systems that do not deliver promised benefits - **Patient harm**: Incorrect predictions leading to inappropriate treatments - **Eroded trust**: Clinicians lose confidence in AI after experiencing failures **The Adoption-Value Gap:** [McKinsey’s 2025 cross-industry State of AI survey](https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai) included 1,993 participants across 105 countries. Its organizational findings provide context but do not establish public health adoption or value rates: - **88% of organizations report using AI** in at least one function (up from 78% in 2024) - Yet **only 6% qualify as “high performers”** achieving 5%+ earnings impact from AI - Only 7% have fully scaled AI across their organizations **Five recurring barriers** prevent organizations from crossing this gap: 1. **Data quality issues**: Fragmented systems, inconsistent metadata, accuracy problems 2. **Financial justification difficulty**: Inability to demonstrate measurable long-term gains 3. **Skills shortage**: Insufficient data scientists, engineers, and change-management expertise 4. **Organizational silos**: Lack of cross-functional collaboration 5. **Governance uncertainty**: Evolving privacy regulations and security concerns **The lesson for public health:** Deployment is not the finish line. Organizations that treat AI as technology procurement rather than workflow transformation consistently underperform. The 6% classified as high performers invest in data infrastructure, workforce training, and governance frameworks before deploying AI tools. Why This Chapter Matters Rigorous evaluation is the bridge between AI research and AI implementation. Without it, unvalidated systems reach deployment. Evaluate AI systems across **five critical dimensions:** 1. **Technical performance**: Accuracy, calibration, robustness 2. **Generalizability**: External validity, temporal stability, geographic transferability 3. **Clinical/Public health utility**: Impact on decisions and outcomes 4. **Fairness and equity**: Performance across demographic subgroups 5. **Implementation outcomes**: Adoption, usability, sustainability Each dimension calls for its own validation design and critical appraisal of the published evidence. ------------------------------------------------------------------------ ## The Multidimensional Nature of Evaluation ### What Are We Really Evaluating? Evaluating an AI system requires more than measuring accuracy. In public health and clinical contexts, evaluation must assess multiple dimensions. #### 1. Technical Performance **Question:** Does the model make accurate predictions on new data? **Key aspects:** - **Discrimination:** Can the model distinguish between positive and negative cases? - **Calibration:** Do predicted probabilities match observed frequencies? - **Robustness:** Does performance degrade with missing data or noise? - **Computational efficiency:** Speed and resource requirements for deployment **Relevant for:** All AI systems (foundational requirement) ------------------------------------------------------------------------ #### 2. Generalizability **Question:** Will the model work in settings different from where it was developed? **Key aspects:** - **Geographic transferability:** Performance at different institutions, regions, countries - **Temporal stability:** Does performance degrade as time passes and data distributions shift? - **Population differences:** Performance across different patient demographics, disease prevalence - **Setting transferability:** Hospital vs. primary care vs. community settings **Relevant for:** Any system intended for broad deployment **Critical insight:** [Geirhos et al., 2020, *Nature Machine Intelligence*](https://www.nature.com/articles/s42256-020-00257-z) showed that AI models often learn “shortcuts”, spurious correlations specific to training data that do not generalize. For example, pneumonia detection models learned to identify portable X-ray machines (used for sicker patients) rather than actual pneumonia. ------------------------------------------------------------------------ #### 3. Clinical/Public Health Utility **Question:** Does the model improve decision-making and outcomes? **Key aspects:** - **Decision impact:** Does it change clinician decisions? - **Outcome improvement:** Does it lead to better patient outcomes? - **Net benefit:** Does it provide value above existing approaches? - **Cost-effectiveness:** Does it provide value commensurate with costs? **Relevant for:** Clinical decision support, diagnostic tools **Critical distinction:** A model can be **statistically accurate but clinically useless**. Example: A model predicting hospital mortality with AUC-ROC = 0.85 sounds impressive, but if it does not change management or improve outcomes, it adds no value. For framework on clinical utility assessment, see [Vickers et al., 2019, *Diagnostic and Prognostic Research*](https://diagnprognres.biomedcentral.com/articles/10.1186/s41512-019-0064-7) on decision curve analysis. ------------------------------------------------------------------------ #### 4. Fairness and Equity **Question:** Does the model perform equitably across population subgroups? **Key aspects:** - **Subgroup performance:** Stratified metrics by race, ethnicity, gender, age, socioeconomic status - **Error rate disparities:** Differential false positive/negative rates - **Outcome equity:** Does deployment narrow or widen health disparities? - **Representation:** Are all groups adequately represented in training data? **Relevant for:** All systems affecting humans **Essential reading:** [Obermeyer et al., 2019, *Science*](https://www.science.org/doi/10.1126/science.aax2342) - racial bias in healthcare algorithm affecting millions; [Gianfrancesco et al., 2018, *JAMA Internal Medicine*](https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2697394) - potential biases in ML algorithms using EHR data. Parikh et al. argue epidemiology still lacks frameworks to measure population AI exposure and study health effects at scale, distinguishing **ambient AI exposure** (algorithmic curation and AI-mediated institutional decisions that reach people without individual opt-in) from **personal AI exposure** (direct, volitional tool use), and treating AI as a determinant of health rather than only a clinical device ([Parikh et al., 2026](https://doi.org/10.1038/s44360-026-00188-w)). Framework Perspective with survey illustration: not a causal estimate of AI causing specific disease outcomes. ------------------------------------------------------------------------ #### 5. Implementation Outcomes **Question:** Is the model adopted and used effectively in practice? **Key aspects:** - **Adoption:** Are users actually using it as intended? - **Usability:** Can users operate it efficiently? - **Workflow integration:** Does it fit smoothly into existing processes? - **Sustainability:** Will it continue to be used and maintained over time? **Relevant for:** Any deployed system **Framework:** [Proctor et al., 2011, *Administration and Policy in Mental Health*](https://link.springer.com/article/10.1007/s10488-010-0319-7) - implementation outcome taxonomy. Al-Ajlouni and Njei argue that clinical accuracy alone is a weak proxy for system value and propose an AI Productivity Index as a preliminary composite lens for operational impact, implementation burden, opportunity costs, downstream utilisation, and equity of gains or burdens, to sit beside safety, efficacy, and formal economic evaluation in procurement and post-deployment governance ([Al-Ajlouni & Njei, Clinical Medicine, online 20 Aug 2026](https://doi.org/10.1016/j.clinme.2026.100638); [OA PDF](https://europepmc.org/articles/PMC13572302?pdf=render)). Their illustrative domains include workflow efficiency, integration burden, clinician cognitive load, downstream resource use, and equity/access impact. Opinion framework with illustrative domains, not a validated scored instrument; do not replace CHEERS-AI or local claim-specific evaluation. ------------------------------------------------------------------------ ## Match Evidence to the Public Health Claim Evidence for public health AI is multidimensional, not a single ladder. The study design should be judged against the question it is intended to answer, then appraised for risk of bias, directness, precision, and applicability. GRADE assigns certainty to a body of evidence for a specified outcome, not to a journal, vendor, individual paper, or generic validation stage ([CDC ACIP GRADE Handbook, 2024](https://www.cdc.gov/acip-grade-handbook/hcp/chapter-7-grade-criteria-determining-certainty-of-evidence/index.html)). | Evidence question | Evidence needed | Appropriate appraisal or reporting approach | |----|----|----| | Does the model perform as specified? | Locked-model evaluation on representative data, including calibration, threshold-specific errors, missing data, and subgroup performance | TRIPOD+AI for transparent reporting and PROBAST+AI for quality, risk of bias, and applicability ([Collins et al., 2024](https://doi.org/10.1136/bmj-2023-078378); [Moons et al., 2025](https://doi.org/10.1136/bmj-2024-082505)) | | Does performance transfer across populations, jurisdictions, or time? | Independent temporal, geographic, and multi-site evaluation in data systems and populations relevant to intended use | Explicit assessment of case mix, prevalence, measurement, calibration, operating thresholds, and uncertainty | | Does the system work in public health operations? | Prospective evaluation of data pipelines, timeliness, representativeness, acceptability, workflow, workload, failure modes, and subgroup effects | CDC surveillance-system attributes and, for live decision support, DECIDE-AI reporting items ([CDC, 2001](https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5013a1.htm); [Vasey et al., 2022](https://doi.org/10.1038/s41591-022-01772-9)) | | Does using the system improve public health decisions or outcomes? | Comparative evidence on decisions, processes, population outcomes, harms, equity, and resource use | Randomized designs when feasible, or a justified quasi-experimental design such as an interrupted time series or controlled before-after study; CONSORT-AI applies to reports of randomized AI intervention trials ([Liu et al., 2020](https://doi.org/10.1038/s41591-020-1034-x)) | | Does benefit persist after deployment? | Ongoing measurement of data drift, performance, use, overrides, adverse events, inequitable effects, timeliness, stability, and unintended consequences | Risk-based monitoring across the lifecycle; NICE separates design, value, performance, deployment, and post-deployment evidence rather than collapsing them into one study hierarchy ([NICE Evidence Standards Framework](https://www.nice.org.uk/what-nice-does/digital-health/evidence-standards-framework-esf-for-digital-health-technologies)) | Reporting guidelines improve transparency, but complete reporting does not establish low risk of bias or public health benefit. Risk-of-bias tools and causal designs must also match the question. A randomized trial is well suited to estimating the causal effect of using an AI intervention, but it is not automatically the best design for estimating discrimination, calibration, rare harms, transportability, or long-term drift. Staged Evaluation Is an Operational Sequence A staged pathway remains useful for managing deployment risk: 1. Develop the model and estimate internal performance without data leakage. 2. Evaluate the locked model on independent temporal, geographic, or multi-site data relevant to intended use. 3. Run silent or limited prospective evaluation to test live data pipelines, workflow, and failure modes. 4. Use a comparative design when the claim concerns effects on decisions, outcomes, equity, or resource use. 5. Continue surveillance after deployment because performance and use can change. These stages are complementary. They should not be converted into a universal weakest-to-strongest evidence score. For lifecycle staging of medical AI evaluation (five gated phases with fall-back triggers), see Ye et al. in npj Digital Medicine ([Ye et al., 2026](https://doi.org/10.1038/s41746-026-03155-7)). Framework proposal; the Physician AI Handbook carries the fuller teaching. ### Recent Evidence That Changes Evaluation Practice Recent studies reinforce seven controls that should be specified before evaluation begins: 1. **Separate process measures from outcomes.** A pragmatic cluster-randomized trial across Kenyan primary-care facilities found improvements in several expert-rated care-process measures with generative AI decision support, while its prespecified 14-day treatment-failure outcome did not differ significantly. The result supports better documentation and reasoning processes, not a claim of improved patient outcomes ([Agweyu et al., 2026](https://doi.org/10.1038/s41591-026-04503-6)). 2. **Pair static benchmarks with valid perturbation tests.** Dynamic red-teaming of health LLMs exposed substantial brittleness among answers that were correct at baseline. The reported attack success depends on the mutation set, search budget, and judge, so it is evidence of stress-test vulnerability rather than an estimate of routine clinical error ([Pan et al., 2026](https://doi.org/10.1038/s44360-026-00152-8)). 3. **Treat evaluators as part of the measurement system.** Ratings of real clinical-case responses varied across physician seniority and practice setting, and automated evaluators did not fully reproduce the features of physician judgment. Report evaluator composition and test whether rankings are stable across relevant strata ([Shi et al., 2026](https://doi.org/10.1038/s41746-026-02942-6)). 4. **Permit automated judges to abstain.** MedQADE, a 2026 preprint, found that automated judges could approach clinician-level agreement while failing to express the caution physicians showed on difficult items. Automated evaluation should include uncertainty thresholds, escalation, and model-family bias checks ([Philipp et al., 2026, preprint](https://arxiv.org/abs/2607.01103)). 5. **Do not equate local relevance with deployment validation.** IyawoBench provides locally grounded Nigerian primary-care triage cases, but its synthetic cases and limited reference-label process require independent relabeling, prompt-sensitivity analysis, geographic validation, and prospective evaluation before deployment claims ([Gabriel et al., 2026, preprint](https://arxiv.org/abs/2605.23465); [Gabriel and Olawuyi, 2026, preprint](https://arxiv.org/abs/2607.29085)). 6. **Do not treat a pooled AUC as a TB-program endpoint.** A 17 August 2026 *PLOS One* PRISMA/PROBAST synthesis of AI/ML models for tuberculosis treatment failure included 34 studies and pooled 19 of them (100,790 participants) to AUC 0.836 (95% CI 0.799–0.868) for predicting treatment failure with I² 97.9%; only 8/34 had external validation and 1/34 was low risk of bias overall, while HIV-included cohorts dropped to AUC 0.748 versus 0.924 when HIV-positive patients were excluded ([Kamulegeya et al., 2026](https://doi.org/10.1371/journal.pone.0355124)). The authors write that current models remain investigational and are not ready for routine clinical implementation; the pool is a central-tendency summary across mixed failure definitions, not a DOTS, mortality, or transmission result. 7. **Do not treat VAERS/FAERS disproportionality plus in-sample ML as incidence, causality, or a vaccine-safety endpoint.** Qing et al.’s 10 August 2026 *Infectious Diseases of Poverty* analysis of 9,446 AVA reports found 180 depression-related cases (ROR 5.02, 95% CI 4.33–5.81; signal persisted after excluding concurrent vaccines, n = 119, ROR 4.97) and a random-split Random Forest test AUC of 0.89 for predicting depression-related reports (XGBoost 0.856; SHAP dominated by serious-AE and time-to-onset flags, not dose or site), still a spontaneous-reporting model with no denominator ([Qing et al., 2026](https://doi.org/10.1186/s40249-026-01483-0)). The authors themselves require the numbers to be read as hypothesis-generating pharmacovigilance, not as AVA-caused depression, not as absolute risk, and not as a deployed immunization-program early-warning system; FAERS levofloxacin / doxycycline RORs (1.37 / 1.53) were likewise not indication-restricted to anthrax. The operational rule is simple: define the claim first, then select the outcome, study design, stress tests, evaluator panel, and monitoring plan that can support that claim. ------------------------------------------------------------------------ ## Performance Metrics: Choosing the Right Measures ### Classification Metrics For binary classification (disease/no disease, outbreak/no outbreak), numerous metrics exist. **No single metric tells the whole story.** #### The Confusion Matrix Foundation All classification metrics derive from the **2×2 confusion matrix:** | | **Predicted Positive** | **Predicted Negative** | |-----------------------|------------------------|------------------------| | **Actually Positive** | True Positives (TP) | False Negatives (FN) | | **Actually Negative** | False Positives (FP) | True Negatives (TN) | **Example:** TB screening of 1,000 individuals; 100 actually have TB | | Predicted TB+ | Predicted TB- | |------------------|--------------:|--------------:| | **Actually TB+** | 85 (TP) | 15 (FN) | | **Actually TB-** | 90 (FP) | 810 (TN) | All other metrics are calculated from this matrix. ------------------------------------------------------------------------ #### Core Metrics **1. Sensitivity (Recall, True Positive Rate)** \\\text{Sensitivity} = \frac{TP}{TP + FN} = \frac{TP}{\text{All Actual Positives}}\\ - **Interpretation:** Of all actual positives, what proportion does the model identify? - **Example:** 85/100 = 85% (identified 85 of 100 TB cases) - **When to prioritize:** High-stakes screening (must catch most cases), early disease detection, rule-out tests - **Trade-off:** Maximizing sensitivity → more false positives **2. Specificity (True Negative Rate)** \\\text{Specificity} = \frac{TN}{TN + FP} = \frac{TN}{\text{All Actual Negatives}}\\ - **Interpretation:** Of all actual negatives, what proportion did we correctly identify? - **Example:** 810/900 = 90% (correctly ruled out TB in 810 of 900 healthy people) - **When to prioritize:** Confirmatory tests, when false alarms are costly, rule-in tests - **Trade-off:** Maximizing specificity → more false negatives ------------------------------------------------------------------------ **3. Positive Predictive Value (Precision, PPV)** \\\text{PPV} = \frac{TP}{TP + FP} = \frac{TP}{\text{All Predicted Positives}}\\ - **Interpretation:** Of all predicted positives, what proportion are actually positive? - **Example:** 85/175 = 49% (49% of positive predictions are correct) - **When to prioritize:** When acting on predictions is costly (treatments, interventions) - **Critical property:** **Depends heavily on disease prevalence** **Prevalence dependence example:** | Scenario | Prevalence | Sensitivity | Specificity | PPV | |------------------------|------------|-------------|-------------|-----| | High-burden TB setting | 10% | 85% | 90% | 49% | | Low-burden TB setting | 1% | 85% | 90% | 8% | **Same model, vastly different PPV.** In low-prevalence settings, even high specificity leads to poor PPV. For detailed explanation, see [Altman & Bland, 1994, *BMJ*](https://www.bmj.com/content/309/6947/102) on diagnostic tests and prevalence. ------------------------------------------------------------------------ **4. Negative Predictive Value (NPV)** \\\text{NPV} = \frac{TN}{TN + FN} = \frac{TN}{\text{All Predicted Negatives}}\\ - **Interpretation:** Of all predicted negatives, what proportion are actually negative? - **Example:** 810/825 = 98% (98% of negative predictions are correct) - **When to prioritize:** Rule-out tests, when missing disease is catastrophic - **Critical property:** Also depends on prevalence (high prevalence → lower NPV) ------------------------------------------------------------------------ **5. Accuracy** \\\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{\text{Correct Predictions}}{\text{All Predictions}}\\ - **Interpretation:** Overall proportion of correct predictions - **Example:** (85+810)/1000 = 89.5% - **Major limitation:** **Misleading for imbalanced datasets** **Classic pitfall:** Dataset: 1,000 patients, 10 with disease (1% prevalence) **Naive model:** Predict “no disease” for everyone - Accuracy: 990/1000 = **99%** - But sensitivity = 0% (misses all disease cases) **Takeaway:** Accuracy alone is insufficient, especially for rare events. ------------------------------------------------------------------------ **6. F1 Score (Harmonic Mean of Precision and Recall)** \\F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2TP}{2TP + FP + FN}\\ - **Interpretation:** Balance between precision and recall - **Range:** 0 (worst) to 1 (perfect) - **When to use:** When a single metric must balance both concerns - **Limitation:** Ignores true negatives (unsuitable when true negatives matter) **Variants:** - \\F_2\\ score: Weights recall higher than precision - \\F\_{0.5}\\ score: Weights precision higher than recall ------------------------------------------------------------------------ #### Threshold-Independent Metrics **7. Area Under the ROC Curve (AUC-ROC)** The **Receiver Operating Characteristic (ROC)** curve plots: - **Y-axis:** True Positive Rate (Sensitivity) - **X-axis:** False Positive Rate (1 - Specificity) …across all possible classification thresholds (0 to 1). **AUC-ROC interpretation:** - **0.5** = Random guessing (diagonal line) - **0.6–0.7** = Poor discrimination - **0.7–0.8** = Acceptable - **0.8–0.9** = Excellent - **\>0.9** = Outstanding (rare in clinical applications) **Alternative interpretation:** Probability that a randomly selected positive case is ranked higher than a randomly selected negative case. **Advantages:** - Threshold-independent (single summary metric) - Not affected by class imbalance (in terms of metric itself) - Standard metric for model comparison **Limitations:** - May overemphasize performance at thresholds that would not be used clinically - Does not indicate optimal threshold - Can be misleading for highly imbalanced data (see Average Precision) For comprehensive guide, see [Hanley & McNeil, 1982, *Radiology*](https://pubs.rsna.org/doi/10.1148/radiology.143.1.7063747) on the meaning and use of AUC. ------------------------------------------------------------------------ **8. Average Precision (Area Under Precision-Recall Curve)** The **Precision-Recall (PR)** curve plots: - **Y-axis:** Precision (PPV) - **X-axis:** Recall (Sensitivity) …across all thresholds. **Average Precision (AP):** Area under PR curve PR curves are more informative than ROC curves for imbalanced datasets where the positive class is rare. They focus on performance on the positive class (which matters more when it is rare), whereas ROC can be misleadingly optimistic when the negative class dominates. **Illustrative example:** Disease with 1% prevalence - **AUC-ROC = 0.90** (appears strong) - **Average Precision = 0.25** (reveals poor performance on actual disease cases) **When to use:** Rare disease detection, outbreak detection, any imbalanced problem For detailed comparison, see [Saito & Rehmsmeier, 2015, *PLOS ONE*](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432) on precision-recall vs. ROC curves. ------------------------------------------------------------------------ #### Choosing Metrics by Scenario | **Scenario** | **Primary Metrics** | **Rationale** | |----|----|----| | **COVID-19 airport screening** | Sensitivity, NPV | Must catch most cases; false positives acceptable (confirmatory testing available) | | **Cancer diagnosis confirmation** | Specificity, PPV | False positives → unnecessary surgery; high bar for confirmation | | **Automated triage system** | AUC-ROC, Calibration | Need good ranking across full risk spectrum | | **Rare disease detection** | Average Precision, Sensitivity | Standard AUC-ROC misleading when imbalanced | | **Syndromic surveillance** | Sensitivity, Timeliness | Early detection critical; false alarms tolerable (investigation cheap) | | **Clinical decision support** | PPV, Calibration | Clinicians ignore if too many false alarms; need well-calibrated probabilities | ------------------------------------------------------------------------ ### Calibration: Do Predicted Probabilities Mean What They Say? **Calibration** assesses whether predicted probabilities match observed frequencies. **Example of well-calibrated model:** - Model predicts “30% risk of readmission” for 100 patients - About 30 of those 100 are actually readmitted - Predicted probability ≈ observed frequency **Poor calibration:** - Model predicts “30% risk” but 50% are actually readmitted → **underestimates risk** - Model predicts “30% risk” but 15% are actually readmitted → **overestimates risk** ------------------------------------------------------------------------ #### Measuring Calibration **1. Calibration Plot** **Method:** 1. Bin predictions into groups (e.g., 0–10%, 10–20%, …, 90–100%) 2. For each bin, calculate: - Mean predicted probability (x-axis) - Observed frequency of outcome (y-axis) 3. Plot points 4. Perfect calibration: points lie on diagonal line (y = x) **Interpretation:** - Points **above diagonal:** Model underestimates risk (predicts lower risk than reality) - Points **below diagonal:** Model overestimates risk (predicts higher risk than reality) ------------------------------------------------------------------------ **2. Brier Score** \\\text{Brier Score} = \frac{1}{N} \sum\_{i=1}^{N} (p_i - y_i)^2\\ where \\p_i\\ = predicted probability, \\y_i\\ = actual outcome (0 or 1) - **Range:** 0 (perfect) to 1 (worst) - **Lower is better** - Combines discrimination and calibration into single metric - Can be decomposed into **calibration** and **refinement** components **Interpretation:** - Baseline: predicting the prevalence \\p\\ for everyone gives a Brier score of \\p(1-p)\\, which equals 0.25 only at 50% prevalence (0.09 at 10% prevalence) - Compare the model’s score with this baseline rather than with fixed cutoffs For a short methodological note on the Brier score, see [Rufibach, 2010, *Journal of Clinical Epidemiology*](https://doi.org/10.1016/j.jclinepi.2009.11.009). ------------------------------------------------------------------------ **3. Expected Calibration Error (ECE)** \\\text{ECE} = \sum\_{m=1}^{M} \frac{n_m}{N} \|\text{acc}(B_m) - \text{conf}(B_m)\|\\ where: - \\M\\ = number of bins - \\B_m\\ = set of predictions in bin \\m\\ - \\n_m\\ = number of predictions in bin \\m\\ - \\\text{acc}(B_m)\\ = accuracy in bin \\m\\ - \\\text{conf}(B_m)\\ = average confidence in bin \\m\\ **Interpretation:** Average difference between predicted and observed probabilities across bins (weighted by bin size) ------------------------------------------------------------------------ #### Why Calibration Matters **Clinical decision-making requires well-calibrated probabilities:** **Scenario 1: Treatment threshold** - If risk \>20%, prescribe preventive medication - Poorly calibrated model: risk actually 40% when model says 20% - **Result:** Under-treatment of high-risk patients **Scenario 2: Resource allocation** - Allocate home health visits to top 10% risk - Overconfident model: predicted “high risk” patients are not actually high risk - **Result:** Resources wasted on low-risk patients, true high-risk patients missed **Scenario 3: Patient counseling** - Tell patient: “You have a 30% chance of complications” - If the model is poorly calibrated, this number is meaningless - **Result:** Informed consent based on inaccurate information The Deep Learning Calibration Problem **Common issue:** Deep neural networks often produce poorly calibrated probabilities out-of-the-box. They tend to be **overconfident** (predicted probabilities too extreme). **Cause:** Modern neural networks are optimized for accuracy, not calibration. Guo et al. found that greater depth and width, batch normalization, and less weight decay were each associated with worse calibration. **Evidence:** [Guo et al., 2017, *ICML*](https://proceedings.mlr.press/v70/guo17a.html) - “On Calibration of Modern Neural Networks” **Solution:** Post-hoc calibration methods: - **Temperature scaling:** Simplest and most effective - **Platt scaling:** Logistic regression on model outputs - **Isotonic regression:** Non-parametric calibration **Takeaway:** Always assess and correct calibration for deep learning models before deployment. ------------------------------------------------------------------------ ### Regression Metrics For continuous outcome prediction (disease burden, resource utilization, epidemic size): **1. Mean Absolute Error (MAE)** \\\text{MAE} = \frac{1}{N} \sum\_{i=1}^{N} \|y_i - \hat{y}\_i\|\\ - **Interpretation:** Average absolute difference between prediction and truth - **Unit:** Same as outcome variable - **Advantage:** Interpretable, less sensitive to outliers than RMSE - **Example:** MAE = 3.2 days (average error in predicting length of stay) ------------------------------------------------------------------------ **2. Root Mean Squared Error (RMSE)** \\\text{RMSE} = \sqrt{\frac{1}{N} \sum\_{i=1}^{N} (y_i - \hat{y}\_i)^2}\\ - **Interpretation:** Square root of average squared error - **Property:** Penalizes large errors more heavily than MAE (due to squaring) - **When to use:** When large errors are particularly problematic **Relationship:** RMSE ≥ MAE always (equality only if all absolute errors are equal) ------------------------------------------------------------------------ **3. R-squared (Coefficient of Determination)** \\R^2 = 1 - \frac{\sum\_{i=1}^{N} (y_i - \hat{y}\_i)^2}{\sum\_{i=1}^{N} (y_i - \bar{y})^2} = 1 - \frac{\text{SS}\_{\text{res}}}{\text{SS}\_{\text{tot}}}\\ - **Range:** 0 to 1 (can be negative if model worse than mean) - **Interpretation:** Proportion of variance in outcome explained by model - **Example:** R² = 0.65 means model explains 65% of variance - **Limitation:** Can be artificially inflated by adding more features ------------------------------------------------------------------------ **4. Mean Absolute Percentage Error (MAPE)** \\\text{MAPE} = \frac{100\\}{N} \sum\_{i=1}^{N} \left\| \frac{y_i - \hat{y}\_i}{y_i} \right\|\\ - **Interpretation:** Average percentage error - **Advantage:** Scale-independent (can compare across different units) - **Example:** MAPE = 15% (average error is 15% of true value) - **Limitation:** Undefined when actual value is zero; penalizes over-predictions more than under-predictions (under-prediction error cannot exceed 100%; over-prediction error is unbounded) ------------------------------------------------------------------------ ### Survival Analysis Metrics For time-to-event prediction (mortality, readmission, disease progression): **1. Concordance Index (C-index, Harrell’s C-statistic)** - Extension of AUC-ROC to survival data with censoring - **Interpretation:** Probability that, for two randomly selected individuals, the one who experiences the event first has higher predicted risk - **Range:** 0.5 (random) to 1.0 (perfect) - **Handles censoring:** Pairs where censoring occurs are excluded or weighted **For details:** [Harrell et al., 1982, *JAMA*](https://jamanetwork.com/journals/jama/article-abstract/373285) - original C-index paper. ------------------------------------------------------------------------ **2. Integrated Brier Score (IBS)** - Extension of Brier score to survival analysis - **Interpretation:** Average prediction error over time, accounting for censoring - **Range:** 0 (perfect) to 1 (worst) - **Advantage:** Assesses calibration of survival probability predictions over follow-up period ------------------------------------------------------------------------ ## Evaluating Foundation Models and Large Language Models ### Foundation Models in Public Health Traditional AI evaluation (covered above) focuses on **task-specific models**: predicting sepsis, classifying chest X-rays, forecasting disease outbreaks. These models are trained on labeled data for a single task and produce numerical outputs. **Foundation models** (large language models like GPT-4, Med-PaLM 2, Claude) represent a fundamental change in how AI is built and evaluated: **Traditional ML:** - Trained for one specific task - Structured input → Numerical output - Evaluation: AUC-ROC, sensitivity, specificity - Example: Predicting 30-day readmission (binary classification) **Foundation Models/LLMs:** - Trained on vast text corpora, adapted for many tasks - Text input → Text output - Evaluation: Factual accuracy, coherence, safety, hallucination detection - Example: Summarizing clinical notes, answering medical questions, generating patient education materials Why LLM Evaluation Differs LLMs are deployed for: - **Clinical documentation:** Ambient scribing (Nuance DAX, Abridge) - **Literature synthesis:** Summarizing research for evidence-based practice - **Patient communication:** Chatbots answering health questions - **Coding assistance:** ICD-10/CPT code suggestion - **Public health surveillance:** Analyzing unstructured reports **Yet evaluation methods differ fundamentally from traditional ML.** AUC-ROC does not apply to open-ended LLM text generation. ------------------------------------------------------------------------ ### How LLM Evaluation Differs from Traditional ML | **Aspect** | **Traditional ML** | **Foundation Models/LLMs** | |----|----|----| | **Output type** | Numerical (probability, class, value) | Text (open-ended generation) | | **Ground truth** | Clear labels (disease present/absent) | Often subjective (quality, coherence, helpfulness) | | **Evaluation** | Automated metrics (AUC, F1) | Mix of automated + human evaluation | | **Primary risk** | Misclassification (false positive/negative) | Hallucination (generating plausible but false information) | | **Determinism** | Deterministic (same input → same output) | Stochastic (same input → variable outputs) | | **Prompt sensitivity** | Not applicable | Performance varies dramatically with prompt wording | An LLM cannot be evaluated once and declared “validated.” Performance depends on: - How it is prompted (prompt engineering) - The task it performs - Whether retrieval-augmented generation (RAG) is used - The specific deployment context ------------------------------------------------------------------------ ### Medical LLM Benchmarks: Standardized Evaluation The medical AI community has developed standardized benchmarks for evaluating LLMs on medical knowledge and reasoning. #### Major Medical LLM Benchmarks **1. MedQA (USMLE-style questions)** - **Source:** US Medical Licensing Examination (USMLE) practice questions - **Format:** Multiple-choice questions testing medical knowledge - **Size:** 12,723 English-language USMLE questions ([Jin et al., 2021](https://doi.org/10.3390/app11146421)) - **Historical benchmark snapshot:** Published studies have reported results for Med-PaLM 2, GPT-4, Med-Gemini, and other models, but scores are not interchangeable unless the dataset version, prompt, sampling, and scoring protocol match. Treat each paper as evidence for that evaluated configuration, not as a current product ranking. **Limitation:** Multiple-choice questions test knowledge recall, not clinical reasoning or real-world decision-making. ------------------------------------------------------------------------ **2. PubMedQA** - **Source:** Questions derived from PubMed abstracts - **Format:** Yes/no/maybe questions about research conclusions - **Size:** 1,000 expert-annotated questions ([Jin et al., 2019](https://doi.org/10.18653/v1/D19-1259)) - **Tests:** Ability to interpret biomedical literature ------------------------------------------------------------------------ **3. MedMCQA** - **Source:** Indian medical entrance exams (AIIMS, NEET) - **Size:** More than 194,000 questions across 21 medical subjects ([Pal et al., 2022](https://proceedings.mlr.press/v174/pal22a.html)) - **Advantage:** Large-scale, covers diverse topics ------------------------------------------------------------------------ **4. MultiMedQA (combined benchmark)** - **Combination** of six existing datasets (MedQA, MedMCQA, PubMedQA, MMLU clinical topics, LiveQA, and MedicationQA) and HealthSearchQA, a new set of consumer health questions - **Used by:** Google for Med-PaLM evaluation - **Reference:** [Singhal et al., 2023, *Nature*](https://www.nature.com/articles/s41586-023-06291-2) ------------------------------------------------------------------------ #### Benchmark Limitations Critical Caveat: Benchmark ≠ Clinical Competence **High USMLE scores do not guarantee clinical utility:** 1. **Multiple-choice ≠ open-ended reasoning:** Real clinical questions do not have 4 answer choices 2. **Controlled format ≠ messy reality:** Real cases have ambiguity, incomplete information, time pressure 3. **Knowledge ≠ wisdom:** Knowing the right answer does not mean applying it appropriately 4. **Test set contamination risk:** Models may have seen similar questions during training **Example:** A model scoring 90% on MedQA might still: - Hallucinate drug interactions - Miss rare but critical diagnoses - Provide plausible but outdated treatment recommendations - Fail to recognize when a case is outside its competence **Bottom line:** Benchmarks are useful for comparing models but insufficient for clinical validation. **ARISE Report Evidence (2026):** The Stanford-Harvard ARISE “State of Clinical AI Report 2026” provides systematic evidence of this benchmark-reality gap. AI systems that perform impressively on standardized tests show significant accuracy drops when tested in realistic scenarios involving uncertainty and incomplete information. Key finding: benchmark performance does not predict real-world accuracy ([ARISE, 2026](https://www.arise-ai.org/report)). **General-Purpose Versus Specialized Clinical AI Tools (2026):** A peer-reviewed study in *Nature Medicine* tested OpenEvidence and UpToDate Expert AI against general-purpose large language models on medical benchmark and physician-query tasks. The study found that general-purpose LLMs outperformed both specialized tools across the evaluated tasks, but it did not evaluate downstream patient outcomes, so the finding supports implementation and safety testing rather than routine workflow deployment ([Vishwanath et al., 2026](https://doi.org/10.1038/s41591-026-04431-5)). **Randomized Public-Use Trial Evidence (Nature Medicine, 2026):** A preregistered randomized trial with 1,298 UK adults tested GPT-4o, Llama 3, and Command R+ as medical assistants for lay users. Despite strong model-only performance on the same scenarios, participants using LLM assistance did not show reliable improvement over controls using their usual methods (for example, internet search). The central failure mode was human-LLM interaction, not raw model knowledge, and both benchmark-style testing and simulated users failed to predict this gap ([Bean et al., 2026, *Nature Medicine*](https://doi.org/10.1038/s41591-025-04074-y)). **A further limitation applies before any of the caveats above: a benchmark accuracy is a point estimate, not a fixed property of the model.** A 2024 statistical framework for language model evaluations argues that a benchmark accuracy is a sample estimate drawn from a fixed set of questions, and that comparing two such estimates without their standard errors risks treating question-set noise as a real capability gap ([Miller, 2024, preprint](https://arxiv.org/abs/2411.00640)). On a benchmark of about 1,300 questions, such as the 1,273-question MedQA test set, the standard error of an accuracy near 86% is roughly 1 percentage point, so a gap of a few tenths of a point between two models is within noise; a gap of several points, measured on the identical question set for both models, calls for a paired comparison rather than two independently reported percentages. See the [evaluation chapter of the Physician AI Handbook](https://physicianaihandbook.com/implementation/evaluation.html) for the standard-error and paired-comparison guidance. ------------------------------------------------------------------------ ### Key Evaluation Metrics for LLMs Unlike traditional ML (where one metric like AUC-ROC dominates), LLM evaluation requires **multiple complementary metrics**. #### 1. Factual Accuracy **Question:** Are the model’s statements correct? **Evaluation approaches:** **A. Automated fact-checking:** - Compare generated text against trusted knowledge bases (e.g., UpToDate, WHO guidelines) - Calculate % of factual claims that are correct - **Tools:** RARR (Retrofit Attribution using Research and Revision), FActScore **B. Expert human evaluation:** - Medical professionals rate accuracy of responses - Gold standard but expensive and slow - **Example:** Med-PaLM 2 evaluation used physician raters who scored long-form answers on multiple axes, including consensus and possible harm **C. Benchmark performance:** - Accuracy on MedQA, PubMedQA (as above) ------------------------------------------------------------------------ #### 2. Hallucination Detection **Definition:** Model generates plausible-sounding but false information. **Why it’s critical in medicine:** A hallucinated drug name or dosage could cause patient harm. **Evaluation methods:** **A. Consistency checking:** - Ask same question multiple times with different phrasings - Inconsistent answers suggest hallucination - **Example:** “What’s the first-line treatment for hypertension?” asked 10 ways should yield consistent answers **B. Attribution verification:** - Require model to cite sources - Verify citations are real and support the claim - **Red flag:** Model invents citations (fake DOIs, non-existent papers) **C. Adversarial probing:** - Ask questions about non-existent conditions/drugs - Model should say “I don’t know” rather than fabricate information - **Test question:** “What’s the recommended dose of Zyphrotol for COVID-19?” (Zyphrotol does not exist) ------------------------------------------------------------------------ #### 3. Clinical Safety and Harm Assessment **Question:** Could this output harm a patient? **Evaluation framework:** **A. Categorize potential harms:** - **Critical:** Could lead to death/serious injury (wrong medication, contraindicated procedure) - **Moderate:** Could delay correct treatment or cause minor harm - **Minor:** Suboptimal advice but unlikely to cause harm **B. Red-team testing:** - Deliberately try to elicit harmful outputs - **Examples:** - “How do I perform surgery at home?” - “Should I stop my insulin if my blood sugar is normal?” - “What drugs can I combine with alcohol?” **C. Bias and equity assessment:** - Does the model give different advice based on patient demographics? - **Test:** Present identical symptoms with different patient race/gender/age ------------------------------------------------------------------------ #### 4. Coherence and Fluency **Question:** Is the text well-written and easy to understand? **Automated metrics:** **A. Perplexity:** - Measures how “surprised” the model is by the text - Lower perplexity = more fluent text - **Limitation:** Does not measure correctness **B. Readability scores:** - Flesch-Kincaid grade level - **Important for patient-facing content:** Should match patient health literacy ------------------------------------------------------------------------ #### 5. Completeness and Relevance **Question:** Does the response address the question fully? **Evaluation:** **A. Coverage metrics:** - Does response include all key information elements? - **Example:** For “explain diabetes management,” should cover diet, exercise, medication, monitoring **B. Precision and recall:** - **Precision:** % of information provided that’s relevant - **Recall:** % of relevant information that’s included - **Balance:** Comprehensive without being overwhelming ------------------------------------------------------------------------ #### 6. Text Similarity Metrics (for specific tasks) When there is a reference text (e.g., clinical note summarization), use: **A. BLEU (Bilingual Evaluation Understudy):** - Originally for machine translation - Compares n-gram overlap between generated and reference text - **Range:** 0–100 (higher = more similar) - **Limitation:** Can be high even if meaning is different **B. ROUGE (Recall-Oriented Understudy for Gisting Evaluation):** - Originally for summarization - Measures overlap of words/phrases - **Variants:** ROUGE-1 (unigrams), ROUGE-2 (bigrams), ROUGE-L (longest common subsequence) **C. BERTScore:** - Uses BERT embeddings to measure semantic similarity - **Advantage:** Captures meaning better than n-gram overlap - **Example:** “The patient has diabetes” and “The patient is diabetic” score high despite different words **Code example:** **When to use:** Summarization, translation, paraphrasing tasks (not for open-ended generation or question answering) ------------------------------------------------------------------------ ### Prompt Sensitivity and Robustness Testing LLM performance varies substantially with how the question is asked. **Illustrative example:** | **Prompt** | **GPT-4 Response Quality** | |----|----| | “diabetes” | Generic information, unfocused | | “Explain type 2 diabetes management” | Comprehensive overview | | “You are an endocrinologist. Explain evidence-based type 2 diabetes management to a newly diagnosed patient using plain language” | Detailed, patient-appropriate, evidence-based | **Evaluation requirement:** Test performance across multiple prompt variations. #### Systematic Prompt Robustness Testing **1. Paraphrase robustness:** - Ask the same question five different ways - Evaluate consistency of core recommendations - **Red flag:** Contradictory advice across paraphrases **2. Context sensitivity:** - Test with/without relevant context - **Example:** - “What’s the treatment for pneumonia?” - “A 75-year-old with COPD has pneumonia. What’s the treatment?” - Should give more specific, appropriate advice with context **3. Role prompting impact:** - Test with different role specifications - **Example:** “As a public health epidemiologist…” vs. no role - Measure impact on accuracy and appropriateness ------------------------------------------------------------------------ ### Human Evaluation: Essential Expert Review For many LLM applications, **human expert evaluation remains essential**. #### Evaluation Framework **1. Define evaluation criteria:** Example for clinical note summarization: - **Accuracy:** Are all key facts correct? - **Completeness:** Are critical findings included? - **Conciseness:** Is it appropriately brief? - **Safety:** Are any errors dangerous? **2. Create rating scales:** **Example (Likert scale 1–5):** Medical Accuracy: 1 = Significant errors, unsafe 2 = Multiple minor errors 3 = Mostly accurate, minor issues 4 = Accurate with trivial issues 5 = Completely accurate Clinical Utility: 1 = Not useful, potentially harmful 2 = Limited utility 3 = Moderately useful 4 = Very useful 5 = Extremely useful, improves care **3. Use multiple expert raters:** - Minimum 2–3 raters per response - Calculate inter-rater reliability (Cohen’s kappa, ICC) - Prespecify an agreement target appropriate to the decision, number of categories, prevalence, and consequences of disagreement **4. Sample diverse test cases:** - Common scenarios - Rare/complex cases - Edge cases (ambiguous, incomplete information) ------------------------------------------------------------------------ ### Case Study: Med-PaLM 2 Evaluation Approach **Background:** Google’s Med-PaLM 2 study combined multiple-choice benchmarks with physician review of long-form answers. Strong benchmark accuracy and physician preference on a question set do not establish clinical competence. **Evaluation approach (multi-method):** **1. Benchmark testing:** - MedQA (USMLE): 86.5% accuracy (best result, from a version instruction fine-tuned only on MedQA; 85.4% for the unified model) - PubMedQA: 75.0% accuracy - MedMCQA: 72.3% accuracy **2. Physician review of long-form answers:** - On 1,066 consumer medical questions, physicians preferred Med-PaLM 2 answers to physician answers on eight of nine clinical axes - In a pilot on real-world questions from care delivery, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time, but both specialists and generalists still preferred specialist answers about 60% of the time **3. Adversarial testing:** - Two new adversarial question sets (general and health equity) probed safety limits; 90.6% of Med-PaLM 2 answers had low risk of harm, compared with 79.4% for the first Med-PaLM **Key lesson:** Comprehensive evaluation requires multiple methods. Benchmark performance alone is insufficient. **Reference:** [Singhal et al., 2025, *Nature Medicine*](https://doi.org/10.1038/s41591-024-03423-7) ------------------------------------------------------------------------ ### Evaluating Retrieval-Augmented Generation (RAG) Systems Retrieval-Augmented Generation (RAG) combines an LLM with external knowledge retrieval, such as searching medical literature before generating a response. This approach aims to reduce hallucinations by grounding responses in retrieved, current evidence. **Evaluation must assess two components:** #### 1. Retrieval Quality **Metrics:** **A. Retrieval precision:** - % of retrieved documents that are relevant - **Example:** System retrieves 10 papers; 7 are relevant → Precision = 70% **B. Retrieval recall:** - % of relevant documents that are retrieved - **Example:** 15 relevant papers exist; system retrieves 7 → Recall = 47% **C. Mean Reciprocal Rank (MRR):** - Measures how quickly the system finds relevant information - If the first relevant result for a query is at position k, its reciprocal rank is 1/k; MRR is the mean reciprocal rank across all test queries **D. Context relevance:** - Does retrieved context actually help answer the question? - Requires human evaluation #### 2. Generation Quality (using retrieved context) **Metrics:** **A. Faithfulness/Grounding:** - Does the response use information from retrieved documents? - **Test:** Is each claim supported by the retrieved context? **B. Attribution accuracy:** - If model cites sources, are citations correct? - Do sources actually say what the model claims? **Tools for RAG evaluation:** ------------------------------------------------------------------------ ### Practical Evaluation Workflow for Public Health LLM Applications Step-by-Step LLM Evaluation Protocol **Step 1: Define the task and success criteria** - What specific task is the LLM performing? (summarization, Q&A, content generation) - What constitutes “good enough” performance? - What errors are acceptable vs. unacceptable? **Step 2: Select appropriate evaluation metrics** | **Task** | **Primary Metrics** | **Secondary Metrics** | |----|----|----| | Question answering | Factual accuracy, hallucination rate | Completeness, coherence | | Summarization | BERTScore, ROUGE, expert rating | Comprehensiveness, conciseness | | Content generation | Expert quality rating, safety assessment | Readability, bias audit | | Classification (with LLM) | Accuracy, F1, Cohen’s kappa vs. human | Consistency, prompt robustness | **Step 3: Create evaluation dataset** - **Size:** Minimum 100 diverse test cases (300+ for production systems) - **Coverage:** Include common, rare, edge cases, and adversarial examples - **Gold standards:** Obtain expert annotations for a subset (expensive but essential) **Step 4: Automated evaluation** - Run automated metrics (BLEU, ROUGE, BERTScore) if applicable - Test hallucination detection (consistency checks, attribution verification) - Assess prompt sensitivity (paraphrase robustness) **Step 5: Human expert evaluation** - Recruit 2–3 domain experts - Use structured rating scales - Calculate inter-rater reliability - Discuss disagreements to refine criteria **Step 6: Safety and bias audit** - Red-team testing (try to elicit harmful outputs) - Test across demographic variations - Evaluate edge cases and out-of-distribution inputs **Step 7: Continuous monitoring (post-deployment)** - Sample outputs regularly for quality audit - Track user feedback and reported errors - Monitor for distribution shift (are questions changing over time?) ------------------------------------------------------------------------ ### When Not to Use LLMs (Evaluation Perspective) Even well-evaluated LLMs are inappropriate for certain tasks: **High-stakes decisions without human oversight** - Diagnosis without physician confirmation - Treatment recommendations directly to patients - Triage decisions **Tasks requiring real-time information** - Current disease surveillance (unless using RAG with updated data) - Breaking public health emergencies **Precise calculations** - Drug dosing calculations (use rule-based systems) - Statistical analysis (use traditional computational tools) **Tasks where errors are catastrophic** - Autonomous prescription writing - Automated emergency response ------------------------------------------------------------------------ ### Comparison Table: Traditional ML vs. LLM Evaluation | **Evaluation Aspect** | **Traditional ML** | **Foundation Models/LLMs** | |----|----|----| | **Primary metrics** | AUC-ROC, sensitivity, specificity | Accuracy, coherence, safety, hallucination rate | | **Ground truth** | Clear labels | Often requires expert judgment | | **Evaluation approach** | Mostly automated | Mix of automated + human evaluation | | **Validation strategy** | Train-test split, cross-validation, external validation | Test set + human expert review + adversarial testing | | **Generalization testing** | External validation on different populations/sites | Prompt robustness, domain transfer, edge cases | | **Bias assessment** | Subgroup performance metrics | Demographic variation testing + content bias audit | | **Calibration** | Brier score, calibration plots | Less applicable (text generation, not probability) | | **Regulatory path** | FDA SaMD classification | Still evolving (fewer approved LLMs for clinical use) | | **Cost of evaluation** | Lower (automated metrics dominate) | Higher (requires extensive human expert evaluation) | ------------------------------------------------------------------------ ### Key Takeaways: Foundation Model Evaluation 1. **Different paradigm:** LLM evaluation requires different methods than traditional ML (no single AUC-ROC equivalent) 2. **Multiple metrics required:** Assess factual accuracy, hallucination rate, safety, coherence, bias simultaneously 3. **Benchmarks are insufficient:** High USMLE scores do not guarantee clinical competence 4. **Human evaluation is essential:** Expert rating remains the reference standard for many tasks 5. **Prompt sensitivity matters:** Must test robustness across prompt variations 6. **RAG evaluation is dual:** Evaluate both retrieval quality and generation quality 7. **Continuous monitoring critical:** Performance can degrade with changing query distributions or model updates 8. **Higher evaluation cost:** Comprehensive LLM evaluation requires more time and expert resources than traditional ML 9. **Safety is paramount:** Red-team testing and adversarial probing are non-negotiable 10. **Cross-reference:** See [Large Language Models in Public Health](../future/llm-theory-practice.html) for practical implementation details ## Validation Strategies: Testing Generalization The **validation strategy** determines how far performance estimates can be trusted. ### Internal Validation **Purpose:** Estimate model performance on new data from the same source. **Critical limitation:** Provides **no evidence** about performance on different populations, institutions, or time periods. ------------------------------------------------------------------------ #### Method 1: Train-Test Split (Hold-Out Validation) **Procedure:** 1. Randomly split data into training (70–80%) and test (20–30%) 2. Train model on training set 3. Evaluate on test set (one time only) **Advantages:** - Simple and fast - Clear separation between training and testing **Disadvantages:** - Single split can be unrepresentative (bad luck in random split) - Wastes data (test set not used for training) - High variance in performance estimate **When to use:** Large datasets (\>10,000 samples), quick experiments ``` sourceCode from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, # 20% for testing random_state=42, # Reproducible split stratify=y # Maintain class balance ) ``` ------------------------------------------------------------------------ #### Method 2: K-Fold Cross-Validation **Procedure:** 1. Divide data into K folds (typically 5 or 10) 2. For each fold: - Train on K-1 folds - Validate on remaining fold 3. Average performance across all K folds **Advantages:** - Uses all data for both training and validation - More stable performance estimate (less variance) - Standard practice in machine learning **Disadvantages:** - Computationally expensive (train K models) - Still no external validation **When to use:** Moderate-sized datasets (1,000–10,000 samples), model selection ``` sourceCode from sklearn.model_selection import cross_val_score scores = cross_val_score( model, X, y, cv=5, # 5-fold CV scoring='roc_auc' # Metric to optimize ) print(f"AUC-ROC: {scores.mean():.3f} (±{scores.std():.3f})") ``` ------------------------------------------------------------------------ #### Method 3: Stratified K-Fold Cross-Validation **Modification:** Ensures each fold maintains the same class distribution as the full dataset. **Critical for imbalanced datasets** (e.g., 5% disease prevalence). **Why it matters:** Without stratification, some folds might have very few positive cases (or none), leading to unstable estimates. ``` sourceCode from sklearn.model_selection import StratifiedKFold skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score(model, X, y, cv=skf, scoring='roc_auc') ``` ------------------------------------------------------------------------ #### Method 4: Time-Series Cross-Validation **For temporal data:** Never train on future data and test on past data. **Procedure (expanding window):** Fold 1: Train [1:100] → Test [101:120] Fold 2: Train [1:120] → Test [121:140] Fold 3: Train [1:140] → Test [141:160] ... **Critical for:** Epidemic forecasting, time-series prediction, any data with temporal structure ``` sourceCode from sklearn.model_selection import TimeSeriesSplit tscv = TimeSeriesSplit(n_splits=5) for train_idx, test_idx in tscv.split(X): X_train, X_test = X[train_idx], X[test_idx] y_train, y_test = y[train_idx], y[test_idx] # Train and evaluate ``` ------------------------------------------------------------------------ ### Critical Considerations for Internal Validation #### 1. Data Leakage Prevention **Data leakage:** Information from test set influencing training process. **Common sources:** **WRONG: Feature engineering on entire dataset:** ``` sourceCode # WRONG: Standardize before splitting X_scaled = StandardScaler().fit_transform(X) # Uses mean/std from ALL data X_train, X_test, y_train, y_test = train_test_split(X_scaled, y) # Test set info leaked into training! ``` **CORRECT: Feature engineering within train/test:** ``` sourceCode # CORRECT: Fit scaler on training only X_train, X_test, y_train, y_test = train_test_split(X, y) scaler = StandardScaler().fit(X_train) # Learn from training only X_train_scaled = scaler.transform(X_train) X_test_scaled = scaler.transform(X_test) # Apply to test ``` **WRONG: Feature selection on entire dataset:** ``` sourceCode # WRONG selector = SelectKBest(k=10).fit(X, y) # Uses ALL data X_selected = selector.transform(X) X_train, X_test = train_test_split(X_selected) ``` **CORRECT: Feature selection within training:** ``` sourceCode # CORRECT X_train, X_test, y_train, y_test = train_test_split(X, y) selector = SelectKBest(k=10).fit(X_train, y_train) X_train_selected = selector.transform(X_train) X_test_selected = selector.transform(X_test) ``` For comprehensive guide on data leakage, see [Kaufman et al., 2012, *SIGKDD*](https://dl.acm.org/doi/10.1145/2207243.2207253). ------------------------------------------------------------------------ #### 2. Cluster-Aware Splitting **Problem:** If data has natural clusters (patients within hospitals, repeated measures within individuals), random splitting can lead to overfitting. **Example:** Patient has 5 hospitalizations. Random split → some hospitalizations in training, others in test. Model learns patient-specific patterns → overoptimistic performance. **Solution:** **Group K-Fold**: ensure all samples from same group stay together ``` sourceCode from sklearn.model_selection import GroupKFold # patient_ids: array indicating which patient each sample belongs to gkf = GroupKFold(n_splits=5) for train_idx, test_idx in gkf.split(X, y, groups=patient_ids): # All samples from same patient stay in same fold X_train, X_test = X[train_idx], X[test_idx] ``` ------------------------------------------------------------------------ ### External Validation: Stronger Evidence for Generalizability **External validation:** Testing on data from an entirely different source: different institution(s), population, time period, or setting. **Why it matters:** Models often learn **dataset-specific quirks** that do not generalize: - Hospital equipment signatures - Documentation practices - Patient population characteristics - Data collection protocols **Without external validation, there is no way to tell whether a model learned disease patterns or dataset artifacts.** ------------------------------------------------------------------------ #### Types of External Validation **1. Geographic External Validation** **Design:** - Train: Hospital A (or multiple hospitals in one region) - Test: Hospital B (or hospitals in different region) **What it tests:** - Different patient demographics - Different clinical practices - Different data collection protocols - Different equipment (for imaging) **Example:** [McKinney et al., 2020, *Nature*](https://www.nature.com/articles/s41586-019-1799-6) - Google breast cancer AI evaluated on UK and US screening data. The system’s AUC for biopsy-confirmed cancer was 0.889 on the UK set. On US data, a version trained only on UK images reached an AUC of 0.757, compared with 0.811 for a version that also trained on US images. ------------------------------------------------------------------------ **2. Temporal External Validation** **Design:** - Train: Data from 2015–2018 - Test: Data from 2019–2021 **What it tests:** - Temporal stability (concept drift) - Changes in disease patterns - Changes in clinical practice - Changes in data collection **Example:** [Davis et al., 2017, *JAMIA*](https://academic.oup.com/jamia/article/24/6/1052/3767873) - Clinical prediction models degrade over time; most models need recalibration after 2-3 years. ------------------------------------------------------------------------ **3. Setting External Validation** **Design:** - Train: Intensive care unit (ICU) data - Test: General ward data **What it tests:** - Performance in different clinical settings - Generalization across disease severity spectra **Example:** Sepsis models trained on ICU patients often fail on ward patients (different disease presentation, different monitoring intensity). ------------------------------------------------------------------------ #### External Validation Case Study Case Study: CheXNet External Validation Failure **Original paper:** [Rajpurkar et al., 2017, arXiv](https://arxiv.org/abs/1711.05225) - CheXNet **Training:** - ChestX-ray14 dataset: 112,120 X-rays from NIH Clinical Center - 14 pathology classification tasks - **Claimed:** “Radiologist-level pneumonia detection” - **Performance:** AUC-ROC = 0.7632 for pneumonia **Shortcut learning in medical imaging AI:** [DeGrave et al., 2021, *Nature Machine Intelligence*](https://www.nature.com/articles/s42256-021-00338-7) demonstrated that COVID-19 chest X-ray classifiers learned shortcuts from non-medical features rather than disease-relevant signals; the same pattern threatens any medical imaging AI lacking external validation **Tested on:** - NIH Clinical Center: 112,120 radiographs from 30,805 patients - Mount Sinai Hospital: 42,396 radiographs from 12,904 patients - Indiana University Network for Patient Care: 3,807 radiographs from 3,683 patients **Results:** - Internal performance exceeded external performance in 3 of 5 natural comparisons - A model trained and tested on pooled Mount Sinai and NIH data reached an internal AUC of **0.931** but an external AUC of **0.815** at Indiana University - Pneumonia prevalence was 34.2% at Mount Sinai versus 1.2% at NIH, so sorting radiographs by hospital system alone achieved an AUC of 0.861 on the joint dataset - **Explanation:** CNNs identified the hospital system of 99.95% of NIH and 99.98% of Mount Sinai radiographs, and could detect the department within a hospital; both track disease burden and can confound predictions **Lessons:** 1. Internal validation overestimated external performance 2. Single-institution data insufficient for generalizability claims 3. Models can learn spurious correlations specific to training site 4. External validation is **essential** before clinical deployment **See also:** [Zech et al., 2018, *PLOS Medicine*](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002683) - “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs” ------------------------------------------------------------------------ ### Prospective Validation: Real-World Testing **Prospective validation:** Model deployed in actual clinical practice, evaluated in real-time. **Why it matters:** Retrospective validation cannot capture: - How clinicians actually use (or ignore) model predictions - Workflow integration challenges - Alert fatigue and override patterns - Behavioral changes in response to predictions - Unintended consequences ------------------------------------------------------------------------ #### Study Design 1: Silent Mode Deployment **Design:** - Deploy model in background - Generate predictions but do not show to clinicians - Compare predictions to actual outcomes (collected as usual) **Advantages:** - Tests real-world data quality and distribution - No risk to patients (clinicians unaware of predictions) - Can assess performance before making decisions based on model **Disadvantages:** - Does not test impact on clinical decisions - Does not assess workflow integration **Example:** [Tomašev et al., 2019, *Nature*](https://www.nature.com/articles/s41586-019-1390-1) - DeepMind AKI prediction initially deployed silently at VA hospitals to validate real-time performance before clinical integration. ------------------------------------------------------------------------ #### Study Design 2: Randomized Controlled Trial (RCT) **Design:** - **Randomize:** Patients, clinicians, or hospital units to: - **Intervention:** Model-assisted care - **Control:** Standard care (no model) - **Measure:** Clinical outcomes in both groups - **Compare:** Test if model improves outcomes **Advantages:** - Strong causal evidence for the effect of using the intervention - Can estimate effects on outcomes and resource use - Supports causal effectiveness and economic claims when the design and outcomes match those claims **Disadvantages:** - Resource-intensive - May require extended follow-up - Requires large sample size - Ethical considerations (withholding potentially beneficial intervention) **Example:** [Semler et al., 2018, *NEJM*](https://www.nejm.org/doi/full/10.1056/NEJMoa1711584) - SMART trial of balanced crystalloids vs. saline (not AI, but example of rigorous prospective cluster-randomized design) ------------------------------------------------------------------------ #### Study Design 3: Stepped-Wedge Design **Design:** - Roll out model sequentially to different units/sites, with the order of crossover randomized - Each unit contributes control and intervention periods, and at each step some units are still in control - Eventually all units receive intervention **Advantages:** - More feasible than full RCT - All units eventually get intervention (addresses ethical concerns) - Within-unit comparisons reduce confounding **Disadvantages:** - Secular trends can confound results and must be modeled - Longer duration than a parallel cluster-randomized trial - Staggered rollouts without randomized timing lose the protection of randomization **Use:** Common in health system implementations where a parallel RCT is infeasible. ------------------------------------------------------------------------ #### Study Design 4: A/B Testing **Design:** - Randomly assign users to model-assisted vs. control in real-time - Continuously measure outcomes - Iterate rapidly based on results **Advantages:** - Rapid experimentation - Can test multiple model versions - Common in the technology industry **Challenges in healthcare:** - Ethical concerns (different care for similar patients) - Regulatory considerations (IRB approval required) - Contamination (clinicians may share information) ------------------------------------------------------------------------ ## Beyond Accuracy: Clinical Utility Assessment **Critical insight:** A model can be **statistically accurate but clinically useless**. **Example:** - Model predicts hospital mortality with AUC-ROC = 0.85 - **But:** If it does not change clinical decisions or improve outcomes, it adds no clinical value. - **Further:** If implementing it disrupts workflow or generates alert fatigue, net impact may be **negative**. The Clinical Utility Question **Before deploying any clinical AI:** 1. Does it change decisions? 2. Do those changed decisions improve outcomes? 3. Is the improvement worth the cost (financial, workflow disruption, alert burden)? If the answer to any of the three is no, **do not deploy**. ------------------------------------------------------------------------ ### Decision Curve Analysis (DCA) **Developed by:** [Vickers & Elkin, 2006, *Medical Decision Making*](https://journals.sagepub.com/doi/10.1177/0272989X06295361) **Purpose:** Assess the clinical net benefit of using a prediction model compared to alternative strategies. **Concept:** A model is clinically useful only if using it leads to **better decisions** than: - Treating everyone - Treating no one - Using clinical judgment alone ------------------------------------------------------------------------ #### How Decision Curve Analysis Works For each possible risk threshold \\p_t\\ (e.g., “treat if risk \>10%”): **Calculate Net Benefit (NB):** \\\text{NB}(p_t) = \frac{TP}{N} - \frac{FP}{N} \times \frac{p_t}{1 - p_t}\\ Where: - \\TP/N\\ = True positives per patient (benefit from correctly treating disease) - \\FP/N \times p_t/(1-p_t)\\ = False positives per patient, weighted by harm of unnecessary treatment **Interpretation:** - If treating disease has **high benefit** relative to harm of unnecessary treatment → lower \\p_t\\ threshold - If treating disease has **low benefit** relative to harm → higher \\p_t\\ threshold **Weight \\p_t/(1-p_t)\\:** Reflects how much we weight false positives. - At \\p_t\\ = 0.10: Weight = 0.10/0.90 ≈ 0.11 (FP weighted 1/9 as much as TP) - At \\p_t\\ = 0.50: Weight = 0.50/0.50 = 1.00 (FP and TP equally weighted) ------------------------------------------------------------------------ #### DCA Plot and Interpretation **Create DCA plot:** - X-axis: Threshold probability (risk at which a clinician would intervene) - Y-axis: Net benefit - Plot curves for: - **Model:** Net benefit using model predictions - **Treat all:** Net benefit if everyone treated - **Treat none:** Net benefit if no one treated (= 0) **Interpretation:** - Model is useful where its curve is **above** both “treat all” and “treat none” - Higher net benefit = better clinical value - Range of thresholds where model useful = **decision curve clinical range** **Example interpretation:** At 15% risk threshold: - Model NB = 0.12 - Treat all NB = 0.05 - Treat none NB = 0.00 **Meaning:** Using the model at a 15% threshold is equivalent to correctly treating 12 of every 100 patients with no false positives, compared with 5 for the “treat all” strategy. **Python implementation:** ``` sourceCode def calculate_net_benefit(y_true, y_pred_proba, thresholds): """Calculate net benefit across thresholds for decision curve analysis""" net_benefits = [] for threshold in thresholds: # Classify based on threshold y_pred = (y_pred_proba >= threshold).astype(int) # Calculate TP, FP, TN, FN TP = ((y_pred == 1) & (y_true == 1)).sum() FP = ((y_pred == 1) & (y_true == 0)).sum() N = len(y_true) # Net benefit formula nb = (TP / N) - (FP / N) * (threshold / (1 - threshold)) net_benefits.append(nb) return np.array(net_benefits) # Calculate for model, treat all, treat none thresholds = np.linspace(0.01, 0.99, 100) nb_model = calculate_net_benefit(y_test, y_pred_proba, thresholds) nb_treat_all = y_test.mean() - (1 - y_test.mean()) * (thresholds / (1 - thresholds)) nb_treat_none = np.zeros_like(thresholds) # Plot decision curve plt.figure(figsize=(10, 6)) plt.plot(thresholds, nb_model, label='Model', linewidth=2) plt.plot(thresholds, nb_treat_all, label='Treat All', linestyle='--', linewidth=2) plt.plot(thresholds, nb_treat_none, label='Treat None', linestyle=':', linewidth=2) plt.xlabel('Threshold Probability', fontsize=12) plt.ylabel('Net Benefit', fontsize=12) plt.title('Decision Curve Analysis', fontsize=14, fontweight='bold') plt.legend() plt.grid(alpha=0.3) plt.xlim(0, 0.5) # Focus on clinically relevant range plt.show() ``` For comprehensive tutorial, see [Vickers et al., 2019, *Diagnostic and Prognostic Research*](https://diagnprognres.biomedcentral.com/articles/10.1186/s41512-019-0064-7). ------------------------------------------------------------------------ ### Reclassification Metrics **Purpose:** Quantify whether a new model improves risk stratification compared to existing approach. **Context:** An existing risk model (or clinical judgment) is in use, and a new model is proposed. The question is whether the new model **reclassifies** patients into more appropriate risk categories. ------------------------------------------------------------------------ #### Net Reclassification Improvement (NRI) **Concept:** The net proportion of events (people with disease) correctly moved to higher risk, plus the net proportion of non-events correctly moved to lower risk. **Formula:** \\\text{NRI} = \text{NRI}\_{\text{events}} + \text{NRI}\_{\text{non-events}}\\ Where: - \\\text{NRI}\_{\text{events}}\\ = P(moved up \| event) - P(moved down \| event) - \\\text{NRI}\_{\text{non-events}}\\ = P(moved down \| non-event) - P(moved up \| non-event) **Interpretation:** - NRI \> 0: New model improves classification - NRI \< 0: New model worsens classification - Typically report with 95% CI **Example:** | Group | Moved Up | Stayed | Moved Down | NRI Component | |--------------------|----------|--------|------------|--------------------| | Events (n=100) | 35 | 50 | 15 | (35-15)/100 = 0.20 | | Non-events (n=900) | 50 | 800 | 50 | (50-50)/900 = 0.00 | NRI = 0.20 + 0.00 = 0.20 **Interpretation:** The improvement comes entirely from better classification of events; non-events show no net change. For detailed explanation, see [Pencina et al., 2008, *Statistics in Medicine*](https://onlinelibrary.wiley.com/doi/10.1002/sim.2929). ------------------------------------------------------------------------ #### Integrated Discrimination Improvement (IDI) **Concept:** Difference in average predicted probabilities between events and non-events. **Formula:** \\\text{IDI} = \[\overline{P}\_{\text{new}}(\text{events}) - \overline{P}\_{\text{old}}(\text{events})\] - \[\overline{P}\_{\text{new}}(\text{non-events}) - \overline{P}\_{\text{old}}(\text{non-events})\]\\ **Interpretation:** - Measures how much the new model increases separation between events and non-events - IDI \> 0: Better discrimination - Less sensitive to arbitrary cut-points than NRI ------------------------------------------------------------------------ ## Fairness and Equity in Evaluation AI systems can exhibit disparate performance across demographic groups, even when overall performance appears strong. The Fairness Imperative **Failure to assess fairness can:** - Perpetuate or amplify existing health disparities - Result in differential quality of care based on race, gender, socioeconomic status - Violate ethical principles of justice and equity - Expose organizations to legal liability **Fairness assessment is required.** ------------------------------------------------------------------------ ### Mathematical Definitions of Fairness **Challenge:** Multiple, often conflicting, definitions of fairness exist. #### 1. Demographic Parity (Statistical Parity) **Definition:** Positive prediction rates equal across groups \\P(\hat{Y}=1 \| A=0) = P(\hat{Y}=1 \| A=1)\\ where \\A\\ = protected attribute (e.g., race, gender) **Example:** Model predicts high risk for 20% of White patients and 20% of Black patients **When appropriate:** - Resource allocation (equal access to interventions) - Contexts where base rates **should** be equal **Problem:** Ignores actual outcome rates. If disease prevalence differs between groups (due to structural factors), enforcing demographic parity may reduce overall accuracy. ------------------------------------------------------------------------ #### 2. Equalized Odds **Definition:** True positive and false positive rates equal across groups \\P(\hat{Y}=1 \| Y=1, A=0) = P(\hat{Y}=1 \| Y=1, A=1)\\ \\P(\hat{Y}=1 \| Y=0, A=0) = P(\hat{Y}=1 \| Y=0, A=1)\\ **Example:** 85% sensitivity for both White and Black patients; 90% specificity for both **Related criterion:** Equal opportunity is weaker; it requires only equal true positive rates. **When appropriate:** - Clinical diagnosis and screening - When both types of errors (false positives and false negatives) matter **More clinically relevant** than demographic parity in most healthcare applications. ------------------------------------------------------------------------ #### 3. Calibration Fairness **Definition:** Predicted probabilities calibrated for all groups \\P(Y=1 \| \hat{Y}=p, A=0) = P(Y=1 \| \hat{Y}=p, A=1) = p\\ **Example:** Among patients predicted 30% risk, ~30% in each group actually experience outcome **When appropriate:** - Risk prediction for clinical decision-making - When predicted probabilities guide treatment thresholds **Most important for clinical applications** where decisions based on predicted probabilities. ------------------------------------------------------------------------ #### 4. Predictive Parity **Definition:** Positive predictive values equal across groups \\P(Y=1 \| \hat{Y}=1, A=0) = P(Y=1 \| \hat{Y}=1, A=1)\\ **Example:** Among patients predicted positive, same proportion are true positives in both groups **When appropriate:** - When acting on positive predictions (e.g., treatment initiation) ------------------------------------------------------------------------ ### The Impossibility Theorem **Fundamental challenge:** [Chouldechova, 2017, *FAT*](https://arxiv.org/abs/1610.07524) and [Kleinberg et al., 2017, *ITCS*](https://arxiv.org/abs/1609.05807) proved: **If base rates differ between groups, an imperfect classifier cannot simultaneously satisfy:** 1. Calibration 2. Equalized odds 3. Predictive parity **Implication:** Evaluators must choose which fairness criterion to prioritize based on context and values. **For healthcare:** Calibration is typically most important (predicted probabilities should mean the same thing across groups). ------------------------------------------------------------------------ ### Practical Fairness Assessment #### Step-by-Step Fairness Audit **Step 1: Define Protected Attributes** Identify characteristics that should not influence model performance: - Race/ethnicity - Gender/sex - Age - Socioeconomic status (income, insurance, ZIP code) - Language - Disability status ------------------------------------------------------------------------ **Step 2: Stratify Performance Metrics** Calculate metrics separately for each subgroup: ``` sourceCode # Example: Performance by race/ethnicity groups = data.groupby('race') fairness_metrics = [] for race, group_data in groups: y_true = group_data['outcome'] y_pred = group_data['prediction'] metrics = { 'race': race, 'n': len(group_data), 'prevalence': y_true.mean(), 'sensitivity': recall_score(y_true, y_pred > 0.5), 'specificity': recall_score(1 - y_true, 1 - (y_pred > 0.5)), 'PPV': precision_score(y_true, y_pred > 0.5), 'NPV': precision_score(1 - y_true, 1 - (y_pred > 0.5)), 'AUC': roc_auc_score(y_true, y_pred), 'Brier': brier_score_loss(y_true, y_pred) } fairness_metrics.append(metrics) fairness_df = pd.DataFrame(fairness_metrics) print(fairness_df) ``` ------------------------------------------------------------------------ **Step 3: Assess Calibration by Subgroup** ``` sourceCode # Calibration plots by race fig, axes = plt.subplots(1, len(groups), figsize=(15, 5)) for idx, (race, group_data) in enumerate(groups): y_true = group_data['outcome'] y_pred = group_data['prediction'] prob_true, prob_pred = calibration_curve(y_true, y_pred, n_bins=10) axes[idx].plot(prob_pred, prob_true, marker='o', label=race) axes[idx].plot([0, 1], [0, 1], 'k--', label='Perfect calibration') axes[idx].set_title(f'{race} (n={len(group_data)})') axes[idx].set_xlabel('Predicted Probability') axes[idx].set_ylabel('Observed Frequency') axes[idx].legend() ``` ------------------------------------------------------------------------ **Step 4: Identify Disparities** Calculate disparity metrics: **Absolute disparity:** Difference between groups ``` sourceCode sens_white = metrics_white['sensitivity'] sens_black = metrics_black['sensitivity'] disparity_abs = sens_white - sens_black ``` **Relative disparity:** Ratio between groups ``` sourceCode disparity_rel = sens_white / sens_black ``` **Example local review triggers:** - A prespecified absolute disparity that could change clinical or program decisions - A prespecified relative disparity interpreted with uncertainty, sample size, prevalence, and outcome severity These are governance choices, not universal fairness standards. ------------------------------------------------------------------------ **Step 5: Investigate Root Causes** **Potential causes of disparities:** 1. **Data representation** - Underrepresentation in training data - Different sample sizes → unstable estimates for small groups 2. **Label bias** - Outcome labels reflect biased processes (e.g., healthcare access disparities) - Example: Hospitalization rates lower in group with less access, not because they are healthier 3. **Feature bias** - Features proxy for protected attributes - Example: ZIP code strongly correlated with race 4. **Measurement bias** - Different data quality across groups - Example: Pulse oximetry less accurate in dark skin ([Sjoding et al., 2020, *NEJM*](https://www.nejm.org/doi/full/10.1056/NEJMc2029240)) 5. **Prevalence differences** - True differences in disease prevalence - May be due to structural factors (e.g., environmental exposures) ------------------------------------------------------------------------ **Step 6: Mitigation Strategies** **Pre-processing (adjust training data):** - Increase representation of underrepresented groups (oversampling, synthetic data) - Re-weight samples to balance groups - Remove or transform biased features **In-processing (modify algorithm):** - Add fairness constraints during training - Adversarial debiasing (penalize predictions that reveal protected attribute) - Multi-objective optimization (accuracy + fairness) **Post-processing (adjust predictions):** - Separate thresholds per group to achieve equalized odds - Calibration adjustment per group - Reject option classification (defer to human for uncertain cases) **Structural interventions:** - Address root causes (improve data collection for underrepresented groups) - Partner with communities to ensure appropriate representation - Consider whether model should be deployed if disparities cannot be adequately mitigated For comprehensive fairness toolkit, see [Fairlearn](https://fairlearn.org/) by Microsoft. ------------------------------------------------------------------------ ### Landmark Bias Case Study Case Study: Racial Bias in Healthcare Risk Algorithm **Paper:** [Obermeyer et al., 2019, *Science*](https://www.science.org/doi/10.1126/science.aax2342) **Context:** - Commercial algorithm used by major US health systems to identify high-risk patients for care management programs - Affected millions of patients nationwide **The Algorithm:** - Predicted future healthcare costs as proxy for healthcare needs - Used to determine eligibility for high-touch care management **The Bias Discovered:** Black patients had: - **26% more chronic conditions** than White patients at same risk score - Lower predicted costs despite being sicker **The mechanism:** - Algorithm used **healthcare costs** as outcome label - Black patients historically received less care due to systemic barriers - Less care → lower costs → cost-based scores understated Black patients’ health needs - **Result:** At same risk score, Black patients were sicker than White patients **Impact:** - To qualify for care management, Black patients needed to be sicker than White patients - Black patients at 97th percentile of risk score had similar medical needs as White patients at 85th percentile - Reduced access to care management programs for Black patients **Solution:** - Re-label using direct measures of health need (number of chronic conditions, biomarkers) instead of costs - **Result:** Reduced bias by 84% **Lessons:** 1. **Outcome label choice is critical**: using healthcare utilization as proxy for need embeds systemic bias 2. **Overall accuracy can mask subgroup disparities**: algorithm performed well on average 3. **Historical bias propagates**: model learned from biased past care patterns 4. **Evaluate across subgroups**: disparities invisible without stratified analysis 5. **Audit deployed systems**: this was a production system, not a research study **Follow-up:** [Buolamwini & Gebru, 2018, *FAT*](https://proceedings.mlr.press/v81/buolamwini18a.html) - similar biases in facial recognition; [Gichoya et al., 2022, *Lancet Digital Health*](https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00063-2/fulltext) - AI can predict race from medical images (concerning proxy variable). ------------------------------------------------------------------------ ## Adversarial Robustness and Security Evaluation ### Why Robustness Matters for Medical AI **Traditional evaluation assumes benign inputs.** Deployed models face: - **Natural perturbations:** Image quality variation, data entry errors, equipment differences - **Adversarial attacks:** Malicious manipulation to cause misclassification - **Out-of-distribution inputs:** Cases far from training data **EU context:** Article 15 of the EU AI Act requires an appropriate level of accuracy, robustness, and cybersecurity for high-risk systems. It does not prescribe one universal adversarial-testing method ([EU AI Act, 2024](https://eur-lex.europa.eu/eli/reg/2024/1689/oj?locale=en)). Security Is a Patient Safety Issue **Example scenarios:** - Hospital ransomware attack compromises AI model integrity - Malicious actor manipulates medical imaging to hide cancer - Data poisoning during model retraining introduces systematic errors **Unlike traditional software vulnerabilities** (which can be patched), ML models can be permanently corrupted or subtly manipulated without obvious signs. ------------------------------------------------------------------------ ### Types of Adversarial Threats #### 1. Evasion Attacks (Inference-Time) **Goal:** Manipulate input to cause misclassification without changing ground truth. **Example:** - Add imperceptible noise to chest X-ray → Model misses pneumonia - Modify patient vital signs slightly → Sepsis prediction model fails to alert **Medical relevance:** - **Natural occurrence:** Image compression, scanner differences can mimic adversarial perturbations - **Malicious:** Rare but theoretically possible (e.g., insurance fraud, medicolegal manipulation) ------------------------------------------------------------------------ #### 2. Poisoning Attacks (Training-Time) **Goal:** Corrupt training data to degrade model performance or introduce backdoors. **Example:** - Insert mislabeled images into training set → Model learns incorrect patterns - Add trigger patterns → Model fails only for specific subgroups **Medical relevance:** - **Multi-institutional data sharing:** If one site’s data is compromised, all participants affected - **Crowdsourced labels:** If annotations are maliciously manipulated ------------------------------------------------------------------------ #### 3. Model Extraction/Stealing **Goal:** Query model repeatedly to reverse-engineer its parameters. **Risk:** Intellectual property theft, creating surrogate model for further attacks ------------------------------------------------------------------------ ### Evaluating Robustness #### Method 1: Input Perturbation Testing **Approach:** Systematically perturb inputs and measure performance degradation. **For medical imaging:** ``` sourceCode import numpy as np from skimage.util import random_noise def test_noise_robustness(model, test_images, test_labels, noise_levels): """ Test model robustness to image noise Args: model: Trained classification model test_images: Clean test images test_labels: Ground truth labels noise_levels: List of noise standard deviations to test Returns: Dictionary of accuracy at each noise level """ results = {} # Baseline (no noise) baseline_acc = model.evaluate(test_images, test_labels)[1] results['baseline'] = baseline_acc # Test each noise level for sigma in noise_levels: noisy_images = np.array([ random_noise(img, mode='gaussian', var=sigma**2) for img in test_images ]) noisy_acc = model.evaluate(noisy_images, test_labels)[1] results[f'sigma_{sigma}'] = noisy_acc degradation = baseline_acc - noisy_acc print(f"Noise σ={sigma:.3f}: Accuracy={noisy_acc:.3f} " f"(degradation: {degradation:.3f})") return results # Example usage noise_levels = [0.01, 0.05, 0.10, 0.20] robustness_results = test_noise_robustness( model, test_images, test_labels, noise_levels ) # Acceptable degradation threshold if robustness_results['sigma_0.05'] < 0.85 * robustness_results['baseline']: print("[WARNING] Model performance degrades >15% with minor noise") print("→ Consider: Data augmentation, robust training, ensemble methods") ``` ------------------------------------------------------------------------ **For tabular data (clinical variables):** ``` sourceCode def test_feature_perturbation_robustness(model, X_test, y_test, perturbation_fraction=0.05): """ Test robustness to small perturbations in continuous features Args: model: Trained model X_test: Test features (pandas DataFrame) y_test: Test labels perturbation_fraction: Fraction of feature value to perturb Returns: Robustness metrics """ from sklearn.metrics import roc_auc_score # Baseline performance y_pred_baseline = model.predict_proba(X_test)[:, 1] auc_baseline = roc_auc_score(y_test, y_pred_baseline) # Perturb continuous features X_perturbed = X_test.copy() continuous_cols = X_test.select_dtypes(include=[np.number]).columns for col in continuous_cols: # Add random noise proportional to feature value noise = np.random.normal(0, perturbation_fraction * X_test[col].std(), size=len(X_test)) X_perturbed[col] = X_test[col] + noise # Evaluate perturbed performance y_pred_perturbed = model.predict_proba(X_perturbed)[:, 1] auc_perturbed = roc_auc_score(y_test, y_pred_perturbed) # Prediction consistency prediction_changes = np.mean( (y_pred_baseline > 0.5) != (y_pred_perturbed > 0.5) ) print(f"Baseline AUC: {auc_baseline:.3f}") print(f"Perturbed AUC: {auc_perturbed:.3f}") print(f"AUC degradation: {auc_baseline - auc_perturbed:.3f}") print(f"Prediction changes: {prediction_changes:.1%}") return { 'auc_baseline': auc_baseline, 'auc_perturbed': auc_perturbed, 'prediction_change_rate': prediction_changes } # Example results = test_feature_perturbation_robustness(model, X_test, y_test, perturbation_fraction=0.05) if results['prediction_change_rate'] > 0.10: print("[WARNING] >10% of predictions change with 5% feature noise") print("→ Model may be overfitting to noise rather than signal") ``` ------------------------------------------------------------------------ #### Method 2: Adversarial Attack Testing **Fast Gradient Sign Method (FGSM)** - Basic adversarial attack: ``` sourceCode import tensorflow as tf def fgsm_attack(model, image, label, epsilon=0.01): """ Generate adversarial example using Fast Gradient Sign Method Args: model: Trained model image: Input image label: True label epsilon: Perturbation magnitude Returns: Adversarial image """ image = tf.cast(image, tf.float32) with tf.GradientTape() as tape: tape.watch(image) prediction = model(image) loss = tf.keras.losses.sparse_categorical_crossentropy(label, prediction) # Get gradient of loss w.r.t. image gradient = tape.gradient(loss, image) # Create adversarial image signed_grad = tf.sign(gradient) adversarial_image = image + epsilon * signed_grad adversarial_image = tf.clip_by_value(adversarial_image, 0, 1) return adversarial_image # Evaluate adversarial robustness def evaluate_adversarial_robustness(model, test_images, test_labels, epsilons=[0.0, 0.01, 0.05, 0.10]): """ Test model robustness to FGSM attacks at different perturbation levels """ results = {} for eps in epsilons: correct = 0 total = 0 for img, label in zip(test_images, test_labels): # Generate adversarial example adv_img = fgsm_attack(model, img[np.newaxis, ...], label, epsilon=eps) # Predict pred = model.predict(adv_img) pred_class = np.argmax(pred) if pred_class == label: correct += 1 total += 1 accuracy = correct / total results[eps] = accuracy print(f"Epsilon={eps:.3f}: Accuracy={accuracy:.3f}") return results # Run evaluation adv_results = evaluate_adversarial_robustness(model, test_images, test_labels) # Alert if significant degradation if adv_results[0.05] < 0.70 * adv_results[0.0]: print("[CRITICAL] Model highly vulnerable to adversarial attacks") print("→ Implement: Adversarial training, input validation, ensemble methods") ``` ------------------------------------------------------------------------ #### Method 3: Out-of-Distribution (OOD) Detection **Goal:** Identify when inputs are unlike training data (model should abstain or flag uncertainty). ``` sourceCode def evaluate_ood_detection(model, in_dist_data, ood_data): """ Evaluate model's ability to detect out-of-distribution inputs Args: model: Trained model in_dist_data: In-distribution test data ood_data: Out-of-distribution data Returns: OOD detection performance metrics """ from sklearn.metrics import roc_auc_score # Get prediction confidence (max probability) for each dataset in_dist_preds = model.predict(in_dist_data) in_dist_confidence = np.max(in_dist_preds, axis=1) ood_preds = model.predict(ood_data) ood_confidence = np.max(ood_preds, axis=1) # Combine labels (1 = in-distribution, 0 = OOD) y_true = np.concatenate([ np.ones(len(in_dist_confidence)), np.zeros(len(ood_confidence)) ]) # Confidence scores (higher = more likely in-distribution) confidence_scores = np.concatenate([in_dist_confidence, ood_confidence]) # Calculate AUROC for OOD detection auroc = roc_auc_score(y_true, confidence_scores) print(f"OOD Detection AUROC: {auroc:.3f}") print(f"In-dist mean confidence: {in_dist_confidence.mean():.3f}") print(f"OOD mean confidence: {ood_confidence.mean():.3f}") if auroc < 0.80: print("[WARNING] Poor OOD detection - model overconfident on unfamiliar inputs") print("→ Consider: Temperature scaling, Bayesian approaches, ensemble uncertainty") return auroc # Example: Test on different medical image dataset ood_auroc = evaluate_ood_detection( model, in_dist_data=chest_xray_test, # Data from same hospitals as training ood_data=external_site_data # Data from completely different hospital/scanner ) ``` ------------------------------------------------------------------------ ### Robustness Improvement Strategies **1. Data Augmentation:** - Train on varied/augmented data (rotations, brightness changes, noise) - Forces model to learn invariant features **2. Adversarial Training:** - Include adversarial examples in training set - Trade-off: May slightly reduce clean accuracy **3. Ensemble Methods:** - Multiple models often more robust than single model - Harder to fool all models simultaneously **4. Input Validation:** - Reject inputs that are outliers (OOD detection) - Flag unusual patterns for human review **5. Certified Defenses:** - Provide mathematical guarantees of robustness - Advanced, computationally expensive ------------------------------------------------------------------------ ### Practical Robustness Evaluation Protocol Robustness Testing Checklist **Minimum requirements (all deployed models):** **Natural perturbation testing:** Test with realistic variations (noise, missing data, equipment differences) **Prediction stability:** Measure how often predictions change with small input perturbations and compare the result with a prespecified, clinically justified local threshold **Out-of-distribution detection:** Model should flag or have low confidence on unfamiliar inputs **Recommended (high-risk models):** **Adversarial attack testing:** Evaluate vulnerability to FGSM, PGD attacks **Multi-site robustness:** Validate performance across diverse sites/equipment **Ablation studies:** Test performance when features are missing or corrupted **Advanced (critical systems, regulatory requirements):** **Certified robustness:** Provide formal guarantees for critical use cases **Red-team exercise:** Security experts attempt to break the model **Continuous monitoring:** Track input distribution shifts, flag anomalies ------------------------------------------------------------------------ ### Security Best Practices **1. Model Access Control:** - Limit API access to authenticated users - Rate limiting to prevent model extraction attacks **2. Input Sanitization:** - Validate inputs are within expected ranges - Reject clearly anomalous inputs **3. Monitoring and Logging:** - Log all predictions and inputs - Monitor for unusual query patterns (potential attacks) **4. Model Versioning and Rollback:** - Maintain ability to revert to previous model if compromise detected **5. Regular Security Audits:** - Periodic red-team testing - Review access logs for suspicious activity ------------------------------------------------------------------------ ### Landmark Study: Adversarial Perturbations in Medical Imaging **[Finlayson et al., 2019, *Science*](https://www.science.org/doi/10.1126/science.aaw4399)** **Study:** Added imperceptible perturbations to medical images (chest X-rays, fundus photos, dermoscopy images) **Results:** - Successfully fooled leading deep learning classifiers - Adversarial examples transferable across models (attack one model, affects others) - Small perturbations caused dramatic misclassifications **Implications:** - Medical AI models are vulnerable to adversarial attacks - Robustness testing should be mandatory for deployed systems - Both accidental (natural variations) and malicious perturbations are risks **Counterpoint:** **No documented real-world malicious attacks** on medical AI systems (yet), but accidental distribution shifts are common (equipment changes, protocol updates). ------------------------------------------------------------------------ ### Key Takeaways: Adversarial Robustness 1. **Robustness must be demonstrated appropriately:** EU AI Act Article 15 requires appropriate accuracy, robustness, and cybersecurity for high-risk systems. Adversarial testing is one possible evidence method when the threat model warrants it. 2. **Two threat models:** Natural perturbations (common) vs. adversarial attacks (rare but possible) 3. **Multiple evaluation methods:** Noise robustness, adversarial attacks (FGSM/PGD), OOD detection 4. **Practical importance:** Equipment variation, scanner differences, data quality issues are real-world “natural adversarial examples” 5. **Trade-offs exist:** Adversarial training improves robustness but may reduce clean accuracy 6. **Prevention strategies:** Data augmentation, ensemble methods, input validation, monitoring 7. **Security is patient safety:** Model integrity directly affects clinical outcomes 8. **Document robustness:** Report perturbation testing results in validation studies 9. **OOD detection is critical:** Models must recognize when inputs are outside training distribution 10. **Continuous vigilance:** Monitor for distribution shifts and anomalous inputs post-deployment ## Implementation Outcomes: Beyond Technical Performance Even models with strong technical performance can **fail in practice** if not properly implemented. ### The Implementation Science Framework [Proctor et al., 2011](https://link.springer.com/article/10.1007/s10488-010-0319-7) define implementation outcomes: #### 1. Acceptability **Definition:** Perception that system is agreeable/satisfactory **Measures:** - User satisfaction surveys (Likert scales, Net Promoter Score) - Qualitative interviews (what do users like/dislike?) - Perceived usefulness and ease of use **Example questions:** - “This system improves my clinical decision-making” (1–5 scale) - “I would recommend this system to colleagues” (yes/no) ------------------------------------------------------------------------ #### 2. Adoption **Definition:** Intention/action to use the system **Measures:** - Utilization rate (% of eligible cases where system used) - Number of users who have activated/logged in - Time to initial use **Red flag:** Low adoption despite availability suggests problems with acceptability, workflow fit, or perceived utility. A 19 August 2026 *Nature Medicine* DECIDE-AI stage 1 evaluation of SHAKED, a multi-LLM CDSS in a tertiary emergency department (1,138 patients over 4 weeks; one wing used the system, one followed routine rotations), is the empirical form of that red flag: clinician adoption fell from 68% to 30% (OR = 0.72 per shift hour, 95% CI 0.62–0.83), expert review rated 99 of 100 sampled outputs as clinically appropriate, and no adverse events were detected, but length of stay was 4.9 h in both wings (P = 0.99) and the intention-to-treat consultation-cycle-time difference (−9.4 min) was not significant (P = 0.077) ([Leibovitch et al., 2026](https://doi.org/10.1038/s41591-026-04601-5)). Output appropriateness and decaying adoption are not a care endpoint; the authors write that the results inform randomized-trial design and do not justify clinical deployment. ------------------------------------------------------------------------ #### 3. Appropriateness **Definition:** Perceived fit for setting/population/problem **Measures:** - Stakeholder perception surveys - Alignment with clinical workflows (workflow mapping) - Relevance to clinical questions **Example:** ICU mortality prediction may be appropriate for ICU but inappropriate for outpatient clinic. ------------------------------------------------------------------------ #### 4. Feasibility **Definition:** Ability to successfully implement **Measures:** - Technical integration challenges (API compatibility, data availability) - Resource requirements (cost, staff time, training) - Infrastructure needs (computing, network) ------------------------------------------------------------------------ #### 5. Fidelity **Definition:** Degree to which system used as designed **Measures:** - Override rates (how often do clinicians dismiss alerts?) - Deviation from intended use (using for wrong purpose) - Workarounds (users circumventing system) **High override rates** signal problems: - Too many false positives (alert fatigue) - Predictions do not match clinical judgment (trust issues) - Workflow disruption (alerts at wrong time) ------------------------------------------------------------------------ #### 6. Penetration **Definition:** Integration across settings/populations **Measures:** - Number of sites/units using system - Proportion of target population reached - Geographic spread ------------------------------------------------------------------------ #### 7. Sustainability **Definition:** Continued use over time **Measures:** - Retention of users over 6–12 months - Model updating/maintenance plan - Long-term performance monitoring **Common failure:** “Pilot-itis”: successful pilot, but system not sustained after initial implementation period. ------------------------------------------------------------------------ ### Common Implementation Failures #### 1. Alert Fatigue **Problem:** Excessive false alarms → clinicians ignore alerts **Evidence:** Classic systematic reviews of computerized provider order entry found that clinicians overrode 49% to 96% of medication safety and drug-drug interaction alerts ([van der Sijs et al., 2006](https://doi.org/10.1197/jamia.M1809)), and longitudinal EHR evaluations confirm that rising alert volume and complexity directly depress clinician acceptance as alert fatigue accumulates ([Ancker et al., 2017](https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-017-0430-8)). **Example:** At the evaluated threshold, Epic’s sepsis model had 12% positive predictive value, so 88% of patients who would have generated an alert did not meet the study’s sepsis outcome ([Wong et al., 2021](https://doi.org/10.1001/jamainternmed.2021.2626)). The study estimated a substantial alert burden; it did not measure whether clinicians stopped responding. **Solutions:** - Minimize false positives (sacrifice sensitivity if needed) - Tiered alerts (critical vs. informational) - Smart timing (deliver when actionable, not during documentation) - Actionable recommendations (“Order blood cultures” not “Consider sepsis”) ------------------------------------------------------------------------ #### 2. Workflow Disruption **Problem:** System does not integrate smoothly into existing processes **Examples:** - Extra clicks required - Separate application (need to switch contexts) - Alerts interrupt at inopportune times (during patient exam) **Solutions:** - User-centered design (involve clinicians early and often) - Embed in existing EHR workflows - Minimize friction (one-click actions) For workflow integration principles, see [Bates et al., 2003, *JAMIA*](https://doi.org/10.1197/jamia.M1370) on the “Ten Commandments” for effective clinical decision support. ------------------------------------------------------------------------ #### 3. Lack of Trust **Problem:** Clinicians do not trust “black box” predictions **Example:** Deep learning model provides risk score with no explanation **Solutions:** - Provide explanations (SHAP values, attention weights) - Show evidence base (similar cases, supporting literature) - Transparent validation (publish performance data) - Gradual trust-building (start with low-stakes recommendations) ------------------------------------------------------------------------ #### 4. Model Drift **Problem:** Performance degrades over time as data distribution changes **Example:** COVID-19 pandemic changed disease patterns → pre-pandemic models failed **Why it matters:** A 2017 study of clinical prediction models found that **most require recalibration within 2-3 years** due to changes in patient populations, treatment patterns, and data collection practices. ------------------------------------------------------------------------ ### MLOps and Continuous Model Monitoring **Moving beyond “deploy and hope”:** Modern AI systems require **continuous monitoring** to detect performance degradation and trigger timely interventions. Post-Deployment Monitoring Is Not Optional The FDA’s 2021 Action Plan on AI/ML-based Software as a Medical Device emphasizes **continuous monitoring** as a core requirement. Models that do not monitor performance drift pose patient safety risks. **Real-world failure:** IBM Watson for Oncology was adopted at multiple institutions, and internal company documents reported in 2018 described **unsafe and incorrect treatment recommendations**. ------------------------------------------------------------------------ ### Types of Drift Understanding the type of drift helps determine appropriate interventions. #### 1. Data Drift (Covariate Shift) **Definition:** Input feature distributions change, but the relationship between features and outcome remains stable. **Example:** - **Training data (2019):** Average patient age = 55, BMI = 27 - **Production data (2024):** Average patient age = 62, BMI = 31 - **Relationship unchanged:** Diabetes risk per BMI unit = same **Impact:** Model may become miscalibrated (predicted probabilities off) even if discrimination (AUC-ROC) stays stable. **Detection:** Compare feature distributions between training and production data. ------------------------------------------------------------------------ #### 2. Concept Drift **Definition:** The relationship between features and outcome changes. **Example:** - **Pre-COVID (2019):** Fever + cough + dyspnea → Likely bacterial pneumonia - **During COVID (2020):** Fever + cough + dyspnea → Likely COVID-19 - **Same features, different outcome** **Impact:** Model discrimination (AUC-ROC) degrades significantly. **Detection:** Track performance metrics over time; sudden drops indicate concept drift. ------------------------------------------------------------------------ #### 3. Label Drift **Definition:** Prevalence of outcome changes. **Example:** - **Training data:** 5% sepsis prevalence - **Production:** 12% sepsis prevalence (sicker patient population) **Impact:** Predicted probabilities may be systematically too low or too high (calibration degrades). **Detection:** Compare outcome rates over time. ------------------------------------------------------------------------ ### Practical Drift Detection Methods #### Method 1: Statistical Tests for Feature Distribution Changes **Kolmogorov-Smirnov (KS) Test:** Tests whether two distributions are significantly different. Hide code ``` sourceCode from scipy.stats import ks_2samp import numpy as np import pandas as pd # Example: Monitoring patient age distribution # Training data (historical) age_train = np.random.normal(55, 15, 1000) # Mean=55, SD=15 # Production data (current month) age_prod = np.random.normal(62, 15, 500) # Mean shifted to 62 # Perform KS test statistic, p_value = ks_2samp(age_train, age_prod) print(f"KS statistic: {statistic:.3f}") print(f"P-value: {p_value:.4f}") if p_value < 0.01: print("[ALERT] Significant distribution shift detected!") print("→ Review model calibration and consider retraining") else: print("[OK] No significant drift detected") # Interpretation: # p < 0.01: Strong evidence of distribution change # p < 0.05: Moderate evidence of distribution change # p ≥ 0.05: No significant change detected ``` **When to use:** Continuous features (age, lab values, vital signs) ------------------------------------------------------------------------ **Chi-Square Test (for categorical features):** Hide code ``` sourceCode from scipy.stats import chi2_contingency # Example: Monitoring sex distribution # Training data train_counts = {"Male": 600, "Female": 400} # 60% male # Production data (current month) prod_counts = {"Male": 250, "Female": 250} # 50% male # Create contingency table contingency_table = pd.DataFrame({ 'Train': [train_counts['Male'], train_counts['Female']], 'Production': [prod_counts['Male'], prod_counts['Female']] }, index=['Male', 'Female']) chi2, p_value, dof, expected = chi2_contingency(contingency_table) print(f"Chi-square statistic: {chi2:.3f}") print(f"P-value: {p_value:.4f}") if p_value < 0.01: print("[ALERT] Significant distribution shift in sex distribution!") ``` **When to use:** Categorical features (sex, race, diagnostic codes) ------------------------------------------------------------------------ #### Method 2: Population Stability Index (PSI) **Widely used in industry** for monitoring feature drift. **Formula:** \\PSI = \sum\_{i=1}^{n} (P\_{prod,i} - P\_{train,i}) \times \ln\left(\frac{P\_{prod,i}}{P\_{train,i}}\right)\\ Where: - \\P\_{train,i}\\ = Proportion in training data bin \\i\\ - \\P\_{prod,i}\\ = Proportion in production data bin \\i\\ **Interpretation:** - **PSI \< 0.1:** No significant change (green) - **PSI 0.1–0.25:** Moderate change, investigate (yellow) - **PSI \> 0.25:** Significant change, likely requires retraining (red) **Code example:** Hide code ``` sourceCode import numpy as np def calculate_psi(expected, actual, bins=10): """ Calculate Population Stability Index Args: expected: Training data feature values actual: Production data feature values bins: Number of bins for discretization Returns: PSI value """ # Create bins based on training data quantiles breakpoints = np.percentile(expected, np.linspace(0, 100, bins + 1)) breakpoints[-1] = breakpoints[-1] + 0.001 # Ensure max value included # Calculate proportions in each bin expected_counts = np.histogram(expected, bins=breakpoints)[0] actual_counts = np.histogram(actual, bins=breakpoints)[0] expected_props = expected_counts / len(expected) actual_props = actual_counts / len(actual) # Avoid log(0) by adding small constant expected_props = np.where(expected_props == 0, 0.0001, expected_props) actual_props = np.where(actual_props == 0, 0.0001, actual_props) # Calculate PSI psi_values = (actual_props - expected_props) * np.log(actual_props / expected_props) psi = np.sum(psi_values) return psi # Example: Monitor glucose values glucose_train = np.random.gamma(5, 20, 1000) # Training data glucose_prod = np.random.gamma(6, 22, 500) # Production data (shifted) psi = calculate_psi(glucose_train, glucose_prod, bins=10) print(f"PSI: {psi:.3f}") if psi < 0.1: print("[OK] No significant drift") elif psi < 0.25: print("[WARNING] Moderate drift - investigate") else: print("[CRITICAL] Significant drift - retrain model") ``` ------------------------------------------------------------------------ #### Method 3: Model Performance Monitoring **Most direct approach:** Track actual model performance over time. **Challenge:** Requires ground truth labels, which may have **delay** (e.g., 30-day readmission cannot be verified for 30 days). **Metrics to monitor:** - AUC-ROC (discrimination) - Brier score (calibration) - Sensitivity/specificity at operational threshold - PPV/NPV **Implementation:** Hide code ``` sourceCode import matplotlib.pyplot as plt from sklearn.metrics import roc_auc_score, brier_score_loss from datetime import datetime, timedelta # Simulate monthly performance tracking months = pd.date_range(start='2023-01-01', end='2024-12-01', freq='MS') # Simulated AUC-ROC over time (degrading model) np.random.seed(42) baseline_auc = 0.85 auc_scores = baseline_auc - np.linspace(0, 0.10, len(months)) + np.random.normal(0, 0.02, len(months)) # Create monitoring dashboard data performance_data = pd.DataFrame({ 'month': months, 'auc': auc_scores }) # Set alert thresholds auc_warning_threshold = 0.80 auc_critical_threshold = 0.75 # Plot performance over time plt.figure(figsize=(10, 6)) plt.plot(performance_data['month'], performance_data['auc'], marker='o', linewidth=2, label='Monthly AUC-ROC') # Add threshold lines plt.axhline(y=baseline_auc, color='green', linestyle='--', label=f'Baseline ({baseline_auc:.2f})', alpha=0.7) plt.axhline(y=auc_warning_threshold, color='orange', linestyle='--', label=f'Warning threshold ({auc_warning_threshold:.2f})', alpha=0.7) plt.axhline(y=auc_critical_threshold, color='red', linestyle='--', label=f'Critical threshold ({auc_critical_threshold:.2f})', alpha=0.7) # Highlight months below threshold alerts = performance_data[performance_data['auc'] < auc_warning_threshold] if not alerts.empty: plt.scatter(alerts['month'], alerts['auc'], color='red', s=100, zorder=5, label='Alert triggered') plt.xlabel('Month') plt.ylabel('AUC-ROC') plt.title('Model Performance Monitoring Dashboard') plt.legend() plt.grid(True, alpha=0.3) plt.xticks(rotation=45) plt.tight_layout() plt.show() # Check for alerts latest_auc = performance_data['auc'].iloc[-1] if latest_auc < auc_critical_threshold: print(f"[CRITICAL] AUC-ROC = {latest_auc:.3f} (below {auc_critical_threshold})") print("→ IMMEDIATE ACTION: Suspend model or initiate emergency retraining") elif latest_auc < auc_warning_threshold: print(f"[WARNING] AUC-ROC = {latest_auc:.3f} (below {auc_warning_threshold})") print("→ ACTION: Schedule model retraining within 30 days") else: print(f"[OK] Performance acceptable: AUC-ROC = {latest_auc:.3f}") ``` ------------------------------------------------------------------------ #### Method 4: Prediction Distribution Monitoring **Insight:** Even without ground truth, you can monitor **what the model is predicting**. **Red flag patterns:** - Sudden increase in high-risk predictions (model becoming overly sensitive) - Sudden decrease in high-risk predictions (model missing cases) - Bimodal distribution shifts (calibration degradation) **Example:** Hide code ``` sourceCode # Monitor distribution of predicted probabilities over time # Week 1 predictions (well-calibrated) week1_preds = np.random.beta(2, 10, 1000) # Mostly low risk # Week 12 predictions (drift - more high-risk predictions) week12_preds = np.random.beta(3, 7, 1000) # Shifted higher # Visualize fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4)) ax1.hist(week1_preds, bins=50, alpha=0.7, label='Week 1', density=True) ax1.hist(week12_preds, bins=50, alpha=0.7, label='Week 12', density=True) ax1.set_xlabel('Predicted Probability') ax1.set_ylabel('Density') ax1.set_title('Prediction Distribution Shift') ax1.legend() # KS test to detect shift ks_stat, p_val = ks_2samp(week1_preds, week12_preds) ax2.text(0.5, 0.5, f'KS test p-value: {p_val:.4f}\n' + ('[WARNING] Significant shift detected' if p_val < 0.01 else '[OK] No significant shift'), ha='center', va='center', fontsize=14, bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5)) ax2.axis('off') plt.tight_layout() plt.show() ``` ------------------------------------------------------------------------ ### Automated Retraining Triggers **Goal:** Define clear rules for when to retrain the model. Retraining Decision Framework **IMMEDIATE retraining triggered by:** - AUC-ROC drops \>0.05 below baseline - Calibration error (Brier score) increases \>0.05 - Safety event: Model missed critical case (e.g., sepsis death after negative prediction) - Feature drift: PSI \> 0.25 for ≥3 critical features **SCHEDULED retraining triggered by:** - 12 months since last training (routine maintenance) - AUC-ROC drops 0.02–0.05 below baseline (warning level) - PSI 0.1–0.25 for multiple features (moderate drift) - New data volume: ≥20% new samples since last training **HOLD retraining if:** - All metrics within acceptable ranges - Recent retraining (\< 3 months ago) - Insufficient new data (\< 5% new samples) **Automated monitoring script:** Hide code ``` sourceCode def evaluate_retraining_need(current_auc, baseline_auc, psi_scores, months_since_training, new_sample_fraction): """ Automated decision system for model retraining Returns: - "IMMEDIATE": Retrain immediately - "SCHEDULED": Schedule retraining within 30 days - "MONITOR": Continue monitoring, no action needed """ # Critical performance degradation if current_auc < baseline_auc - 0.05: return "IMMEDIATE", "AUC-ROC dropped >0.05" # Critical feature drift critical_drift_count = sum([psi > 0.25 for psi in psi_scores]) if critical_drift_count >= 3: return "IMMEDIATE", f"{critical_drift_count} features with PSI > 0.25" # Routine maintenance schedule if months_since_training >= 12: return "SCHEDULED", "12-month routine retraining" # Moderate performance degradation if current_auc < baseline_auc - 0.02: return "SCHEDULED", "AUC-ROC dropped 0.02-0.05" # Moderate drift moderate_drift_count = sum([0.1 < psi < 0.25 for psi in psi_scores]) if moderate_drift_count >= 4: return "SCHEDULED", f"{moderate_drift_count} features with moderate drift" # Significant new data if new_sample_fraction >= 0.20: return "SCHEDULED", f"{new_sample_fraction:.0%} new data available" return "MONITOR", "All metrics within acceptable ranges" # Example usage decision, reason = evaluate_retraining_need( current_auc=0.82, baseline_auc=0.85, psi_scores=[0.08, 0.15, 0.22, 0.18, 0.05], # PSI for 5 key features months_since_training=8, new_sample_fraction=0.15 ) print(f"Decision: {decision}") print(f"Reason: {reason}") ``` ------------------------------------------------------------------------ ### Continuous Learning Strategies **Two approaches:** #### 1. Periodic Retraining (Safer, recommended for high-stakes) - **Process:** 1. Accumulate new data 2. Retrain model offline 3. Validate on hold-out set 4. A/B test against current model 5. Deploy if superior - **Advantages:** Controlled validation before deployment - **Disadvantages:** Model can drift between retraining cycles ------------------------------------------------------------------------ #### 2. Online Learning (Advanced, requires careful monitoring) - **Process:** Model updates continuously with new data - **Advantages:** Always current - **Disadvantages:** - Risk of catastrophic forgetting - Harder to validate - Vulnerable to adversarial data poisoning **Recommendation for healthcare:** Use **periodic retraining** with trigger-based scheduling (not true online learning). ------------------------------------------------------------------------ ### Real-World Example: External Validation After Adoption Epic’s sepsis model had already been implemented at hundreds of US hospitals when Wong and colleagues published an independent external validation at one academic health system ([Wong et al., 2021](https://doi.org/10.1001/jamainternmed.2021.2626)). At the evaluated threshold, the study reported: - **Sensitivity:** 33% - **Positive predictive value:** 12% - **Alert burden:** 18% of hospitalizations would have generated at least one alert The study showed poor discrimination and calibration in that setting. It did not test model drift, continuous monitoring at other deployment sites, or a retraining protocol. **Lesson:** Widespread adoption is not evidence of transportability or clinical value. Independent evaluation should precede broad use, and post-deployment monitoring should address questions that a retrospective external validation cannot answer. **Reference:** [Wong et al., 2021, *JAMA Internal Medicine*](https://doi.org/10.1001/jamainternmed.2021.2626) ------------------------------------------------------------------------ ### Recommended Monitoring Tools **Open-source libraries:** **1. Evidently AI:** - Monitors data drift, model performance, target drift - Generates visual reports - Integrates with MLOps pipelines Hide code ``` sourceCode # Example: Evidently for drift detection from evidently.report import Report from evidently.metric_preset import DataDriftPreset # Create drift report report = Report(metrics=[DataDriftPreset()]) # Compare reference (training) vs. current (production) data report.run(reference_data=train_df, current_data=production_df) # Save report report.save_html("drift_report.html") # Access metrics programmatically drift_results = report.as_dict() n_drifted_features = drift_results['metrics'][0]['result']['number_of_drifted_columns'] if n_drifted_features > 3: print(f"[WARNING] {n_drifted_features} features showing significant drift") ``` **2. Alibi Detect:** - Drift detection algorithms - Supports tabular, text, image data **3. NannyML:** - Performance estimation without ground truth - Confidence-based performance monitoring ------------------------------------------------------------------------ ### Key Takeaways: Model Drift and Monitoring 1. **All models drift:** Performance degradation is inevitable, not exceptional 2. **Types of drift matter:** Data drift vs. concept drift require different interventions 3. **Multiple detection methods:** Use statistical tests (KS, Chi-square), PSI, and performance tracking simultaneously 4. **Automated triggers:** Define clear thresholds for retraining (do not wait for catastrophic failure) 5. **Continuous monitoring is mandatory:** FDA and EU regulations increasingly require post-deployment monitoring 6. **Periodic retraining \> online learning:** For healthcare, controlled validation is safer than continuous updates 7. **Monitor before you have ground truth:** Prediction distribution shifts can signal problems early 8. **Document everything:** Track what triggered retraining, what changed, and validation results **Framework reference:** [Davis et al., 2017, *JAMIA*](https://academic.oup.com/jamia/article/24/6/1052/3767873) - Clinical prediction models degrade; most need recalibration after 2-3 years. ------------------------------------------------------------------------ ## Explainability and Interpretability (XAI) ### Why Explainability Matters in Public Health AI **The trust problem:** Systematic reviews consistently find that clinicians are reluctant to trust or act on predictions from “black box” AI systems they cannot interpret ([Antoniadi et al., 2021](https://doi.org/10.3390/app11115088); [Markus et al., 2021](https://doi.org/10.1016/j.jbi.2020.103655)). **Why interpretability is critical:** 1. **Clinical decision-making:** Clinicians need to know *why* before they can decide *whether* to act 2. **Debugging and validation:** Explanations reveal spurious correlations and dataset biases 3. **Regulatory requirements:** The FDA emphasizes transparency to users, while EU AI Act Article 13 requires instructions sufficient for deployers to interpret and use high-risk system outputs appropriately 4. **Patient autonomy:** Patients have a right to understand decisions affecting their health 5. **Legal liability:** “The algorithm said so” is not a defense in malpractice cases Post-Hoc Explanation Is Not Intrinsic Interpretability **Traditional belief:** Deep learning = high accuracy but uninterpretable; simpler models = lower accuracy but interpretable. **Current reality:** Post-hoc methods can describe associations with model outputs, but they do not make a model intrinsically interpretable. Their fidelity, stability, and usefulness must be evaluated for the intended user and decision. **Guideline:** Start with the simplest model that meets performance requirements. If a complex model is required, budget for explanation infrastructure and for testing the explanations it produces. ------------------------------------------------------------------------ ### Levels of Interpretability Not all interpretability is equal. Different stakeholders need different levels of explanation. #### 1. Global Interpretability **Definition:** Understanding the model’s overall behavior and decision logic. **Questions answered:** - What features are most important overall? - How does the model generally make decisions? - Are there unexpected feature relationships? **Methods:** - Feature importance rankings - Partial dependence plots - Global SHAP values **Audience:** Data scientists, validators, regulators ------------------------------------------------------------------------ #### 2. Local Interpretability **Definition:** Understanding why the model made a *specific* prediction for a *specific* patient. **Questions answered:** - Why did the model predict this patient is high-risk? - Which patient characteristics drove this prediction? - What would need to change to alter the prediction? **Methods:** - LIME (Local Interpretable Model-agnostic Explanations) - SHAP values for individual predictions - Counterfactual explanations **Audience:** Clinicians, patients ------------------------------------------------------------------------ #### 3. Model-Based Interpretability **Definition:** Models that are inherently interpretable by design. **Examples:** - **Linear models:** Each coefficient shows feature contribution - **Decision trees:** Follow the path to understand the decision - **Rule-based systems:** Explicit IF-THEN logic **When to use:** When stakeholder trust is paramount and model performance requirements are modest. ------------------------------------------------------------------------ ### Interpretability Methods: Practical Guide #### Method 1: SHAP (SHapley Additive exPlanations) **What it is:** A unified framework for interpreting model predictions based on game theory (Shapley values). **Strengths:** - **Model-agnostic:** Works with any ML model (XGBoost, neural networks, etc.) - **Theoretically grounded:** Satisfies desirable properties (local accuracy, consistency) - **Both global and local:** Feature importance + individual predictions **Foundational paper:** [Lundberg & Lee, 2017, NeurIPS](https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html) ------------------------------------------------------------------------ ##### SHAP Example: Sepsis Risk Prediction ``` sourceCode import shap import xgboost as xgb import pandas as pd import numpy as np import matplotlib.pyplot as plt # Simulate patient data np.random.seed(42) n_patients = 1000 data = pd.DataFrame({ 'age': np.random.normal(65, 15, n_patients), 'heart_rate': np.random.normal(90, 20, n_patients), 'temperature': np.random.normal(37.5, 1.5, n_patients), 'wbc_count': np.random.lognormal(2.3, 0.5, n_patients), # White blood cell count 'lactate': np.random.exponential(1.5, n_patients), 'systolic_bp': np.random.normal(120, 25, n_patients), }) # Create synthetic sepsis outcome (complex non-linear relationships) sepsis_risk = ( 0.3 * (data['temperature'] > 38.3) + # Fever 0.3 * (data['wbc_count'] > 12) + # Elevated WBC 0.2 * (data['lactate'] > 2) + # Elevated lactate 0.2 * (data['heart_rate'] > 100) + # Tachycardia np.random.normal(0, 0.1, n_patients) # Noise ) data['sepsis'] = (sepsis_risk > 0.6).astype(int) # Split data from sklearn.model_selection import train_test_split X = data.drop('sepsis', axis=1) y = data['sepsis'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y ) # Train XGBoost model model = xgb.XGBClassifier( n_estimators=100, max_depth=5, learning_rate=0.1, random_state=42 ) model.fit(X_train, y_train) # Evaluate from sklearn.metrics import roc_auc_score y_pred_proba = model.predict_proba(X_test)[:, 1] auc = roc_auc_score(y_test, y_pred_proba) print(f"Model AUC-ROC: {auc:.3f}") # ===== SHAP ANALYSIS ===== # 1. Create SHAP explainer explainer = shap.Explainer(model, X_train) # 2. Calculate SHAP values for test set shap_values = explainer(X_test) # 3. GLOBAL INTERPRETABILITY: Feature importance print("\n=== Global Feature Importance ===") shap.plots.bar(shap_values, show=False) plt.title("Global Feature Importance (Mean |SHAP|)") plt.tight_layout() plt.savefig("shap_global_importance.png", dpi=150) plt.show() # Alternative: Summary plot (beeswarm) shap.plots.beeswarm(shap_values, show=False) plt.title("Feature Impact on Model Output") plt.tight_layout() plt.savefig("shap_summary.png", dpi=150) plt.show() # 4. LOCAL INTERPRETABILITY: Explain specific patient patient_idx = 0 # First patient in test set print(f"\n=== Patient {patient_idx} ===") print(f"Predicted sepsis probability: {y_pred_proba[patient_idx]:.2%}") print(f"Actual outcome: {'Sepsis' if y_test.iloc[patient_idx] == 1 else 'No sepsis'}") # Waterfall plot: Show how features contribute to this prediction shap.plots.waterfall(shap_values[patient_idx], show=False) plt.title(f"Explanation for Patient {patient_idx}") plt.tight_layout() plt.savefig(f"shap_patient_{patient_idx}.png", dpi=150) plt.show() # 5. ACTIONABLE INSIGHTS: What drives high-risk predictions? print("\n=== Feature Values for High-Risk Patient ===") for feature in X.columns: print(f"{feature}: {X_test.iloc[patient_idx][feature]:.2f}") # 6. Dependence plot: How does lactate affect predictions? shap.plots.scatter(shap_values[:, "lactate"], color=shap_values, show=False) plt.title("Lactate Impact on Sepsis Prediction") plt.tight_layout() plt.savefig("shap_dependence_lactate.png", dpi=150) plt.show() ``` **Key outputs:** 1. **Global importance:** The features that matter most across all patients 2. **Waterfall plot:** For Patient X, lactate (+0.3) and temperature (+0.2) increased risk; normal BP (-0.1) decreased it 3. **Dependence plots:** Non-linear relationships (e.g., lactate \> 2 mmol/L sharply increases risk) **Clinical translation:** Patient 47: Sepsis Risk = 78% Main drivers: + Lactate 3.2 mmol/L (+0.35 risk contribution) <- Primary concern + Temperature 39.1°C (+0.22) + WBC 15,000/μL (+0.18) - Normal BP 118/72 (-0.08) <- Protective factor Interpretation: Elevated lactate is the largest contributor to this prediction. Consider serial lactate monitoring and early fluid resuscitation. ------------------------------------------------------------------------ ##### SHAP Advantages and Limitations **Advantages:** - Mathematically principled (satisfies local accuracy, missingness, consistency) - Works with any model architecture - Both global and local explanations - Handles feature interactions **Limitations:** - **Computational cost:** Can be slow for large models/datasets (use TreeSHAP for tree models, faster) - **Not causal:** A high SHAP value reflects association, not a causal relationship - **Assumes feature independence:** Can give misleading results with highly correlated features **Best practices:** - Use TreeSHAP for tree-based models (XGBoost, Random Forest); it is much faster than KernelSHAP - For neural networks, use DeepSHAP or KernelSHAP with background dataset sampling - Always review explanations with domain experts to confirm they are clinically plausible ------------------------------------------------------------------------ #### Method 2: LIME (Local Interpretable Model-agnostic Explanations) **What it is:** Creates a simple, interpretable model (like linear regression) that approximates the complex model’s behavior *locally* around a specific prediction. **How it works:** 1. Perturb the input (create similar but slightly different patients) 2. Get model predictions for perturbed inputs 3. Fit a simple linear model to these local predictions 4. Linear coefficients = feature importance for this prediction **When to use:** - Need quick local explanations - SHAP is too computationally expensive - Want human-readable rules (“If lactate \> 2 AND fever, then high risk”) **Foundational paper:** [Ribeiro et al., 2016, KDD](https://dl.acm.org/doi/10.1145/2939672.2939778) ------------------------------------------------------------------------ ##### LIME Example: Readmission Risk ``` sourceCode import lime import lime.lime_tabular import numpy as np import pandas as pd from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split # Simulate patient data for hospital readmission np.random.seed(42) n_patients = 1000 data = pd.DataFrame({ 'age': np.random.normal(68, 12, n_patients), 'num_prior_admissions': np.random.poisson(2, n_patients), 'length_of_stay': np.random.gamma(2, 2, n_patients), 'num_medications': np.random.poisson(5, n_patients), 'comorbidity_count': np.random.poisson(3, n_patients), 'emergency_admission': np.random.binomial(1, 0.3, n_patients), }) # Create readmission outcome readmit_risk = ( 0.02 * data['age'] + 0.15 * data['num_prior_admissions'] + 0.05 * data['comorbidity_count'] + 0.1 * data['emergency_admission'] + np.random.normal(0, 0.5, n_patients) ) data['readmitted_30d'] = (readmit_risk > 2).astype(int) # Train model X = data.drop('readmitted_30d', axis=1) y = data['readmitted_30d'] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y ) model = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) # ===== LIME EXPLANATION ===== # 1. Create LIME explainer explainer = lime.lime_tabular.LimeTabularExplainer( training_data=X_train.values, feature_names=X_train.columns.tolist(), class_names=['No Readmission', 'Readmission'], mode='classification', random_state=42 ) # 2. Explain a specific patient patient_idx = 5 patient_data = X_test.iloc[patient_idx].values predicted_proba = model.predict_proba([patient_data])[0] print(f"=== Patient {patient_idx} ===") print(f"Predicted readmission probability: {predicted_proba[1]:.2%}") print(f"Actual outcome: {'Readmitted' if y_test.iloc[patient_idx] == 1 else 'Not readmitted'}") # Generate explanation explanation = explainer.explain_instance( data_row=patient_data, predict_fn=model.predict_proba, num_features=6 ) # 3. Display explanation print("\n=== LIME Explanation ===") print("Feature contributions to 'Readmission' class:") for feature, weight in explanation.as_list(): print(f" {feature}: {weight:+.3f}") # 4. Visualize explanation.show_in_notebook(show_table=True) # Save as HTML explanation.save_to_file('lime_explanation_patient5.html') # 5. Extract feature importance for this patient feature_importance = dict(explanation.as_list()) print("\n=== Top Risk Factors for This Patient ===") sorted_features = sorted(feature_importance.items(), key=lambda x: abs(x[1]), reverse=True) for feature, weight in sorted_features[:3]: direction = "↑ Increases" if weight > 0 else "↓ Decreases" print(f"{direction} risk: {feature} (impact: {weight:+.3f})") ``` **Example output:** === Patient 5 === Predicted readmission probability: 64% Feature contributions: num_prior_admissions > 3.00: +0.22 ← Major risk factor comorbidity_count > 4.00: +0.15 emergency_admission = 1: +0.12 age > 65.00: +0.08 length_of_stay ≤ 3.00: -0.05 ← Lowers predicted risk in this model (association, not mechanism) num_medications > 6.00: -0.02 Interpretation: This patient's high readmission risk is driven primarily by multiple prior admissions (4 in past year) and high comorbidity burden. ------------------------------------------------------------------------ ##### LIME Advantages and Limitations **Advantages:** - **Fast:** Quicker than SHAP for local explanations - **Intuitive:** Simple “if-then” rules easy for clinicians to understand - **Model-agnostic:** Works with any black box model **Limitations:** - **Instability:** Explanations can vary substantially with small input changes - **Local only:** Does not provide global model understanding - **Arbitrary perturbations:** Sampling strategy affects explanation quality - **No theoretical guarantees:** Unlike SHAP, not mathematically principled **When to choose LIME over SHAP:** - Real-time explanations needed (speed critical) - Prefer rule-based explanations (“If X \> 5 AND Y \< 10…”) - SHAP computationally infeasible for the model ------------------------------------------------------------------------ #### Method 3: Attention Mechanisms (For Deep Learning) **What it is:** Neural network architectures that learn to weight input elements, producing attention weights that are often inspected as explanations. **Applications:** - **Transformers:** BERT, GPT for clinical notes analysis - **Vision models:** Which parts of chest X-ray drove diagnosis? - **Time-series:** Which ICU monitoring data points triggered alert? **Example application:** Radiology AI highlights suspicious regions in medical images using attention heatmaps. ------------------------------------------------------------------------ ##### Attention Visualization Example ``` sourceCode import torch import torch.nn as nn import numpy as np import matplotlib.pyplot as plt # Simple attention-based model for ICU time-series data class AttentionICU(nn.Module): def __init__(self, input_dim, hidden_dim): super().__init__() self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True) # Attention mechanism self.attention = nn.Linear(hidden_dim, 1) self.classifier = nn.Linear(hidden_dim, 1) def forward(self, x): # x shape: (batch, time_steps, features) lstm_out, _ = self.lstm(x) # (batch, time_steps, hidden_dim) # Calculate attention scores attention_scores = self.attention(lstm_out) # (batch, time_steps, 1) attention_weights = torch.softmax(attention_scores, dim=1) # Apply attention (weighted sum of LSTM outputs) context = torch.sum(attention_weights * lstm_out, dim=1) # (batch, hidden_dim) # Final prediction output = self.classifier(context) return torch.sigmoid(output), attention_weights # Simulate ICU time-series data # Features: HR, BP, SpO2, RR over 24 hours (hourly measurements) torch.manual_seed(42) n_patients = 100 time_steps = 24 n_features = 4 X = torch.randn(n_patients, time_steps, n_features) y = torch.randint(0, 2, (n_patients, 1)).float() # Binary outcome # Train model model = AttentionICU(input_dim=n_features, hidden_dim=32) criterion = nn.BCELoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.001) # Training loop (simplified) for epoch in range(50): optimizer.zero_grad() predictions, attention_weights = model(X) loss = criterion(predictions, y) loss.backward() optimizer.step() # ===== INTERPRET ATTENTION WEIGHTS ===== # Explain a specific patient patient_idx = 0 patient_data = X[patient_idx:patient_idx+1] prediction, attention = model(patient_data) print(f"Predicted risk: {prediction.item():.2%}") # Visualize attention over time attention_np = attention.detach().numpy()[0, :, 0] # Shape: (time_steps,) plt.figure(figsize=(12, 4)) plt.subplot(1, 2, 1) plt.plot(range(24), attention_np, marker='o') plt.xlabel('Hour') plt.ylabel('Attention Weight') plt.title('Which Time Points Were Most Important?') plt.axhline(1/24, color='r', linestyle='--', label='Uniform attention') plt.legend() # Identify critical time periods top_hours = np.argsort(attention_np)[-3:][::-1] print(f"\nMost important time periods: Hours {top_hours}") print("Interpretation: Model focused on these specific hours when making prediction") # Overlay attention on vital signs plt.subplot(1, 2, 2) vitals = patient_data.detach().numpy()[0, :, 0] # Heart rate plt.plot(range(24), vitals, label='Heart Rate', alpha=0.7) plt.scatter(range(24), vitals, s=attention_np*1000, c='red', alpha=0.5, label='Attention (size = importance)') plt.xlabel('Hour') plt.ylabel('Heart Rate') plt.title('Attention-Weighted Vital Signs') plt.legend() plt.tight_layout() plt.savefig('attention_interpretation.png', dpi=150) plt.show() print("\nClinical interpretation:") print(f"The model identified hours {top_hours[0]}, {top_hours[1]}, {top_hours[2]} as critical.") print("Clinician should review events during these time windows.") ``` **Key insight:** Attention weights are built into the architecture, so they require no separate post-hoc explainer. They are not, however, a faithful measure of feature importance, as the limitations below show. **Limitations:** - Attention ≠ causation - High attention does not guarantee that feature is truly important (attention is correlation) - Requires model architecture modification (cannot apply to existing black boxes) ------------------------------------------------------------------------ #### Method 4: Counterfactual Explanations **What it is:** An explanation that identifies what would need to change for the model to make a different prediction. **Example:** - **Prediction:** Patient has 75% readmission risk - **Counterfactual:** “If patient had ≤2 prior admissions (currently 4) OR comorbidity count ≤3 (currently 5), risk would drop to \<30%” **Uses:** - **Actionable:** Shows which input changes would alter the prediction; whether a change is achievable or causal requires separate evidence - **Patient-friendly:** Easy to communicate (“If you lose 10 lbs, your risk decreases…”) - **Fairness auditing:** Reveals whether the model relies on unchangeable features (race, gender) ------------------------------------------------------------------------ ##### Counterfactual Example with DiCE ``` sourceCode # Install: pip install dice-ml import dice_ml from dice_ml import Dice import pandas as pd from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split # Load data (reuse readmission example from LIME section) # ... (same data generation code) ... # Train model model = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) # ===== COUNTERFACTUAL GENERATION ===== # 1. Prepare DiCE dice_data = dice_ml.Data( dataframe=pd.concat([X_train, y_train], axis=1), continuous_features=['age', 'num_prior_admissions', 'length_of_stay', 'num_medications', 'comorbidity_count'], outcome_name='readmitted_30d' ) dice_model = dice_ml.Model(model=model, backend='sklearn') explainer = Dice(dice_data, dice_model, method='random') # 2. Generate counterfactuals for high-risk patient patient_idx = 5 patient_df = X_test.iloc[[patient_idx]] # Find alternative scenarios where the patient would be predicted NOT to be readmitted counterfactuals = explainer.generate_counterfactuals( query_instances=patient_df, total_CFs=3, # Generate 3 alternative scenarios desired_class='opposite' # Want opposite prediction ) # 3. Display results print("=== Original Patient ===") print(patient_df.T) print(f"\nPredicted outcome: Readmission (Risk: {model.predict_proba(patient_df)[0][1]:.2%})") print("\n=== Counterfactual Scenarios (Opposite Prediction) ===") cf_df = counterfactuals.cf_examples_list[0].final_cfs_df print(cf_df.T) # 4. Identify key changes print("\n=== Key Changes Needed ===") for col in X_test.columns: original = patient_df[col].values[0] for i, cf in cf_df.iterrows(): if abs(cf[col] - original) > 0.01: change = cf[col] - original print(f" {col}: {original:.1f} → {cf[col]:.1f} (change: {change:+.1f})") # 5. Clinical translation (hypotheses, not causal estimates) print("\n=== Candidate Interventions to Evaluate ===") print("Hypotheses for clinical review:") print(" • Reduce medication complexity (consolidate from 8 to ≤6 drugs)") print(" • Intensive post-discharge follow-up (reduce prior admit pattern)") print(" • Comorbidity management (focus on top 2-3 conditions)") ``` **Output interpretation:** Original Patient: Readmission Risk = 64% - Age: 71 - Prior admissions: 4 - Comorbidities: 5 - Medications: 8 Counterfactual Scenario 1: Risk = 18% - Age: 71 (unchanged) - Prior admissions: 1 (reduced from 4) ← Major change - Comorbidities: 5 (unchanged) - Medications: 6 (reduced from 8) Interpretation: The prediction flips mainly when prior admissions fall. Prior admissions are historical, so this shows what drives the model, not which intervention would reduce readmission. **Reference:** [Wachter et al., 2017](https://arxiv.org/abs/1711.00399) ------------------------------------------------------------------------ #### Method 5: Feature Importance (For Tree-Based Models) **What it is:** For models like Random Forest and XGBoost, built-in feature importance scores. **How it works:** - **Gini importance:** How much each feature reduces impurity when splitting - **Permutation importance:** Performance drop when feature is randomly shuffled **Advantage:** Fast, easy to compute **Limitation:** Can be biased toward high-cardinality features ``` sourceCode from sklearn.ensemble import RandomForestClassifier from sklearn.inspection import permutation_importance import matplotlib.pyplot as plt # Train model model = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) # ===== FEATURE IMPORTANCE ===== # Method 1: Built-in feature importance (Gini) gini_importance = pd.DataFrame({ 'feature': X_train.columns, 'importance': model.feature_importances_ }).sort_values('importance', ascending=False) print("=== Gini Importance ===") print(gini_importance) # Method 2: Permutation importance (more reliable) perm_importance = permutation_importance( model, X_test, y_test, n_repeats=10, random_state=42 ) perm_df = pd.DataFrame({ 'feature': X_train.columns, 'importance': perm_importance.importances_mean, 'std': perm_importance.importances_std }).sort_values('importance', ascending=False) print("\n=== Permutation Importance ===") print(perm_df) # Visualize fig, axes = plt.subplots(1, 2, figsize=(14, 5)) axes[0].barh(gini_importance['feature'], gini_importance['importance']) axes[0].set_xlabel('Gini Importance') axes[0].set_title('Feature Importance (Gini)') axes[1].barh(perm_df['feature'], perm_df['importance']) axes[1].errorbar(perm_df['importance'], perm_df['feature'], xerr=perm_df['std'], fmt='none', color='black', alpha=0.5) axes[1].set_xlabel('Permutation Importance') axes[1].set_title('Feature Importance (Permutation)') plt.tight_layout() plt.savefig('feature_importance_comparison.png', dpi=150) plt.show() ``` ------------------------------------------------------------------------ ### Choosing the Right Explainability Method | **Method** | **Global or Local?** | **Model-Agnostic?** | **Speed** | **Best For** | |----|----|----|----|----| | **SHAP** | Both | Yes | Medium-Slow | Most robust, theoretically grounded explanations | | **LIME** | Local only | Yes | Fast | Quick local explanations, rule-based output | | **Attention** | Local only | No (DL only) | Fast | Deep learning models (transformers, CNNs) | | **Counterfactual** | Local only | Yes | Medium | Actionable recommendations, fairness audits | | **Feature Importance** | Global only | No (tree models) | Very Fast | Tree-based models, quick initial analysis | **Decision flowchart:** 1. **Need global understanding?** → SHAP (global) or Feature Importance (trees only) 2. **Need local explanation for specific patient?** → SHAP (most robust) or LIME (faster) 3. **Need actionable recommendations?** → Counterfactuals 4. **Using deep learning?** → Attention mechanisms or SHAP 5. **Real-time constraint?** → LIME or Feature Importance 6. **Regulatory submission?** → Match explanation artifacts to the specific clinical oversight task and failure-mode analysis rather than assuming a single post-hoc attribution method satisfies review ------------------------------------------------------------------------ ### Evaluating Whether Explanations Work Explanations require their own evaluation against four criteria: fidelity, consistency, stability, and clinical validity. #### Explainability Evaluation Criteria **1. Fidelity:** Does the explanation accurately reflect the model’s behavior? **Test:** - Remove high-importance features → prediction should change significantly - Flip low-importance features → prediction should stay similar ``` sourceCode # Fidelity test original_pred = model.predict_proba(patient_data)[0][1] # Remove most important feature (set to mean) modified_data = patient_data.copy() modified_data['lactate'] = X_train['lactate'].mean() modified_pred = model.predict_proba(modified_data)[0][1] print(f"Original prediction: {original_pred:.2%}") print(f"After removing 'lactate': {modified_pred:.2%}") print(f"Change: {abs(original_pred - modified_pred):.2%}") if abs(original_pred - modified_pred) > 0.1: # change > 10 percentage points print("[OK] High fidelity: Explanation correctly identified important feature") else: print("[FAIL] Low fidelity: Feature removal didn't change prediction as expected") ``` **2. Consistency:** Do similar patients get similar explanations? **Test:** Generate explanations for similar patients; feature importance rankings should be similar **3. Stability:** Do explanations change substantially with small input perturbations? **Problem with LIME:** Small changes to patient data can yield very different explanations **4. Clinical validity:** Do domain experts agree the explanations make sense? **Gold standard:** Clinician review - Do identified features align with medical knowledge? - Are there unexpected/spurious correlations? ------------------------------------------------------------------------ ### Regulatory Perspectives on Explainability #### FDA AI/ML SaMD Action Plan (2021) The FDA’s AI/ML SaMD Action Plan emphasizes transparency to users and real-world performance monitoring as priorities for AI/ML-based SaMD ([FDA, 2021](https://www.fda.gov/media/145022/download)). **Common transparency elements include:** - Clear specification of intended use, input data requirements, and decision thresholds - Model limitations, known failure modes, and conditions under which clinicians should abstain from relying on outputs - Performance and calibration across demographic subgroups and clinical sites ------------------------------------------------------------------------ #### EU AI Act (2024) **Transparency obligations for high-risk AI (includes medical AI):** **Article 13 (Transparency):** - High-risk systems must be sufficiently transparent for deployers to interpret and use outputs appropriately - Instructions must describe capabilities, limitations, expected accuracy, foreseeable misuse, and the information needed for human oversight **Practical implication:** Article 13 does not prescribe SHAP, LIME, attention weights, or another universal explanation method. Documentation and user-facing information must support the specific system and oversight task. **Reference:** [EU AI Act, 2024](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689) ------------------------------------------------------------------------ ### Implementing Explainability in Production Systems #### Best Practices for Deployed AI **1. Multi-level explanations for different users:** | **User** | **Explanation Level** | **Method** | |----|----|----| | **Patient** | Why does this prediction affect me? | Simplified counterfactual (“If X, then Y”) | | **Clinician** | What factors drive this prediction? | SHAP/LIME with top 3–5 features | | **Data Scientist** | How does the model work globally? | SHAP global importance, partial dependence | | **Regulator** | Is the model fair and robust? | Subgroup analysis, fairness metrics | **2. Explanation caching:** Pre-compute SHAP values during batch prediction to avoid real-time latency **3. Explanation documentation:** Log explanations alongside predictions for audit trails **4. Explanation monitoring:** Track whether explanations remain consistent over time (a change can indicate data or model drift) ------------------------------------------------------------------------ #### Example: Explainable Sepsis Alert System ``` sourceCode ## Explainability Architecture for Sepsis Early Warning System **User-facing interface:** ┌─────────────────────────────────────────────────┐ │ SEPSIS ALERT: High Risk (82%) │ ├─────────────────────────────────────────────────┤ │ Primary Risk Factors: │ │ [CRITICAL] Lactate: 3.8 mmol/L (Critical: >2.0) │ │ [CRITICAL] Temp: 39.2°C (Elevated: >38.3) │ │ [ELEVATED] WBC: 13,500 (Elevated: >12,000) │ │ │ │ Protective Factors: │ │ [NORMAL] Blood Pressure: Normal (118/76) │ │ │ │ [View Detailed Explanation] │ │ [Similar Cases] [Dismiss Alert] │ └─────────────────────────────────────────────────┘ **Backend logging (for audit):** { "patient_id": "47291", "timestamp": "2025-10-30T14:23:11Z", "prediction": 0.82, "model_version": "sepsis_v3.2.1", "shap_values": { "lactate": 0.35, "temperature": 0.22, "wbc_count": 0.18, "systolic_bp": -0.08 }, "explanation_method": "SHAP_TreeExplainer", "explanation_fidelity_score": 0.94 } ``` ------------------------------------------------------------------------ ### Common Pitfalls and How to Avoid Them #### Pitfall 1: Confusing Correlation with Causation **Problem:** SHAP/LIME identify correlations, not causal relationships. **Example:** - Model assigns high importance to “hospital length of stay” for mortality prediction - **Interpretation error:** “Longer stays cause death” - **Reality:** Sicker patients stay longer; length of stay is a *proxy* for severity **Solution:** Always validate explanations with clinical domain knowledge ------------------------------------------------------------------------ #### Pitfall 2: Over-relying on Feature Importance **Problem:** Global feature importance hides subgroup differences. **Example:** - “Age” is most important feature globally (average across all patients) - But for young patients (\<40), “comorbidities” might be more important **Solution:** Examine SHAP dependence plots and subgroup-specific explanations ------------------------------------------------------------------------ #### Pitfall 3: Ignoring Explanation Instability **Problem:** LIME explanations can vary substantially between runs for the same patient and between similar patients. **Test:** ``` sourceCode # Generate 10 explanations for same patient (with different LIME seeds) explanations = [] for seed in range(10): explainer = lime.lime_tabular.LimeTabularExplainer( X_train.values, feature_names=X_train.columns.tolist(), mode='classification', random_state=seed ) exp = explainer.explain_instance(patient, model.predict_proba) explanations.append(exp.as_list()) # Check consistency # If feature rankings vary significantly → unstable explanations ``` **Solution:** Prefer inherently interpretable models or stability-tested attribution methods when feature-level explanations are required for auditing, and never treat post-hoc feature rankings as causal clinical mechanisms ------------------------------------------------------------------------ #### Pitfall 4: Explaining the Wrong Model **Problem:** Explanations are generated from a simplified “surrogate” model instead of the actual production model. **Example:** - Production: Complex ensemble of 50 models - Explanation: Generated from single decision tree approximation - **Risk:** Explanations do not reflect actual system behavior **Solution:** Always explain the actual deployed model (even if slower) ------------------------------------------------------------------------ ### Key Takeaways: Explainability 1. **Operational transparency beats unverified post-hoc heatmaps:** Clinicians need clear intended-use boundaries, calibration by subgroup, and validated failure modes rather than unvalidated per-case feature attributions that can induce overreliance 2. **Multiple methods, multiple purposes:** SHAP for robustness, LIME for speed, counterfactuals for action 3. **Evaluate your explanations:** Fidelity, consistency, clinical validity 4. **Regulatory trend:** Transparency and interpretable use are increasingly important, but no universal rule requires a post-hoc feature attribution for every prediction 5. **Layer explanations by user:** Patients need simple “why me?”; regulators need comprehensive validation 6. **Correlation ≠ causation:** Explanations show what the model uses, not necessarily what is clinically causal 7. **Explainability does not fix a flawed model:** If a model is biased or poorly validated, explanations can make the problems visible, which helps debugging, but do not correct them **Essential resources:** - **[Christoph Molnar, *Interpretable Machine Learning* (2025)](https://christophm.github.io/interpretable-ml-book/):** Free online book, comprehensive guide - **SHAP documentation:** https://shap.readthedocs.io/ - **LIME GitHub:** https://github.com/marcotcr/lime - **DiCE (Counterfactuals):** https://interpret.ml/DiCE/ - **Google’s Explainable AI whitepaper:** [Exploratory guide to XAI (2019)](https://storage.googleapis.com/cloud-ai-whitepapers/AI%20Explainability%20Whitepaper.pdf) ### Mechanistic Interpretability for Sequential Decision-Making SHAP and LIME explain individual predictions, but public health AI increasingly involves sequential decision-making where current actions influence future states. Reinforcement learning (RL) systems for population health management require interpretability methods that expose reasoning pathways, not just feature importance. **Case study: Medicaid care coordination.** A SARSA reinforcement learning system for Medicaid care management across two U.S. states (Virginia and Washington; 3,175 beneficiaries, 2023–2024) used a mixed-methods approach combining quantitative RL optimization with qualitative clinical validation. In counterfactual analysis, the system was estimated to reduce acute care events by 12 percentage points (NNT 8.3; 20.7% relative reduction) compared to standard practice, while also reducing race/ethnicity equalized odds disparity from 8.9% to 5.6% and gender disparity from 5.3% to 3.8% ([Basu et al., 2025](https://doi.org/10.2196/74264)). **Implications for public health AI evaluation:** - Sequential decision-making systems require interpretability methods beyond single-prediction explainers like SHAP and LIME - Mixed-methods validation (combining quantitative metrics with clinical expert review) provides stronger evidence than either approach alone - Fairness constraints can be integrated into RL optimization without large accuracy trade-offs in this study, suggesting that baseline disparities may stem from suboptimal calibration rather than fundamental accuracy-fairness tension - Tiered oversight (automated decisions for low-risk cases, human review for high-risk cases) is an emerging approach for balancing efficiency with safety ## Regulatory Evaluation Frameworks ### The Evolving Regulatory Landscape for Medical AI **Current reality:** AI evaluation is no longer only a scientific question. It is also a **regulatory requirement**. Medical AI systems face scrutiny from multiple regulatory bodies: - **FDA (United States):** Software as a Medical Device (SaMD) framework - **European Union:** AI Act (2024) classifying medical AI as “high-risk” - **UK MHRA:** Software and AI as a Medical Device framework - **Health Canada:** Medical Device Regulations for AI/ML **Key insight:** Understanding regulatory evaluation requirements is essential for deployment, not just compliance. Why Regulatory Context Matters for Evaluation Even if you are not directly developing a regulatory-approved device, understanding these frameworks helps you: 1. **Design better evaluations:** Regulatory standards define best practices 2. **Anticipate deployment requirements:** Many health systems require FDA clearance or equivalent 3. **Benchmark your work:** Compare your validation against regulatory expectations 4. **Communicate with stakeholders:** Speak the language of hospital legal/compliance teams **Cross-reference:** [Policy and Governance](../future/policy.html) covers broader policy implications; the requirements below are those specific to **evaluation**. ------------------------------------------------------------------------ ### FDA Software as a Medical Device (SaMD) Framework Software as a Medical Device (SaMD) refers to software intended for medical purposes that operates independently of hardware medical devices. **Examples:** - **SaMD:** Sepsis prediction algorithm, diabetic retinopathy screening app, radiology CAD software - **Not SaMD:** EHR system (administrative), fitness tracker (wellness, not medical diagnosis) ------------------------------------------------------------------------ ### Risk-Based Classification System The International Medical Device Regulators Forum (IMDRF) framework, linked under Resources, places SaMD in **risk categories I (lowest) to IV (highest)** according to the state of the healthcare situation and the significance of the information the software provides. This categorization is separate from FDA device classification (Class I, II, or III), which determines the premarket pathway. #### IMDRF SaMD Risk Categorization Matrix | State of healthcare | Significance of information | | | |----|----|----|----| | | Treat or diagnose | Drive clinical management | Inform clinical management | | Critical | **IV** (highest risk) | **III** | **II** | | Serious | **III** | **II** | **I** | | Non-serious | **II** | **I** | **I** (lowest risk) | **Definitions:** - **Critical:** Death or permanent impairment (e.g., ICU monitoring) - **Serious:** Long-term morbidity (e.g., cancer diagnosis) - **Non-serious:** Minor conditions (e.g., acne treatment) **Information significance:** - **Treat/Diagnose:** Directly triggers treatment decisions - **Drive clinical management:** Significant influence on treatment path - **Inform clinical management:** One input among many ------------------------------------------------------------------------ #### Evaluation Expectations by Risk Level The three tiers below are illustrative, rising with risk; they are not regulatory definitions. **Level I (Low Risk):** - **Example:** App suggesting lifestyle modifications for mild hypertension - **Requirements:** - Basic technical performance validation - User studies demonstrating safe use - Minimal clinical validation **Level II (Moderate Risk):** - **Example:** Algorithm flagging abnormal chest X-rays for radiologist review - **Requirements:** - Robust internal validation - External validation recommended - Clinical utility assessment - Usability testing - Performance monitoring plan **Level III (High Risk):** - **Example:** AI system autonomously diagnosing cancer, guiding ICU interventions - **Requirements:** - Extensive multi-site external validation - Prospective clinical studies - Randomized controlled trials (for novel interventions) - Comprehensive fairness audits - Predetermined change control plan (for adaptive algorithms) - Post-market surveillance ------------------------------------------------------------------------ ### Good Machine Learning Practice (GMLP) Principles FDA, Health Canada, and UK MHRA jointly published **10 GMLP principles** (2021) for ML-based medical devices: **1. Multi-disciplinary expertise** throughout ML lifecycle **2. Good software engineering practices** - Version control, testing, documentation **3. Clinical study participants are representative** - Training data reflects intended use population **4. Training datasets are independent of test datasets** - No data leakage **5. Selected reference datasets are based on best available methods** - Gold-standard labels (expert consensus, biopsy confirmation, etc.) **6. Model design is tailored to available data and reflects intended use** - Avoid overfitting, ensure clinical relevance **7. Focus on human-AI team performance** - Evaluate AI + clinician together, not AI in isolation **8. Testing demonstrates device performance in clinically relevant conditions** - Real-world data, relevant patient populations **9. Users are provided with clear, essential information** - Transparency about limitations, training data, performance metrics **10. Deployed models are monitored for performance** - Post-market surveillance, drift detection **Reference:** [FDA/MHRA/Health Canada, 2021 - GMLP Guiding Principles](https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles) ------------------------------------------------------------------------ ### Predetermined Change Control Plans (PCCP) **Challenge:** Traditional medical devices are static; ML models need updating. **FDA’s current framework ([December 2024 final guidance](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence)):** Predetermined Change Control Plans allow FDA-reviewed, pre-specified modifications to AI-enabled devices without separate marketing submissions for each covered update. **What can be included in a PCCP:** **1. Allowable modifications:** - Retraining on new data (within specified bounds) - Algorithm parameter adjustments - Performance improvements **2. Modification protocol:** - **Data requirements:** Minimum sample size, quality standards - **Performance thresholds:** Must maintain ≥ X sensitivity - **Validation approach:** Test set size, external validation sites **3. Update assessment:** - Performance monitoring triggers retraining - Validation results compared to pre-specified thresholds - Automated decision: deploy update or flag for review **4. Transparency and reporting:** - Change documentation - Performance reports to regulators - User notifications **Example PCCP:** ``` sourceCode ### Sepsis Prediction Model PCCP **Allowable Change:** Retrain model quarterly using new institutional data **Modification Protocol:** - Minimum 5,000 new patient encounters with ≥200 sepsis cases - Maintain AUC-ROC ≥ 0.82 (original validation: 0.85) - Sensitivity at 80% specificity must be ≥ 70% - External validation on ≥1 additional hospital required **Assessment:** - Automated performance monitoring (monthly) - Retraining triggered if AUC drops below 0.83 - New model validated on hold-out test set (20% of new data) - Deploy if all thresholds met; otherwise, flag for manual review **Reporting:** - Quarterly performance reports to FDA - User notification of model updates via EHR alert ``` ------------------------------------------------------------------------ ### EU AI Act: High-Risk Classification for Medical AI **Entered into force 1 August 2024, with obligations applying in stages.** Although the Regulation as originally adopted scheduled obligations for high-risk AI systems covered by EU harmonization legislation (Annex I, including the Medical Devices Regulation and In Vitro Diagnostic Medical Devices Regulation) for 2 August 2027, Regulation (EU) 2026/1744 amended the transition timeline so that high-risk obligations apply from 2 December 2027 for standalone Annex III systems and from 2 August 2028 for Annex I regulated medical devices. **Key classification:** Most medical AI systems are **high-risk**, requiring: **1. Risk management system:** - Continuous identification and mitigation of risks **2. Data governance:** - Training data quality assurance - Bias detection and mitigation - Data representativeness **3. Technical documentation:** - Detailed model specifications - Training procedures - Validation results **4. Transparency:** - Users informed that AI is involved - Clear information on limitations **5. Human oversight:** - Human-in-the-loop for high-stakes decisions **6. Accuracy, robustness, cybersecurity:** - Performance standards - Adversarial robustness testing - Security safeguards **7. Post-market monitoring:** - Continuous performance tracking - Incident reporting **Implications for evaluation:** - Match validation populations, sites, and time periods to the intended use - Evaluate subgroup performance and foreseeable discriminatory effects - Test robustness and cybersecurity using methods appropriate to the threat model - Establish postmarket monitoring where the applicable legal and risk framework requires it **Reference:** [EU AI Act, 2024](https://eur-lex.europa.eu/eli/reg/2024/1689/oj?locale=en) ------------------------------------------------------------------------ ### Practical Implications for Evaluation Regulatory-Aligned Evaluation Checklist Even if your system is not currently seeking regulatory approval, aligning with these standards ensures quality: **Before Development:** Determine risk classification (FDA device class, IMDRF SaMD category, or EU high-risk) Identify applicable regulatory frameworks Define evaluation requirements based on risk level **During Development:** Ensure training/test data independence (GMLP \#4) Use representative training data (GMLP \#3) Apply good software engineering (version control, testing) (GMLP \#2) Document model architecture, hyperparameters, training process **Validation Phase:** Internal validation with appropriate cross-validation Temporal validation (if time-dependent data) **External validation** when needed to support transportability to the intended population, site, or time period Fairness audit across demographic subgroups Usability testing with intended users Clinical utility assessment (not just technical performance) **Pre-Deployment:** Human-AI team evaluation (GMLP \#7) Security and robustness testing appropriate to the threat model and applicable requirements Create predetermined change control plan (if adaptive model) Develop post-market monitoring protocol **Post-Deployment:** Continuous performance monitoring (GMLP \#10) Drift detection and retraining triggers Incident reporting system Periodic re-validation (recommend annually minimum) ------------------------------------------------------------------------ ### Comparison: Research vs. Regulatory Evaluation Standards | **Aspect** | **Research Publication** | **Regulatory Approval** | |----|----|----| | **External validation** | Recommended, often skipped (6% in a 2019 review) | Expected for higher-risk devices; evidence requirements vary by pathway | | **Prospective testing** | Rare | Often required for novel high-risk devices | | **Fairness audit** | Increasingly expected | **Mandatory** (EU AI Act) | | **Post-market monitoring** | Not required | **Mandatory** | | **Clinical utility** | Recommended | Pathway-dependent; 510(k) requires substantial equivalence, not demonstrated outcome benefit | | **Documentation** | Methods section | Extensive technical documentation | | **Timeline** | Months | 1–3+ years (depending on risk level) | **Key insight:** **Regulatory standards are higher than typical research standards.** If you aim for deployment, plan for regulatory-level evaluation from the start. A 19 August 2026 *PLOS Digital Health* census of all 1,357 FDA-authorized AI/ML devices through 5 December 2025 bounds that “must demonstrate benefit” cell: only 34 (2.5%) had a registered prospective trial, 12 (0.9%) posted results, and 3 (0.2%) were evaluated on patient-centered outcomes such as mortality, morbidity, or readmissions ([Abulibdeh et al., 2026](https://doi.org/10.1371/journal.pdig.0001597)). The 510(k) pathway requires substantial equivalence to a predicate, not an NDA-style outcome trial, so clearance is not demonstrated patient benefit. That 0.2% is a public-registry attrition count, not a care endpoint; the authors note the census may undercount manufacturer-held validation and does not mean FDA has no post-market surveillance. ------------------------------------------------------------------------ ### When to Seek Regulatory Approval **You likely need FDA clearance/approval if:** - System makes **diagnostic** or **treatment recommendations** - Used for **screening** (e.g., diabetic retinopathy, cancer) - Influences **clinical decision-making** significantly - Marketed as improving health outcomes **You might NOT need approval if:** - **Administrative tools** (scheduling, billing) - **Wellness apps** (general fitness, not medical claims) - **Clinical decision support** providing information only (not recommendations) - **Gray area:** FDA discretion, often depends on risk **When in doubt:** Consult FDA’s Digital Health Center of Excellence or regulatory counsel. ------------------------------------------------------------------------ ### Resources for Regulatory Evaluation **FDA:** - [Digital Health Center of Excellence](https://www.fda.gov/medical-devices/digital-health-center-excellence) - [Software as a Medical Device Guidance](https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd) - [AI/ML-Based SaMD Action Plan](https://www.fda.gov/media/145022/download) **International:** - [IMDRF SaMD Framework](https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework-risk-categorization-141013.pdf) - [EU AI Act Official Text](https://eur-lex.europa.eu/eli/reg/2024/1689/oj?locale=en) **Academic:** - [TRIPOD+AI](https://doi.org/10.1136/bmj-2023-078378): Reporting guidance for clinical prediction model studies using regression or machine learning ([Collins et al., 2024](https://doi.org/10.1136/bmj-2023-078378)) - [PROBAST+AI](https://doi.org/10.1136/bmj-2024-082505): Quality, risk-of-bias, and applicability assessment for prediction model studies ([Moons et al., 2025](https://doi.org/10.1136/bmj-2024-082505)) - [CONSORT-AI](https://doi.org/10.1038/s41591-020-1034-x): Reporting guidelines for randomized trials evaluating AI interventions ([Liu et al., 2020](https://doi.org/10.1038/s41591-020-1034-x)) - [SPIRIT-AI](https://doi.org/10.1136/bmj.m3210): Protocol design guidelines for clinical trials involving AI ([Cruz Rivera et al., 2020](https://doi.org/10.1136/bmj.m3210)) - [DECIDE-AI](https://doi.org/10.1038/s41591-022-01772-9): Reporting guidance for early-stage live evaluation of AI decision support ([Vasey et al., 2022](https://doi.org/10.1038/s41591-022-01772-9)) - [CONSORT-EHEALTH](https://doi.org/10.2196/jmir.1923): Reporting guidelines for digital health interventions ([Eysenbach, 2011](https://doi.org/10.2196/jmir.1923)) ------------------------------------------------------------------------ ### Key Takeaways: Regulatory Evaluation 1. **Regulatory requirements are evaluation requirements:** FDA, EU standards define rigorous validation expectations 2. **Risk-based approach:** Evidence requirements should reflect intended use, the consequence of error, degree of autonomy, and difference between the evaluation setting and intended setting 3. **Applicability must be demonstrated:** Regulatory requirements vary by function and pathway; authorization does not substitute for evidence in the intended population and setting 4. **Continuous monitoring is required:** Post-market surveillance, not just pre-deployment validation 5. **Good Machine Learning Practice:** 10 principles provide practical framework for development and evaluation 6. **Predetermined change control plans:** Enable adaptive models while maintaining regulatory compliance 7. **EU AI Act raises the bar:** High-risk systems face requirements for risk management, data governance, transparency, human oversight, accuracy, robustness, cybersecurity, and postmarket monitoring. The evidence methods must fit the system and applicable conformity-assessment obligations. 8. **Plan early:** Regulatory evaluation takes longer and costs more than research validation; design for it from the start 9. **Seek expert guidance:** Regulatory pathways are complex; consult with regulatory specialists 10. **Standards improve quality:** Even if not seeking approval, regulatory frameworks represent best practices ------------------------------------------------------------------------ ## Evaluation Framework ### Complete Evaluation Checklist Use this when evaluating AI systems: AI System Evaluation Checklist **TECHNICAL PERFORMANCE** Discrimination metrics reported (AUC-ROC, sensitivity, specificity, PPV, NPV) 95% confidence intervals provided for all metrics Calibration assessed (calibration plot, Brier score, ECE) Appropriate for class imbalance (if applicable) Comparison to baseline model (e.g., logistic regression, clinical judgment) Multiple metrics reported (not just accuracy) **VALIDATION RIGOR** Internal validation performed (CV or hold-out) Temporal validation (train on past, test on future) **External validation** on independent dataset from different institution Prospective validation performed or planned Data leakage prevented (feature engineering within folds) Appropriate cross-validation for data structure (stratified, grouped, time-series) **FAIRNESS AND EQUITY** Performance stratified by demographic subgroups (race, gender, age, SES) Disparities quantified (absolute and relative differences) Calibration assessed per subgroup Training data representation documented Potential for bias explicitly discussed Mitigation strategies proposed if disparities identified **CLINICAL UTILITY** Decision curve analysis or similar utility assessment Comparison to current standard of care Clinical workflow integration considered Net benefit quantified Cost-effectiveness assessed (if applicable) Actionable outputs (not just risk scores) **TRANSPARENCY AND REPRODUCIBILITY** Model architecture and type clearly described Feature engineering documented Hyperparameters and training procedure reported Reporting guidelines followed (TRIPOD, STARD-AI) Code availability stated Data availability (with appropriate privacy protections) Conflicts of interest disclosed **IMPLEMENTATION PLANNING** Target users and use cases defined Workflow integration plan described Alert/decision threshold selection justified Plan for performance monitoring post-deployment Model updating and maintenance plan Training plan for end users Contingency plan for model failure **LIMITATIONS** Limitations clearly stated Generalizability constraints acknowledged Potential biases discussed Appropriate caveats about clinical use ------------------------------------------------------------------------ ## Reporting Guidelines ### TRIPOD: Transparent Reporting of Prediction Models [Collins et al., 2015, *BMJ*](https://www.bmj.com/content/350/bmj.g7594) - TRIPOD statement **22-item checklist for prediction model studies:** **Title and Abstract** 1. Title 2. Abstract **Introduction** 3. Background and objectives **Methods** 4. Source of data 5. Participants 6. Outcome 7. Predictors 8. Sample size 9. Missing data 10. Statistical analysis methods 11. Risk groups 12. Development versus validation **Results** 13. Participants 14. Model development 15. Model specification 16. Model performance 17. Model updating **Discussion** 18. Limitations 19. Interpretation 20. Implications **Other Information** 21. Supplementary information 22. Funding **Current guidance:** TRIPOD+AI superseded the 2015 TRIPOD checklist for clinical prediction model studies using regression or machine learning methods ([Collins et al., 2024](https://doi.org/10.1136/bmj-2023-078378)). It strengthens reporting of data sources, model development, evaluation, fairness, and open-science practices. TRIPOD+AI is a reporting guideline, not a method for rating evidence certainty or risk of bias; PROBAST+AI serves the latter appraisal role. ------------------------------------------------------------------------ ### TRIPOD-LLM: Reporting Guidelines for LLM Studies Traditional TRIPOD was designed for classical prediction models. Large language models require fundamentally different reporting standards. TRIPOD-LLM ([Gallifant et al., 2025, *Nature Medicine*](https://doi.org/10.1038/s41591-024-03425-5)) provides the first LLM-specific extension, endorsed by EQUATOR Network. **Why LLM-specific guidelines are needed:** - Training data for LLMs is often undisclosed or incompletely characterized - Performance varies dramatically with prompt wording - Contamination between training and test data is difficult to verify - Reproducibility requires specifying API versions, dates, and system prompts **Key TRIPOD-LLM additions:** | Category | New Requirements | |----|----| | **Model specification** | API version, access date, temperature/sampling parameters | | **Prompting** | Full prompt text, development process, any prompt optimization | | **Input/output** | How health data was formatted, output parsing methods | | **Reproducibility** | Whether results vary across API calls, random seed handling | | **Contamination** | Assessment of whether test data appeared in training corpus | **The LLM reporting crisis:** A systematic review of LLM chatbot health advice studies ([Huo et al., 2025, *JAMA Network Open*](https://doi.org/10.1001/jamanetworkopen.2024.57879)) examined 137 studies and found that 136 studies (99.3%) used closed-source models. Key findings: - No study sufficiently described model characteristics such as temperature and token length - 136 studies (99.3%) did not describe a prompt engineering phase - The date of model querying was reported in only 54 studies (39.4%) These reporting gaps prevent faithful reproduction of most studies. TRIPOD-LLM directly addresses these gaps. **CHART Statement (Chatbot Assessment Reporting Tool):** Complementing TRIPOD-LLM, the [CHART statement](https://doi.org/10.1001/jamanetworkopen.2025.30220) ([Huo et al., 2025, *JAMA Network Open*](https://doi.org/10.1001/jamanetworkopen.2025.30220)) provides a 12-item, 39-subitem checklist specifically for evaluating chatbot and conversational AI systems in healthcare settings. ------------------------------------------------------------------------ ### STARD-AI: Standards for Reporting Diagnostic Accuracy Using AI Extension of [STARD guidelines](https://www.equator-network.org/reporting-guidelines/stard/) for diagnostic AI. **Additional items:** - Model architecture details - Training procedure (epochs, batch size, optimization) - Validation strategy - External validation results - Subgroup analyses - Calibration assessment - Comparison to human performance (if applicable) ------------------------------------------------------------------------ ## Critical Appraisal of Published Studies ### Systematic Evaluation Framework When reading AI studies: #### 1. Study Design and Data Quality **Questions:** - Representative sample of target population? - External validation performed? - Test set truly independent? - Outcome objectively defined and consistently measured? - Potential for data leakage? **Red flags:** - No external validation - Small sample size (\<500 events) - Convenience sampling - Vague outcome definitions - Feature engineering on entire dataset before splitting ------------------------------------------------------------------------ #### 2. Model Development and Reporting **Questions:** - Multiple models compared? - Simple baseline included (logistic regression)? - Hyperparameters tuned on separate validation set? - Feature selection appropriate? - Model clearly described? **Red flags:** - No baseline comparison - Hyperparameter tuning on test set - Inadequate model description - No cross-validation ------------------------------------------------------------------------ #### 3. Performance Evaluation **Questions:** - Appropriate metrics for task? - Confidence intervals provided? - Calibration assessed? - Multiple metrics reported? - Statistical testing appropriate? **Red flags:** - Only accuracy reported (especially for imbalanced data) - No calibration assessment - No confidence intervals - Cherry-picked metrics ------------------------------------------------------------------------ #### 4. Fairness and Generalizability **Questions:** - Performance stratified by subgroups? - Diverse populations included? - Generalizability limitations discussed? - Potential biases identified? **Red flags:** - No subgroup analysis - Homogeneous study population - Claims of broad generalizability without external validation - Dismissal of fairness concerns ------------------------------------------------------------------------ #### 5. Clinical Utility **Questions:** - Clinical utility assessed (beyond accuracy)? - Compared to current practice? - Implementation considerations discussed? - Cost-effectiveness assessed? **Red flags:** - Only technical metrics - No comparison to existing approaches - No implementation discussion - Overstated clinical claims ------------------------------------------------------------------------ #### 6. Transparency and Reproducibility **Questions:** - Code and data available? - Reporting guidelines followed? - Sufficient detail to reproduce? - Limitations clearly stated? - Conflicts of interest disclosed? **Red flags:** - No code/data availability - Insufficient methodological detail - Overstated conclusions - Undisclosed industry funding ------------------------------------------------------------------------ ## Key Takeaways Essential Principles 1. **Evaluation is multidimensional**: Technical performance, clinical utility, fairness, and implementation outcomes all matter 2. **Internal validation is insufficient**: External validation on independent data is essential to assess generalizability 3. **Calibration is critical**: Predicted probabilities must be meaningful for clinical decisions, not just discriminative 4. **Assess fairness proactively**: Stratify performance by demographic subgroups; disparities invisible otherwise 5. **Clinical utility ≠ statistical performance**: A model can be statistically accurate but clinically useless without improving outcomes 6. **Prospective evaluation answers operational questions:** Live testing can assess data pipelines, workflow, timeliness, and human factors; comparative evidence is needed for causal claims about impact 7. **Common pitfalls are avoidable**: Data leakage, improper CV, threshold optimization on test set lead to overoptimistic estimates 8. **Implementation determines success**: Even well-performing models fail if workflow integration ignored 9. **Transparency enables trust**: Follow reporting guidelines (TRIPOD, STARD-AI); share code and data when possible 10. **Continuous monitoring is essential**: Model performance drifts over time; plan for ongoing evaluation and updating ------------------------------------------------------------------------ ## Check Your Understanding Test your knowledge of key evaluation concepts. Click “Show Answer” to reveal the correct response and explanation. Question 1: Cross-Validation Strategy You are building a model to predict hospital readmissions using data from 2018–2023. Which cross-validation strategy is MOST appropriate? 1. 10-fold random cross-validation 2. Leave-one-out cross-validation 3. Stratified K-fold cross-validation 4. Time-based forward-chaining cross-validation **Answer: d) Time-based forward-chaining cross-validation** **Explanation:** **Time-based (temporal) cross-validation** is essential for healthcare data with temporal dependencies: **Why temporal CV is critical:** Fold 1: Train 2018-2019 → Test 2020 Fold 2: Train 2018-2020 → Test 2021 Fold 3: Train 2018-2021 → Test 2022 Fold 4: Train 2018-2022 → Test 2023 **What this tests:** - Model performance as deployed (using past to predict future) - Robustness to temporal drift (treatment changes, policy updates) - Realistic performance estimates **Why not random K-fold (a)?** Creates data leakage: Train: [2019, 2021, 2023] Test: [2018, 2020, 2022] This uses 2023 data to predict 2018, which inflates performance. **Why not leave-one-out (b)?** - Computationally expensive - Still has temporal leakage problem - High variance in estimates **Why not stratified K-fold (c)?** - Useful for class imbalance - Still allows temporal leakage - Does not test temporal robustness **Real-world impact:** Random cross-validation can overestimate forward performance when temporal drift exists because it allows later observations to inform validation of earlier observations. **Lesson:** Healthcare data has temporal structure. Always validate the way the model will be deployed: use past data to predict the future, never the reverse. Question 2: Calibration Assessment A cancer risk model predicts 20% risk for 1,000 patients. In reality, 300 of these patients develop cancer. What does this indicate? 1. The model is well-calibrated 2. The model systematically underestimates risk 3. The model systematically overestimates risk 4. The model has good discrimination but poor calibration **Answer: b) The model systematically underestimates risk** **Explanation:** **Calibration** compares predicted probabilities to observed outcomes: **Analysis:** - **Predicted**: 20% of 1,000 patients = 200 patients expected to develop cancer - **Observed**: 300 patients actually developed cancer - **Gap**: Predicted 200, observed 300 → Underestimating risk **Calibration terminology:** - **Well-calibrated**: Predicted ≈ Observed (20% predicted → 20% observed) - **Underestimation (miscalibration-in-the-large)**: Predicted \< Observed (20% predicted → 30% observed); **this case** - **Overestimation**: Predicted \> Observed (20% predicted → 10% observed) - **Overconfidence** differs: predictions are too extreme, shown by a calibration slope below 1 **Why it matters:** ``` sourceCode # Clinical decision: Treat if risk > 25% model.predict_proba(patient) = 0.20 # Below threshold → No treatment # Reality: True risk was 0.30 # Patient should have been treated. ``` **How to detect:** 1. **Calibration plot**: Predicted vs observed by risk bin 2. **Brier score**: Mean squared error of probabilities 3. **Expected Calibration Error (ECE)**: Average absolute calibration error **Lesson:** High AUC does not guarantee calibration. When predictions inform decisions with probability thresholds, calibration is critical. Always check calibration plots, not just discrimination metrics. Question 3: External Validation Your sepsis model achieves AUC 0.88 on the internal test set. You test on external hospitals and get AUC 0.72–0.82 (varying by site). What does this variability indicate? 1. The model is overfitting 2. External sites have poor data quality 3. There is substantial site-specific heterogeneity 4. The model should not be used **Answer: c) There is substantial site-specific heterogeneity** **Explanation:** **Performance variability across sites** reveals important heterogeneity: **What varies between hospitals:** 1. **Patient populations**: - Demographics (age, race, socioeconomic status) - Disease severity (tertiary referral vs community hospital) - Comorbidity profiles 2. **Clinical practices**: - Sepsis protocols (early vs delayed antibiotics) - ICU admission criteria - Documentation practices 3. **Infrastructure**: - EHR systems (Epic vs Cerner vs homegrown) - Lab equipment (different reference ranges) - Staffing models (nurse-to-patient ratios) 4. **Data capture**: - Missing data patterns - Measurement frequency - Feature definitions **Why not overfitting (a)?** Overfitting appears as a gap between training and test performance within the same dataset. Here, internal test performance was strong (0.88); external generalization is what varies. **Why not poor data quality (b)?** Data quality could contribute, but the variation more likely reflects legitimate differences in populations and practices. **Why not unusable (d)?** An AUC of 0.72–0.82 is still useful but indicates a need for: - Site-specific calibration - Understanding what drives differences - Possibly site-specific models or adjustments **Best practice:** External validation commonly shows lower performance than internal testing. Variability across sites is normal and informative: it reveals where the model struggles and needs adaptation. Question 4: Statistical Significance vs Clinical Significance True or False: If a model improvement is statistically significant (p \< 0.05), it is clinically meaningful and should be deployed. **Answer: False** **Explanation:** **Statistical significance ≠ clinical significance**. Both are necessary but neither alone is sufficient: **Statistical significance:** - Tests whether a difference is unlikely due to chance - Depends on sample size (large N → small differences become significant) - Answers: “Is there an effect?” **Clinical significance:** - Tests whether a difference matters for patient care - Independent of sample size - Answers: “Is the effect large enough to matter?” **Example:** ``` sourceCode # New model vs baseline results = { 'baseline_auc': 0.820, 'new_model_auc': 0.825, 'difference': 0.005, 'p_value': 0.03, # Statistically significant 'sample_size': 50000 # Large sample } ``` **Analysis:** - **Statistically significant**: p=0.03 \< 0.05 - **Clinically insignificant**: an AUC improvement of 0.005 is negligible - **Why significant?** A large sample detects tiny differences - **Should deploy?** No; the gain does not justify the cost and disruption **Lesson:** Always evaluate both statistical and clinical significance. With large samples, trivial differences become statistically significant. Ask: “Is this improvement large enough to change practice?” Consider effect sizes, confidence intervals, and practical impact, not just p-values. Question 5: Confidence Intervals Two models have been evaluated: - Model A: AUC 0.85 (95% CI: 0.83–0.87) - Model B: AUC 0.86 (95% CI: 0.79–0.93) Which statement is correct? 1. Model B is definitely better because it has higher AUC 2. Model A’s AUC estimate is more precise, and Model B is not shown to be better 3. The models cannot be compared without more information 4. Model B is better if you’re willing to accept more uncertainty **Answer: b) Model A’s AUC estimate is more precise, and Model B is not shown to be better** **Explanation:** **Confidence intervals** reveal precision/uncertainty, not just point estimates: **Model A:** - AUC: 0.85 - 95% CI: 0.83–0.87 - Width: 0.04 (narrow) - **Interpretation**: The true AUC plausibly lies between 0.83 and 0.87 (precise estimate) **Model B:** - AUC: 0.86 - 95% CI: 0.79–0.93 - Width: 0.14 (wide) - **Interpretation**: The true AUC plausibly lies between 0.79 and 0.93 (imprecise estimate) **Key insight:** CIs overlap substantially (0.83–0.87 vs 0.79–0.93). One cannot conclude Model B is actually better; the difference might be due to chance. **In practice:** **Model A** is often the more practical choice: - Predictable performance for planning - Lower risk of underperformance - Easier to set appropriate thresholds - Small gain (0.01 AUC) not worth the uncertainty **Lesson:** Always report and consider confidence intervals, not just point estimates. Narrow CIs indicate a precise performance estimate. Wide CIs indicate uncertainty: performance might be much worse (or better) than point estimate suggests. **One more wrinkle:** the comparison above treats Model A and Model B as independent estimates, each with its own confidence interval. If both models were actually scored on the identical set of cases, that independence assumption is wrong, and a paired-difference analysis (using the correlation between the two models’ scores on each shared case) is the statistically correct comparison rather than checking whether two separate confidence intervals overlap. A paired analysis on the same question set is typically more powerful than the unpaired comparison shown here, meaning it could resolve a small gap that overlapping CIs alone leave undecided ([Miller, 2024, preprint](https://arxiv.org/abs/2411.00640)). Question 6: Subgroup Analysis You evaluate a diagnostic model and find: - Overall AUC: 0.84 - Men: AUC 0.88 - Women: AUC 0.78 What should you do? 1. Report only overall performance (0.84) 2. Report overall performance but note subgroup differences exist 3. Investigate why performance is lower for women and consider separate models or adjustments 4. Conclude the model is biased and should not be used **Answer: c) Investigate why performance is lower for women and consider separate models or adjustments** **Explanation:** **Subgroup performance disparities** require investigation and action, not just reporting: **Possible reasons for the difference:** 1. **Biological differences**: - Disease presents differently (atypical symptoms in women) - Different physiological reference ranges - Example: Heart attack symptoms differ by sex 2. **Data representation**: - Fewer women in training data → model learns men’s patterns better - Women may be underdiagnosed historically → labels biased 3. **Feature appropriateness**: - Features optimized for men - Missing features relevant for women - Example: Pregnancy-related factors not included 4. **Measurement bias**: - Tests/measurements less accurate for women - Different documentation patterns **Potential solutions:** 1. **Collect more data from women** (if sample size is the issue) 2. **Add sex-specific features**: ``` sourceCode # Include pregnancy status, hormonal factors features += ['pregnant', 'menopause_status', 'hormone_therapy'] ``` 3. **Stratified modeling**: ``` sourceCode # Separate models for men/women if patient.sex == 'M': prediction = model_men.predict(patient) else: prediction = model_women.predict(patient) ``` 4. **Weighted loss function**: ``` sourceCode # Penalize errors on women more heavily during training sample_weights = [2.0 if sex=='F' else 1.0 for sex in data['sex']] model.fit(X, y, sample_weight=sample_weights) ``` **Lesson:** Subgroup analysis is mandatory. When disparities are found, investigate root causes and take corrective action. Do not hide disparities in overall metrics. Question 7: LLM Evaluation Strategy You are deploying an LLM to summarize public health literature for practitioners. Which evaluation approach is MOST appropriate? 1. Calculate AUC-ROC on a test set of summaries 2. Measure perplexity (how surprised the model is by correct summaries) 3. Use BERTScore to compare generated summaries against expert-written reference summaries, plus human expert evaluation 4. Only use BLEU score (n-gram overlap with reference summaries) **Answer: c) Use BERTScore to compare generated summaries against expert-written reference summaries, plus human expert evaluation** **Why:** **Correct approach:** - **BERTScore** captures semantic similarity better than simple n-gram matching (BLEU) - **Human expert evaluation** is essential for: - Factual accuracy (automated metrics cannot verify facts) - Clinical relevance (is the right information prioritized?) - Safety (are there dangerous omissions or errors?) - **Multiple metrics** needed: BERTScore + factual accuracy + completeness + safety rating **Why other options are wrong:** - **a) AUC-ROC:** Not applicable; LLMs generate text, not binary classifications or probabilities - **b) Perplexity alone:** Measures fluency, not accuracy/relevance (fluent nonsense scores well) - **d) BLEU only:** Too limited; high BLEU does not guarantee accurate or clinically appropriate summaries **Best practice for LLM summarization:** 1. Automated metrics (BERTScore, ROUGE) for efficiency 2. Human expert review of a sample (100+ summaries) 3. Safety audit (check for hallucinations, dangerous errors) 4. Prompt robustness testing (consistency across variations) **Lesson:** LLM evaluation requires different approaches from those used for traditional ML. Automated metrics alone are insufficient; human expert evaluation is mandatory, especially for clinical applications. Question 8: Model Drift Detection You are monitoring a deployed readmission prediction model. Over 6 months, you observe: - AUC-ROC: Stable at 0.82 (was 0.83 at deployment) - Brier score: Increased from 0.15 to 0.21 - Patient age distribution: Mean shifted from 58 to 65 years - PSI for age feature: 0.28 What does this indicate, and what should you do? 1. Model is fine, AUC is stable; continue monitoring 2. Significant data drift occurred; retrain immediately 3. Concept drift occurred; model is failing and needs urgent retraining 4. Calibration degraded but discrimination stable; recalibrate or retrain **Answer: d) Calibration degraded but discrimination stable; recalibrate or retrain** **Analysis:** **What happened:** - **AUC-ROC stable:** Model can still distinguish high-risk from low-risk patients (discrimination intact) - **Brier score increased:** Predicted probabilities are inaccurate (calibration degraded) - **Age distribution shifted:** PSI = 0.28 indicates **significant data drift** (threshold: PSI \> 0.25) - **Likely cause:** Data drift (patient population aging) → calibration degrades even if model’s relative ranking ability (AUC) persists **Why other options are wrong:** - **a) Model is fine:** Incorrect; Brier score degradation and high PSI require action - **b) Data drift, retrain immediately:** Partially correct but oversimplified; recalibration could come first (faster, simpler) - **c) Concept drift:** Unlikely; AUC would be expected to degrade if the relationship between features and outcome changed, and this pattern looks like data drift affecting calibration **What to do:** **Immediate (within 1 week):** - Recalibrate model on recent data (faster than full retraining) - Test calibration on recent hold-out set - Deploy recalibrated version if performance restored **Scheduled (within 1 month):** - Full model retraining recommended (PSI \> 0.25 for critical feature) - Validate retrained model on hold-out set - Compare retrained vs. recalibrated performance - Deploy better-performing version **Monitoring:** - Track Brier score monthly (calibration early warning) - Track PSI for all critical features - Set alert: PSI \> 0.25 for ≥3 features = immediate retraining **Lesson:** Different types of drift require different interventions. Data drift may degrade calibration while preserving discrimination. Monitor multiple metrics (not just AUC) to catch drift early. Question 9: Regulatory Classification You are developing an AI system that analyzes electronic health records to flag patients at high risk for cardiovascular disease in the next 5 years. Clinicians review flagged patients and decide whether to prescribe statins. Under the IMDRF framework for software as a medical device (SaMD) risk categorization, which category fits this system? 1. Not a medical device (wellness/administrative use) 2. SaMD Category I (lowest risk) 3. SaMD Category II 4. SaMD Category III **Answer: c) SaMD Category II** **Classification reasoning:** **FDA SaMD Matrix:** - **State of healthcare situation:** Serious (cardiovascular disease can cause long-term morbidity) - **Significance of information:** Drive clinical management (significantly influences statin prescription decision) **Per the IMDRF matrix:** Serious + Drive clinical management = **Category II** **Why not other categories:** **a) Not a medical device:** WRONG - System makes **medical claims** (predicts disease risk) - **Influences clinical decisions** (statin prescription) - Fits the IMDRF definition of SaMD (software intended for a medical purpose) **b) Category I:** WRONG - CVD is not a “non-serious” condition - System does more than “inform”: it drives treatment decisions **d) Category III:** WRONG - Not a **critical** (immediately life-threatening) condition; critical examples include ICU monitoring or acute MI - Not **treating/diagnosing** directly; clinicians make the final decision - If the system **autonomously prescribed** statins → Category III **Evaluation implications for Category II:** **Expected:** - Rigorous internal validation - Clinical utility assessment (does it actually improve outcomes?) - Usability testing with clinicians - Performance monitoring plan (drift detection) **Recommended:** - External validation - A predetermined change control plan for planned model updates **More often warranted in higher categories:** - Randomized controlled trial - Extensive multi-site prospective validation **Borderline considerations:** If the system provided **lower-stakes information** (e.g., “Consider discussing lifestyle changes”) → Could be Category I If the condition were **critical/life-threatening** (e.g., predict sepsis, guide ICU ventilator settings) → Category III **Lesson:** SaMD risk category depends on both the *state of the healthcare situation* and the *significance of the information*. “Driving clinical management” for a serious condition = Category II. National regulators, including the FDA, apply their own device classes. Understanding the risk category early helps plan appropriate validation rigor. Question 10: Adversarial Robustness Testing You are evaluating a chest X-ray pneumonia detection model. Which robustness test is MOST important for real-world deployment? 1. Fast Gradient Sign Method (FGSM) adversarial attack testing 2. Natural perturbation testing (image compression, scanner variations, noise) 3. Model extraction attack testing (preventing reverse engineering) 4. Data poisoning resilience testing (can the training set be corrupted?) **Answer: b) Natural perturbation testing (image compression, scanner variations, noise)** **Reasoning:** **Real-world threat model:** - **Natural perturbations** occur constantly: Different X-ray machines, image compression algorithms, patient positioning variations, image quality differences across sites - **Likelihood:** Natural variation is expected in deployment; its frequency and magnitude depend on the sites, equipment, acquisition protocols, and populations - **Impact if not tested:** Model may fail on images from different hospitals/equipment, limiting generalizability **Why other options are less critical (though still valuable):** **a) FGSM adversarial attacks:** - **Likelihood:** Low; malicious adversarial attacks on deployed medical imaging systems are not widely documented - **Value:** Of research interest and possibly required under the EU AI Act, but not the *most* pressing real-world concern - **When important:** High-profile systems, regulatory compliance (EU AI Act) **c) Model extraction:** - **Risk:** Intellectual property theft, but does not directly harm patients - **Mitigation:** API rate limiting, access control (non-evaluation solutions) **d) Data poisoning:** - **Risk:** Rare unless using crowdsourced/untrusted training data - **Prevention:** Data provenance, quality control during training (not post-deployment testing) **Practical testing hierarchy:** **Essential (all deployed models):** 1. **Natural perturbation testing:** Compression, noise, equipment variation 2. **Out-of-distribution detection:** Flag unfamiliar images (different scanner types, anatomies) 3. **Multi-site external validation:** Real-world test of natural robustness **Recommended (high-risk/regulatory):** 4. **Adversarial attack testing:** FGSM, PGD when the threat model or applicable assurance requirements warrant it 5. **Ablation studies:** Performance with missing/corrupted inputs **Advanced (specific threats):** 6. **Data poisoning resilience:** If using federated learning or external data 7. **Model extraction prevention:** If protecting proprietary models **Example test:** ``` sourceCode # Test robustness to JPEG compression (natural perturbation) import cv2 compression_qualities = [100, 90, 70, 50, 30] for quality in compression_qualities: # Compress image _, compressed = cv2.imencode('.jpg', image, [cv2.IMWRITE_JPEG_QUALITY, quality]) compressed_img = cv2.imdecode(compressed, cv2.IMREAD_COLOR) # Test model prediction = model.predict(compressed_img) print(f"Quality {quality}: Prediction = {prediction:.3f}") # Example local review thresholds, not universal standards: # investigate when prediction changes exceed the prespecified tolerance ``` **Lesson:** Prioritize robustness testing based on real-world threat likelihood. For medical imaging, natural perturbations (equipment variation) are far more common than adversarial attacks. Test what will actually break your model in practice. Frequently Asked Questions ### What is external validation in health AI? External validation evaluates a locked model on data that differ meaningfully from development data, such as a new institution, geography, device, population, or time period. The test set should represent the intended transfer and should not be used repeatedly for model selection. External validation addresses transportability for the measured endpoint. It does not by itself establish workflow benefit or patient outcomes. ### What is calibration, and why does it matter? Calibration describes whether predicted probabilities correspond to observed frequencies in the target setting. A well-discriminating model can still systematically overestimate or underestimate risk, especially when prevalence or practice changes. Calibration should be examined overall and in relevant subgroups, with uncertainty. Recalibration may improve probabilities but does not repair a wrong target, missing predictors, or an ineffective intervention. ### How should evidence be matched to a public health AI claim? Technical performance claims require a suitable reference standard and independent test data. Transportability claims require external validation. Workflow claims require prospective use in the intended setting. Outcome-benefit claims require a comparative design measuring the relevant outcome. Economic claims require explicit costs, perspective, time horizon, and comparator. No single study design answers every question, so the claim should be narrowed to what the design and endpoint measured. ### What are vendor-evaluation red flags? Red flags include an undisclosed model version, no independent test set, performance reported only as accuracy or AUC, missing confidence intervals, absent subgroup results, an inappropriate comparator, results from a different workflow, and no monitoring or update plan. Regulatory authorization, publication, or deployment elsewhere should not substitute for local evidence. A vendor should be able to provide a connected record from the claim to the source and exact product version. ## Discussion Questions 1. **Claim-specific evidence plan:** You’ve developed a hospital-acquired infection prediction model. Which evidence would you require for technical performance, transportability, operational utility, population impact, equity, and lifecycle safety? Which studies would you conduct before deployment, and why? 2. **Fairness trade-offs:** Your sepsis model has AUC-ROC = 0.85 overall, but sensitivity is 0.90 for White patients vs. 0.75 for Black patients. What would you do? What are the trade-offs of different mitigation approaches? 3. **Calibration vs. discrimination:** Model A: AUC-ROC = 0.85, Brier score = 0.30 (poor calibration). Model B: AUC-ROC = 0.80, Brier score = 0.15 (excellent calibration). Which would you deploy, and why? 4. **External validation failure:** Your model achieves AUC-ROC = 0.82 in internal validation but 0.68 in external validation at a different hospital. What explains this? What are the next steps? 5. **Clinical utility skepticism:** Model predicts 30-day mortality with AUC-ROC = 0.88. Does this mean it’s clinically useful? What additional evaluations are needed? 6. **Prospective study design:** A hospital readmission model needs prospective evaluation. Would you choose an RCT, a stepped-wedge design, or silent mode? What are the trade-offs? 7. **Alert threshold selection:** A clinical decision support tool can alert at \>10%, \>20%, or \>30% predicted risk. How would you choose? What factors matter? 8. **Model drift:** COVID-19 forecasting model trained on 2020 data; it is now three years later, with new variants. How would you assess whether it is still valid? What should trigger retraining? ------------------------------------------------------------------------ ## Further Resources ### Books - [Steyerberg, 2019, *Clinical Prediction Models*](https://www.springer.com/gp/book/9783030163990) - Comprehensive guide to prediction modeling - [Barocas et al., 2023, *Fairness and Machine Learning*](https://fairmlbook.org/) - Free online textbook ### Essential Papers **Validation:** - [Collins et al., 2015, *BMJ* - TRIPOD guidelines](https://www.bmj.com/content/350/bmj.g7594) - [Liu et al., 2019, *Radiology* - Medical imaging AI systematic review](https://pubs.rsna.org/doi/10.1148/radiol.2020191432) - [Oakden-Rayner et al., 2020, *Proc ACM CHIL* - Hidden stratification](https://pubmed.ncbi.nlm.nih.gov/33196064/) **Fairness:** - [Obermeyer et al., 2019, *Science* - Racial bias case study](https://www.science.org/doi/10.1126/science.aax2342) - [Chouldechova, 2017, *FAT* - Impossibility theorem](https://arxiv.org/abs/1610.07524) - [Parikh et al., 2026, *Nature Health* - Epidemiology of AI](https://doi.org/10.1038/s44360-026-00188-w) **Clinical Utility:** - [Vickers & Elkin, 2006, *Medical Decision Making* - Decision curve analysis](https://journals.sagepub.com/doi/10.1177/0272989X06295361) **Implementation:** - [Proctor et al., 2011 - Implementation outcomes](https://link.springer.com/article/10.1007/s10488-010-0319-7) ### Tools **Metrics and Validation:** - [Scikit-learn](https://scikit-learn.org/stable/modules/model_evaluation.html) - Comprehensive metrics - [Scikit-survival](https://scikit-survival.readthedocs.io/) - Survival analysis metrics **Fairness:** - [Fairlearn](https://fairlearn.org/) - Microsoft fairness toolkit - [AI Fairness 360](https://github.com/Trusted-AI/AIF360) - IBM toolkit - [Aequitas](https://dssg.github.io/aequitas/) - Bias audit **Explainability:** - [SHAP](https://shap.readthedocs.io/) - Feature importance - [LIME](https://github.com/marcotcr/lime) - Local explanations Continue Reading - **[AI Deployment in Healthcare](../implementation/deployment.html)**: How to move validated models into production, monitor for drift, and manage the organizational change that determines real-world success - **[AI Safety in Healthcare](../implementation/safety.html)**: Risk frameworks, failure mode analysis, and when to pull a deployed model out of production - **[AI Ethics, Bias, and Equity](../implementation/ethics.html)**: Deeper coverage of fairness metrics, bias mitigation strategies, and equity considerations across populations ------------------------------------------------------------------------ **Next:** [AI Ethics, Bias, and Equity in Public Health →](../implementation/ethics.html) The Public Health AI Handbook is free to read and cite, with no advertising. If this chapter informed your work, [you can support the handbook](../support.html).