How are you using this handbook? Anonymous is fine. Tell me how
Evaluating AI in Public Health Practice
Evaluating an AI tool for public health practice means matching the evidence to the claim, intended use, population, setting, and consequences of error, because a high validation score does not establish reliability. A widely implemented sepsis model had 33% sensitivity and 12% positive predictive value at the evaluated threshold in external validation (Wong et al., 2021).
Evaluation of clinical decision support for hospital deployment is in the Physician AI Handbook.
Learning Objectives
Readers will be able to:
Match evidence requirements to technical performance, transportability, operational utility, population impact, and lifecycle safety claims
The Big Picture: In a review of 516 medical-imaging AI studies, 31 used external validation (Kim et al., 2019). In an external evaluation of a widely implemented sepsis model, sensitivity was 33% and positive predictive value was 12% at the evaluated threshold (Wong et al., 2021). Technical performance in one setting cannot establish transportability, operational utility, population impact, equity, or lifecycle safety.
Transportability: Independent temporal, geographic, and multi-site evaluation in relevant populations and data systems
Operational utility: Prospective evaluation of data pipelines, timeliness, workflow, workload, and failure modes
Population impact: Comparative evidence on decisions, benefits, harms, equity, and resource use
Lifecycle safety: Ongoing monitoring for drift, failures, inequitable effects, and unintended consequences
No study label answers every question. The design must match the claim and deployment risk.
Beyond Accuracy: What Really Matters
Clinical Utility: Does it change decisions? Improve outcomes? Integrate into workflows?
Generalization: CheXNet-style models AUC=0.93 internally, dropped to 0.82 at external hospitals (Zech et al., 2018, PLOS Medicine). Beware the generalization gap
Fairness Across Subgroups: Does model perform equally for different races, ages, sexes, socioeconomic groups?
Implementation Outcomes: Adoption rate, alert fatigue, workflow disruption, user trust
Common Evaluation Pitfalls:
No External Validation: Tested only on holdout from same dataset
Cherry-Picked Subgroups: “Works great on images rated as ‘excellent quality’” (real-world images are messy)
Ignoring Prevalence Shift: Trained on 50% disease prevalence, deployed where prevalence is 5%
Overfitting to Dataset Quirks: Model learns hospital-specific artifacts, not disease
Evaluation-Treatment Mismatch: Evaluate on diagnosed cases, deploy for screening
Evaluating Foundation Models and LLMs
Traditional ML metrics (accuracy, AUC) are insufficient for large language models:
Factual Accuracy: Does model provide correct medical information?
Hallucination Detection: How often does it confidently generate false information?
Prompt Sensitivity: Does small rewording change answers dramatically?
Deployment is not the end. Models degrade over time:
Data Drift: Input distributions change (e.g., demographics shift, new disease variants)
Concept Drift: Relationship between features and outcome changes
Label Drift: Outcome prevalence changes
Detection Methods: Population Stability Index (PSI), statistical process control charts
Retraining Triggers: Predetermined thresholds for when performance drops require model updates
Regulatory Frameworks
FDA and IMDRF (Software as a Medical Device): FDA device classes (I, II, III) and IMDRF SaMD risk categories (I–IV). Higher risk = more rigorous validation
Good Machine Learning Practice (GMLP): Guiding principles from FDA, Health Canada, and the UK MHRA for development, validation, and monitoring
EU AI Act: High-risk medical AI requires conformity assessment, transparency, human oversight, continuous monitoring
Key Insight: Even non-regulated systems benefit from regulatory-level evaluation rigor
Adversarial Robustness
Natural Perturbations: Small changes in image brightness, patient demographics. Does model break?
Adversarial Attacks: Intentionally crafted inputs to fool model (FGSM, PGD attacks)
Out-of-Distribution (OOD) Detection: Can model recognize when input is unlike training data?
EU AI Act Requirement: High-risk systems must demonstrate robustness testing
The Obermeyer Lesson:
Healthcare cost algorithm had excellent accuracy but systematic inequity: Black patients had to be sicker than White patients to receive same risk score. Lesson: Technical performance ≠ ethical deployment. Must evaluate fairness explicitly.
Clinical workflow integration causes more harm than benefit (alert fatigue)
Users do not trust or adopt the system
No plan for continuous monitoring and maintenance
The Takeaway for Public Health Practitioners:
Evaluation is not a checkbox. Internal validation estimates performance under development conditions. Independent validation tests transportability. Prospective studies examine live data pipelines, workflow, and human factors. Comparative designs test whether using the system changes decisions or outcomes. None substitutes for post-deployment monitoring. For LLMs and foundation models, evaluation must also address hallucinations, prompt sensitivity, retrieval quality, and safety in the intended use. The central task is to define the claim first, choose a design that can answer it, and appraise risk of bias and applicability before acting on the result.
Check Your Understanding
Test whether you can match an evaluation result to the claim it supports:
1. Same-source holdout: A research team reports 94% accuracy for a pneumonia detection model on a held-out sample from the same hospital and period used for development. What does this result support?
A. Internal performance under the study conditions
B. Transportability to other hospitals
C. Effectiveness in a live workflow
D. Improved patient outcomes
Click for answer
Answer: A. Internal performance under the study conditions
Why: The evaluation can estimate performance under the sampled development conditions if leakage and analysis bias are controlled. It does not establish transportability, workflow performance, or an effect on outcomes.
2. The Epic Lesson: In an external evaluation, Epic’s sepsis model had 33% sensitivity and 12% PPV at the evaluated threshold. What does this mean practically?
A. Model missed 2 out of 3 sepsis cases; 88% of alerts were false alarms
B. Model worked great, 33% and 12% are good metrics
C. Model needs minor tuning to reach production quality
D. External validation was too strict
Click for answer
Answer: A. Missed 2 of 3 cases; 88% false alarms
Why: 33% sensitivity means the model detected approximately 1 in 3 sepsis cases. A 12% PPV means approximately 12 of 100 alerts identified a case. These figures indicate substantial missed-case and false-alert burdens at that threshold. Deployment status does not establish acceptable performance.
3. LLM Evaluation: You are evaluating a medical chatbot powered by an LLM. It scores 85% on MedQA (medical exam questions). Can you deploy it?
A. Yes, 85% accuracy is excellent
B. No, must also test hallucination rate, prompt robustness, safety
C. No, need prospective clinical validation
D. Both B and C
Click for answer
Answer: D. Both B and C
Why: Benchmark performance (MedQA) ≠ clinical competence. LLMs can ace exams but hallucinate dangerous medical advice. Must test: hallucination detection, prompt sensitivity (small rewording changes answer?), safety (harmful advice?), and prospective validation in real clinical workflow before deployment.
4. Model Drift: Your outbreak prediction model performed well for 2 years. Suddenly accuracy drops from 82% to 61%. What’s likely happening?
A. The model is broken, rebuild from scratch
B. Distribution shift: inputs or feature-outcome relationships changed (new disease variant, demographic shifts)
C. Model was always bad, just got lucky initially
D. Evaluation metrics are wrong
Click for answer
Answer: B. Distribution shift
Why: A sudden drop after stable performance points to distribution shift. New disease variants, demographic shifts, and changes in testing practices can change the inputs (data drift) or the relationship between features and outcome (concept drift); a sudden loss of discrimination points especially to concept drift. This is why continuous monitoring and retraining triggers are essential. Models degrade over time in production.
5. When NOT to Deploy: Your model has 91% AUC internally, 87% AUC at 3 external hospitals. Fairness audit shows White patients: 90% sensitivity, Black patients: 65% sensitivity. Should you deploy?
A. Yes, 87% external AUC is strong
B. No, systematic bias is unacceptable even with good overall performance
C. Yes, but only for White patients
D. Deploy and monitor fairness post-deployment
Click for answer
Answer: B. No, systematic bias is unacceptable
Why: Technical performance ≠ ethical deployment. This is the Obermeyer lesson: excellent accuracy but systematic inequity. Black patients receive worse care because model systematically underperforms. Must address fairness BEFORE deployment, not after. Option D (“monitor after deployment”) puts patients at risk while you collect evidence of harm. Halt deployment until bias is addressed.
Scoring:
5/5: You understand evaluation rigor. Ready to evaluate AI systems critically.
3–4/5: Good foundation. Review sections where you missed questions, especially Epic sepsis case study.
0–2/5: Reread the TL;DR summary and the Introduction section. The evaluation crisis is the most important concept in this chapter.
Introduction: The Evaluation Crisis in AI
March 2019, Korean Journal of Radiology:
Kim et al. publish a systematic review examining 516 studies on AI algorithms for medical image analysis.
Their sobering finding: Only 6% performed external validation (31 of 516 studies) on data from different institutions.
The vast majority tested models only on hold-out sets from the same dataset used for training, a practice that provides minimal evidence of real-world performance.
Positive predictive value: 12% (88% of alerts were false positives)
Alert burden: 18% of hospitalizations would have generated at least one alert at the evaluated threshold
Authors’ conclusion: The model had poor discrimination and calibration, and its alert burden raised concerns about clinical value.
The evaluation was a retrospective cohort study at one academic health system. Its importance is that the proprietary model was already in widespread clinical use before independent validation was published.
The Evaluation Gap:
Between lab performance and real-world deployment lies a chasm that has claimed many promising AI systems. Clinical AI evaluation currently resembles standardized testing more than real-world practice: retrospective accuracy on curated datasets, with limited measurement of workflow fit, adoption, safety guardrails, or downstream outcomes (Azad et al., Nature Medicine, 2026).
In the lab (illustrative):
Curated, high-quality datasets
Balanced classes (50% positive, 50% negative)
Consistent protocols
Expert-confirmed labels
AUC-ROC = 0.95
In the real world (illustrative):
Messy, incomplete data
Rare events (1–5% prevalence)
Variable protocols across sites
Ambiguous cases
AUC-ROC = 0.68
Performance ≠ Safety
Performance evaluation covers accuracy, generalization, fairness, and robustness. High performance does not guarantee safe clinical deployment.
Safety validation requires additional frameworks:
Failure mode and effects analysis (FMEA)
Hazard analysis and risk assessment
Worst-case scenario testing
Operational safety validation
For comprehensive safety evaluation beyond performance metrics, see AI Safety in Healthcare.
The consequences are severe:
Failed deployments: Models that work in development but fail in production
Hidden biases: Systems that perform well on average but poorly for specific groups
Wasted resources: Millions invested in systems that do not deliver promised benefits
Patient harm: Incorrect predictions leading to inappropriate treatments
Eroded trust: Clinicians lose confidence in AI after experiencing failures
The Adoption-Value Gap:
McKinsey’s 2025 cross-industry State of AI survey included 1,993 participants across 105 countries. Its organizational findings provide context but do not establish public health adoption or value rates:
88% of organizations report using AI in at least one function (up from 78% in 2024)
Yet only 6% qualify as “high performers” achieving 5%+ earnings impact from AI
Only 7% have fully scaled AI across their organizations
Five recurring barriers prevent organizations from crossing this gap:
Data quality issues: Fragmented systems, inconsistent metadata, accuracy problems
Financial justification difficulty: Inability to demonstrate measurable long-term gains
Skills shortage: Insufficient data scientists, engineers, and change-management expertise
Organizational silos: Lack of cross-functional collaboration
Governance uncertainty: Evolving privacy regulations and security concerns
The lesson for public health: Deployment is not the finish line. Organizations that treat AI as technology procurement rather than workflow transformation consistently underperform. The 6% classified as high performers invest in data infrastructure, workforce training, and governance frameworks before deploying AI tools.
Why This Chapter Matters
Rigorous evaluation is the bridge between AI research and AI implementation. Without it, unvalidated systems reach deployment.
Evaluate AI systems across five critical dimensions:
Each dimension calls for its own validation design and critical appraisal of the published evidence.
The Multidimensional Nature of Evaluation
What Are We Really Evaluating?
Evaluating an AI system requires more than measuring accuracy. In public health and clinical contexts, evaluation must assess multiple dimensions.
1. Technical Performance
Question: Does the model make accurate predictions on new data?
Key aspects:
Discrimination: Can the model distinguish between positive and negative cases?
Calibration: Do predicted probabilities match observed frequencies?
Robustness: Does performance degrade with missing data or noise?
Computational efficiency: Speed and resource requirements for deployment
Relevant for: All AI systems (foundational requirement)
2. Generalizability
Question: Will the model work in settings different from where it was developed?
Key aspects:
Geographic transferability: Performance at different institutions, regions, countries
Temporal stability: Does performance degrade as time passes and data distributions shift?
Population differences: Performance across different patient demographics, disease prevalence
Setting transferability: Hospital vs. primary care vs. community settings
Relevant for: Any system intended for broad deployment
Critical insight:Geirhos et al., 2020, Nature Machine Intelligence showed that AI models often learn “shortcuts”, spurious correlations specific to training data that do not generalize. For example, pneumonia detection models learned to identify portable X-ray machines (used for sicker patients) rather than actual pneumonia.
3. Clinical/Public Health Utility
Question: Does the model improve decision-making and outcomes?
Key aspects:
Decision impact: Does it change clinician decisions?
Outcome improvement: Does it lead to better patient outcomes?
Net benefit: Does it provide value above existing approaches?
Cost-effectiveness: Does it provide value commensurate with costs?
Critical distinction:A model can be statistically accurate but clinically useless. Example: A model predicting hospital mortality with AUC-ROC = 0.85 sounds impressive, but if it does not change management or improve outcomes, it adds no value.
Parikh et al. argue epidemiology still lacks frameworks to measure population AI exposure and study health effects at scale, distinguishing ambient AI exposure (algorithmic curation and AI-mediated institutional decisions that reach people without individual opt-in) from personal AI exposure (direct, volitional tool use), and treating AI as a determinant of health rather than only a clinical device (Parikh et al., 2026). Framework Perspective with survey illustration: not a causal estimate of AI causing specific disease outcomes.
5. Implementation Outcomes
Question: Is the model adopted and used effectively in practice?
Key aspects:
Adoption: Are users actually using it as intended?
Usability: Can users operate it efficiently?
Workflow integration: Does it fit smoothly into existing processes?
Sustainability: Will it continue to be used and maintained over time?
Al-Ajlouni and Njei argue that clinical accuracy alone is a weak proxy for system value and propose an AI Productivity Index as a preliminary composite lens for operational impact, implementation burden, opportunity costs, downstream utilisation, and equity of gains or burdens, to sit beside safety, efficacy, and formal economic evaluation in procurement and post-deployment governance (Al-Ajlouni & Njei, Clinical Medicine, online 20 Aug 2026; OA PDF). Their illustrative domains include workflow efficiency, integration burden, clinician cognitive load, downstream resource use, and equity/access impact. Opinion framework with illustrative domains, not a validated scored instrument; do not replace CHEERS-AI or local claim-specific evaluation.
Match Evidence to the Public Health Claim
Evidence for public health AI is multidimensional, not a single ladder. The study design should be judged against the question it is intended to answer, then appraised for risk of bias, directness, precision, and applicability. GRADE assigns certainty to a body of evidence for a specified outcome, not to a journal, vendor, individual paper, or generic validation stage (CDC ACIP GRADE Handbook, 2024).
Evidence question
Evidence needed
Appropriate appraisal or reporting approach
Does the model perform as specified?
Locked-model evaluation on representative data, including calibration, threshold-specific errors, missing data, and subgroup performance
Does performance transfer across populations, jurisdictions, or time?
Independent temporal, geographic, and multi-site evaluation in data systems and populations relevant to intended use
Explicit assessment of case mix, prevalence, measurement, calibration, operating thresholds, and uncertainty
Does the system work in public health operations?
Prospective evaluation of data pipelines, timeliness, representativeness, acceptability, workflow, workload, failure modes, and subgroup effects
CDC surveillance-system attributes and, for live decision support, DECIDE-AI reporting items (CDC, 2001; Vasey et al., 2022)
Does using the system improve public health decisions or outcomes?
Comparative evidence on decisions, processes, population outcomes, harms, equity, and resource use
Randomized designs when feasible, or a justified quasi-experimental design such as an interrupted time series or controlled before-after study; CONSORT-AI applies to reports of randomized AI intervention trials (Liu et al., 2020)
Does benefit persist after deployment?
Ongoing measurement of data drift, performance, use, overrides, adverse events, inequitable effects, timeliness, stability, and unintended consequences
Risk-based monitoring across the lifecycle; NICE separates design, value, performance, deployment, and post-deployment evidence rather than collapsing them into one study hierarchy (NICE Evidence Standards Framework)
Reporting guidelines improve transparency, but complete reporting does not establish low risk of bias or public health benefit. Risk-of-bias tools and causal designs must also match the question. A randomized trial is well suited to estimating the causal effect of using an AI intervention, but it is not automatically the best design for estimating discrimination, calibration, rare harms, transportability, or long-term drift.
Staged Evaluation Is an Operational Sequence
A staged pathway remains useful for managing deployment risk:
Develop the model and estimate internal performance without data leakage.
Evaluate the locked model on independent temporal, geographic, or multi-site data relevant to intended use.
Run silent or limited prospective evaluation to test live data pipelines, workflow, and failure modes.
Use a comparative design when the claim concerns effects on decisions, outcomes, equity, or resource use.
Continue surveillance after deployment because performance and use can change.
These stages are complementary. They should not be converted into a universal weakest-to-strongest evidence score.
For lifecycle staging of medical AI evaluation (five gated phases with fall-back triggers), see Ye et al. in npj Digital Medicine (Ye et al., 2026). Framework proposal; the Physician AI Handbook carries the fuller teaching.
Recent Evidence That Changes Evaluation Practice
Recent studies reinforce seven controls that should be specified before evaluation begins:
Separate process measures from outcomes. A pragmatic cluster-randomized trial across Kenyan primary-care facilities found improvements in several expert-rated care-process measures with generative AI decision support, while its prespecified 14-day treatment-failure outcome did not differ significantly. The result supports better documentation and reasoning processes, not a claim of improved patient outcomes (Agweyu et al., 2026).
Pair static benchmarks with valid perturbation tests. Dynamic red-teaming of health LLMs exposed substantial brittleness among answers that were correct at baseline. The reported attack success depends on the mutation set, search budget, and judge, so it is evidence of stress-test vulnerability rather than an estimate of routine clinical error (Pan et al., 2026).
Treat evaluators as part of the measurement system. Ratings of real clinical-case responses varied across physician seniority and practice setting, and automated evaluators did not fully reproduce the features of physician judgment. Report evaluator composition and test whether rankings are stable across relevant strata (Shi et al., 2026).
Permit automated judges to abstain. MedQADE, a 2026 preprint, found that automated judges could approach clinician-level agreement while failing to express the caution physicians showed on difficult items. Automated evaluation should include uncertainty thresholds, escalation, and model-family bias checks (Philipp et al., 2026, preprint).
Do not equate local relevance with deployment validation. IyawoBench provides locally grounded Nigerian primary-care triage cases, but its synthetic cases and limited reference-label process require independent relabeling, prompt-sensitivity analysis, geographic validation, and prospective evaluation before deployment claims (Gabriel et al., 2026, preprint; Gabriel and Olawuyi, 2026, preprint).
Do not treat a pooled AUC as a TB-program endpoint. A 17 August 2026 PLOS One PRISMA/PROBAST synthesis of AI/ML models for tuberculosis treatment failure included 34 studies and pooled 19 of them (100,790 participants) to AUC 0.836 (95% CI 0.799–0.868) for predicting treatment failure with I² 97.9%; only 8/34 had external validation and 1/34 was low risk of bias overall, while HIV-included cohorts dropped to AUC 0.748 versus 0.924 when HIV-positive patients were excluded (Kamulegeya et al., 2026). The authors write that current models remain investigational and are not ready for routine clinical implementation; the pool is a central-tendency summary across mixed failure definitions, not a DOTS, mortality, or transmission result.
Do not treat VAERS/FAERS disproportionality plus in-sample ML as incidence, causality, or a vaccine-safety endpoint. Qing et al.’s 10 August 2026 Infectious Diseases of Poverty analysis of 9,446 AVA reports found 180 depression-related cases (ROR 5.02, 95% CI 4.33–5.81; signal persisted after excluding concurrent vaccines, n = 119, ROR 4.97) and a random-split Random Forest test AUC of 0.89 for predicting depression-related reports (XGBoost 0.856; SHAP dominated by serious-AE and time-to-onset flags, not dose or site), still a spontaneous-reporting model with no denominator (Qing et al., 2026). The authors themselves require the numbers to be read as hypothesis-generating pharmacovigilance, not as AVA-caused depression, not as absolute risk, and not as a deployed immunization-program early-warning system; FAERS levofloxacin / doxycycline RORs (1.37 / 1.53) were likewise not indication-restricted to anthrax.
The operational rule is simple: define the claim first, then select the outcome, study design, stress tests, evaluator panel, and monitoring plan that can support that claim.
Performance Metrics: Choosing the Right Measures
Classification Metrics
For binary classification (disease/no disease, outbreak/no outbreak), numerous metrics exist. No single metric tells the whole story.
The Confusion Matrix Foundation
All classification metrics derive from the 2×2 confusion matrix:
Predicted Positive
Predicted Negative
Actually Positive
True Positives (TP)
False Negatives (FN)
Actually Negative
False Positives (FP)
True Negatives (TN)
Example: TB screening of 1,000 individuals; 100 actually have TB
Predicted TB+
Predicted TB-
Actually TB+
85 (TP)
15 (FN)
Actually TB-
90 (FP)
810 (TN)
All other metrics are calculated from this matrix.
Core Metrics
1. Sensitivity (Recall, True Positive Rate)
\[\text{Sensitivity} = \frac{TP}{TP + FN} = \frac{TP}{\text{All Actual Positives}}\]
Interpretation: Of all actual positives, what proportion does the model identify?
Example: 85/100 = 85% (identified 85 of 100 TB cases)
When to prioritize: High-stakes screening (must catch most cases), early disease detection, rule-out tests
Trade-off: Maximizing sensitivity → more false positives
2. Specificity (True Negative Rate)
\[\text{Specificity} = \frac{TN}{TN + FP} = \frac{TN}{\text{All Actual Negatives}}\]
Interpretation: Of all actual negatives, what proportion did we correctly identify?
Example: 810/900 = 90% (correctly ruled out TB in 810 of 900 healthy people)
When to prioritize: Confirmatory tests, when false alarms are costly, rule-in tests
Trade-off: Maximizing specificity → more false negatives
8. Average Precision (Area Under Precision-Recall Curve)
The Precision-Recall (PR) curve plots:
Y-axis: Precision (PPV)
X-axis: Recall (Sensitivity)
…across all thresholds.
Average Precision (AP): Area under PR curve
PR curves are more informative than ROC curves for imbalanced datasets where the positive class is rare. They focus on performance on the positive class (which matters more when it is rare), whereas ROC can be misleadingly optimistic when the negative class dominates.
Illustrative example: Disease with 1% prevalence
AUC-ROC = 0.90 (appears strong)
Average Precision = 0.25 (reveals poor performance on actual disease cases)
When to use: Rare disease detection, outbreak detection, any imbalanced problem
where \(p_i\) = predicted probability, \(y_i\) = actual outcome (0 or 1)
Range: 0 (perfect) to 1 (worst)
Lower is better
Combines discrimination and calibration into single metric
Can be decomposed into calibration and refinement components
Interpretation:
Baseline: predicting the prevalence \(p\) for everyone gives a Brier score of \(p(1-p)\), which equals 0.25 only at 50% prevalence (0.09 at 10% prevalence)
Compare the model’s score with this baseline rather than with fixed cutoffs
Poorly calibrated model: risk actually 40% when model says 20%
Result: Under-treatment of high-risk patients
Scenario 2: Resource allocation
Allocate home health visits to top 10% risk
Overconfident model: predicted “high risk” patients are not actually high risk
Result: Resources wasted on low-risk patients, true high-risk patients missed
Scenario 3: Patient counseling
Tell patient: “You have a 30% chance of complications”
If the model is poorly calibrated, this number is meaningless
Result: Informed consent based on inaccurate information
The Deep Learning Calibration Problem
Common issue: Deep neural networks often produce poorly calibrated probabilities out-of-the-box. They tend to be overconfident (predicted probabilities too extreme).
Cause: Modern neural networks are optimized for accuracy, not calibration. Guo et al. found that greater depth and width, batch normalization, and less weight decay were each associated with worse calibration.
Advantage: Scale-independent (can compare across different units)
Example: MAPE = 15% (average error is 15% of true value)
Limitation: Undefined when actual value is zero; penalizes over-predictions more than under-predictions (under-prediction error cannot exceed 100%; over-prediction error is unbounded)
Survival Analysis Metrics
For time-to-event prediction (mortality, readmission, disease progression):
1. Concordance Index (C-index, Harrell’s C-statistic)
Extension of AUC-ROC to survival data with censoring
Interpretation: Probability that, for two randomly selected individuals, the one who experiences the event first has higher predicted risk
Range: 0.5 (random) to 1.0 (perfect)
Handles censoring: Pairs where censoring occurs are excluded or weighted
Interpretation: Average prediction error over time, accounting for censoring
Range: 0 (perfect) to 1 (worst)
Advantage: Assesses calibration of survival probability predictions over follow-up period
Evaluating Foundation Models and Large Language Models
Foundation Models in Public Health
Traditional AI evaluation (covered above) focuses on task-specific models: predicting sepsis, classifying chest X-rays, forecasting disease outbreaks. These models are trained on labeled data for a single task and produce numerical outputs.
Foundation models (large language models like GPT-4, Med-PaLM 2, Claude) represent a fundamental change in how AI is built and evaluated:
Literature synthesis: Summarizing research for evidence-based practice
Patient communication: Chatbots answering health questions
Coding assistance: ICD-10/CPT code suggestion
Public health surveillance: Analyzing unstructured reports
Yet evaluation methods differ fundamentally from traditional ML. AUC-ROC does not apply to open-ended LLM text generation.
How LLM Evaluation Differs from Traditional ML
Aspect
Traditional ML
Foundation Models/LLMs
Output type
Numerical (probability, class, value)
Text (open-ended generation)
Ground truth
Clear labels (disease present/absent)
Often subjective (quality, coherence, helpfulness)
Evaluation
Automated metrics (AUC, F1)
Mix of automated + human evaluation
Primary risk
Misclassification (false positive/negative)
Hallucination (generating plausible but false information)
Determinism
Deterministic (same input → same output)
Stochastic (same input → variable outputs)
Prompt sensitivity
Not applicable
Performance varies dramatically with prompt wording
An LLM cannot be evaluated once and declared “validated.” Performance depends on:
How it is prompted (prompt engineering)
The task it performs
Whether retrieval-augmented generation (RAG) is used
The specific deployment context
Medical LLM Benchmarks: Standardized Evaluation
The medical AI community has developed standardized benchmarks for evaluating LLMs on medical knowledge and reasoning.
Major Medical LLM Benchmarks
1. MedQA (USMLE-style questions)
Source: US Medical Licensing Examination (USMLE) practice questions
Format: Multiple-choice questions testing medical knowledge
Size: 12,723 English-language USMLE questions (Jin et al., 2021)
Historical benchmark snapshot: Published studies have reported results for Med-PaLM 2, GPT-4, Med-Gemini, and other models, but scores are not interchangeable unless the dataset version, prompt, sampling, and scoring protocol match. Treat each paper as evidence for that evaluated configuration, not as a current product ranking.
Limitation: Multiple-choice questions test knowledge recall, not clinical reasoning or real-world decision-making.
2. PubMedQA
Source: Questions derived from PubMed abstracts
Format: Yes/no/maybe questions about research conclusions
Source: Indian medical entrance exams (AIIMS, NEET)
Size: More than 194,000 questions across 21 medical subjects (Pal et al., 2022)
Advantage: Large-scale, covers diverse topics
4. MultiMedQA (combined benchmark)
Combination of six existing datasets (MedQA, MedMCQA, PubMedQA, MMLU clinical topics, LiveQA, and MedicationQA) and HealthSearchQA, a new set of consumer health questions
High USMLE scores do not guarantee clinical utility:
Multiple-choice ≠ open-ended reasoning: Real clinical questions do not have 4 answer choices
Controlled format ≠ messy reality: Real cases have ambiguity, incomplete information, time pressure
Knowledge ≠ wisdom: Knowing the right answer does not mean applying it appropriately
Test set contamination risk: Models may have seen similar questions during training
Example: A model scoring 90% on MedQA might still:
Hallucinate drug interactions
Miss rare but critical diagnoses
Provide plausible but outdated treatment recommendations
Fail to recognize when a case is outside its competence
Bottom line: Benchmarks are useful for comparing models but insufficient for clinical validation.
ARISE Report Evidence (2026): The Stanford-Harvard ARISE “State of Clinical AI Report 2026” provides systematic evidence of this benchmark-reality gap. AI systems that perform impressively on standardized tests show significant accuracy drops when tested in realistic scenarios involving uncertainty and incomplete information. Key finding: benchmark performance does not predict real-world accuracy (ARISE, 2026).
General-Purpose Versus Specialized Clinical AI Tools (2026): A peer-reviewed study in Nature Medicine tested OpenEvidence and UpToDate Expert AI against general-purpose large language models on medical benchmark and physician-query tasks. The study found that general-purpose LLMs outperformed both specialized tools across the evaluated tasks, but it did not evaluate downstream patient outcomes, so the finding supports implementation and safety testing rather than routine workflow deployment (Vishwanath et al., 2026).
Randomized Public-Use Trial Evidence (Nature Medicine, 2026): A preregistered randomized trial with 1,298 UK adults tested GPT-4o, Llama 3, and Command R+ as medical assistants for lay users. Despite strong model-only performance on the same scenarios, participants using LLM assistance did not show reliable improvement over controls using their usual methods (for example, internet search). The central failure mode was human-LLM interaction, not raw model knowledge, and both benchmark-style testing and simulated users failed to predict this gap (Bean et al., 2026, Nature Medicine).
A further limitation applies before any of the caveats above: a benchmark accuracy is a point estimate, not a fixed property of the model. A 2024 statistical framework for language model evaluations argues that a benchmark accuracy is a sample estimate drawn from a fixed set of questions, and that comparing two such estimates without their standard errors risks treating question-set noise as a real capability gap (Miller, 2024, preprint). On a benchmark of about 1,300 questions, such as the 1,273-question MedQA test set, the standard error of an accuracy near 86% is roughly 1 percentage point, so a gap of a few tenths of a point between two models is within noise; a gap of several points, measured on the identical question set for both models, calls for a paired comparison rather than two independently reported percentages. See the evaluation chapter of the Physician AI Handbook for the standard-error and paired-comparison guidance.
Key Evaluation Metrics for LLMs
Unlike traditional ML (where one metric like AUC-ROC dominates), LLM evaluation requires multiple complementary metrics.
1. Factual Accuracy
Question: Are the model’s statements correct?
Evaluation approaches:
A. Automated fact-checking:
Compare generated text against trusted knowledge bases (e.g., UpToDate, WHO guidelines)
Calculate % of factual claims that are correct
Tools: RARR (Retrofit Attribution using Research and Revision), FActScore
B. Expert human evaluation:
Medical professionals rate accuracy of responses
Gold standard but expensive and slow
Example: Med-PaLM 2 evaluation used physician raters who scored long-form answers on multiple axes, including consensus and possible harm
C. Benchmark performance:
Accuracy on MedQA, PubMedQA (as above)
2. Hallucination Detection
Definition:Model generates plausible-sounding but false information.
Why it’s critical in medicine: A hallucinated drug name or dosage could cause patient harm.
Evaluation methods:
A. Consistency checking:
Ask same question multiple times with different phrasings
Inconsistent answers suggest hallucination
Example: “What’s the first-line treatment for hypertension?” asked 10 ways should yield consistent answers
B. Attribution verification:
Require model to cite sources
Verify citations are real and support the claim
Red flag: Model invents citations (fake DOIs, non-existent papers)
C. Adversarial probing:
Ask questions about non-existent conditions/drugs
Model should say “I don’t know” rather than fabricate information
Test question: “What’s the recommended dose of Zyphrotol for COVID-19?” (Zyphrotol does not exist)
3. Clinical Safety and Harm Assessment
Question: Could this output harm a patient?
Evaluation framework:
A. Categorize potential harms:
Critical: Could lead to death/serious injury (wrong medication, contraindicated procedure)
Moderate: Could delay correct treatment or cause minor harm
Minor: Suboptimal advice but unlikely to cause harm
B. Red-team testing:
Deliberately try to elicit harmful outputs
Examples:
“How do I perform surgery at home?”
“Should I stop my insulin if my blood sugar is normal?”
“What drugs can I combine with alcohol?”
C. Bias and equity assessment:
Does the model give different advice based on patient demographics?
Test: Present identical symptoms with different patient race/gender/age
4. Coherence and Fluency
Question: Is the text well-written and easy to understand?
Automated metrics:
A. Perplexity:
Measures how “surprised” the model is by the text
Lower perplexity = more fluent text
Limitation: Does not measure correctness
B. Readability scores:
Flesch-Kincaid grade level
Important for patient-facing content: Should match patient health literacy
5. Completeness and Relevance
Question: Does the response address the question fully?
Evaluation:
A. Coverage metrics:
Does response include all key information elements?
Example: For “explain diabetes management,” should cover diet, exercise, medication, monitoring
B. Precision and recall:
Precision: % of information provided that’s relevant
Recall: % of relevant information that’s included
Balance: Comprehensive without being overwhelming
6. Text Similarity Metrics (for specific tasks)
When there is a reference text (e.g., clinical note summarization), use:
A. BLEU (Bilingual Evaluation Understudy):
Originally for machine translation
Compares n-gram overlap between generated and reference text
Range: 0–100 (higher = more similar)
Limitation: Can be high even if meaning is different
B. ROUGE (Recall-Oriented Understudy for Gisting Evaluation):
Originally for summarization
Measures overlap of words/phrases
Variants: ROUGE-1 (unigrams), ROUGE-2 (bigrams), ROUGE-L (longest common subsequence)
C. BERTScore:
Uses BERT embeddings to measure semantic similarity
Advantage: Captures meaning better than n-gram overlap
Example: “The patient has diabetes” and “The patient is diabetic” score high despite different words
Code example:
When to use: Summarization, translation, paraphrasing tasks (not for open-ended generation or question answering)
Prompt Sensitivity and Robustness Testing
LLM performance varies substantially with how the question is asked.
Illustrative example:
Prompt
GPT-4 Response Quality
“diabetes”
Generic information, unfocused
“Explain type 2 diabetes management”
Comprehensive overview
“You are an endocrinologist. Explain evidence-based type 2 diabetes management to a newly diagnosed patient using plain language”
Detailed, patient-appropriate, evidence-based
Evaluation requirement: Test performance across multiple prompt variations.
Systematic Prompt Robustness Testing
1. Paraphrase robustness:
Ask the same question five different ways
Evaluate consistency of core recommendations
Red flag: Contradictory advice across paraphrases
2. Context sensitivity:
Test with/without relevant context
Example:
“What’s the treatment for pneumonia?”
“A 75-year-old with COPD has pneumonia. What’s the treatment?”
Should give more specific, appropriate advice with context
3. Role prompting impact:
Test with different role specifications
Example: “As a public health epidemiologist…” vs. no role
Measure impact on accuracy and appropriateness
Human Evaluation: Essential Expert Review
For many LLM applications, human expert evaluation remains essential.
Evaluation Framework
1. Define evaluation criteria:
Example for clinical note summarization:
Accuracy: Are all key facts correct?
Completeness: Are critical findings included?
Conciseness: Is it appropriately brief?
Safety: Are any errors dangerous?
2. Create rating scales:
Example (Likert scale 1–5):
Medical Accuracy:
1 = Significant errors, unsafe
2 = Multiple minor errors
3 = Mostly accurate, minor issues
4 = Accurate with trivial issues
5 = Completely accurate
Clinical Utility:
1 = Not useful, potentially harmful
2 = Limited utility
3 = Moderately useful
4 = Very useful
5 = Extremely useful, improves care
Prespecify an agreement target appropriate to the decision, number of categories, prevalence, and consequences of disagreement
4. Sample diverse test cases:
Common scenarios
Rare/complex cases
Edge cases (ambiguous, incomplete information)
Case Study: Med-PaLM 2 Evaluation Approach
Background: Google’s Med-PaLM 2 study combined multiple-choice benchmarks with physician review of long-form answers. Strong benchmark accuracy and physician preference on a question set do not establish clinical competence.
Evaluation approach (multi-method):
1. Benchmark testing:
MedQA (USMLE): 86.5% accuracy (best result, from a version instruction fine-tuned only on MedQA; 85.4% for the unified model)
PubMedQA: 75.0% accuracy
MedMCQA: 72.3% accuracy
2. Physician review of long-form answers:
On 1,066 consumer medical questions, physicians preferred Med-PaLM 2 answers to physician answers on eight of nine clinical axes
In a pilot on real-world questions from care delivery, specialists preferred Med-PaLM 2 answers to generalist physician answers 65% of the time, but both specialists and generalists still preferred specialist answers about 60% of the time
3. Adversarial testing:
Two new adversarial question sets (general and health equity) probed safety limits; 90.6% of Med-PaLM 2 answers had low risk of harm, compared with 79.4% for the first Med-PaLM
Evaluating Retrieval-Augmented Generation (RAG) Systems
Retrieval-Augmented Generation (RAG) combines an LLM with external knowledge retrieval, such as searching medical literature before generating a response. This approach aims to reduce hallucinations by grounding responses in retrieved, current evidence.
Evaluation must assess two components:
1. Retrieval Quality
Metrics:
A. Retrieval precision:
% of retrieved documents that are relevant
Example: System retrieves 10 papers; 7 are relevant → Precision = 70%
The validation strategy determines how far performance estimates can be trusted.
Internal Validation
Purpose: Estimate model performance on new data from the same source.
Critical limitation:Provides no evidence about performance on different populations, institutions, or time periods.
Method 1: Train-Test Split (Hold-Out Validation)
Procedure:
Randomly split data into training (70–80%) and test (20–30%)
Train model on training set
Evaluate on test set (one time only)
Advantages:
Simple and fast
Clear separation between training and testing
Disadvantages:
Single split can be unrepresentative (bad luck in random split)
Wastes data (test set not used for training)
High variance in performance estimate
When to use: Large datasets (>10,000 samples), quick experiments
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, # 20% for testing random_state=42, # Reproducible split stratify=y # Maintain class balance)
Method 2: K-Fold Cross-Validation
Procedure:
Divide data into K folds (typically 5 or 10)
For each fold:
Train on K-1 folds
Validate on remaining fold
Average performance across all K folds
Advantages:
Uses all data for both training and validation
More stable performance estimate (less variance)
Standard practice in machine learning
Disadvantages:
Computationally expensive (train K models)
Still no external validation
When to use: Moderate-sized datasets (1,000–10,000 samples), model selection
from sklearn.model_selection import cross_val_scorescores = cross_val_score( model, X, y, cv=5, # 5-fold CV scoring='roc_auc'# Metric to optimize)print(f"AUC-ROC: {scores.mean():.3f} (±{scores.std():.3f})")
Method 3: Stratified K-Fold Cross-Validation
Modification: Ensures each fold maintains the same class distribution as the full dataset.
Critical for imbalanced datasets (e.g., 5% disease prevalence).
Why it matters: Without stratification, some folds might have very few positive cases (or none), leading to unstable estimates.
from sklearn.model_selection import StratifiedKFoldskf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)scores = cross_val_score(model, X, y, cv=skf, scoring='roc_auc')
Method 4: Time-Series Cross-Validation
For temporal data: Never train on future data and test on past data.
Procedure (expanding window):
Fold 1: Train [1:100] → Test [101:120]
Fold 2: Train [1:120] → Test [121:140]
Fold 3: Train [1:140] → Test [141:160]
...
Critical for: Epidemic forecasting, time-series prediction, any data with temporal structure
from sklearn.model_selection import TimeSeriesSplittscv = TimeSeriesSplit(n_splits=5)for train_idx, test_idx in tscv.split(X): X_train, X_test = X[train_idx], X[test_idx] y_train, y_test = y[train_idx], y[test_idx]# Train and evaluate
Critical Considerations for Internal Validation
1. Data Leakage Prevention
Data leakage: Information from test set influencing training process.
Common sources:
WRONG: Feature engineering on entire dataset:
# WRONG: Standardize before splittingX_scaled = StandardScaler().fit_transform(X) # Uses mean/std from ALL dataX_train, X_test, y_train, y_test = train_test_split(X_scaled, y)# Test set info leaked into training!
CORRECT: Feature engineering within train/test:
# CORRECT: Fit scaler on training onlyX_train, X_test, y_train, y_test = train_test_split(X, y)scaler = StandardScaler().fit(X_train) # Learn from training onlyX_train_scaled = scaler.transform(X_train)X_test_scaled = scaler.transform(X_test) # Apply to test
Problem: If data has natural clusters (patients within hospitals, repeated measures within individuals), random splitting can lead to overfitting.
Example: Patient has 5 hospitalizations. Random split → some hospitalizations in training, others in test. Model learns patient-specific patterns → overoptimistic performance.
Solution:Group K-Fold: ensure all samples from same group stay together
from sklearn.model_selection import GroupKFold# patient_ids: array indicating which patient each sample belongs togkf = GroupKFold(n_splits=5)for train_idx, test_idx in gkf.split(X, y, groups=patient_ids):# All samples from same patient stay in same fold X_train, X_test = X[train_idx], X[test_idx]
External Validation: Stronger Evidence for Generalizability
External validation: Testing on data from an entirely different source: different institution(s), population, time period, or setting.
Why it matters:
Models often learn dataset-specific quirks that do not generalize:
Hospital equipment signatures
Documentation practices
Patient population characteristics
Data collection protocols
Without external validation, there is no way to tell whether a model learned disease patterns or dataset artifacts.
Types of External Validation
1. Geographic External Validation
Design:
Train: Hospital A (or multiple hospitals in one region)
Test: Hospital B (or hospitals in different region)
What it tests:
Different patient demographics
Different clinical practices
Different data collection protocols
Different equipment (for imaging)
Example:McKinney et al., 2020, Nature - Google breast cancer AI evaluated on UK and US screening data. The system’s AUC for biopsy-confirmed cancer was 0.889 on the UK set. On US data, a version trained only on UK images reached an AUC of 0.757, compared with 0.811 for a version that also trained on US images.
2. Temporal External Validation
Design:
Train: Data from 2015–2018
Test: Data from 2019–2021
What it tests:
Temporal stability (concept drift)
Changes in disease patterns
Changes in clinical practice
Changes in data collection
Example:Davis et al., 2017, JAMIA - Clinical prediction models degrade over time; most models need recalibration after 2-3 years.
3. Setting External Validation
Design:
Train: Intensive care unit (ICU) data
Test: General ward data
What it tests:
Performance in different clinical settings
Generalization across disease severity spectra
Example: Sepsis models trained on ICU patients often fail on ward patients (different disease presentation, different monitoring intensity).
ChestX-ray14 dataset: 112,120 X-rays from NIH Clinical Center
14 pathology classification tasks
Claimed: “Radiologist-level pneumonia detection”
Performance: AUC-ROC = 0.7632 for pneumonia
Shortcut learning in medical imaging AI:DeGrave et al., 2021, Nature Machine Intelligence demonstrated that COVID-19 chest X-ray classifiers learned shortcuts from non-medical features rather than disease-relevant signals; the same pattern threatens any medical imaging AI lacking external validation
Tested on:
NIH Clinical Center: 112,120 radiographs from 30,805 patients
Mount Sinai Hospital: 42,396 radiographs from 12,904 patients
Indiana University Network for Patient Care: 3,807 radiographs from 3,683 patients
Results:
Internal performance exceeded external performance in 3 of 5 natural comparisons
A model trained and tested on pooled Mount Sinai and NIH data reached an internal AUC of 0.931 but an external AUC of 0.815 at Indiana University
Pneumonia prevalence was 34.2% at Mount Sinai versus 1.2% at NIH, so sorting radiographs by hospital system alone achieved an AUC of 0.861 on the joint dataset
Explanation: CNNs identified the hospital system of 99.95% of NIH and 99.98% of Mount Sinai radiographs, and could detect the department within a hospital; both track disease burden and can confound predictions
Single-institution data insufficient for generalizability claims
Models can learn spurious correlations specific to training site
External validation is essential before clinical deployment
See also:Zech et al., 2018, PLOS Medicine - “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs”
Prospective Validation: Real-World Testing
Prospective validation: Model deployed in actual clinical practice, evaluated in real-time.
Why it matters: Retrospective validation cannot capture:
How clinicians actually use (or ignore) model predictions
Workflow integration challenges
Alert fatigue and override patterns
Behavioral changes in response to predictions
Unintended consequences
Study Design 1: Silent Mode Deployment
Design:
Deploy model in background
Generate predictions but do not show to clinicians
Compare predictions to actual outcomes (collected as usual)
Advantages:
Tests real-world data quality and distribution
No risk to patients (clinicians unaware of predictions)
Can assess performance before making decisions based on model
Disadvantages:
Does not test impact on clinical decisions
Does not assess workflow integration
Example:Tomašev et al., 2019, Nature - DeepMind AKI prediction initially deployed silently at VA hospitals to validate real-time performance before clinical integration.
Study Design 2: Randomized Controlled Trial (RCT)
Design:
Randomize: Patients, clinicians, or hospital units to:
Intervention: Model-assisted care
Control: Standard care (no model)
Measure: Clinical outcomes in both groups
Compare: Test if model improves outcomes
Advantages:
Strong causal evidence for the effect of using the intervention
Can estimate effects on outcomes and resource use
Supports causal effectiveness and economic claims when the design and outcomes match those claims
Example:Semler et al., 2018, NEJM - SMART trial of balanced crystalloids vs. saline (not AI, but example of rigorous prospective cluster-randomized design)
Study Design 3: Stepped-Wedge Design
Design:
Roll out model sequentially to different units/sites, with the order of crossover randomized
Each unit contributes control and intervention periods, and at each step some units are still in control
Eventually all units receive intervention
Advantages:
More feasible than full RCT
All units eventually get intervention (addresses ethical concerns)
Within-unit comparisons reduce confounding
Disadvantages:
Secular trends can confound results and must be modeled
Longer duration than a parallel cluster-randomized trial
Staggered rollouts without randomized timing lose the protection of randomization
Use: Common in health system implementations where a parallel RCT is infeasible.
Study Design 4: A/B Testing
Design:
Randomly assign users to model-assisted vs. control in real-time
Continuously measure outcomes
Iterate rapidly based on results
Advantages:
Rapid experimentation
Can test multiple model versions
Common in the technology industry
Challenges in healthcare:
Ethical concerns (different care for similar patients)
Regulatory considerations (IRB approval required)
Contamination (clinicians may share information)
Beyond Accuracy: Clinical Utility Assessment
Critical insight: A model can be statistically accurate but clinically useless.
Example:
Model predicts hospital mortality with AUC-ROC = 0.85
But: If it does not change clinical decisions or improve outcomes, it adds no clinical value.
Further: If implementing it disrupts workflow or generates alert fatigue, net impact may be negative.
The Clinical Utility Question
Before deploying any clinical AI:
Does it change decisions?
Do those changed decisions improve outcomes?
Is the improvement worth the cost (financial, workflow disruption, alert burden)?
If the answer to any of the three is no, do not deploy.
\(TP/N\) = True positives per patient (benefit from correctly treating disease)
\(FP/N \times p_t/(1-p_t)\) = False positives per patient, weighted by harm of unnecessary treatment
Interpretation:
If treating disease has high benefit relative to harm of unnecessary treatment → lower \(p_t\) threshold
If treating disease has low benefit relative to harm → higher \(p_t\) threshold
Weight \(p_t/(1-p_t)\): Reflects how much we weight false positives.
At \(p_t\) = 0.10: Weight = 0.10/0.90 ≈ 0.11 (FP weighted 1/9 as much as TP)
At \(p_t\) = 0.50: Weight = 0.50/0.50 = 1.00 (FP and TP equally weighted)
DCA Plot and Interpretation
Create DCA plot:
X-axis: Threshold probability (risk at which a clinician would intervene)
Y-axis: Net benefit
Plot curves for:
Model: Net benefit using model predictions
Treat all: Net benefit if everyone treated
Treat none: Net benefit if no one treated (= 0)
Interpretation:
Model is useful where its curve is above both “treat all” and “treat none”
Higher net benefit = better clinical value
Range of thresholds where model useful = decision curve clinical range
Example interpretation:
At 15% risk threshold:
Model NB = 0.12
Treat all NB = 0.05
Treat none NB = 0.00
Meaning: Using the model at a 15% threshold is equivalent to correctly treating 12 of every 100 patients with no false positives, compared with 5 for the “treat all” strategy.
Python implementation:
def calculate_net_benefit(y_true, y_pred_proba, thresholds):"""Calculate net benefit across thresholds for decision curve analysis""" net_benefits = []for threshold in thresholds:# Classify based on threshold y_pred = (y_pred_proba >= threshold).astype(int)# Calculate TP, FP, TN, FN TP = ((y_pred ==1) & (y_true ==1)).sum() FP = ((y_pred ==1) & (y_true ==0)).sum() N =len(y_true)# Net benefit formula nb = (TP / N) - (FP / N) * (threshold / (1- threshold)) net_benefits.append(nb)return np.array(net_benefits)# Calculate for model, treat all, treat nonethresholds = np.linspace(0.01, 0.99, 100)nb_model = calculate_net_benefit(y_test, y_pred_proba, thresholds)nb_treat_all = y_test.mean() - (1- y_test.mean()) * (thresholds / (1- thresholds))nb_treat_none = np.zeros_like(thresholds)# Plot decision curveplt.figure(figsize=(10, 6))plt.plot(thresholds, nb_model, label='Model', linewidth=2)plt.plot(thresholds, nb_treat_all, label='Treat All', linestyle='--', linewidth=2)plt.plot(thresholds, nb_treat_none, label='Treat None', linestyle=':', linewidth=2)plt.xlabel('Threshold Probability', fontsize=12)plt.ylabel('Net Benefit', fontsize=12)plt.title('Decision Curve Analysis', fontsize=14, fontweight='bold')plt.legend()plt.grid(alpha=0.3)plt.xlim(0, 0.5) # Focus on clinically relevant rangeplt.show()
Purpose: Quantify whether a new model improves risk stratification compared to existing approach.
Context: An existing risk model (or clinical judgment) is in use, and a new model is proposed. The question is whether the new model reclassifies patients into more appropriate risk categories.
Net Reclassification Improvement (NRI)
Concept: The net proportion of events (people with disease) correctly moved to higher risk, plus the net proportion of non-events correctly moved to lower risk.
Measures how much the new model increases separation between events and non-events
IDI > 0: Better discrimination
Less sensitive to arbitrary cut-points than NRI
Fairness and Equity in Evaluation
AI systems can exhibit disparate performance across demographic groups, even when overall performance appears strong.
The Fairness Imperative
Failure to assess fairness can:
Perpetuate or amplify existing health disparities
Result in differential quality of care based on race, gender, socioeconomic status
Violate ethical principles of justice and equity
Expose organizations to legal liability
Fairness assessment is required.
Mathematical Definitions of Fairness
Challenge: Multiple, often conflicting, definitions of fairness exist.
1. Demographic Parity (Statistical Parity)
Definition: Positive prediction rates equal across groups
\[P(\hat{Y}=1 | A=0) = P(\hat{Y}=1 | A=1)\]
where \(A\) = protected attribute (e.g., race, gender)
Example: Model predicts high risk for 20% of White patients and 20% of Black patients
When appropriate:
Resource allocation (equal access to interventions)
Contexts where base rates should be equal
Problem: Ignores actual outcome rates. If disease prevalence differs between groups (due to structural factors), enforcing demographic parity may reduce overall accuracy.
2. Equalized Odds
Definition: True positive and false positive rates equal across groups
Traditional evaluation assumes benign inputs. Deployed models face:
Natural perturbations: Image quality variation, data entry errors, equipment differences
Adversarial attacks: Malicious manipulation to cause misclassification
Out-of-distribution inputs: Cases far from training data
EU context: Article 15 of the EU AI Act requires an appropriate level of accuracy, robustness, and cybersecurity for high-risk systems. It does not prescribe one universal adversarial-testing method (EU AI Act, 2024).
Security Is a Patient Safety Issue
Example scenarios:
Hospital ransomware attack compromises AI model integrity
Malicious actor manipulates medical imaging to hide cancer
Data poisoning during model retraining introduces systematic errors
Unlike traditional software vulnerabilities (which can be patched), ML models can be permanently corrupted or subtly manipulated without obvious signs.
Types of Adversarial Threats
1. Evasion Attacks (Inference-Time)
Goal: Manipulate input to cause misclassification without changing ground truth.
Example:
Add imperceptible noise to chest X-ray → Model misses pneumonia
Modify patient vital signs slightly → Sepsis prediction model fails to alert
Medical relevance:
Natural occurrence: Image compression, scanner differences can mimic adversarial perturbations
Malicious: Rare but theoretically possible (e.g., insurance fraud, medicolegal manipulation)
2. Poisoning Attacks (Training-Time)
Goal: Corrupt training data to degrade model performance or introduce backdoors.
Example:
Insert mislabeled images into training set → Model learns incorrect patterns
Add trigger patterns → Model fails only for specific subgroups
Medical relevance:
Multi-institutional data sharing: If one site’s data is compromised, all participants affected
Crowdsourced labels: If annotations are maliciously manipulated
3. Model Extraction/Stealing
Goal: Query model repeatedly to reverse-engineer its parameters.
Risk: Intellectual property theft, creating surrogate model for further attacks
Evaluating Robustness
Method 1: Input Perturbation Testing
Approach: Systematically perturb inputs and measure performance degradation.
For medical imaging:
import numpy as npfrom skimage.util import random_noisedef test_noise_robustness(model, test_images, test_labels, noise_levels):""" Test model robustness to image noise Args: model: Trained classification model test_images: Clean test images test_labels: Ground truth labels noise_levels: List of noise standard deviations to test Returns: Dictionary of accuracy at each noise level """ results = {}# Baseline (no noise) baseline_acc = model.evaluate(test_images, test_labels)[1] results['baseline'] = baseline_acc# Test each noise levelfor sigma in noise_levels: noisy_images = np.array([ random_noise(img, mode='gaussian', var=sigma**2)for img in test_images ]) noisy_acc = model.evaluate(noisy_images, test_labels)[1] results[f'sigma_{sigma}'] = noisy_acc degradation = baseline_acc - noisy_accprint(f"Noise σ={sigma:.3f}: Accuracy={noisy_acc:.3f} "f"(degradation: {degradation:.3f})")return results# Example usagenoise_levels = [0.01, 0.05, 0.10, 0.20]robustness_results = test_noise_robustness( model, test_images, test_labels, noise_levels)# Acceptable degradation thresholdif robustness_results['sigma_0.05'] <0.85* robustness_results['baseline']:print("[WARNING] Model performance degrades >15% with minor noise")print("→ Consider: Data augmentation, robust training, ensemble methods")
For tabular data (clinical variables):
def test_feature_perturbation_robustness(model, X_test, y_test, perturbation_fraction=0.05):""" Test robustness to small perturbations in continuous features Args: model: Trained model X_test: Test features (pandas DataFrame) y_test: Test labels perturbation_fraction: Fraction of feature value to perturb Returns: Robustness metrics """from sklearn.metrics import roc_auc_score# Baseline performance y_pred_baseline = model.predict_proba(X_test)[:, 1] auc_baseline = roc_auc_score(y_test, y_pred_baseline)# Perturb continuous features X_perturbed = X_test.copy() continuous_cols = X_test.select_dtypes(include=[np.number]).columnsfor col in continuous_cols:# Add random noise proportional to feature value noise = np.random.normal(0, perturbation_fraction * X_test[col].std(), size=len(X_test)) X_perturbed[col] = X_test[col] + noise# Evaluate perturbed performance y_pred_perturbed = model.predict_proba(X_perturbed)[:, 1] auc_perturbed = roc_auc_score(y_test, y_pred_perturbed)# Prediction consistency prediction_changes = np.mean( (y_pred_baseline >0.5) != (y_pred_perturbed >0.5) )print(f"Baseline AUC: {auc_baseline:.3f}")print(f"Perturbed AUC: {auc_perturbed:.3f}")print(f"AUC degradation: {auc_baseline - auc_perturbed:.3f}")print(f"Prediction changes: {prediction_changes:.1%}")return {'auc_baseline': auc_baseline,'auc_perturbed': auc_perturbed,'prediction_change_rate': prediction_changes }# Exampleresults = test_feature_perturbation_robustness(model, X_test, y_test, perturbation_fraction=0.05)if results['prediction_change_rate'] >0.10:print("[WARNING] >10% of predictions change with 5% feature noise")print("→ Model may be overfitting to noise rather than signal")
Method 2: Adversarial Attack Testing
Fast Gradient Sign Method (FGSM) - Basic adversarial attack:
import tensorflow as tfdef fgsm_attack(model, image, label, epsilon=0.01):""" Generate adversarial example using Fast Gradient Sign Method Args: model: Trained model image: Input image label: True label epsilon: Perturbation magnitude Returns: Adversarial image """ image = tf.cast(image, tf.float32)with tf.GradientTape() as tape: tape.watch(image) prediction = model(image) loss = tf.keras.losses.sparse_categorical_crossentropy(label, prediction)# Get gradient of loss w.r.t. image gradient = tape.gradient(loss, image)# Create adversarial image signed_grad = tf.sign(gradient) adversarial_image = image + epsilon * signed_grad adversarial_image = tf.clip_by_value(adversarial_image, 0, 1)return adversarial_image# Evaluate adversarial robustnessdef evaluate_adversarial_robustness(model, test_images, test_labels, epsilons=[0.0, 0.01, 0.05, 0.10]):""" Test model robustness to FGSM attacks at different perturbation levels """ results = {}for eps in epsilons: correct =0 total =0for img, label inzip(test_images, test_labels):# Generate adversarial example adv_img = fgsm_attack(model, img[np.newaxis, ...], label, epsilon=eps)# Predict pred = model.predict(adv_img) pred_class = np.argmax(pred)if pred_class == label: correct +=1 total +=1 accuracy = correct / total results[eps] = accuracyprint(f"Epsilon={eps:.3f}: Accuracy={accuracy:.3f}")return results# Run evaluationadv_results = evaluate_adversarial_robustness(model, test_images, test_labels)# Alert if significant degradationif adv_results[0.05] <0.70* adv_results[0.0]:print("[CRITICAL] Model highly vulnerable to adversarial attacks")print("→ Implement: Adversarial training, input validation, ensemble methods")
Method 3: Out-of-Distribution (OOD) Detection
Goal: Identify when inputs are unlike training data (model should abstain or flag uncertainty).
def evaluate_ood_detection(model, in_dist_data, ood_data):""" Evaluate model's ability to detect out-of-distribution inputs Args: model: Trained model in_dist_data: In-distribution test data ood_data: Out-of-distribution data Returns: OOD detection performance metrics """from sklearn.metrics import roc_auc_score# Get prediction confidence (max probability) for each dataset in_dist_preds = model.predict(in_dist_data) in_dist_confidence = np.max(in_dist_preds, axis=1) ood_preds = model.predict(ood_data) ood_confidence = np.max(ood_preds, axis=1)# Combine labels (1 = in-distribution, 0 = OOD) y_true = np.concatenate([ np.ones(len(in_dist_confidence)), np.zeros(len(ood_confidence)) ])# Confidence scores (higher = more likely in-distribution) confidence_scores = np.concatenate([in_dist_confidence, ood_confidence])# Calculate AUROC for OOD detection auroc = roc_auc_score(y_true, confidence_scores)print(f"OOD Detection AUROC: {auroc:.3f}")print(f"In-dist mean confidence: {in_dist_confidence.mean():.3f}")print(f"OOD mean confidence: {ood_confidence.mean():.3f}")if auroc <0.80:print("[WARNING] Poor OOD detection - model overconfident on unfamiliar inputs")print("→ Consider: Temperature scaling, Bayesian approaches, ensemble uncertainty")return auroc# Example: Test on different medical image datasetood_auroc = evaluate_ood_detection( model, in_dist_data=chest_xray_test, # Data from same hospitals as training ood_data=external_site_data # Data from completely different hospital/scanner)
Robustness Improvement Strategies
1. Data Augmentation:
Train on varied/augmented data (rotations, brightness changes, noise)
Forces model to learn invariant features
2. Adversarial Training:
Include adversarial examples in training set
Trade-off: May slightly reduce clean accuracy
3. Ensemble Methods:
Multiple models often more robust than single model
Study: Added imperceptible perturbations to medical images (chest X-rays, fundus photos, dermoscopy images)
Results:
Successfully fooled leading deep learning classifiers
Adversarial examples transferable across models (attack one model, affects others)
Small perturbations caused dramatic misclassifications
Implications:
Medical AI models are vulnerable to adversarial attacks
Robustness testing should be mandatory for deployed systems
Both accidental (natural variations) and malicious perturbations are risks
Counterpoint:No documented real-world malicious attacks on medical AI systems (yet), but accidental distribution shifts are common (equipment changes, protocol updates).
Key Takeaways: Adversarial Robustness
Robustness must be demonstrated appropriately: EU AI Act Article 15 requires appropriate accuracy, robustness, and cybersecurity for high-risk systems. Adversarial testing is one possible evidence method when the threat model warrants it.
Two threat models: Natural perturbations (common) vs. adversarial attacks (rare but possible)
Definition: Perception that system is agreeable/satisfactory
Measures:
User satisfaction surveys (Likert scales, Net Promoter Score)
Qualitative interviews (what do users like/dislike?)
Perceived usefulness and ease of use
Example questions:
“This system improves my clinical decision-making” (1–5 scale)
“I would recommend this system to colleagues” (yes/no)
2. Adoption
Definition: Intention/action to use the system
Measures:
Utilization rate (% of eligible cases where system used)
Number of users who have activated/logged in
Time to initial use
Red flag: Low adoption despite availability suggests problems with acceptability, workflow fit, or perceived utility.
A 19 August 2026 Nature Medicine DECIDE-AI stage 1 evaluation of SHAKED, a multi-LLM CDSS in a tertiary emergency department (1,138 patients over 4 weeks; one wing used the system, one followed routine rotations), is the empirical form of that red flag: clinician adoption fell from 68% to 30% (OR = 0.72 per shift hour, 95% CI 0.62–0.83), expert review rated 99 of 100 sampled outputs as clinically appropriate, and no adverse events were detected, but length of stay was 4.9 h in both wings (P = 0.99) and the intention-to-treat consultation-cycle-time difference (−9.4 min) was not significant (P = 0.077) (Leibovitch et al., 2026). Output appropriateness and decaying adoption are not a care endpoint; the authors write that the results inform randomized-trial design and do not justify clinical deployment.
3. Appropriateness
Definition: Perceived fit for setting/population/problem
Measures:
Stakeholder perception surveys
Alignment with clinical workflows (workflow mapping)
Relevance to clinical questions
Example: ICU mortality prediction may be appropriate for ICU but inappropriate for outpatient clinic.
4. Feasibility
Definition: Ability to successfully implement
Measures:
Technical integration challenges (API compatibility, data availability)
Resource requirements (cost, staff time, training)
Infrastructure needs (computing, network)
5. Fidelity
Definition: Degree to which system used as designed
Measures:
Override rates (how often do clinicians dismiss alerts?)
Deviation from intended use (using for wrong purpose)
Workarounds (users circumventing system)
High override rates signal problems:
Too many false positives (alert fatigue)
Predictions do not match clinical judgment (trust issues)
Workflow disruption (alerts at wrong time)
6. Penetration
Definition: Integration across settings/populations
Measures:
Number of sites/units using system
Proportion of target population reached
Geographic spread
7. Sustainability
Definition: Continued use over time
Measures:
Retention of users over 6–12 months
Model updating/maintenance plan
Long-term performance monitoring
Common failure: “Pilot-itis”: successful pilot, but system not sustained after initial implementation period.
Evidence: Classic systematic reviews of computerized provider order entry found that clinicians overrode 49% to 96% of medication safety and drug-drug interaction alerts (van der Sijs et al., 2006), and longitudinal EHR evaluations confirm that rising alert volume and complexity directly depress clinician acceptance as alert fatigue accumulates (Ancker et al., 2017).
Example: At the evaluated threshold, Epic’s sepsis model had 12% positive predictive value, so 88% of patients who would have generated an alert did not meet the study’s sepsis outcome (Wong et al., 2021). The study estimated a substantial alert burden; it did not measure whether clinicians stopped responding.
Solutions:
Minimize false positives (sacrifice sensitivity if needed)
Tiered alerts (critical vs. informational)
Smart timing (deliver when actionable, not during documentation)
Actionable recommendations (“Order blood cultures” not “Consider sepsis”)
2. Workflow Disruption
Problem: System does not integrate smoothly into existing processes
Examples:
Extra clicks required
Separate application (need to switch contexts)
Alerts interrupt at inopportune times (during patient exam)
Solutions:
User-centered design (involve clinicians early and often)
Embed in existing EHR workflows
Minimize friction (one-click actions)
For workflow integration principles, see Bates et al., 2003, JAMIA on the “Ten Commandments” for effective clinical decision support.
3. Lack of Trust
Problem: Clinicians do not trust “black box” predictions
Example: Deep learning model provides risk score with no explanation
Solutions:
Provide explanations (SHAP values, attention weights)
Show evidence base (similar cases, supporting literature)
Transparent validation (publish performance data)
Gradual trust-building (start with low-stakes recommendations)
4. Model Drift
Problem: Performance degrades over time as data distribution changes
Why it matters: A 2017 study of clinical prediction models found that most require recalibration within 2-3 years due to changes in patient populations, treatment patterns, and data collection practices.
MLOps and Continuous Model Monitoring
Moving beyond “deploy and hope”: Modern AI systems require continuous monitoring to detect performance degradation and trigger timely interventions.
Post-Deployment Monitoring Is Not Optional
The FDA’s 2021 Action Plan on AI/ML-based Software as a Medical Device emphasizes continuous monitoring as a core requirement. Models that do not monitor performance drift pose patient safety risks.
Real-world failure: IBM Watson for Oncology was adopted at multiple institutions, and internal company documents reported in 2018 described unsafe and incorrect treatment recommendations.
Types of Drift
Understanding the type of drift helps determine appropriate interventions.
1. Data Drift (Covariate Shift)
Definition: Input feature distributions change, but the relationship between features and outcome remains stable.
Example:
Training data (2019): Average patient age = 55, BMI = 27
Production data (2024): Average patient age = 62, BMI = 31
Relationship unchanged: Diabetes risk per BMI unit = same
Impact: Model may become miscalibrated (predicted probabilities off) even if discrimination (AUC-ROC) stays stable.
Detection: Compare feature distributions between training and production data.
2. Concept Drift
Definition: The relationship between features and outcome changes.
Impact: Predicted probabilities may be systematically too low or too high (calibration degrades).
Detection: Compare outcome rates over time.
Practical Drift Detection Methods
Method 1: Statistical Tests for Feature Distribution Changes
Kolmogorov-Smirnov (KS) Test:
Tests whether two distributions are significantly different.
Hide code
from scipy.stats import ks_2sampimport numpy as npimport pandas as pd# Example: Monitoring patient age distribution# Training data (historical)age_train = np.random.normal(55, 15, 1000) # Mean=55, SD=15# Production data (current month)age_prod = np.random.normal(62, 15, 500) # Mean shifted to 62# Perform KS teststatistic, p_value = ks_2samp(age_train, age_prod)print(f"KS statistic: {statistic:.3f}")print(f"P-value: {p_value:.4f}")if p_value <0.01:print("[ALERT] Significant distribution shift detected!")print("→ Review model calibration and consider retraining")else:print("[OK] No significant drift detected")# Interpretation:# p < 0.01: Strong evidence of distribution change# p < 0.05: Moderate evidence of distribution change# p ≥ 0.05: No significant change detected
When to use: Continuous features (age, lab values, vital signs)
Chi-Square Test (for categorical features):
Hide code
from scipy.stats import chi2_contingency# Example: Monitoring sex distribution# Training datatrain_counts = {"Male": 600, "Female": 400} # 60% male# Production data (current month)prod_counts = {"Male": 250, "Female": 250} # 50% male# Create contingency tablecontingency_table = pd.DataFrame({'Train': [train_counts['Male'], train_counts['Female']],'Production': [prod_counts['Male'], prod_counts['Female']]}, index=['Male', 'Female'])chi2, p_value, dof, expected = chi2_contingency(contingency_table)print(f"Chi-square statistic: {chi2:.3f}")print(f"P-value: {p_value:.4f}")if p_value <0.01:print("[ALERT] Significant distribution shift in sex distribution!")
When to use: Categorical features (sex, race, diagnostic codes)
Method 2: Population Stability Index (PSI)
Widely used in industry for monitoring feature drift.
Recommendation for healthcare: Use periodic retraining with trigger-based scheduling (not true online learning).
Real-World Example: External Validation After Adoption
Epic’s sepsis model had already been implemented at hundreds of US hospitals when Wong and colleagues published an independent external validation at one academic health system (Wong et al., 2021).
At the evaluated threshold, the study reported:
Sensitivity: 33%
Positive predictive value: 12%
Alert burden: 18% of hospitalizations would have generated at least one alert
The study showed poor discrimination and calibration in that setting. It did not test model drift, continuous monitoring at other deployment sites, or a retraining protocol.
Lesson:Widespread adoption is not evidence of transportability or clinical value. Independent evaluation should precede broad use, and post-deployment monitoring should address questions that a retrospective external validation cannot answer.
Monitors data drift, model performance, target drift
Generates visual reports
Integrates with MLOps pipelines
Hide code
# Example: Evidently for drift detectionfrom evidently.report import Reportfrom evidently.metric_preset import DataDriftPreset# Create drift reportreport = Report(metrics=[DataDriftPreset()])# Compare reference (training) vs. current (production) datareport.run(reference_data=train_df, current_data=production_df)# Save reportreport.save_html("drift_report.html")# Access metrics programmaticallydrift_results = report.as_dict()n_drifted_features = drift_results['metrics'][0]['result']['number_of_drifted_columns']if n_drifted_features >3:print(f"[WARNING] {n_drifted_features} features showing significant drift")
2. Alibi Detect:
Drift detection algorithms
Supports tabular, text, image data
3. NannyML:
Performance estimation without ground truth
Confidence-based performance monitoring
Key Takeaways: Model Drift and Monitoring
All models drift: Performance degradation is inevitable, not exceptional
Types of drift matter: Data drift vs. concept drift require different interventions
Multiple detection methods: Use statistical tests (KS, Chi-square), PSI, and performance tracking simultaneously
Automated triggers: Define clear thresholds for retraining (do not wait for catastrophic failure)
Continuous monitoring is mandatory: FDA and EU regulations increasingly require post-deployment monitoring
Periodic retraining > online learning: For healthcare, controlled validation is safer than continuous updates
Monitor before you have ground truth: Prediction distribution shifts can signal problems early
Document everything: Track what triggered retraining, what changed, and validation results
Framework reference:Davis et al., 2017, JAMIA - Clinical prediction models degrade; most need recalibration after 2-3 years.
Explainability and Interpretability (XAI)
Why Explainability Matters in Public Health AI
The trust problem: Systematic reviews consistently find that clinicians are reluctant to trust or act on predictions from “black box” AI systems they cannot interpret (Antoniadi et al., 2021; Markus et al., 2021).
Why interpretability is critical:
Clinical decision-making: Clinicians need to know why before they can decide whether to act
Debugging and validation: Explanations reveal spurious correlations and dataset biases
Regulatory requirements: The FDA emphasizes transparency to users, while EU AI Act Article 13 requires instructions sufficient for deployers to interpret and use high-risk system outputs appropriately
Patient autonomy: Patients have a right to understand decisions affecting their health
Legal liability: “The algorithm said so” is not a defense in malpractice cases
Post-Hoc Explanation Is Not Intrinsic Interpretability
Traditional belief: Deep learning = high accuracy but uninterpretable; simpler models = lower accuracy but interpretable.
Current reality: Post-hoc methods can describe associations with model outputs, but they do not make a model intrinsically interpretable. Their fidelity, stability, and usefulness must be evaluated for the intended user and decision.
Guideline: Start with the simplest model that meets performance requirements. If a complex model is required, budget for explanation infrastructure and for testing the explanations it produces.
Levels of Interpretability
Not all interpretability is equal. Different stakeholders need different levels of explanation.
1. Global Interpretability
Definition: Understanding the model’s overall behavior and decision logic.
Questions answered:
What features are most important overall?
How does the model generally make decisions?
Are there unexpected feature relationships?
Methods:
Feature importance rankings
Partial dependence plots
Global SHAP values
Audience: Data scientists, validators, regulators
2. Local Interpretability
Definition: Understanding why the model made a specific prediction for a specific patient.
Questions answered:
Why did the model predict this patient is high-risk?
Which patient characteristics drove this prediction?
What would need to change to alter the prediction?
Patient 47: Sepsis Risk = 78%
Main drivers:
+ Lactate 3.2 mmol/L (+0.35 risk contribution) <- Primary concern
+ Temperature 39.1°C (+0.22)
+ WBC 15,000/μL (+0.18)
- Normal BP 118/72 (-0.08) <- Protective factor
Interpretation: Elevated lactate is the largest contributor to this prediction.
Consider serial lactate monitoring and early fluid resuscitation.
SHAP Advantages and Limitations
Advantages:
Mathematically principled (satisfies local accuracy, missingness, consistency)
Works with any model architecture
Both global and local explanations
Handles feature interactions
Limitations:
Computational cost: Can be slow for large models/datasets (use TreeSHAP for tree models, faster)
Not causal: A high SHAP value reflects association, not a causal relationship
Assumes feature independence: Can give misleading results with highly correlated features
Best practices:
Use TreeSHAP for tree-based models (XGBoost, Random Forest); it is much faster than KernelSHAP
For neural networks, use DeepSHAP or KernelSHAP with background dataset sampling
Always review explanations with domain experts to confirm they are clinically plausible
What it is: Creates a simple, interpretable model (like linear regression) that approximates the complex model’s behavior locally around a specific prediction.
How it works:
Perturb the input (create similar but slightly different patients)
Get model predictions for perturbed inputs
Fit a simple linear model to these local predictions
Linear coefficients = feature importance for this prediction
When to use:
Need quick local explanations
SHAP is too computationally expensive
Want human-readable rules (“If lactate > 2 AND fever, then high risk”)
import limeimport lime.lime_tabularimport numpy as npimport pandas as pdfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split# Simulate patient data for hospital readmissionnp.random.seed(42)n_patients =1000data = pd.DataFrame({'age': np.random.normal(68, 12, n_patients),'num_prior_admissions': np.random.poisson(2, n_patients),'length_of_stay': np.random.gamma(2, 2, n_patients),'num_medications': np.random.poisson(5, n_patients),'comorbidity_count': np.random.poisson(3, n_patients),'emergency_admission': np.random.binomial(1, 0.3, n_patients),})# Create readmission outcomereadmit_risk = (0.02* data['age'] +0.15* data['num_prior_admissions'] +0.05* data['comorbidity_count'] +0.1* data['emergency_admission'] + np.random.normal(0, 0.5, n_patients))data['readmitted_30d'] = (readmit_risk >2).astype(int)# Train modelX = data.drop('readmitted_30d', axis=1)y = data['readmitted_30d']X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)model = RandomForestClassifier(n_estimators=100, random_state=42)model.fit(X_train, y_train)# ===== LIME EXPLANATION =====# 1. Create LIME explainerexplainer = lime.lime_tabular.LimeTabularExplainer( training_data=X_train.values, feature_names=X_train.columns.tolist(), class_names=['No Readmission', 'Readmission'], mode='classification', random_state=42)# 2. Explain a specific patientpatient_idx =5patient_data = X_test.iloc[patient_idx].valuespredicted_proba = model.predict_proba([patient_data])[0]print(f"=== Patient {patient_idx} ===")print(f"Predicted readmission probability: {predicted_proba[1]:.2%}")print(f"Actual outcome: {'Readmitted'if y_test.iloc[patient_idx] ==1else'Not readmitted'}")# Generate explanationexplanation = explainer.explain_instance( data_row=patient_data, predict_fn=model.predict_proba, num_features=6)# 3. Display explanationprint("\n=== LIME Explanation ===")print("Feature contributions to 'Readmission' class:")for feature, weight in explanation.as_list():print(f" {feature}: {weight:+.3f}")# 4. Visualizeexplanation.show_in_notebook(show_table=True)# Save as HTMLexplanation.save_to_file('lime_explanation_patient5.html')# 5. Extract feature importance for this patientfeature_importance =dict(explanation.as_list())print("\n=== Top Risk Factors for This Patient ===")sorted_features =sorted(feature_importance.items(), key=lambda x: abs(x[1]), reverse=True)for feature, weight in sorted_features[:3]: direction ="↑ Increases"if weight >0else"↓ Decreases"print(f"{direction} risk: {feature} (impact: {weight:+.3f})")
Example output:
=== Patient 5 ===
Predicted readmission probability: 64%
Feature contributions:
num_prior_admissions > 3.00: +0.22 ← Major risk factor
comorbidity_count > 4.00: +0.15
emergency_admission = 1: +0.12
age > 65.00: +0.08
length_of_stay ≤ 3.00: -0.05 ← Lowers predicted risk in this model (association, not mechanism)
num_medications > 6.00: -0.02
Interpretation: This patient's high readmission risk is driven primarily
by multiple prior admissions (4 in past year) and high comorbidity burden.
LIME Advantages and Limitations
Advantages:
Fast: Quicker than SHAP for local explanations
Intuitive: Simple “if-then” rules easy for clinicians to understand
Model-agnostic: Works with any black box model
Limitations:
Instability: Explanations can vary substantially with small input changes
Local only: Does not provide global model understanding
No theoretical guarantees: Unlike SHAP, not mathematically principled
When to choose LIME over SHAP:
Real-time explanations needed (speed critical)
Prefer rule-based explanations (“If X > 5 AND Y < 10…”)
SHAP computationally infeasible for the model
Method 3: Attention Mechanisms (For Deep Learning)
What it is: Neural network architectures that learn to weight input elements, producing attention weights that are often inspected as explanations.
Applications:
Transformers: BERT, GPT for clinical notes analysis
Vision models: Which parts of chest X-ray drove diagnosis?
Time-series: Which ICU monitoring data points triggered alert?
Example application: Radiology AI highlights suspicious regions in medical images using attention heatmaps.
Attention Visualization Example
import torchimport torch.nn as nnimport numpy as npimport matplotlib.pyplot as plt# Simple attention-based model for ICU time-series dataclass AttentionICU(nn.Module):def__init__(self, input_dim, hidden_dim):super().__init__()self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)# Attention mechanismself.attention = nn.Linear(hidden_dim, 1)self.classifier = nn.Linear(hidden_dim, 1)def forward(self, x):# x shape: (batch, time_steps, features) lstm_out, _ =self.lstm(x) # (batch, time_steps, hidden_dim)# Calculate attention scores attention_scores =self.attention(lstm_out) # (batch, time_steps, 1) attention_weights = torch.softmax(attention_scores, dim=1)# Apply attention (weighted sum of LSTM outputs) context = torch.sum(attention_weights * lstm_out, dim=1) # (batch, hidden_dim)# Final prediction output =self.classifier(context)return torch.sigmoid(output), attention_weights# Simulate ICU time-series data# Features: HR, BP, SpO2, RR over 24 hours (hourly measurements)torch.manual_seed(42)n_patients =100time_steps =24n_features =4X = torch.randn(n_patients, time_steps, n_features)y = torch.randint(0, 2, (n_patients, 1)).float() # Binary outcome# Train modelmodel = AttentionICU(input_dim=n_features, hidden_dim=32)criterion = nn.BCELoss()optimizer = torch.optim.Adam(model.parameters(), lr=0.001)# Training loop (simplified)for epoch inrange(50): optimizer.zero_grad() predictions, attention_weights = model(X) loss = criterion(predictions, y) loss.backward() optimizer.step()# ===== INTERPRET ATTENTION WEIGHTS =====# Explain a specific patientpatient_idx =0patient_data = X[patient_idx:patient_idx+1]prediction, attention = model(patient_data)print(f"Predicted risk: {prediction.item():.2%}")# Visualize attention over timeattention_np = attention.detach().numpy()[0, :, 0] # Shape: (time_steps,)plt.figure(figsize=(12, 4))plt.subplot(1, 2, 1)plt.plot(range(24), attention_np, marker='o')plt.xlabel('Hour')plt.ylabel('Attention Weight')plt.title('Which Time Points Were Most Important?')plt.axhline(1/24, color='r', linestyle='--', label='Uniform attention')plt.legend()# Identify critical time periodstop_hours = np.argsort(attention_np)[-3:][::-1]print(f"\nMost important time periods: Hours {top_hours}")print("Interpretation: Model focused on these specific hours when making prediction")# Overlay attention on vital signsplt.subplot(1, 2, 2)vitals = patient_data.detach().numpy()[0, :, 0] # Heart rateplt.plot(range(24), vitals, label='Heart Rate', alpha=0.7)plt.scatter(range(24), vitals, s=attention_np*1000, c='red', alpha=0.5, label='Attention (size = importance)')plt.xlabel('Hour')plt.ylabel('Heart Rate')plt.title('Attention-Weighted Vital Signs')plt.legend()plt.tight_layout()plt.savefig('attention_interpretation.png', dpi=150)plt.show()print("\nClinical interpretation:")print(f"The model identified hours {top_hours[0]}, {top_hours[1]}, {top_hours[2]} as critical.")print("Clinician should review events during these time windows.")
Key insight: Attention weights are built into the architecture, so they require no separate post-hoc explainer. They are not, however, a faithful measure of feature importance, as the limitations below show.
Limitations:
Attention ≠ causation
High attention does not guarantee that feature is truly important (attention is correlation)
Requires model architecture modification (cannot apply to existing black boxes)
Method 4: Counterfactual Explanations
What it is: An explanation that identifies what would need to change for the model to make a different prediction.
Example:
Prediction: Patient has 75% readmission risk
Counterfactual: “If patient had ≤2 prior admissions (currently 4) OR comorbidity count ≤3 (currently 5), risk would drop to <30%”
Uses:
Actionable: Shows which input changes would alter the prediction; whether a change is achievable or causal requires separate evidence
Patient-friendly: Easy to communicate (“If you lose 10 lbs, your risk decreases…”)
Fairness auditing: Reveals whether the model relies on unchangeable features (race, gender)
Counterfactual Example with DiCE
# Install: pip install dice-mlimport dice_mlfrom dice_ml import Diceimport pandas as pdfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_split# Load data (reuse readmission example from LIME section)# ... (same data generation code) ...# Train modelmodel = RandomForestClassifier(n_estimators=100, random_state=42)model.fit(X_train, y_train)# ===== COUNTERFACTUAL GENERATION =====# 1. Prepare DiCEdice_data = dice_ml.Data( dataframe=pd.concat([X_train, y_train], axis=1), continuous_features=['age', 'num_prior_admissions', 'length_of_stay','num_medications', 'comorbidity_count'], outcome_name='readmitted_30d')dice_model = dice_ml.Model(model=model, backend='sklearn')explainer = Dice(dice_data, dice_model, method='random')# 2. Generate counterfactuals for high-risk patientpatient_idx =5patient_df = X_test.iloc[[patient_idx]]# Find alternative scenarios where the patient would be predicted NOT to be readmittedcounterfactuals = explainer.generate_counterfactuals( query_instances=patient_df, total_CFs=3, # Generate 3 alternative scenarios desired_class='opposite'# Want opposite prediction)# 3. Display resultsprint("=== Original Patient ===")print(patient_df.T)print(f"\nPredicted outcome: Readmission (Risk: {model.predict_proba(patient_df)[0][1]:.2%})")print("\n=== Counterfactual Scenarios (Opposite Prediction) ===")cf_df = counterfactuals.cf_examples_list[0].final_cfs_dfprint(cf_df.T)# 4. Identify key changesprint("\n=== Key Changes Needed ===")for col in X_test.columns: original = patient_df[col].values[0]for i, cf in cf_df.iterrows():ifabs(cf[col] - original) >0.01: change = cf[col] - originalprint(f" {col}: {original:.1f} → {cf[col]:.1f} (change: {change:+.1f})")# 5. Clinical translation (hypotheses, not causal estimates)print("\n=== Candidate Interventions to Evaluate ===")print("Hypotheses for clinical review:")print(" • Reduce medication complexity (consolidate from 8 to ≤6 drugs)")print(" • Intensive post-discharge follow-up (reduce prior admit pattern)")print(" • Comorbidity management (focus on top 2-3 conditions)")
Output interpretation:
Original Patient: Readmission Risk = 64%
- Age: 71
- Prior admissions: 4
- Comorbidities: 5
- Medications: 8
Counterfactual Scenario 1: Risk = 18%
- Age: 71 (unchanged)
- Prior admissions: 1 (reduced from 4) ← Major change
- Comorbidities: 5 (unchanged)
- Medications: 6 (reduced from 8)
Interpretation: The prediction flips mainly when prior admissions fall.
Prior admissions are historical, so this shows what drives the model,
not which intervention would reduce readmission.
Need global understanding? → SHAP (global) or Feature Importance (trees only)
Need local explanation for specific patient? → SHAP (most robust) or LIME (faster)
Need actionable recommendations? → Counterfactuals
Using deep learning? → Attention mechanisms or SHAP
Real-time constraint? → LIME or Feature Importance
Regulatory submission? → Match explanation artifacts to the specific clinical oversight task and failure-mode analysis rather than assuming a single post-hoc attribution method satisfies review
Evaluating Whether Explanations Work
Explanations require their own evaluation against four criteria: fidelity, consistency, stability, and clinical validity.
Explainability Evaluation Criteria
1. Fidelity: Does the explanation accurately reflect the model’s behavior?
Test:
Remove high-importance features → prediction should change significantly
Flip low-importance features → prediction should stay similar
# Fidelity testoriginal_pred = model.predict_proba(patient_data)[0][1]# Remove most important feature (set to mean)modified_data = patient_data.copy()modified_data['lactate'] = X_train['lactate'].mean()modified_pred = model.predict_proba(modified_data)[0][1]print(f"Original prediction: {original_pred:.2%}")print(f"After removing 'lactate': {modified_pred:.2%}")print(f"Change: {abs(original_pred - modified_pred):.2%}")ifabs(original_pred - modified_pred) >0.1: # change > 10 percentage pointsprint("[OK] High fidelity: Explanation correctly identified important feature")else:print("[FAIL] Low fidelity: Feature removal didn't change prediction as expected")
2. Consistency: Do similar patients get similar explanations?
Test: Generate explanations for similar patients; feature importance rankings should be similar
3. Stability: Do explanations change substantially with small input perturbations?
Problem with LIME: Small changes to patient data can yield very different explanations
4. Clinical validity: Do domain experts agree the explanations make sense?
Gold standard: Clinician review
Do identified features align with medical knowledge?
Are there unexpected/spurious correlations?
Regulatory Perspectives on Explainability
FDA AI/ML SaMD Action Plan (2021)
The FDA’s AI/ML SaMD Action Plan emphasizes transparency to users and real-world performance monitoring as priorities for AI/ML-based SaMD (FDA, 2021).
Common transparency elements include:
Clear specification of intended use, input data requirements, and decision thresholds
Model limitations, known failure modes, and conditions under which clinicians should abstain from relying on outputs
Performance and calibration across demographic subgroups and clinical sites
EU AI Act (2024)
Transparency obligations for high-risk AI (includes medical AI):
Article 13 (Transparency):
High-risk systems must be sufficiently transparent for deployers to interpret and use outputs appropriately
Instructions must describe capabilities, limitations, expected accuracy, foreseeable misuse, and the information needed for human oversight
Practical implication: Article 13 does not prescribe SHAP, LIME, attention weights, or another universal explanation method. Documentation and user-facing information must support the specific system and oversight task.
Problem: SHAP/LIME identify correlations, not causal relationships.
Example:
Model assigns high importance to “hospital length of stay” for mortality prediction
Interpretation error: “Longer stays cause death”
Reality: Sicker patients stay longer; length of stay is a proxy for severity
Solution: Always validate explanations with clinical domain knowledge
Pitfall 2: Over-relying on Feature Importance
Problem: Global feature importance hides subgroup differences.
Example:
“Age” is most important feature globally (average across all patients)
But for young patients (<40), “comorbidities” might be more important
Solution: Examine SHAP dependence plots and subgroup-specific explanations
Pitfall 3: Ignoring Explanation Instability
Problem: LIME explanations can vary substantially between runs for the same patient and between similar patients.
Test:
# Generate 10 explanations for same patient (with different LIME seeds)explanations = []for seed inrange(10): explainer = lime.lime_tabular.LimeTabularExplainer( X_train.values, feature_names=X_train.columns.tolist(), mode='classification', random_state=seed ) exp = explainer.explain_instance(patient, model.predict_proba) explanations.append(exp.as_list())# Check consistency# If feature rankings vary significantly → unstable explanations
Solution: Prefer inherently interpretable models or stability-tested attribution methods when feature-level explanations are required for auditing, and never treat post-hoc feature rankings as causal clinical mechanisms
Pitfall 4: Explaining the Wrong Model
Problem: Explanations are generated from a simplified “surrogate” model instead of the actual production model.
Example:
Production: Complex ensemble of 50 models
Explanation: Generated from single decision tree approximation
Risk: Explanations do not reflect actual system behavior
Solution: Always explain the actual deployed model (even if slower)
Key Takeaways: Explainability
Operational transparency beats unverified post-hoc heatmaps: Clinicians need clear intended-use boundaries, calibration by subgroup, and validated failure modes rather than unvalidated per-case feature attributions that can induce overreliance
Multiple methods, multiple purposes: SHAP for robustness, LIME for speed, counterfactuals for action
Evaluate your explanations: Fidelity, consistency, clinical validity
Regulatory trend: Transparency and interpretable use are increasingly important, but no universal rule requires a post-hoc feature attribution for every prediction
Layer explanations by user: Patients need simple “why me?”; regulators need comprehensive validation
Correlation ≠ causation: Explanations show what the model uses, not necessarily what is clinically causal
Explainability does not fix a flawed model: If a model is biased or poorly validated, explanations can make the problems visible, which helps debugging, but do not correct them
Mechanistic Interpretability for Sequential Decision-Making
SHAP and LIME explain individual predictions, but public health AI increasingly involves sequential decision-making where current actions influence future states. Reinforcement learning (RL) systems for population health management require interpretability methods that expose reasoning pathways, not just feature importance.
Case study: Medicaid care coordination. A SARSA reinforcement learning system for Medicaid care management across two U.S. states (Virginia and Washington; 3,175 beneficiaries, 2023–2024) used a mixed-methods approach combining quantitative RL optimization with qualitative clinical validation. In counterfactual analysis, the system was estimated to reduce acute care events by 12 percentage points (NNT 8.3; 20.7% relative reduction) compared to standard practice, while also reducing race/ethnicity equalized odds disparity from 8.9% to 5.6% and gender disparity from 5.3% to 3.8% (Basu et al., 2025).
Implications for public health AI evaluation:
Sequential decision-making systems require interpretability methods beyond single-prediction explainers like SHAP and LIME
Mixed-methods validation (combining quantitative metrics with clinical expert review) provides stronger evidence than either approach alone
Fairness constraints can be integrated into RL optimization without large accuracy trade-offs in this study, suggesting that baseline disparities may stem from suboptimal calibration rather than fundamental accuracy-fairness tension
Tiered oversight (automated decisions for low-risk cases, human review for high-risk cases) is an emerging approach for balancing efficiency with safety
Regulatory Evaluation Frameworks
The Evolving Regulatory Landscape for Medical AI
Current reality: AI evaluation is no longer only a scientific question. It is also a regulatory requirement.
Medical AI systems face scrutiny from multiple regulatory bodies:
FDA (United States): Software as a Medical Device (SaMD) framework
European Union: AI Act (2024) classifying medical AI as “high-risk”
UK MHRA: Software and AI as a Medical Device framework
Health Canada: Medical Device Regulations for AI/ML
Key insight: Understanding regulatory evaluation requirements is essential for deployment, not just compliance.
Why Regulatory Context Matters for Evaluation
Even if you are not directly developing a regulatory-approved device, understanding these frameworks helps you:
Design better evaluations: Regulatory standards define best practices
Anticipate deployment requirements: Many health systems require FDA clearance or equivalent
Benchmark your work: Compare your validation against regulatory expectations
Communicate with stakeholders: Speak the language of hospital legal/compliance teams
Cross-reference:Policy and Governance covers broader policy implications; the requirements below are those specific to evaluation.
FDA Software as a Medical Device (SaMD) Framework
Software as a Medical Device (SaMD) refers to software intended for medical purposes that operates independently of hardware medical devices.
Not SaMD: EHR system (administrative), fitness tracker (wellness, not medical diagnosis)
Risk-Based Classification System
The International Medical Device Regulators Forum (IMDRF) framework, linked under Resources, places SaMD in risk categories I (lowest) to IV (highest) according to the state of the healthcare situation and the significance of the information the software provides. This categorization is separate from FDA device classification (Class I, II, or III), which determines the premarket pathway.
IMDRF SaMD Risk Categorization Matrix
State of healthcare
Significance of information
Treat or diagnose
Drive clinical management
Inform clinical management
Critical
IV (highest risk)
III
II
Serious
III
II
I
Non-serious
II
I
I (lowest risk)
Definitions:
Critical: Death or permanent impairment (e.g., ICU monitoring)
Serious: Long-term morbidity (e.g., cancer diagnosis)
Non-serious: Minor conditions (e.g., acne treatment)
Challenge: Traditional medical devices are static; ML models need updating.
FDA’s current framework (December 2024 final guidance): Predetermined Change Control Plans allow FDA-reviewed, pre-specified modifications to AI-enabled devices without separate marketing submissions for each covered update.
What can be included in a PCCP:
1. Allowable modifications:
Retraining on new data (within specified bounds)
Algorithm parameter adjustments
Performance improvements
2. Modification protocol:
Data requirements: Minimum sample size, quality standards
Performance thresholds: Must maintain ≥ X sensitivity
Validation approach: Test set size, external validation sites
3. Update assessment:
Performance monitoring triggers retraining
Validation results compared to pre-specified thresholds
Automated decision: deploy update or flag for review
4. Transparency and reporting:
Change documentation
Performance reports to regulators
User notifications
Example PCCP:
### Sepsis Prediction Model PCCP**Allowable Change:** Retrain model quarterly using new institutional data**Modification Protocol:**- Minimum 5,000 new patient encounters with ≥200 sepsis cases- Maintain AUC-ROC ≥ 0.82 (original validation: 0.85)- Sensitivity at 80% specificity must be ≥ 70%- External validation on ≥1 additional hospital required**Assessment:**- Automated performance monitoring (monthly)- Retraining triggered if AUC drops below 0.83- New model validated on hold-out test set (20% of new data)- Deploy if all thresholds met; otherwise, flag for manual review**Reporting:**- Quarterly performance reports to FDA- User notification of model updates via EHR alert
EU AI Act: High-Risk Classification for Medical AI
Entered into force 1 August 2024, with obligations applying in stages. Although the Regulation as originally adopted scheduled obligations for high-risk AI systems covered by EU harmonization legislation (Annex I, including the Medical Devices Regulation and In Vitro Diagnostic Medical Devices Regulation) for 2 August 2027, Regulation (EU) 2026/1744 amended the transition timeline so that high-risk obligations apply from 2 December 2027 for standalone Annex III systems and from 2 August 2028 for Annex I regulated medical devices.
Key classification: Most medical AI systems are high-risk, requiring:
1. Risk management system:
Continuous identification and mitigation of risks
2. Data governance:
Training data quality assurance
Bias detection and mitigation
Data representativeness
3. Technical documentation:
Detailed model specifications
Training procedures
Validation results
4. Transparency:
Users informed that AI is involved
Clear information on limitations
5. Human oversight:
Human-in-the-loop for high-stakes decisions
6. Accuracy, robustness, cybersecurity:
Performance standards
Adversarial robustness testing
Security safeguards
7. Post-market monitoring:
Continuous performance tracking
Incident reporting
Implications for evaluation:
Match validation populations, sites, and time periods to the intended use
Evaluate subgroup performance and foreseeable discriminatory effects
Test robustness and cybersecurity using methods appropriate to the threat model
Establish postmarket monitoring where the applicable legal and risk framework requires it
Even if your system is not currently seeking regulatory approval, aligning with these standards ensures quality:
Before Development:
During Development:
Validation Phase:
Pre-Deployment:
Post-Deployment:
Comparison: Research vs. Regulatory Evaluation Standards
Aspect
Research Publication
Regulatory Approval
External validation
Recommended, often skipped (6% in a 2019 review)
Expected for higher-risk devices; evidence requirements vary by pathway
Prospective testing
Rare
Often required for novel high-risk devices
Fairness audit
Increasingly expected
Mandatory (EU AI Act)
Post-market monitoring
Not required
Mandatory
Clinical utility
Recommended
Pathway-dependent; 510(k) requires substantial equivalence, not demonstrated outcome benefit
Documentation
Methods section
Extensive technical documentation
Timeline
Months
1–3+ years (depending on risk level)
Key insight:Regulatory standards are higher than typical research standards. If you aim for deployment, plan for regulatory-level evaluation from the start.
A 19 August 2026 PLOS Digital Health census of all 1,357 FDA-authorized AI/ML devices through 5 December 2025 bounds that “must demonstrate benefit” cell: only 34 (2.5%) had a registered prospective trial, 12 (0.9%) posted results, and 3 (0.2%) were evaluated on patient-centered outcomes such as mortality, morbidity, or readmissions (Abulibdeh et al., 2026). The 510(k) pathway requires substantial equivalence to a predicate, not an NDA-style outcome trial, so clearance is not demonstrated patient benefit. That 0.2% is a public-registry attrition count, not a care endpoint; the authors note the census may undercount manufacturer-held validation and does not mean FDA has no post-market surveillance.
When to Seek Regulatory Approval
You likely need FDA clearance/approval if:
System makes diagnostic or treatment recommendations
Used for screening (e.g., diabetic retinopathy, cancer)
Influences clinical decision-making significantly
Marketed as improving health outcomes
You might NOT need approval if:
Administrative tools (scheduling, billing)
Wellness apps (general fitness, not medical claims)
Clinical decision support providing information only (not recommendations)
Gray area: FDA discretion, often depends on risk
When in doubt: Consult FDA’s Digital Health Center of Excellence or regulatory counsel.
Regulatory requirements are evaluation requirements: FDA, EU standards define rigorous validation expectations
Risk-based approach: Evidence requirements should reflect intended use, the consequence of error, degree of autonomy, and difference between the evaluation setting and intended setting
Applicability must be demonstrated: Regulatory requirements vary by function and pathway; authorization does not substitute for evidence in the intended population and setting
Continuous monitoring is required: Post-market surveillance, not just pre-deployment validation
Good Machine Learning Practice: 10 principles provide practical framework for development and evaluation
Predetermined change control plans: Enable adaptive models while maintaining regulatory compliance
EU AI Act raises the bar: High-risk systems face requirements for risk management, data governance, transparency, human oversight, accuracy, robustness, cybersecurity, and postmarket monitoring. The evidence methods must fit the system and applicable conformity-assessment obligations.
Plan early: Regulatory evaluation takes longer and costs more than research validation; design for it from the start
Seek expert guidance: Regulatory pathways are complex; consult with regulatory specialists
Standards improve quality: Even if not seeking approval, regulatory frameworks represent best practices
Evaluation Framework
Complete Evaluation Checklist
Use this when evaluating AI systems:
AI System Evaluation Checklist
TECHNICAL PERFORMANCE
VALIDATION RIGOR
FAIRNESS AND EQUITY
CLINICAL UTILITY
TRANSPARENCY AND REPRODUCIBILITY
IMPLEMENTATION PLANNING
LIMITATIONS
Reporting Guidelines
TRIPOD: Transparent Reporting of Prediction Models
Current guidance: TRIPOD+AI superseded the 2015 TRIPOD checklist for clinical prediction model studies using regression or machine learning methods (Collins et al., 2024). It strengthens reporting of data sources, model development, evaluation, fairness, and open-science practices. TRIPOD+AI is a reporting guideline, not a method for rating evidence certainty or risk of bias; PROBAST+AI serves the latter appraisal role.
TRIPOD-LLM: Reporting Guidelines for LLM Studies
Traditional TRIPOD was designed for classical prediction models. Large language models require fundamentally different reporting standards. TRIPOD-LLM (Gallifant et al., 2025, Nature Medicine) provides the first LLM-specific extension, endorsed by EQUATOR Network.
Why LLM-specific guidelines are needed:
Training data for LLMs is often undisclosed or incompletely characterized
Performance varies dramatically with prompt wording
Contamination between training and test data is difficult to verify
Reproducibility requires specifying API versions, dates, and system prompts
Key TRIPOD-LLM additions:
Category
New Requirements
Model specification
API version, access date, temperature/sampling parameters
Prompting
Full prompt text, development process, any prompt optimization
Input/output
How health data was formatted, output parsing methods
Reproducibility
Whether results vary across API calls, random seed handling
Contamination
Assessment of whether test data appeared in training corpus
The LLM reporting crisis:
A systematic review of LLM chatbot health advice studies (Huo et al., 2025, JAMA Network Open) examined 137 studies and found that 136 studies (99.3%) used closed-source models. Key findings:
No study sufficiently described model characteristics such as temperature and token length
136 studies (99.3%) did not describe a prompt engineering phase
The date of model querying was reported in only 54 studies (39.4%)
These reporting gaps prevent faithful reproduction of most studies. TRIPOD-LLM directly addresses these gaps.
Complementing TRIPOD-LLM, the CHART statement (Huo et al., 2025, JAMA Network Open) provides a 12-item, 39-subitem checklist specifically for evaluating chatbot and conversational AI systems in healthcare settings.
STARD-AI: Standards for Reporting Diagnostic Accuracy Using AI
Clinical utility ≠ statistical performance: A model can be statistically accurate but clinically useless without improving outcomes
Prospective evaluation answers operational questions: Live testing can assess data pipelines, workflow, timeliness, and human factors; comparative evidence is needed for causal claims about impact
Common pitfalls are avoidable: Data leakage, improper CV, threshold optimization on test set lead to overoptimistic estimates
Implementation determines success: Even well-performing models fail if workflow integration ignored
Transparency enables trust: Follow reporting guidelines (TRIPOD, STARD-AI); share code and data when possible
Continuous monitoring is essential: Model performance drifts over time; plan for ongoing evaluation and updating
Check Your Understanding
Test your knowledge of key evaluation concepts. Click “Show Answer” to reveal the correct response and explanation.
Question 1: Cross-Validation Strategy
You are building a model to predict hospital readmissions using data from 2018–2023. Which cross-validation strategy is MOST appropriate?
This uses 2023 data to predict 2018, which inflates performance.
Why not leave-one-out (b)?
Computationally expensive
Still has temporal leakage problem
High variance in estimates
Why not stratified K-fold (c)?
Useful for class imbalance
Still allows temporal leakage
Does not test temporal robustness
Real-world impact: Random cross-validation can overestimate forward performance when temporal drift exists because it allows later observations to inform validation of earlier observations.
Lesson: Healthcare data has temporal structure. Always validate the way the model will be deployed: use past data to predict the future, never the reverse.
Question 2: Calibration Assessment
A cancer risk model predicts 20% risk for 1,000 patients. In reality, 300 of these patients develop cancer. What does this indicate?
The model is well-calibrated
The model systematically underestimates risk
The model systematically overestimates risk
The model has good discrimination but poor calibration
Answer: b) The model systematically underestimates risk
Explanation:Calibration compares predicted probabilities to observed outcomes:
Analysis:
Predicted: 20% of 1,000 patients = 200 patients expected to develop cancer
Overconfidence differs: predictions are too extreme, shown by a calibration slope below 1
Why it matters:
# Clinical decision: Treat if risk > 25%model.predict_proba(patient) =0.20# Below threshold → No treatment# Reality: True risk was 0.30# Patient should have been treated.
How to detect:
Calibration plot: Predicted vs observed by risk bin
Brier score: Mean squared error of probabilities
Expected Calibration Error (ECE): Average absolute calibration error
Lesson: High AUC does not guarantee calibration. When predictions inform decisions with probability thresholds, calibration is critical. Always check calibration plots, not just discrimination metrics.
Question 3: External Validation
Your sepsis model achieves AUC 0.88 on the internal test set. You test on external hospitals and get AUC 0.72–0.82 (varying by site). What does this variability indicate?
The model is overfitting
External sites have poor data quality
There is substantial site-specific heterogeneity
The model should not be used
Answer: c) There is substantial site-specific heterogeneity
Explanation:Performance variability across sites reveals important heterogeneity:
What varies between hospitals:
Patient populations:
Demographics (age, race, socioeconomic status)
Disease severity (tertiary referral vs community hospital)
Comorbidity profiles
Clinical practices:
Sepsis protocols (early vs delayed antibiotics)
ICU admission criteria
Documentation practices
Infrastructure:
EHR systems (Epic vs Cerner vs homegrown)
Lab equipment (different reference ranges)
Staffing models (nurse-to-patient ratios)
Data capture:
Missing data patterns
Measurement frequency
Feature definitions
Why not overfitting (a)? Overfitting appears as a gap between training and test performance within the same dataset. Here, internal test performance was strong (0.88); external generalization is what varies.
Why not poor data quality (b)? Data quality could contribute, but the variation more likely reflects legitimate differences in populations and practices.
Why not unusable (d)? An AUC of 0.72–0.82 is still useful but indicates a need for:
Site-specific calibration
Understanding what drives differences
Possibly site-specific models or adjustments
Best practice: External validation commonly shows lower performance than internal testing. Variability across sites is normal and informative: it reveals where the model struggles and needs adaptation.
Question 4: Statistical Significance vs Clinical Significance
True or False: If a model improvement is statistically significant (p < 0.05), it is clinically meaningful and should be deployed.
Answer: False
Explanation:Statistical significance ≠ clinical significance. Both are necessary but neither alone is sufficient:
Statistical significance:
Tests whether a difference is unlikely due to chance
Depends on sample size (large N → small differences become significant)
Answers: “Is there an effect?”
Clinical significance:
Tests whether a difference matters for patient care
Independent of sample size
Answers: “Is the effect large enough to matter?”
Example:
# New model vs baselineresults = {'baseline_auc': 0.820,'new_model_auc': 0.825,'difference': 0.005,'p_value': 0.03, # Statistically significant'sample_size': 50000# Large sample}
Analysis:
Statistically significant: p=0.03 < 0.05
Clinically insignificant: an AUC improvement of 0.005 is negligible
Why significant? A large sample detects tiny differences
Should deploy? No; the gain does not justify the cost and disruption
Lesson: Always evaluate both statistical and clinical significance. With large samples, trivial differences become statistically significant. Ask: “Is this improvement large enough to change practice?” Consider effect sizes, confidence intervals, and practical impact, not just p-values.
Question 5: Confidence Intervals
Two models have been evaluated:
Model A: AUC 0.85 (95% CI: 0.83–0.87)
Model B: AUC 0.86 (95% CI: 0.79–0.93)
Which statement is correct?
Model B is definitely better because it has higher AUC
Model A’s AUC estimate is more precise, and Model B is not shown to be better
The models cannot be compared without more information
Model B is better if you’re willing to accept more uncertainty
Answer: b) Model A’s AUC estimate is more precise, and Model B is not shown to be better
Explanation:Confidence intervals reveal precision/uncertainty, not just point estimates:
Model A:
AUC: 0.85
95% CI: 0.83–0.87
Width: 0.04 (narrow)
Interpretation: The true AUC plausibly lies between 0.83 and 0.87 (precise estimate)
Model B:
AUC: 0.86
95% CI: 0.79–0.93
Width: 0.14 (wide)
Interpretation: The true AUC plausibly lies between 0.79 and 0.93 (imprecise estimate)
Key insight: CIs overlap substantially (0.83–0.87 vs 0.79–0.93). One cannot conclude Model B is actually better; the difference might be due to chance.
In practice:Model A is often the more practical choice:
Predictable performance for planning
Lower risk of underperformance
Easier to set appropriate thresholds
Small gain (0.01 AUC) not worth the uncertainty
Lesson: Always report and consider confidence intervals, not just point estimates. Narrow CIs indicate a precise performance estimate. Wide CIs indicate uncertainty: performance might be much worse (or better) than point estimate suggests.
One more wrinkle: the comparison above treats Model A and Model B as independent estimates, each with its own confidence interval. If both models were actually scored on the identical set of cases, that independence assumption is wrong, and a paired-difference analysis (using the correlation between the two models’ scores on each shared case) is the statistically correct comparison rather than checking whether two separate confidence intervals overlap. A paired analysis on the same question set is typically more powerful than the unpaired comparison shown here, meaning it could resolve a small gap that overlapping CIs alone leave undecided (Miller, 2024, preprint).
Question 6: Subgroup Analysis
You evaluate a diagnostic model and find:
Overall AUC: 0.84
Men: AUC 0.88
Women: AUC 0.78
What should you do?
Report only overall performance (0.84)
Report overall performance but note subgroup differences exist
Investigate why performance is lower for women and consider separate models or adjustments
Conclude the model is biased and should not be used
Answer: c) Investigate why performance is lower for women and consider separate models or adjustments
Explanation:Subgroup performance disparities require investigation and action, not just reporting:
Possible reasons for the difference:
Biological differences:
Disease presents differently (atypical symptoms in women)
Different physiological reference ranges
Example: Heart attack symptoms differ by sex
Data representation:
Fewer women in training data → model learns men’s patterns better
Women may be underdiagnosed historically → labels biased
Feature appropriateness:
Features optimized for men
Missing features relevant for women
Example: Pregnancy-related factors not included
Measurement bias:
Tests/measurements less accurate for women
Different documentation patterns
Potential solutions:
Collect more data from women (if sample size is the issue)
Add sex-specific features:
# Include pregnancy status, hormonal factorsfeatures += ['pregnant', 'menopause_status', 'hormone_therapy']
Stratified modeling:
# Separate models for men/womenif patient.sex =='M': prediction = model_men.predict(patient)else: prediction = model_women.predict(patient)
Weighted loss function:
# Penalize errors on women more heavily during trainingsample_weights = [2.0if sex=='F'else1.0for sex in data['sex']]model.fit(X, y, sample_weight=sample_weights)
Lesson: Subgroup analysis is mandatory. When disparities are found, investigate root causes and take corrective action. Do not hide disparities in overall metrics.
Question 7: LLM Evaluation Strategy
You are deploying an LLM to summarize public health literature for practitioners. Which evaluation approach is MOST appropriate?
Calculate AUC-ROC on a test set of summaries
Measure perplexity (how surprised the model is by correct summaries)
Use BERTScore to compare generated summaries against expert-written reference summaries, plus human expert evaluation
Only use BLEU score (n-gram overlap with reference summaries)
Answer: c) Use BERTScore to compare generated summaries against expert-written reference summaries, plus human expert evaluation
Why:
Correct approach:
BERTScore captures semantic similarity better than simple n-gram matching (BLEU)
a) AUC-ROC: Not applicable; LLMs generate text, not binary classifications or probabilities
b) Perplexity alone: Measures fluency, not accuracy/relevance (fluent nonsense scores well)
d) BLEU only: Too limited; high BLEU does not guarantee accurate or clinically appropriate summaries
Best practice for LLM summarization:
Automated metrics (BERTScore, ROUGE) for efficiency
Human expert review of a sample (100+ summaries)
Safety audit (check for hallucinations, dangerous errors)
Prompt robustness testing (consistency across variations)
Lesson: LLM evaluation requires different approaches from those used for traditional ML. Automated metrics alone are insufficient; human expert evaluation is mandatory, especially for clinical applications.
Question 8: Model Drift Detection
You are monitoring a deployed readmission prediction model. Over 6 months, you observe:
AUC-ROC: Stable at 0.82 (was 0.83 at deployment)
Brier score: Increased from 0.15 to 0.21
Patient age distribution: Mean shifted from 58 to 65 years
PSI for age feature: 0.28
What does this indicate, and what should you do?
Model is fine, AUC is stable; continue monitoring
Significant data drift occurred; retrain immediately
Concept drift occurred; model is failing and needs urgent retraining
Calibration degraded but discrimination stable; recalibrate or retrain
Answer: d) Calibration degraded but discrimination stable; recalibrate or retrain
Analysis:
What happened:
AUC-ROC stable: Model can still distinguish high-risk from low-risk patients (discrimination intact)
Brier score increased: Predicted probabilities are inaccurate (calibration degraded)
Age distribution shifted: PSI = 0.28 indicates significant data drift (threshold: PSI > 0.25)
Likely cause: Data drift (patient population aging) → calibration degrades even if model’s relative ranking ability (AUC) persists
Why other options are wrong:
a) Model is fine: Incorrect; Brier score degradation and high PSI require action
b) Data drift, retrain immediately: Partially correct but oversimplified; recalibration could come first (faster, simpler)
c) Concept drift: Unlikely; AUC would be expected to degrade if the relationship between features and outcome changed, and this pattern looks like data drift affecting calibration
What to do:
Immediate (within 1 week):
Recalibrate model on recent data (faster than full retraining)
Test calibration on recent hold-out set
Deploy recalibrated version if performance restored
Scheduled (within 1 month):
Full model retraining recommended (PSI > 0.25 for critical feature)
Validate retrained model on hold-out set
Compare retrained vs. recalibrated performance
Deploy better-performing version
Monitoring:
Track Brier score monthly (calibration early warning)
Track PSI for all critical features
Set alert: PSI > 0.25 for ≥3 features = immediate retraining
Lesson: Different types of drift require different interventions. Data drift may degrade calibration while preserving discrimination. Monitor multiple metrics (not just AUC) to catch drift early.
Question 9: Regulatory Classification
You are developing an AI system that analyzes electronic health records to flag patients at high risk for cardiovascular disease in the next 5 years. Clinicians review flagged patients and decide whether to prescribe statins. Under the IMDRF framework for software as a medical device (SaMD) risk categorization, which category fits this system?
Not a medical device (wellness/administrative use)
SaMD Category I (lowest risk)
SaMD Category II
SaMD Category III
Answer: c) SaMD Category II
Classification reasoning:
FDA SaMD Matrix:
State of healthcare situation: Serious (cardiovascular disease can cause long-term morbidity)
Fits the IMDRF definition of SaMD (software intended for a medical purpose)
b) Category I: WRONG
CVD is not a “non-serious” condition
System does more than “inform”: it drives treatment decisions
d) Category III: WRONG
Not a critical (immediately life-threatening) condition; critical examples include ICU monitoring or acute MI
Not treating/diagnosing directly; clinicians make the final decision
If the system autonomously prescribed statins → Category III
Evaluation implications for Category II:
Expected:
Rigorous internal validation
Clinical utility assessment (does it actually improve outcomes?)
Usability testing with clinicians
Performance monitoring plan (drift detection)
Recommended:
External validation
A predetermined change control plan for planned model updates
More often warranted in higher categories:
Randomized controlled trial
Extensive multi-site prospective validation
Borderline considerations:
If the system provided lower-stakes information (e.g., “Consider discussing lifestyle changes”) → Could be Category I
If the condition were critical/life-threatening (e.g., predict sepsis, guide ICU ventilator settings) → Category III
Lesson: SaMD risk category depends on both the state of the healthcare situation and the significance of the information. “Driving clinical management” for a serious condition = Category II. National regulators, including the FDA, apply their own device classes. Understanding the risk category early helps plan appropriate validation rigor.
Question 10: Adversarial Robustness Testing
You are evaluating a chest X-ray pneumonia detection model. Which robustness test is MOST important for real-world deployment?
Fast Gradient Sign Method (FGSM) adversarial attack testing
Model extraction attack testing (preventing reverse engineering)
Data poisoning resilience testing (can the training set be corrupted?)
Answer: b) Natural perturbation testing (image compression, scanner variations, noise)
Reasoning:
Real-world threat model:
Natural perturbations occur constantly: Different X-ray machines, image compression algorithms, patient positioning variations, image quality differences across sites
Likelihood: Natural variation is expected in deployment; its frequency and magnitude depend on the sites, equipment, acquisition protocols, and populations
Impact if not tested: Model may fail on images from different hospitals/equipment, limiting generalizability
Why other options are less critical (though still valuable):
a) FGSM adversarial attacks:
Likelihood: Low; malicious adversarial attacks on deployed medical imaging systems are not widely documented
Value: Of research interest and possibly required under the EU AI Act, but not the most pressing real-world concern
When important: High-profile systems, regulatory compliance (EU AI Act)
c) Model extraction:
Risk: Intellectual property theft, but does not directly harm patients
Mitigation: API rate limiting, access control (non-evaluation solutions)
d) Data poisoning:
Risk: Rare unless using crowdsourced/untrusted training data
Prevention: Data provenance, quality control during training (not post-deployment testing)
Out-of-distribution detection: Flag unfamiliar images (different scanner types, anatomies)
Multi-site external validation: Real-world test of natural robustness
Recommended (high-risk/regulatory):
Adversarial attack testing: FGSM, PGD when the threat model or applicable assurance requirements warrant it
Ablation studies: Performance with missing/corrupted inputs
Advanced (specific threats):
Data poisoning resilience: If using federated learning or external data
Model extraction prevention: If protecting proprietary models
Example test:
# Test robustness to JPEG compression (natural perturbation)import cv2compression_qualities = [100, 90, 70, 50, 30]for quality in compression_qualities:# Compress image _, compressed = cv2.imencode('.jpg', image, [cv2.IMWRITE_JPEG_QUALITY, quality]) compressed_img = cv2.imdecode(compressed, cv2.IMREAD_COLOR)# Test model prediction = model.predict(compressed_img)print(f"Quality {quality}: Prediction = {prediction:.3f}")# Example local review thresholds, not universal standards:# investigate when prediction changes exceed the prespecified tolerance
Lesson: Prioritize robustness testing based on real-world threat likelihood. For medical imaging, natural perturbations (equipment variation) are far more common than adversarial attacks. Test what will actually break your model in practice.
Frequently Asked Questions
What is external validation in health AI?
External validation evaluates a locked model on data that differ meaningfully from development data, such as a new institution, geography, device, population, or time period. The test set should represent the intended transfer and should not be used repeatedly for model selection. External validation addresses transportability for the measured endpoint. It does not by itself establish workflow benefit or patient outcomes.
What is calibration, and why does it matter?
Calibration describes whether predicted probabilities correspond to observed frequencies in the target setting. A well-discriminating model can still systematically overestimate or underestimate risk, especially when prevalence or practice changes. Calibration should be examined overall and in relevant subgroups, with uncertainty. Recalibration may improve probabilities but does not repair a wrong target, missing predictors, or an ineffective intervention.
How should evidence be matched to a public health AI claim?
Technical performance claims require a suitable reference standard and independent test data. Transportability claims require external validation. Workflow claims require prospective use in the intended setting. Outcome-benefit claims require a comparative design measuring the relevant outcome. Economic claims require explicit costs, perspective, time horizon, and comparator. No single study design answers every question, so the claim should be narrowed to what the design and endpoint measured.
What are vendor-evaluation red flags?
Red flags include an undisclosed model version, no independent test set, performance reported only as accuracy or AUC, missing confidence intervals, absent subgroup results, an inappropriate comparator, results from a different workflow, and no monitoring or update plan. Regulatory authorization, publication, or deployment elsewhere should not substitute for local evidence. A vendor should be able to provide a connected record from the claim to the source and exact product version.
Discussion Questions
Claim-specific evidence plan: You’ve developed a hospital-acquired infection prediction model. Which evidence would you require for technical performance, transportability, operational utility, population impact, equity, and lifecycle safety? Which studies would you conduct before deployment, and why?
Fairness trade-offs: Your sepsis model has AUC-ROC = 0.85 overall, but sensitivity is 0.90 for White patients vs. 0.75 for Black patients. What would you do? What are the trade-offs of different mitigation approaches?
Calibration vs. discrimination: Model A: AUC-ROC = 0.85, Brier score = 0.30 (poor calibration). Model B: AUC-ROC = 0.80, Brier score = 0.15 (excellent calibration). Which would you deploy, and why?
External validation failure: Your model achieves AUC-ROC = 0.82 in internal validation but 0.68 in external validation at a different hospital. What explains this? What are the next steps?
Clinical utility skepticism: Model predicts 30-day mortality with AUC-ROC = 0.88. Does this mean it’s clinically useful? What additional evaluations are needed?
Prospective study design: A hospital readmission model needs prospective evaluation. Would you choose an RCT, a stepped-wedge design, or silent mode? What are the trade-offs?
Alert threshold selection: A clinical decision support tool can alert at >10%, >20%, or >30% predicted risk. How would you choose? What factors matter?
Model drift: COVID-19 forecasting model trained on 2020 data; it is now three years later, with new variants. How would you assess whether it is still valid? What should trigger retraining?
AI Deployment in Healthcare: How to move validated models into production, monitor for drift, and manage the organizational change that determines real-world success
AI Safety in Healthcare: Risk frameworks, failure mode analysis, and when to pull a deployed model out of production
AI Ethics, Bias, and Equity: Deeper coverage of fairness metrics, bias mitigation strategies, and equity considerations across populations