A high validation score does not establish reliability in public health practice. A widely implemented sepsis model had 33% sensitivity and 12% positive predictive value at the evaluated threshold in external validation (Wong et al., 2021). Adoption decisions therefore require evidence matched to the claim, intended use, population, setting, and consequences of error.
Learning Objectives
This chapter addresses the evaluation crisis in AI deployment. You will learn to:
Match evidence requirements to technical performance, transportability, operational utility, population impact, and lifecycle safety claims
The Big Picture: In a review of 516 medical-imaging AI studies, 31 used external validation (Kim et al., 2019). In an external evaluation of a widely implemented sepsis model, sensitivity was 33% and positive predictive value was 12% at the evaluated threshold (Wong et al., 2021). Technical performance in one setting cannot establish transportability, operational utility, population impact, equity, or lifecycle safety.
Transportability: Independent temporal, geographic, and multi-site evaluation in relevant populations and data systems
Operational utility: Prospective evaluation of data pipelines, timeliness, workflow, workload, and failure modes
Population impact: Comparative evidence on decisions, benefits, harms, equity, and resource use
Lifecycle safety: Ongoing monitoring for drift, failures, inequitable effects, and unintended consequences
No study label answers every question. The design must match the claim and deployment risk.
Beyond Accuracy: What Really Matters
Clinical Utility: Does it change decisions? Improve outcomes? Integrate into workflows?
Generalization: CheXNet-style models AUC=0.93 internally, dropped to 0.82 at external hospitals (Zech et al., 2018, PLOS Medicine). Beware the generalization gap
Fairness Across Subgroups: Does model perform equally for different races, ages, sexes, socioeconomic groups?
Implementation Outcomes: Adoption rate, alert fatigue, workflow disruption, user trust
Common Evaluation Pitfalls:
No External Validation: Tested only on holdout from same dataset
Cherry-Picked Subgroups: “Works great on images rated as ‘excellent quality’” (real-world images are messy)
Ignoring Prevalence Shift: Trained on 50% disease prevalence, deployed where prevalence is 5%
Overfitting to Dataset Quirks: Model learns hospital-specific artifacts, not disease
Evaluation-Treatment Mismatch: Evaluate on diagnosed cases, deploy for screening
NEW for 2025: Evaluating Foundation Models and LLMs
Traditional ML metrics (accuracy, AUC) insufficient for large language models:
Factual Accuracy: Does model provide correct medical information?
Hallucination Detection: How often does it confidently generate false information?
Prompt Sensitivity: Does small rewording change answers dramatically?
Deployment is not the end. Models degrade over time:
Data Drift: Input distributions change (e.g., demographics shift, new disease variants)
Concept Drift: Relationship between features and outcome changes
Label Drift: Definition of outcome evolves
Detection Methods: Population Stability Index (PSI), statistical process control charts
Retraining Triggers: Predetermined thresholds for when performance drops require model updates
NEW for 2025: Regulatory Frameworks
FDA SaMD (Software as Medical Device): Risk-based classification (I, II, III). Higher risk = more rigorous validation
Good Machine Learning Practice (GMLP): Industry standards for development, validation, monitoring
EU AI Act: High-risk medical AI requires conformity assessment, transparency, human oversight, continuous monitoring
Key Insight: Even non-regulated systems benefit from regulatory-level evaluation rigor
NEW for 2025: Adversarial Robustness
Natural Perturbations: Small changes in image brightness, patient demographics. Does model break?
Adversarial Attacks: Intentionally crafted inputs to fool model (FGSM, PGD attacks)
Out-of-Distribution (OOD) Detection: Can model recognize when input is unlike training data?
EU AI Act Requirement: High-risk systems must demonstrate robustness testing
The Obermeyer Lesson:
Healthcare cost algorithm had excellent accuracy but systematic inequity: Black patients had to be sicker than White patients to receive same risk score. Lesson: Technical performance ≠ ethical deployment. Must evaluate fairness explicitly.
Red flags that should halt deployment: 1. External validation shows poor generalization 2. Fairness audit reveals systematic bias 3. Clinical workflow integration causes more harm than benefit (alert fatigue) 4. Users do not trust or adopt the system 5. No plan for continuous monitoring and maintenance
The Takeaway for Public Health Practitioners:
Evaluation is not a checkbox. Internal validation estimates performance under development conditions. Independent validation tests transportability. Prospective studies examine live data pipelines, workflow, and human factors. Comparative designs test whether using the system changes decisions or outcomes. None substitutes for post-deployment monitoring. For LLMs and foundation models, evaluation must also address hallucinations, prompt sensitivity, retrieval quality, and safety in the intended use. The central task is to define the claim first, choose a design that can answer it, and appraise risk of bias and applicability before acting on the result.
Check Your Understanding
Test whether you can match an evaluation result to the claim it supports:
1. Same-source holdout: A research team reports 94% accuracy for a pneumonia detection model on a held-out sample from the same hospital and period used for development. What does this result support? - A. Internal performance under the study conditions - B. Transportability to other hospitals - C. Effectiveness in a live workflow - D. Improved patient outcomes
Click for answer
Answer: A. Internal performance under the study conditions
Why: The evaluation can estimate performance under the sampled development conditions if leakage and analysis bias are controlled. It does not establish transportability, workflow performance, or an effect on outcomes.
2. The Epic Lesson: In an external evaluation, Epic’s sepsis model had 33% sensitivity and 12% PPV at the evaluated threshold. What does this mean practically? - A. Model missed 2 out of 3 sepsis cases; 88% of alerts were false alarms - B. Model worked great, 33% and 12% are good metrics - C. Model needs minor tuning to reach production quality - D. External validation was too strict
Click for answer
Answer: A. Missed 2 of 3 cases; 88% false alarms
Why: 33% sensitivity means the model detected approximately 1 in 3 sepsis cases. A 12% PPV means approximately 12 of 100 alerts identified a case. These figures indicate substantial missed-case and false-alert burdens at that threshold. Deployment status does not establish acceptable performance.
3. LLM Evaluation: You’re evaluating a medical chatbot powered by an LLM. It scores 85% on MedQA (medical exam questions). Can you deploy it? - A. Yes, 85% accuracy is excellent - B. No, must also test hallucination rate, prompt robustness, safety - C. No, need prospective clinical validation - D. Both B and C
Click for answer
Answer: D. Both B and C
Why: Benchmark performance (MedQA) ≠ clinical competence. LLMs can ace exams but hallucinate dangerous medical advice. Must test: hallucination detection, prompt sensitivity (small rewording changes answer?), safety (harmful advice?), and prospective validation in real clinical workflow before deployment.
4. Model Drift: Your outbreak prediction model performed well for 2 years. Suddenly accuracy drops from 82% to 61%. What’s likely happening? - A. The model is broken, rebuild from scratch - B. Data drift: input distributions changed (new disease variant, demographic shifts) - C. Model was always bad, just got lucky initially - D. Evaluation metrics are wrong
Click for answer
Answer: B. Data drift
Why: Sudden performance drops indicate data/concept drift. New disease variants, demographic shifts, changes in testing practices can make training data unrepresentative. This is why continuous monitoring and retraining triggers are essential. Models degrade over time in production.
5. When NOT to Deploy: Your model has 91% AUC internally, 87% AUC at 3 external hospitals. Fairness audit shows White patients: 90% sensitivity, Black patients: 65% sensitivity. Should you deploy? - A. Yes, 87% external AUC is strong - B. No, systematic bias is unacceptable even with good overall performance - C. Yes, but only for White patients - D. Deploy and monitor fairness post-deployment
Click for answer
Answer: B. No, systematic bias is unacceptable
Why: Technical performance ≠ ethical deployment. This is the Obermeyer lesson: excellent accuracy but systematic inequity. Black patients receive worse care because model systematically underperforms. Must address fairness BEFORE deployment, not after. Option D (“monitor after deployment”) puts patients at risk while you collect evidence of harm. Halt deployment until bias is addressed.
Scoring: - 5/5: Excellent! You understand evaluation rigor. Ready to evaluate AI systems critically. - 3-4/5: Good foundation. Review sections where you missed questions, especially Epic sepsis case study. - 0-2/5: Reread the TL;DR summary and the Introduction section. The evaluation crisis is the most important concept in this chapter.
Introduction: The Evaluation Crisis in AI
March 2019, Korean Journal of Radiology:
Kim et al. publish a systematic review examining 516 studies on AI algorithms for medical image analysis.
Their sobering finding: Only 6% performed external validation (31 of 516 studies) on data from different institutions.
The vast majority tested models only on hold-out sets from the same dataset used for training, a practice that provides minimal evidence of real-world performance.
The model’s performance: - Sensitivity: 33% (missed 2 out of 3 sepsis cases) - Positive predictive value: 12% (88% of alerts were false positives) - Alert burden: 18% of hospitalizations would have generated at least one alert at the evaluated threshold
Authors’ conclusion: The model had poor discrimination and calibration, and its alert burden raised concerns about clinical value.
The evaluation was a retrospective cohort study at one academic health system. Its importance is that the proprietary model was already in widespread clinical use before independent validation was published.
The Evaluation Gap:
Between lab performance and real-world deployment lies a chasm that has claimed many promising AI systems. Clinical AI evaluation currently resembles standardized testing more than real-world practice: retrospective accuracy on curated datasets, with limited measurement of workflow fit, adoption, safety guardrails, or downstream outcomes (Azad et al., Nature Medicine, 2026).
In the real world: - Messy, incomplete data - Rare events (1-5% prevalence) - Variable protocols across sites - Ambiguous cases - AUC-ROC = 0.68
Performance ≠ Safety
This chapter focuses on evaluating model performance: accuracy, generalization, fairness, and robustness. However, high performance does not guarantee safe clinical deployment.
For comprehensive safety evaluation beyond performance metrics, see AI Safety in Healthcare.
The consequences are severe:
Failed deployments: Models that work in development but fail in production Hidden biases: Systems that perform well on average but poorly for specific groups Wasted resources: Millions invested in systems that do not deliver promised benefits Patient harm: Incorrect predictions leading to inappropriate treatments Eroded trust: Clinicians lose confidence in AI after experiencing failures
The Adoption-Value Gap:
McKinsey’s 2025 cross-industry State of AI survey included 1,993 participants across 105 countries. Its organizational findings provide context but do not establish public health adoption or value rates:
88% of organizations report using AI in at least one function (up from 78% in 2024)
Yet only 6% qualify as “high performers” achieving 5%+ earnings impact from AI
Only 7% have fully scaled AI across their organizations
Five recurring barriers prevent organizations from crossing this gap:
Data quality issues: Fragmented systems, inconsistent metadata, accuracy problems
Financial justification difficulty: Inability to demonstrate measurable long-term gains
Skills shortage: Insufficient data scientists, engineers, and change-management expertise
Organizational silos: Lack of cross-functional collaboration
Governance uncertainty: Evolving privacy regulations and security concerns
The lesson for public health: Deployment is not the finish line. Organizations that treat AI as technology procurement rather than workflow transformation consistently underperform. The 6% who succeed invest in data infrastructure, workforce training, and governance frameworks before deploying AI tools.
Why This Chapter Matters
Rigorous evaluation is the bridge between AI research and AI implementation. Without it, we’re deploying unvalidated systems and hoping for the best.
Evaluate AI systems across five critical dimensions:
You’ll learn how to evaluate AI systems rigorously, design validation studies, and critically appraise published research.
The Multidimensional Nature of Evaluation
What Are We Really Evaluating?
Evaluating an AI system is not just about measuring accuracy. In public health and clinical contexts, we need to assess multiple dimensions.
1. Technical Performance
Question: Does the model make accurate predictions on new data?
Key aspects: - Discrimination: Can the model distinguish between positive and negative cases? - Calibration: Do predicted probabilities match observed frequencies? - Robustness: Does performance degrade with missing data or noise? - Computational efficiency: Speed and resource requirements for deployment
Relevant for: All AI systems (foundational requirement)
2. Generalizability
Question: Will the model work in settings different from where it was developed?
Key aspects: - Geographic transferability: Performance at different institutions, regions, countries - Temporal stability: Does performance degrade as time passes and data distributions shift? - Population differences: Performance across different patient demographics, disease prevalence - Setting transferability: Hospital vs. primary care vs. community settings
Relevant for: Any system intended for broad deployment
Critical insight:Geirhos et al., 2020, Nature Machine Intelligence showed that AI models often learn “shortcuts”, spurious correlations specific to training data that do not generalize. For example, pneumonia detection models learned to identify portable X-ray machines (used for sicker patients) rather than actual pneumonia.
3. Clinical/Public Health Utility
Question: Does the model improve decision-making and outcomes?
Key aspects: - Decision impact: Does it change clinician decisions? - Outcome improvement: Does it lead to better patient outcomes? - Net benefit: Does it provide value above existing approaches? - Cost-effectiveness: Does it provide value commensurate with costs?
Critical distinction:A model can be statistically accurate but clinically useless. Example: A model predicting hospital mortality with AUC-ROC = 0.85 sounds impressive, but if it does not change management or improve outcomes, it adds no value.
Question: Does the model perform equitably across population subgroups?
Key aspects: - Subgroup performance: Stratified metrics by race, ethnicity, gender, age, socioeconomic status - Error rate disparities: Differential false positive/negative rates - Outcome equity: Does deployment narrow or widen health disparities? - Representation: Are all groups adequately represented in training data?
Question: Is the model adopted and used effectively in practice?
Key aspects: - Adoption: Are users actually using it as intended? - Usability: Can users operate it efficiently? - Workflow integration: Does it fit smoothly into existing processes? - Sustainability: Will it continue to be used and maintained over time?
Evidence for public health AI is multidimensional, not a single ladder. The study design should be judged against the question it is intended to answer, then appraised for risk of bias, directness, precision, and applicability. GRADE assigns certainty to a body of evidence for a specified outcome, not to a journal, vendor, individual paper, or generic validation stage (CDC ACIP GRADE Handbook, 2024).
Evidence question
Evidence needed
Appropriate appraisal or reporting approach
Does the model perform as specified?
Locked-model evaluation on representative data, including calibration, threshold-specific errors, missing data, and subgroup performance
Does performance transfer across populations, jurisdictions, or time?
Independent temporal, geographic, and multi-site evaluation in data systems and populations relevant to intended use
Explicit assessment of case mix, prevalence, measurement, calibration, operating thresholds, and uncertainty
Does the system work in public health operations?
Prospective evaluation of data pipelines, timeliness, representativeness, acceptability, workflow, workload, failure modes, and subgroup effects
CDC surveillance-system attributes and, for live decision support, DECIDE-AI reporting items (CDC, 2001; Vasey et al., 2022)
Does using the system improve public health decisions or outcomes?
Comparative evidence on decisions, processes, population outcomes, harms, equity, and resource use
Randomized designs when feasible, or a justified quasi-experimental design such as an interrupted time series or controlled before-after study; CONSORT-AI applies to reports of randomized AI intervention trials (Liu et al., 2020)
Does benefit persist after deployment?
Ongoing measurement of data drift, performance, use, overrides, adverse events, inequitable effects, timeliness, stability, and unintended consequences
Risk-based monitoring across the lifecycle; NICE separates design, value, performance, deployment, and post-deployment evidence rather than collapsing them into one study hierarchy (NICE Evidence Standards Framework)
Reporting guidelines improve transparency, but complete reporting does not establish low risk of bias or public health benefit. Risk-of-bias tools and causal designs must also match the question. A randomized trial is well suited to estimating the causal effect of using an AI intervention, but it is not automatically the best design for estimating discrimination, calibration, rare harms, transportability, or long-term drift.
Staged Evaluation Is an Operational Sequence
A staged pathway remains useful for managing deployment risk:
Develop the model and estimate internal performance without data leakage.
Evaluate the locked model on independent temporal, geographic, or multi-site data relevant to intended use.
Run silent or limited prospective evaluation to test live data pipelines, workflow, and failure modes.
Use a comparative design when the claim concerns effects on decisions, outcomes, equity, or resource use.
Continue surveillance after deployment because performance and use can change.
These stages are complementary. They should not be converted into a universal weakest-to-strongest evidence score.
Recent Evidence That Changes Evaluation Practice
Recent studies reinforce five controls that should be specified before evaluation begins:
Separate process measures from outcomes. A pragmatic cluster-randomized trial across Kenyan primary-care facilities found improvements in several expert-rated care-process measures with generative AI decision support, while its prespecified 14-day treatment-failure outcome did not differ significantly. The result supports better documentation and reasoning processes, not a claim of improved patient outcomes (Agweyu et al., 2026).
Pair static benchmarks with valid perturbation tests. Dynamic red-teaming of health LLMs exposed substantial brittleness among answers that were correct at baseline. The reported attack success depends on the mutation set, search budget, and judge, so it is evidence of stress-test vulnerability rather than an estimate of routine clinical error (Pan et al., 2026).
Treat evaluators as part of the measurement system. Ratings of real clinical-case responses varied across physician seniority and practice setting, and automated evaluators did not fully reproduce the features of physician judgment. Report evaluator composition and test whether rankings are stable across relevant strata (Shi et al., 2026).
Permit automated judges to abstain. MedQADE, a 2026 preprint, found that automated judges could approach clinician-level agreement while failing to express the caution physicians showed on difficult items. Automated evaluation should include uncertainty thresholds, escalation, and model-family bias checks (Philipp et al., 2026, preprint).
Do not equate local relevance with deployment validation. IyawoBench provides locally grounded Nigerian primary-care triage cases, but its synthetic cases and limited reference-label process require independent relabeling, prompt-sensitivity analysis, geographic validation, and prospective evaluation before deployment claims (Gabriel et al., 2026, preprint; Gabriel and Olawuyi, 2026, preprint).
The operational rule is simple: define the claim first, then select the outcome, study design, stress tests, evaluator panel, and monitoring plan that can support that claim.
Definition: Perception that system is agreeable/satisfactory
Measures: - User satisfaction surveys (Likert scales, Net Promoter Score) - Qualitative interviews (what do users like/dislike?) - Perceived usefulness and ease of use
Example questions: - “This system improves my clinical decision-making” (1-5 scale) - “I would recommend this system to colleagues” (yes/no)
2. Adoption
Definition: Intention/action to use the system
Measures: - Utilization rate (% of eligible cases where system used) - Number of users who have activated/logged in - Time to initial use
Red flag: Low adoption despite availability suggests problems with acceptability, workflow fit, or perceived utility.
3. Appropriateness
Definition: Perceived fit for setting/population/problem
Measures: - Stakeholder perception surveys - Alignment with clinical workflows (workflow mapping) - Relevance to clinical questions
Example: ICU mortality prediction may be appropriate for ICU but inappropriate for outpatient clinic.
Definition: Degree to which system used as designed
Measures: - Override rates (how often do clinicians dismiss alerts?) - Deviation from intended use (using for wrong purpose) - Workarounds (users circumventing system)
High override rates signal problems: - Too many false positives (alert fatigue) - Predictions do not match clinical judgment (trust issues) - Workflow disruption (alerts at wrong time)
6. Penetration
Definition: Integration across settings/populations
Measures: - Number of sites/units using system - Proportion of target population reached - Geographic spread
7. Sustainability
Definition: Continued use over time
Measures: - Retention of users over 6-12 months - Model updating/maintenance plan - Long-term performance monitoring
Common failure: “Pilot-itis” , successful pilot, but system not sustained after initial implementation period.
Example: At the evaluated threshold, Epic’s sepsis model had 12% positive predictive value, so 88% of patients who would have generated an alert did not meet the study’s sepsis outcome (Wong et al., 2021). The study estimated a substantial alert burden; it did not measure whether clinicians stopped responding.
Solutions: - Minimize false positives (sacrifice sensitivity if needed) - Tiered alerts (critical vs. informational) - Smart timing (deliver when actionable, not during documentation) - Actionable recommendations (“Order blood cultures” not “Consider sepsis”)
2. Workflow Disruption
Problem: System does not integrate smoothly into existing processes
Examples: - Extra clicks required - Separate application (need to switch contexts) - Alerts interrupt at inopportune times (during patient exam)
Solutions: - User-centered design (involve clinicians early and often) - Embed in existing EHR workflows - Minimize friction (one-click actions)
For workflow integration principles, see Bates et al., 2003, JAMIA on the “Ten Commandments” for effective clinical decision support.
3. Lack of Trust
Problem: Clinicians do not trust “black box” predictions
Example: Deep learning model provides risk score with no explanation
Solutions: - Provide explanations (SHAP values, attention weights) - Show evidence base (similar cases, supporting literature) - Transparent validation (publish performance data) - Gradual trust-building (start with low-stakes recommendations)
4. Model Drift
Problem: Performance degrades over time as data distribution changes
Why it matters: A 2017 study of clinical prediction models found that most require recalibration within 2-3 years due to changes in patient populations, treatment patterns, and data collection practices.
MLOps and Continuous Model Monitoring
Moving beyond “deploy and hope”: Modern AI systems require continuous monitoring to detect performance degradation and trigger timely interventions.
Post-Deployment Monitoring is Not Optional
The FDA’s 2021 Action Plan on AI/ML-based Software as a Medical Device emphasizes continuous monitoring as a core requirement. Models that do not monitor performance drift pose patient safety risks.
Real-world failure: IBM Watson for Oncology was deployed at multiple institutions but provided unsafe treatment recommendations that were not detected for years due to inadequate monitoring.
Types of Drift
Understanding the type of drift helps determine appropriate interventions.
1. Data Drift (Covariate Shift)
Definition: Input feature distributions change, but the relationship between features and outcome remains stable.
Example: - Training data (2019): Average patient age = 55, BMI = 27 - Production data (2024): Average patient age = 62, BMI = 31 - Relationship unchanged: Diabetes risk per BMI unit = same
Impact: Model may become miscalibrated (predicted probabilities off) even if discrimination (AUC-ROC) stays stable.
Detection: Compare feature distributions between training and production data.
2. Concept Drift
Definition: The relationship between features and outcome changes.
Example: - Pre-COVID (2019): Fever + cough + dyspnea → Likely bacterial pneumonia - During COVID (2020): Fever + cough + dyspnea → Likely COVID-19 - Same features, different outcome
Impact: Model discrimination (AUC-ROC) degrades significantly.
Impact: Predicted probabilities may be systematically too low or too high (calibration degrades).
Detection: Compare outcome rates over time.
Practical Drift Detection Methods
Method 1: Statistical Tests for Feature Distribution Changes
Kolmogorov-Smirnov (KS) Test:
Tests whether two distributions are significantly different.
Hide code
from scipy.stats import ks_2sampimport numpy as npimport pandas as pd# Example: Monitoring patient age distribution# Training data (historical)age_train = np.random.normal(55, 15, 1000) # Mean=55, SD=15# Production data (current month)age_prod = np.random.normal(62, 15, 500) # Mean shifted to 62# Perform KS teststatistic, p_value = ks_2samp(age_train, age_prod)print(f"KS statistic: {statistic:.3f}")print(f"P-value: {p_value:.4f}")if p_value <0.01:print("[ALERT] Significant distribution shift detected!")print("→ Review model calibration and consider retraining")else:print("[OK] No significant drift detected")# Interpretation:# p < 0.01: Strong evidence of distribution change# p < 0.05: Moderate evidence of distribution change# p ≥ 0.05: No significant change detected
When to use: Continuous features (age, lab values, vital signs)
Chi-Square Test (for categorical features):
Hide code
from scipy.stats import chi2_contingency# Example: Monitoring sex distribution# Training datatrain_counts = {"Male": 600, "Female": 400} # 60% male# Production data (current month)prod_counts = {"Male": 250, "Female": 250} # 50% male# Create contingency tablecontingency_table = pd.DataFrame({'Train': [train_counts['Male'], train_counts['Female']],'Production': [prod_counts['Male'], prod_counts['Female']]}, index=['Male', 'Female'])chi2, p_value, dof, expected = chi2_contingency(contingency_table)print(f"Chi-square statistic: {chi2:.3f}")print(f"P-value: {p_value:.4f}")if p_value <0.01:print("[ALERT] Significant distribution shift in sex distribution!")
When to use: Categorical features (sex, race, diagnostic codes)
Method 2: Population Stability Index (PSI)
Widely used in industry for monitoring feature drift.
Insight: Even without ground truth, you can monitor what the model is predicting.
Red flag patterns: - Sudden increase in high-risk predictions (model becoming overly sensitive) - Sudden decrease in high-risk predictions (model missing cases) - Bimodal distribution shifts (calibration degradation)
Example:
Hide code
# Monitor distribution of predicted probabilities over time# Week 1 predictions (well-calibrated)week1_preds = np.random.beta(2, 10, 1000) # Mostly low risk# Week 12 predictions (drift - more high-risk predictions)week12_preds = np.random.beta(3, 7, 1000) # Shifted higher# Visualizefig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))ax1.hist(week1_preds, bins=50, alpha=0.7, label='Week 1', density=True)ax1.hist(week12_preds, bins=50, alpha=0.7, label='Week 12', density=True)ax1.set_xlabel('Predicted Probability')ax1.set_ylabel('Density')ax1.set_title('Prediction Distribution Shift')ax1.legend()# KS test to detect shiftks_stat, p_val = ks_2samp(week1_preds, week12_preds)ax2.text(0.5, 0.5, f'KS test p-value: {p_val:.4f}\n'+ ('[WARNING] Significant shift detected'if p_val <0.01else'[OK] No significant shift'), ha='center', va='center', fontsize=14, bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))ax2.axis('off')plt.tight_layout()plt.show()
Automated Retraining Triggers
Goal: Define clear rules for when to retrain the model.
Retraining Decision Framework
IMMEDIATE retraining triggered by: - AUC-ROC drops >0.05 below baseline - Calibration error (Brier score) increases >0.05 - Safety event: Model missed critical case (e.g., sepsis death after negative prediction) - Feature drift: PSI > 0.25 for ≥3 critical features
SCHEDULED retraining triggered by: - 12 months since last training (routine maintenance) - AUC-ROC drops 0.02-0.05 below baseline (warning level) - PSI 0.1-0.25 for multiple features (moderate drift) - New data volume: ≥20% new samples since last training
HOLD retraining if: - All metrics within acceptable ranges - Recent retraining (< 3 months ago) - Insufficient new data (< 5% new samples)
Automated monitoring script:
Hide code
def evaluate_retraining_need(current_auc, baseline_auc, psi_scores, months_since_training, new_sample_fraction):""" Automated decision system for model retraining Returns: - "IMMEDIATE": Retrain immediately - "SCHEDULED": Schedule retraining within 30 days - "MONITOR": Continue monitoring, no action needed """# Critical performance degradationif current_auc < baseline_auc -0.05:return"IMMEDIATE", "AUC-ROC dropped >0.05"# Critical feature drift critical_drift_count =sum([psi >0.25for psi in psi_scores])if critical_drift_count >=3:return"IMMEDIATE", f"{critical_drift_count} features with PSI > 0.25"# Routine maintenance scheduleif months_since_training >=12:return"SCHEDULED", "12-month routine retraining"# Moderate performance degradationif current_auc < baseline_auc -0.02:return"SCHEDULED", "AUC-ROC dropped 0.02-0.05"# Moderate drift moderate_drift_count =sum([0.1< psi <0.25for psi in psi_scores])if moderate_drift_count >=4:return"SCHEDULED", f"{moderate_drift_count} features with moderate drift"# Significant new dataif new_sample_fraction >=0.20:return"SCHEDULED", f"{new_sample_fraction:.0%} new data available"return"MONITOR", "All metrics within acceptable ranges"# Example usagedecision, reason = evaluate_retraining_need( current_auc=0.82, baseline_auc=0.85, psi_scores=[0.08, 0.15, 0.22, 0.18, 0.05], # PSI for 5 key features months_since_training=8, new_sample_fraction=0.15)print(f"Decision: {decision}")print(f"Reason: {reason}")
Continuous Learning Strategies
Two approaches:
1. Periodic Retraining (Safer, recommended for high-stakes)
Process:
Accumulate new data
Retrain model offline
Validate on hold-out set
A/B test against current model
Deploy if superior
Advantages: Controlled validation before deployment
Disadvantages: Model can drift between retraining cycles
Recommendation for healthcare: Use periodic retraining with trigger-based scheduling (not true online learning).
Real-World Example: External Validation After Adoption
Epic’s sepsis model had already been implemented at hundreds of US hospitals when Wong and colleagues published an independent external validation at one academic health system (Wong et al., 2021).
At the evaluated threshold, the study reported:
Sensitivity: 33%
Positive predictive value: 12%
Alert burden: 18% of hospitalizations would have generated at least one alert
The study showed poor discrimination and calibration in that setting. It did not test model drift, continuous monitoring at other deployment sites, or a retraining protocol.
Lesson:Widespread adoption is not evidence of transportability or clinical value. Independent evaluation should precede broad use, and post-deployment monitoring should address questions that a retrospective external validation cannot answer.
2024-2025 reality: AI evaluation is no longer just a scientific question. It’s a regulatory requirement.
By 2025, medical AI systems face scrutiny from multiple regulatory bodies: - FDA (United States): Software as a Medical Device (SaMD) framework - European Union: AI Act (2024) classifying medical AI as “high-risk” - UK MHRA: Software and AI as a Medical Device framework - Health Canada: Medical Device Regulations for AI/ML
Key insight: Understanding regulatory evaluation requirements is essential for deployment, not just compliance.
Why Regulatory Context Matters for Evaluation
Even if you’re not directly developing a regulatory-approved device, understanding these frameworks helps you:
Design better evaluations: Regulatory standards define best practices
Anticipate deployment requirements: Many health systems require FDA clearance or equivalent
Benchmark your work: Compare your validation against regulatory expectations
Communicate with stakeholders: Speak the language of hospital legal/compliance teams
Cross-reference:Policy and Governance covers broader policy implications; this section focuses on evaluation-specific regulatory requirements.
FDA Software as a Medical Device (SaMD) Framework
Software as a Medical Device (SaMD) refers to software intended for medical purposes that operates independently of hardware medical devices.
Examples: - SaMD: Sepsis prediction algorithm, diabetic retinopathy screening app, radiology CAD software - Not SaMD: EHR system (administrative), fitness tracker (wellness, not medical diagnosis)
Risk-Based Classification System
The FDA classifies SaMD by risk level, which determines evaluation rigor required.
Risk Categorization Matrix
State of healthcare
Significance of information
Treat or diagnose
Drive clinical management
Inform clinical management
Critical
III (highest risk)
III
II
Serious
III
II
II
Non-serious
II
II
I (lowest risk)
Definitions: - Critical: Death or permanent impairment (e.g., ICU monitoring) - Serious: Long-term morbidity (e.g., cancer diagnosis) - Non-serious: Minor conditions (e.g., acne treatment)
Information significance: - Treat/Diagnose: Directly triggers treatment decisions - Drive clinical management: Significant influence on treatment path - Inform clinical management: One input among many
Evaluation Requirements by Risk Level
Level I (Low Risk): - Example: App suggesting lifestyle modifications for mild hypertension - Requirements: - Basic technical performance validation - User studies demonstrating safe use - Minimal clinical validation
Challenge: Traditional medical devices are static; ML models need updating.
FDA’s current framework (December 2024 final guidance): Predetermined Change Control Plans allow FDA-reviewed, pre-specified modifications to AI-enabled devices without separate marketing submissions for each covered update.
What can be included in a PCCP:
1. Allowable modifications: - Retraining on new data (within specified bounds) - Algorithm parameter adjustments - Performance improvements
2. Modification protocol: - Data requirements: Minimum sample size, quality standards - Performance thresholds: Must maintain ≥ X sensitivity - Validation approach: Test set size, external validation sites
3. Update assessment: - Performance monitoring triggers retraining - Validation results compared to pre-specified thresholds - Automated decision: deploy update or flag for review
4. Transparency and reporting: - Change documentation - Performance reports to regulators - User notifications
Example PCCP:
### Sepsis Prediction Model PCCP**Allowable Change:** Retrain model quarterly using new institutional data**Modification Protocol:**- Minimum 5,000 new patient encounters with ≥200 sepsis cases- Maintain AUC-ROC ≥ 0.82 (original validation: 0.85)- Sensitivity at 80% specificity must be ≥ 70%- External validation on ≥1 additional hospital required**Assessment:**- Automated performance monitoring (monthly)- Retraining triggered if AUC drops below 0.83- New model validated on hold-out test set (20% of new data)- Deploy if all thresholds met; otherwise, flag for manual review**Reporting:**- Quarterly performance reports to FDA- User notification of model updates via EHR alert
EU AI Act: High-Risk Classification for Medical AI
Adopted 2024, fully enforced by 2026.
Key classification: Most medical AI systems are high-risk, requiring:
1. Risk management system: - Continuous identification and mitigation of risks
2. Data governance: - Training data quality assurance - Bias detection and mitigation - Data representativeness
3. Technical documentation: - Detailed model specifications - Training procedures - Validation results
4. Transparency: - Users informed that AI is involved - Clear information on limitations
5. Human oversight: - Human-in-the-loop for high-stakes decisions
Implications for evaluation: - Match validation populations, sites, and time periods to the intended use - Evaluate subgroup performance and foreseeable discriminatory effects - Test robustness and cybersecurity using methods appropriate to the threat model - Establish postmarket monitoring where the applicable legal and risk framework requires it
Even if your system is not currently seeking regulatory approval, aligning with these standards ensures quality:
Before Development: - [ ] Determine risk classification (SaMD Level I/II/III or EU high-risk) - [ ] Identify applicable regulatory frameworks - [ ] Define evaluation requirements based on risk level
During Development: - [ ] Ensure training/test data independence (GMLP #4) - [ ] Use representative training data (GMLP #3) - [ ] Apply good software engineering (version control, testing) (GMLP #2) - [ ] Document model architecture, hyperparameters, training process
Validation Phase: - [ ] Internal validation with appropriate cross-validation - [ ] Temporal validation (if time-dependent data) - [ ] External validation when needed to support transportability to the intended population, site, or time period - [ ] Fairness audit across demographic subgroups - [ ] Usability testing with intended users - [ ] Clinical utility assessment (not just technical performance)
Pre-Deployment: - [ ] Human-AI team evaluation (GMLP #7) - [ ] Security and robustness testing appropriate to the threat model and applicable requirements - [ ] Create predetermined change control plan (if adaptive model) - [ ] Develop post-market monitoring protocol
Comparison: Research vs. Regulatory Evaluation Standards
Aspect
Research Publication
Regulatory Approval
External validation
Recommended, often skipped (6% in 2020 study)
Mandatory for Level II/III, EU high-risk
Prospective testing
Rare
Often required for novel high-risk devices
Fairness audit
Increasingly expected
Mandatory (EU AI Act)
Post-market monitoring
Not required
Mandatory
Clinical utility
Recommended
Required (must demonstrate benefit)
Documentation
Methods section
Extensive technical documentation
Timeline
Months
1-3+ years (depending on risk level)
Key insight:Regulatory standards are higher than typical research standards. If you aim for deployment, plan for regulatory-level evaluation from the start.
When to Seek Regulatory Approval
You likely need FDA clearance/approval if: - System makes diagnostic or treatment recommendations - Used for screening (e.g., diabetic retinopathy, cancer) - Influences clinical decision-making significantly - Marketed as improving health outcomes
You might NOT need approval if: - Administrative tools (scheduling, billing) - Wellness apps (general fitness, not medical claims) - Clinical decision support providing information only (not recommendations) - Gray area: FDA discretion, often depends on risk
When in doubt: Consult FDA’s Digital Health Center of Excellence or regulatory counsel.
Regulatory requirements are evaluation requirements: FDA, EU standards define rigorous validation expectations
Risk-based approach: Evidence requirements should reflect intended use, the consequence of error, degree of autonomy, and difference between the evaluation setting and intended setting
Applicability must be demonstrated: Regulatory requirements vary by function and pathway; authorization does not substitute for evidence in the intended population and setting
Continuous monitoring is required: Post-market surveillance, not just pre-deployment validation
Good Machine Learning Practice: 10 principles provide practical framework for development and evaluation
Predetermined change control plans: Enable adaptive models while maintaining regulatory compliance
EU AI Act raises the bar: High-risk systems face requirements for risk management, data governance, transparency, human oversight, accuracy, robustness, cybersecurity, and postmarket monitoring. The evidence methods must fit the system and applicable conformity-assessment obligations.
Plan early: Regulatory evaluation takes longer and costs more than research validation; design for it from the start
Seek expert guidance: Regulatory pathways are complex; consult with regulatory specialists
Standards improve quality: Even if not seeking approval, regulatory frameworks represent best practices
Comprehensive Evaluation Framework
Complete Evaluation Checklist
Use this when evaluating AI systems:
AI System Evaluation Checklist
TECHNICAL PERFORMANCE - [ ] Discrimination metrics reported (AUC-ROC, sensitivity, specificity, PPV, NPV) - [ ] 95% confidence intervals provided for all metrics - [ ] Calibration assessed (calibration plot, Brier score, ECE) - [ ] Appropriate for class imbalance (if applicable) - [ ] Comparison to baseline model (e.g., logistic regression, clinical judgment) - [ ] Multiple metrics reported (not just accuracy)
VALIDATION RIGOR - [ ] Internal validation performed (CV or hold-out) - [ ] Temporal validation (train on past, test on future) - [ ] External validation on independent dataset from different institution - [ ] Prospective validation performed or planned - [ ] Data leakage prevented (feature engineering within folds) - [ ] Appropriate cross-validation for data structure (stratified, grouped, time-series)
FAIRNESS AND EQUITY - [ ] Performance stratified by demographic subgroups (race, gender, age, SES) - [ ] Disparities quantified (absolute and relative differences) - [ ] Calibration assessed per subgroup - [ ] Training data representation documented - [ ] Potential for bias explicitly discussed - [ ] Mitigation strategies proposed if disparities identified
CLINICAL UTILITY - [ ] Decision curve analysis or similar utility assessment - [ ] Comparison to current standard of care - [ ] Clinical workflow integration considered - [ ] Net benefit quantified - [ ] Cost-effectiveness assessed (if applicable) - [ ] Actionable outputs (not just risk scores)
TRANSPARENCY AND REPRODUCIBILITY - [ ] Model architecture and type clearly described - [ ] Feature engineering documented - [ ] Hyperparameters and training procedure reported - [ ] Reporting guidelines followed (TRIPOD, STARD-AI) - [ ] Code availability stated - [ ] Data availability (with appropriate privacy protections) - [ ] Conflicts of interest disclosed
IMPLEMENTATION PLANNING - [ ] Target users and use cases defined - [ ] Workflow integration plan described - [ ] Alert/decision threshold selection justified - [ ] Plan for performance monitoring post-deployment - [ ] Model updating and maintenance plan - [ ] Training plan for end users - [ ] Contingency plan for model failure
Results - Model Specification 17. Model specification (all parameters) 18. Model performance (discrimination and calibration)
Discussion 19. Interpretation (clinical meaning, implications) 20. Limitations 21. Implications for practice
Other 22. Funding and conflicts of interest
Current guidance: TRIPOD+AI superseded the 2015 TRIPOD checklist for clinical prediction model studies using regression or machine learning methods (Collins et al., 2024). It strengthens reporting of data sources, model development, evaluation, fairness, and open-science practices. TRIPOD+AI is a reporting guideline, not a method for rating evidence certainty or risk of bias; PROBAST+AI serves the latter appraisal role.
TRIPOD-LLM: Reporting Guidelines for LLM Studies
Traditional TRIPOD was designed for classical prediction models. Large language models require fundamentally different reporting standards. TRIPOD-LLM (Gallifant et al., 2025, Nature Medicine) provides the first LLM-specific extension, endorsed by EQUATOR Network.
Why LLM-specific guidelines are needed:
Training data for LLMs is often undisclosed or incompletely characterized
Performance varies dramatically with prompt wording
Contamination between training and test data is difficult to verify
Reproducibility requires specifying API versions, dates, and system prompts
Key TRIPOD-LLM additions:
Category
New Requirements
Model specification
API version, access date, temperature/sampling parameters
Prompting
Full prompt text, development process, any prompt optimization
Input/output
How health data was formatted, output parsing methods
Reproducibility
Whether results vary across API calls, random seed handling
Contamination
Assessment of whether test data appeared in training corpus
The LLM reporting crisis:
A systematic review of LLM chatbot health advice studies (Huo et al., 2025, JAMA Network Open) examined 137 studies and found that 136 studies (99.3%) used closed-source models. Key findings:
Only 2 studies (1.5%) used accessible or open models, and only 1 study (0.7%) fully described the model version
Prompts were incompletely reported in most studies
Temperature and sampling parameters rarely disclosed
API access dates (critical for versioning) almost never reported
These reporting gaps mean that faithful reproduction was not possible for most studies. TRIPOD-LLM directly addresses these gaps.
Complementing TRIPOD-LLM, the CHART statement (Huo et al., 2025, JAMA Network Open) provides a 12-item, 39-subitem checklist specifically for evaluating chatbot and conversational AI systems in healthcare settings.
STARD-AI: Standards for Reporting Diagnostic Accuracy Using AI
Additional items: - Model architecture details - Training procedure (epochs, batch size, optimization) - Validation strategy - External validation results - Subgroup analyses - Calibration assessment - Comparison to human performance (if applicable)
Critical Appraisal of Published Studies
Systematic Evaluation Framework
When reading AI studies:
1. Study Design and Data Quality
Questions: - Representative sample of target population? - External validation performed? - Test set truly independent? - Outcome objectively defined and consistently measured? - Potential for data leakage?
Red flags: - No external validation - Small sample size (<500 events) - Convenience sampling - Vague outcome definitions - Feature engineering on entire dataset before splitting
2. Model Development and Reporting
Questions: - Multiple models compared? - Simple baseline included (logistic regression)? - Hyperparameters tuned on separate validation set? - Feature selection appropriate? - Model clearly described?
Red flags: - No baseline comparison - Hyperparameter tuning on test set - Inadequate model description - No cross-validation
Red flags: - Only accuracy reported (especially for imbalanced data) - No calibration assessment - No confidence intervals - Cherry-picked metrics
4. Fairness and Generalizability
Questions: - Performance stratified by subgroups? - Diverse populations included? - Generalizability limitations discussed? - Potential biases identified?
Red flags: - No subgroup analysis - Homogeneous study population - Claims of broad generalizability without external validation - Dismissal of fairness concerns
5. Clinical Utility
Questions: - Clinical utility assessed (beyond accuracy)? - Compared to current practice? - Implementation considerations discussed? - Cost-effectiveness assessed?
Red flags: - Only technical metrics - No comparison to existing approaches - No implementation discussion - Overstated clinical claims
6. Transparency and Reproducibility
Questions: - Code and data available? - Reporting guidelines followed? - Sufficient detail to reproduce? - Limitations clearly stated? - Conflicts of interest disclosed?
Red flags: - No code/data availability - Insufficient methodological detail - Overstated conclusions - Undisclosed industry funding
Key Takeaways
Essential Principles
Evaluation is multidimensional , Technical performance, clinical utility, fairness, and implementation outcomes all matter
Internal validation is insufficient , External validation on independent data is essential to assess generalizability
Calibration is critical , Predicted probabilities must be meaningful for clinical decisions, not just discriminative
Clinical utility ≠ statistical performance , A model can be statistically accurate but clinically useless without improving outcomes
Prospective evaluation answers operational questions: Live testing can assess data pipelines, workflow, timeliness, and human factors; comparative evidence is needed for causal claims about impact
Common pitfalls are avoidable , Data leakage, improper CV, threshold optimization on test set lead to overoptimistic estimates
Implementation determines success , Even well-performing models fail if workflow integration ignored
Transparency enables trust , Follow reporting guidelines (TRIPOD, STARD-AI); share code and data when possible
Continuous monitoring is essential , Model performance drifts over time; plan for ongoing evaluation and updating
External validation evaluates a locked model on data that differ meaningfully from development data, such as a new institution, geography, device, population, or time period. The test set should represent the intended transfer and should not be used repeatedly for model selection. External validation addresses transportability for the measured endpoint. It does not by itself establish workflow benefit or patient outcomes.
What is calibration, and why does it matter?
Calibration describes whether predicted probabilities correspond to observed frequencies in the target setting. A well-discriminating model can still systematically overestimate or underestimate risk, especially when prevalence or practice changes. Calibration should be examined overall and in relevant subgroups, with uncertainty. Recalibration may improve probabilities but does not repair a wrong target, missing predictors, or an ineffective intervention.
How should evidence be matched to a public health AI claim?
Technical performance claims require a suitable reference standard and independent test data. Transportability claims require external validation. Workflow claims require prospective use in the intended setting. Outcome-benefit claims require a comparative design measuring the relevant outcome. Economic claims require explicit costs, perspective, time horizon, and comparator. No single study design answers every question, so the claim should be narrowed to what the design and endpoint measured.
What are vendor-evaluation red flags?
Red flags include an undisclosed model version, no independent test set, performance reported only as accuracy or AUC, missing confidence intervals, absent subgroup results, an inappropriate comparator, results from a different workflow, and no monitoring or update plan. Regulatory authorization, publication, or deployment elsewhere should not substitute for local evidence. A vendor should be able to provide a connected record from the claim to the source and exact product version.
Discussion Questions
Claim-specific evidence plan: You’ve developed a hospital-acquired infection prediction model. Which evidence would you require for technical performance, transportability, operational utility, population impact, equity, and lifecycle safety? Which studies would you conduct before deployment, and why?
Fairness trade-offs: Your sepsis model has AUC-ROC = 0.85 overall, but sensitivity is 0.90 for White patients vs. 0.75 for Black patients. What would you do? What are trade-offs of different mitigation approaches?
Calibration vs. discrimination: Model A: AUC-ROC = 0.85, Brier score = 0.30 (poor calibration). Model B: AUC-ROC = 0.80, Brier score = 0.15 (excellent calibration). Which deploy? Why?
External validation failure: Your model achieves AUC-ROC = 0.82 internal validation but 0.68 external validation at different hospital. What explains this? What next steps?
Clinical utility skepticism: Model predicts 30-day mortality with AUC-ROC = 0.88. Does this mean it’s clinically useful? What additional evaluations needed?
Prospective study design: Evaluate hospital readmission model prospectively. RCT, stepped-wedge, or silent mode? What are trade-offs?
Alert threshold selection: Clinical decision support tool can alert at >10%, >20%, or >30% predicted risk. How choose? What factors matter?
Model drift: COVID-19 forecasting model trained on 2020 data; now 2023 with new variants. How assess if still valid? What triggers retraining?
AI Deployment in Healthcare: How to move validated models into production, monitor for drift, and manage the organizational change that determines real-world success
AI Safety and Risk Management: Risk frameworks, failure mode analysis, and when to pull a deployed model out of production
AI Ethics, Bias, and Equity: Deeper coverage of fairness metrics, bias mitigation strategies, and equity considerations across populations