Evaluating AI Systems for Healthcare

A high validation score does not establish reliability in public health practice. A widely implemented sepsis model had 33% sensitivity and 12% positive predictive value at the evaluated threshold in external validation (Wong et al., 2021). Adoption decisions therefore require evidence matched to the claim, intended use, population, setting, and consequences of error.

Learning Objectives

Readers will be able to:

  • Match evidence requirements to technical performance, transportability, operational utility, population impact, and lifecycle safety claims
  • Evaluate clinical utility beyond accuracy (decisions, outcomes, workflow integration, fairness)
  • Assess LLMs and foundation models (hallucination detection, prompt robustness, RAG systems)
  • Detect and monitor drift in production systems (data, concept, and label drift)
  • Apply regulatory frameworks (FDA SaMD, EU AI Act, GMLP principles)
  • Test adversarial robustness and out-of-distribution performance
  • Recognize when NOT to deploy despite good technical metrics

Prerequisites: Machine Learning Fundamentals, The Data Problem, Diagnostic and Clinical Decision Support. For LLM evaluation details, see Large Language Models in Public Health.

The Big Picture: In a review of 516 medical-imaging AI studies, 31 used external validation (Kim et al., 2019). In an external evaluation of a widely implemented sepsis model, sensitivity was 33% and positive predictive value was 12% at the evaluated threshold (Wong et al., 2021). Technical performance in one setting cannot establish transportability, operational utility, population impact, equity, or lifecycle safety.

Match Evidence to the Claim:

  1. Technical performance: Discrimination, calibration, threshold-specific errors, missing data, and subgroup performance
  2. Transportability: Independent temporal, geographic, and multi-site evaluation in relevant populations and data systems
  3. Operational utility: Prospective evaluation of data pipelines, timeliness, workflow, workload, and failure modes
  4. Population impact: Comparative evidence on decisions, benefits, harms, equity, and resource use
  5. Lifecycle safety: Ongoing monitoring for drift, failures, inequitable effects, and unintended consequences

No study label answers every question. The design must match the claim and deployment risk.

Beyond Accuracy: What Really Matters

  • Clinical Utility: Does it change decisions? Improve outcomes? Integrate into workflows?
  • Generalization: CheXNet-style models AUC=0.93 internally, dropped to 0.82 at external hospitals (Zech et al., 2018, PLOS Medicine). Beware the generalization gap
  • Fairness Across Subgroups: Does model perform equally for different races, ages, sexes, socioeconomic groups?
  • Implementation Outcomes: Adoption rate, alert fatigue, workflow disruption, user trust

Common Evaluation Pitfalls:

  1. No External Validation: Tested only on holdout from same dataset
  2. Cherry-Picked Subgroups: “Works great on images rated as ‘excellent quality’” (real-world images are messy)
  3. Ignoring Prevalence Shift: Trained on 50% disease prevalence, deployed where prevalence is 5%
  4. Overfitting to Dataset Quirks: Model learns hospital-specific artifacts, not disease
  5. Evaluation-Treatment Mismatch: Evaluate on diagnosed cases, deploy for screening

Evaluating Foundation Models and LLMs

Traditional ML metrics (accuracy, AUC) are insufficient for large language models:

  • Factual Accuracy: Does model provide correct medical information?
  • Hallucination Detection: How often does it confidently generate false information?
  • Prompt Sensitivity: Does small rewording change answers dramatically?
  • Safety: Harmful advice, biased responses, privacy leaks
  • Medical Benchmarks: MedQA, PubMedQA, USMLE-style questions (but benchmarks do not equal clinical competence)
  • RAG Evaluation: For retrieval-augmented generation, evaluate retrieval quality AND generation quality separately

→ See also: Large Language Models in Public Health for LLM evaluation frameworks and practical validation strategies

Continuous Monitoring (ML Ops)

Deployment is not the end. Models degrade over time:

  • Data Drift: Input distributions change (e.g., demographics shift, new disease variants)
  • Concept Drift: Relationship between features and outcome changes
  • Label Drift: Outcome prevalence changes
  • Detection Methods: Population Stability Index (PSI), statistical process control charts
  • Retraining Triggers: Predetermined thresholds for when performance drops require model updates

Regulatory Frameworks

  • FDA and IMDRF (Software as a Medical Device): FDA device classes (I, II, III) and IMDRF SaMD risk categories (I–IV). Higher risk = more rigorous validation
  • Good Machine Learning Practice (GMLP): Guiding principles from FDA, Health Canada, and the UK MHRA for development, validation, and monitoring
  • EU AI Act: High-risk medical AI requires conformity assessment, transparency, human oversight, continuous monitoring
  • Key Insight: Even non-regulated systems benefit from regulatory-level evaluation rigor

Adversarial Robustness

  • Natural Perturbations: Small changes in image brightness, patient demographics. Does model break?
  • Adversarial Attacks: Intentionally crafted inputs to fool model (FGSM, PGD attacks)
  • Out-of-Distribution (OOD) Detection: Can model recognize when input is unlike training data?
  • EU AI Act Requirement: High-risk systems must demonstrate robustness testing

The Obermeyer Lesson:

Healthcare cost algorithm had excellent accuracy but systematic inequity: Black patients had to be sicker than White patients to receive same risk score. Lesson: Technical performance ≠ ethical deployment. Must evaluate fairness explicitly.

→ See also: Ethics, Bias, and Equity in Healthcare AI for comprehensive frameworks on evaluating AI fairness

When NOT to Deploy (Despite Good Performance):

Red flags that should halt deployment:

  1. External validation shows poor generalization
  2. Fairness audit reveals systematic bias
  3. Clinical workflow integration causes more harm than benefit (alert fatigue)
  4. Users do not trust or adopt the system
  5. No plan for continuous monitoring and maintenance

The Takeaway for Public Health Practitioners:

Evaluation is not a checkbox. Internal validation estimates performance under development conditions. Independent validation tests transportability. Prospective studies examine live data pipelines, workflow, and human factors. Comparative designs test whether using the system changes decisions or outcomes. None substitutes for post-deployment monitoring. For LLMs and foundation models, evaluation must also address hallucinations, prompt sensitivity, retrieval quality, and safety in the intended use. The central task is to define the claim first, choose a design that can answer it, and appraise risk of bias and applicability before acting on the result.

Test whether you can match an evaluation result to the claim it supports:

1. Same-source holdout: A research team reports 94% accuracy for a pneumonia detection model on a held-out sample from the same hospital and period used for development. What does this result support?

  • A. Internal performance under the study conditions
  • B. Transportability to other hospitals
  • C. Effectiveness in a live workflow
  • D. Improved patient outcomes
Click for answer

Answer: A. Internal performance under the study conditions

Why: The evaluation can estimate performance under the sampled development conditions if leakage and analysis bias are controlled. It does not establish transportability, workflow performance, or an effect on outcomes.

2. The Epic Lesson: In an external evaluation, Epic’s sepsis model had 33% sensitivity and 12% PPV at the evaluated threshold. What does this mean practically?

  • A. Model missed 2 out of 3 sepsis cases; 88% of alerts were false alarms
  • B. Model worked great, 33% and 12% are good metrics
  • C. Model needs minor tuning to reach production quality
  • D. External validation was too strict
Click for answer

Answer: A. Missed 2 of 3 cases; 88% false alarms

Why: 33% sensitivity means the model detected approximately 1 in 3 sepsis cases. A 12% PPV means approximately 12 of 100 alerts identified a case. These figures indicate substantial missed-case and false-alert burdens at that threshold. Deployment status does not establish acceptable performance.

3. LLM Evaluation: You are evaluating a medical chatbot powered by an LLM. It scores 85% on MedQA (medical exam questions). Can you deploy it?

  • A. Yes, 85% accuracy is excellent
  • B. No, must also test hallucination rate, prompt robustness, safety
  • C. No, need prospective clinical validation
  • D. Both B and C
Click for answer

Answer: D. Both B and C

Why: Benchmark performance (MedQA) ≠ clinical competence. LLMs can ace exams but hallucinate dangerous medical advice. Must test: hallucination detection, prompt sensitivity (small rewording changes answer?), safety (harmful advice?), and prospective validation in real clinical workflow before deployment.

4. Model Drift: Your outbreak prediction model performed well for 2 years. Suddenly accuracy drops from 82% to 61%. What’s likely happening?

  • A. The model is broken, rebuild from scratch
  • B. Distribution shift: inputs or feature-outcome relationships changed (new disease variant, demographic shifts)
  • C. Model was always bad, just got lucky initially
  • D. Evaluation metrics are wrong
Click for answer

Answer: B. Distribution shift

Why: A sudden drop after stable performance points to distribution shift. New disease variants, demographic shifts, and changes in testing practices can change the inputs (data drift) or the relationship between features and outcome (concept drift); a sudden loss of discrimination points especially to concept drift. This is why continuous monitoring and retraining triggers are essential. Models degrade over time in production.

5. When NOT to Deploy: Your model has 91% AUC internally, 87% AUC at 3 external hospitals. Fairness audit shows White patients: 90% sensitivity, Black patients: 65% sensitivity. Should you deploy?

  • A. Yes, 87% external AUC is strong
  • B. No, systematic bias is unacceptable even with good overall performance
  • C. Yes, but only for White patients
  • D. Deploy and monitor fairness post-deployment
Click for answer

Answer: B. No, systematic bias is unacceptable

Why: Technical performance ≠ ethical deployment. This is the Obermeyer lesson: excellent accuracy but systematic inequity. Black patients receive worse care because model systematically underperforms. Must address fairness BEFORE deployment, not after. Option D (“monitor after deployment”) puts patients at risk while you collect evidence of harm. Halt deployment until bias is addressed.

Scoring:

  • 5/5: You understand evaluation rigor. Ready to evaluate AI systems critically.
  • 3–4/5: Good foundation. Review sections where you missed questions, especially Epic sepsis case study.
  • 0–2/5: Reread the TL;DR summary and the Introduction section. The evaluation crisis is the most important concept in this chapter.

Introduction: The Evaluation Crisis in AI

March 2019, Korean Journal of Radiology:

Kim et al. publish a systematic review examining 516 studies on AI algorithms for medical image analysis.

Their sobering finding: Only 6% performed external validation (31 of 516 studies) on data from different institutions.

The vast majority tested models only on hold-out sets from the same dataset used for training, a practice that provides minimal evidence of real-world performance.

June 2021, JAMA Internal Medicine:

Wong et al. publish an external validation of Epic’s sepsis prediction model, which was implemented at hundreds of US hospitals.

The model’s performance:

  • Sensitivity: 33% (missed 2 out of 3 sepsis cases)
  • Positive predictive value: 12% (88% of alerts were false positives)
  • Alert burden: 18% of hospitalizations would have generated at least one alert at the evaluated threshold

Authors’ conclusion: The model had poor discrimination and calibration, and its alert burden raised concerns about clinical value.

The evaluation was a retrospective cohort study at one academic health system. Its importance is that the proprietary model was already in widespread clinical use before independent validation was published.

The Evaluation Gap:

Between lab performance and real-world deployment lies a chasm that has claimed many promising AI systems. Clinical AI evaluation currently resembles standardized testing more than real-world practice: retrospective accuracy on curated datasets, with limited measurement of workflow fit, adoption, safety guardrails, or downstream outcomes (Azad et al., Nature Medicine, 2026).

In the lab (illustrative):

  • Curated, high-quality datasets
  • Balanced classes (50% positive, 50% negative)
  • Consistent protocols
  • Expert-confirmed labels
  • AUC-ROC = 0.95

In the real world (illustrative):

  • Messy, incomplete data
  • Rare events (1–5% prevalence)
  • Variable protocols across sites
  • Ambiguous cases
  • AUC-ROC = 0.68
Performance ≠ Safety

Performance evaluation covers accuracy, generalization, fairness, and robustness. High performance does not guarantee safe clinical deployment.

Safety validation requires additional frameworks:

  • Failure mode and effects analysis (FMEA)
  • Hazard analysis and risk assessment
  • Worst-case scenario testing
  • Operational safety validation

For comprehensive safety evaluation beyond performance metrics, see AI Safety in Healthcare.

The consequences are severe:

  • Failed deployments: Models that work in development but fail in production
  • Hidden biases: Systems that perform well on average but poorly for specific groups
  • Wasted resources: Millions invested in systems that do not deliver promised benefits
  • Patient harm: Incorrect predictions leading to inappropriate treatments
  • Eroded trust: Clinicians lose confidence in AI after experiencing failures

The Adoption-Value Gap:

McKinsey’s 2025 cross-industry State of AI survey included 1,993 participants across 105 countries. Its organizational findings provide context but do not establish public health adoption or value rates:

  • 88% of organizations report using AI in at least one function (up from 78% in 2024)
  • Yet only 6% qualify as “high performers” achieving 5%+ earnings impact from AI
  • Only 7% have fully scaled AI across their organizations

Five recurring barriers prevent organizations from crossing this gap:

  1. Data quality issues: Fragmented systems, inconsistent metadata, accuracy problems
  2. Financial justification difficulty: Inability to demonstrate measurable long-term gains
  3. Skills shortage: Insufficient data scientists, engineers, and change-management expertise
  4. Organizational silos: Lack of cross-functional collaboration
  5. Governance uncertainty: Evolving privacy regulations and security concerns

The lesson for public health: Deployment is not the finish line. Organizations that treat AI as technology procurement rather than workflow transformation consistently underperform. The 6% classified as high performers invest in data infrastructure, workforce training, and governance frameworks before deploying AI tools.

Why This Chapter Matters

Rigorous evaluation is the bridge between AI research and AI implementation. Without it, unvalidated systems reach deployment.

Evaluate AI systems across five critical dimensions:

  1. Technical performance: Accuracy, calibration, robustness
  2. Generalizability: External validity, temporal stability, geographic transferability
  3. Clinical/Public health utility: Impact on decisions and outcomes
  4. Fairness and equity: Performance across demographic subgroups
  5. Implementation outcomes: Adoption, usability, sustainability

Each dimension calls for its own validation design and critical appraisal of the published evidence.

The Multidimensional Nature of Evaluation

What Are We Really Evaluating?

Evaluating an AI system requires more than measuring accuracy. In public health and clinical contexts, evaluation must assess multiple dimensions.

1. Technical Performance

Question: Does the model make accurate predictions on new data?

Key aspects:

  • Discrimination: Can the model distinguish between positive and negative cases?
  • Calibration: Do predicted probabilities match observed frequencies?
  • Robustness: Does performance degrade with missing data or noise?
  • Computational efficiency: Speed and resource requirements for deployment

Relevant for: All AI systems (foundational requirement)

2. Generalizability

Question: Will the model work in settings different from where it was developed?

Key aspects:

  • Geographic transferability: Performance at different institutions, regions, countries
  • Temporal stability: Does performance degrade as time passes and data distributions shift?
  • Population differences: Performance across different patient demographics, disease prevalence
  • Setting transferability: Hospital vs. primary care vs. community settings

Relevant for: Any system intended for broad deployment

Critical insight: Geirhos et al., 2020, Nature Machine Intelligence showed that AI models often learn “shortcuts”, spurious correlations specific to training data that do not generalize. For example, pneumonia detection models learned to identify portable X-ray machines (used for sicker patients) rather than actual pneumonia.

3. Clinical/Public Health Utility

Question: Does the model improve decision-making and outcomes?

Key aspects:

  • Decision impact: Does it change clinician decisions?
  • Outcome improvement: Does it lead to better patient outcomes?
  • Net benefit: Does it provide value above existing approaches?
  • Cost-effectiveness: Does it provide value commensurate with costs?

Relevant for: Clinical decision support, diagnostic tools

Critical distinction: A model can be statistically accurate but clinically useless. Example: A model predicting hospital mortality with AUC-ROC = 0.85 sounds impressive, but if it does not change management or improve outcomes, it adds no value.

For framework on clinical utility assessment, see Vickers et al., 2019, Diagnostic and Prognostic Research on decision curve analysis.

4. Fairness and Equity

Question: Does the model perform equitably across population subgroups?

Key aspects:

  • Subgroup performance: Stratified metrics by race, ethnicity, gender, age, socioeconomic status
  • Error rate disparities: Differential false positive/negative rates
  • Outcome equity: Does deployment narrow or widen health disparities?
  • Representation: Are all groups adequately represented in training data?

Relevant for: All systems affecting humans

Essential reading: Obermeyer et al., 2019, Science - racial bias in healthcare algorithm affecting millions; Gianfrancesco et al., 2018, JAMA Internal Medicine - potential biases in ML algorithms using EHR data.

Parikh et al. argue epidemiology still lacks frameworks to measure population AI exposure and study health effects at scale, distinguishing ambient AI exposure (algorithmic curation and AI-mediated institutional decisions that reach people without individual opt-in) from personal AI exposure (direct, volitional tool use), and treating AI as a determinant of health rather than only a clinical device (Parikh et al., 2026). Framework Perspective with survey illustration: not a causal estimate of AI causing specific disease outcomes.

5. Implementation Outcomes

Question: Is the model adopted and used effectively in practice?

Key aspects:

  • Adoption: Are users actually using it as intended?
  • Usability: Can users operate it efficiently?
  • Workflow integration: Does it fit smoothly into existing processes?
  • Sustainability: Will it continue to be used and maintained over time?

Relevant for: Any deployed system

Framework: Proctor et al., 2011, Administration and Policy in Mental Health - implementation outcome taxonomy.

Al-Ajlouni and Njei argue that clinical accuracy alone is a weak proxy for system value and propose an AI Productivity Index as a preliminary composite lens for operational impact, implementation burden, opportunity costs, downstream utilisation, and equity of gains or burdens, to sit beside safety, efficacy, and formal economic evaluation in procurement and post-deployment governance (Al-Ajlouni & Njei, Clinical Medicine, online 20 Aug 2026; OA PDF). Their illustrative domains include workflow efficiency, integration burden, clinician cognitive load, downstream resource use, and equity/access impact. Opinion framework with illustrative domains, not a validated scored instrument; do not replace CHEERS-AI or local claim-specific evaluation.

Match Evidence to the Public Health Claim

Evidence for public health AI is multidimensional, not a single ladder. The study design should be judged against the question it is intended to answer, then appraised for risk of bias, directness, precision, and applicability. GRADE assigns certainty to a body of evidence for a specified outcome, not to a journal, vendor, individual paper, or generic validation stage (CDC ACIP GRADE Handbook, 2024).

Evidence question Evidence needed Appropriate appraisal or reporting approach
Does the model perform as specified? Locked-model evaluation on representative data, including calibration, threshold-specific errors, missing data, and subgroup performance TRIPOD+AI for transparent reporting and PROBAST+AI for quality, risk of bias, and applicability (Collins et al., 2024; Moons et al., 2025)
Does performance transfer across populations, jurisdictions, or time? Independent temporal, geographic, and multi-site evaluation in data systems and populations relevant to intended use Explicit assessment of case mix, prevalence, measurement, calibration, operating thresholds, and uncertainty
Does the system work in public health operations? Prospective evaluation of data pipelines, timeliness, representativeness, acceptability, workflow, workload, failure modes, and subgroup effects CDC surveillance-system attributes and, for live decision support, DECIDE-AI reporting items (CDC, 2001; Vasey et al., 2022)
Does using the system improve public health decisions or outcomes? Comparative evidence on decisions, processes, population outcomes, harms, equity, and resource use Randomized designs when feasible, or a justified quasi-experimental design such as an interrupted time series or controlled before-after study; CONSORT-AI applies to reports of randomized AI intervention trials (Liu et al., 2020)
Does benefit persist after deployment? Ongoing measurement of data drift, performance, use, overrides, adverse events, inequitable effects, timeliness, stability, and unintended consequences Risk-based monitoring across the lifecycle; NICE separates design, value, performance, deployment, and post-deployment evidence rather than collapsing them into one study hierarchy (NICE Evidence Standards Framework)

Reporting guidelines improve transparency, but complete reporting does not establish low risk of bias or public health benefit. Risk-of-bias tools and causal designs must also match the question. A randomized trial is well suited to estimating the causal effect of using an AI intervention, but it is not automatically the best design for estimating discrimination, calibration, rare harms, transportability, or long-term drift.

Staged Evaluation Is an Operational Sequence

A staged pathway remains useful for managing deployment risk:

  1. Develop the model and estimate internal performance without data leakage.
  2. Evaluate the locked model on independent temporal, geographic, or multi-site data relevant to intended use.
  3. Run silent or limited prospective evaluation to test live data pipelines, workflow, and failure modes.
  4. Use a comparative design when the claim concerns effects on decisions, outcomes, equity, or resource use.
  5. Continue surveillance after deployment because performance and use can change.

These stages are complementary. They should not be converted into a universal weakest-to-strongest evidence score.

For lifecycle staging of medical AI evaluation (five gated phases with fall-back triggers), see Ye et al. in npj Digital Medicine (Ye et al., 2026). Framework proposal; the Physician AI Handbook carries the fuller teaching.

Recent Evidence That Changes Evaluation Practice

Recent studies reinforce seven controls that should be specified before evaluation begins:

  1. Separate process measures from outcomes. A pragmatic cluster-randomized trial across Kenyan primary-care facilities found improvements in several expert-rated care-process measures with generative AI decision support, while its prespecified 14-day treatment-failure outcome did not differ significantly. The result supports better documentation and reasoning processes, not a claim of improved patient outcomes (Agweyu et al., 2026).
  2. Pair static benchmarks with valid perturbation tests. Dynamic red-teaming of health LLMs exposed substantial brittleness among answers that were correct at baseline. The reported attack success depends on the mutation set, search budget, and judge, so it is evidence of stress-test vulnerability rather than an estimate of routine clinical error (Pan et al., 2026).
  3. Treat evaluators as part of the measurement system. Ratings of real clinical-case responses varied across physician seniority and practice setting, and automated evaluators did not fully reproduce the features of physician judgment. Report evaluator composition and test whether rankings are stable across relevant strata (Shi et al., 2026).
  4. Permit automated judges to abstain. MedQADE, a 2026 preprint, found that automated judges could approach clinician-level agreement while failing to express the caution physicians showed on difficult items. Automated evaluation should include uncertainty thresholds, escalation, and model-family bias checks (Philipp et al., 2026, preprint).
  5. Do not equate local relevance with deployment validation. IyawoBench provides locally grounded Nigerian primary-care triage cases, but its synthetic cases and limited reference-label process require independent relabeling, prompt-sensitivity analysis, geographic validation, and prospective evaluation before deployment claims (Gabriel et al., 2026, preprint; Gabriel and Olawuyi, 2026, preprint).
  6. Do not treat a pooled AUC as a TB-program endpoint. A 17 August 2026 PLOS One PRISMA/PROBAST synthesis of AI/ML models for tuberculosis treatment failure included 34 studies and pooled 19 of them (100,790 participants) to AUC 0.836 (95% CI 0.799–0.868) with I² 97.9%; only 8/34 had external validation and 1/34 was low risk of bias overall, while HIV-included cohorts dropped to AUC 0.748 versus 0.924 when HIV-positive patients were excluded (Kamulegeya et al., 2026). The authors write that current models remain investigational and are not ready for routine clinical implementation; the pool is a central-tendency summary across mixed failure definitions, not a DOTS, mortality, or transmission result.
  7. Do not treat VAERS/FAERS disproportionality plus in-sample ML as incidence, causality, or a vaccine-safety endpoint. Qing et al.’s 10 August 2026 Infectious Diseases of Poverty analysis of 9,446 AVA reports found 180 depression-related cases (ROR 5.02, 95% CI 4.33–5.81; signal persisted after excluding concurrent vaccines, n = 119, ROR 4.97) and a random-split Random Forest test AUC of 0.89 (XGBoost 0.856; SHAP dominated by serious-AE and time-to-onset flags, not dose or site), still a spontaneous-reporting model with no denominator (Qing et al., 2026). The authors themselves require the numbers to be read as hypothesis-generating pharmacovigilance, not as AVA-caused depression, not as absolute risk, and not as a deployed immunization-program early-warning system; FAERS levofloxacin / doxycycline RORs (1.37 / 1.53) were likewise not indication-restricted to anthrax. Reporting signal ≠ vaccine-safety endpoint.

The operational rule is simple: define the claim first, then select the outcome, study design, stress tests, evaluator panel, and monitoring plan that can support that claim.

Performance Metrics: Choosing the Right Measures

See Performance Metrics for Public Health AI.

Validation Strategies: Testing Generalization

See Validation, Equity, and Security Testing.

Implementation Outcomes: Beyond Technical Performance

Even models with strong technical performance can fail in practice if not properly implemented.

The Implementation Science Framework

Proctor et al., 2011 define implementation outcomes:

1. Acceptability

Definition: Perception that system is agreeable/satisfactory

Measures:

  • User satisfaction surveys (Likert scales, Net Promoter Score)
  • Qualitative interviews (what do users like/dislike?)
  • Perceived usefulness and ease of use

Example questions:

  • “This system improves my clinical decision-making” (1–5 scale)
  • “I would recommend this system to colleagues” (yes/no)

2. Adoption

Definition: Intention/action to use the system

Measures:

  • Utilization rate (% of eligible cases where system used)
  • Number of users who have activated/logged in
  • Time to initial use

Red flag: Low adoption despite availability suggests problems with acceptability, workflow fit, or perceived utility.

A 19 August 2026 Nature Medicine DECIDE-AI stage 1 evaluation of SHAKED, a multi-LLM CDSS in a tertiary emergency department (1,138 patients over 4 weeks; one wing used the system, one followed routine rotations), is the empirical form of that red flag: clinician adoption fell from 68% to 30% (OR = 0.72 per shift hour, 95% CI 0.62–0.83), expert review rated 99 of 100 sampled outputs as clinically appropriate, and no adverse events were detected, but length of stay was 4.9 h in both wings (P = 0.99) and the intention-to-treat consultation-cycle-time difference (−9.4 min) was not significant (P = 0.077) (Leibovitch et al., 2026). Output appropriateness and decaying adoption are not a care endpoint; the authors write that the results inform randomized-trial design and do not justify clinical deployment. Adoption / appropriateness ≠ care endpoint.

3. Appropriateness

Definition: Perceived fit for setting/population/problem

Measures:

  • Stakeholder perception surveys
  • Alignment with clinical workflows (workflow mapping)
  • Relevance to clinical questions

Example: ICU mortality prediction may be appropriate for ICU but inappropriate for outpatient clinic.

4. Feasibility

Definition: Ability to successfully implement

Measures:

  • Technical integration challenges (API compatibility, data availability)
  • Resource requirements (cost, staff time, training)
  • Infrastructure needs (computing, network)

5. Fidelity

Definition: Degree to which system used as designed

Measures:

  • Override rates (how often do clinicians dismiss alerts?)
  • Deviation from intended use (using for wrong purpose)
  • Workarounds (users circumventing system)

High override rates signal problems:

  • Too many false positives (alert fatigue)
  • Predictions do not match clinical judgment (trust issues)
  • Workflow disruption (alerts at wrong time)

6. Penetration

Definition: Integration across settings/populations

Measures:

  • Number of sites/units using system
  • Proportion of target population reached
  • Geographic spread

7. Sustainability

Definition: Continued use over time

Measures:

  • Retention of users over 6–12 months
  • Model updating/maintenance plan
  • Long-term performance monitoring

Common failure: “Pilot-itis”: successful pilot, but system not sustained after initial implementation period.

Common Implementation Failures

1. Alert Fatigue

Problem: Excessive false alarms → clinicians ignore alerts

Evidence: Ancker et al., 2017, BMC Medical Informatics and Decision Making - Drug-drug interaction alerts overridden 49-96% of time.

Example: At the evaluated threshold, Epic’s sepsis model had 12% positive predictive value, so 88% of patients who would have generated an alert did not meet the study’s sepsis outcome (Wong et al., 2021). The study estimated a substantial alert burden; it did not measure whether clinicians stopped responding.

Solutions:

  • Minimize false positives (sacrifice sensitivity if needed)
  • Tiered alerts (critical vs. informational)
  • Smart timing (deliver when actionable, not during documentation)
  • Actionable recommendations (“Order blood cultures” not “Consider sepsis”)

2. Workflow Disruption

Problem: System does not integrate smoothly into existing processes

Examples:

  • Extra clicks required
  • Separate application (need to switch contexts)
  • Alerts interrupt at inopportune times (during patient exam)

Solutions:

  • User-centered design (involve clinicians early and often)
  • Embed in existing EHR workflows
  • Minimize friction (one-click actions)

For workflow integration principles, see Bates et al., 2003, JAMIA on the “Ten Commandments” for effective clinical decision support.

3. Lack of Trust

Problem: Clinicians do not trust “black box” predictions

Example: Deep learning model provides risk score with no explanation

Solutions:

  • Provide explanations (SHAP values, attention weights)
  • Show evidence base (similar cases, supporting literature)
  • Transparent validation (publish performance data)
  • Gradual trust-building (start with low-stakes recommendations)

4. Model Drift

Problem: Performance degrades over time as data distribution changes

Example: COVID-19 pandemic changed disease patterns → pre-pandemic models failed

Why it matters: A 2017 study of clinical prediction models found that most require recalibration within 2-3 years due to changes in patient populations, treatment patterns, and data collection practices.

MLOps and Continuous Model Monitoring

Moving beyond “deploy and hope”: Modern AI systems require continuous monitoring to detect performance degradation and trigger timely interventions.

Post-Deployment Monitoring Is Not Optional

The FDA’s 2021 Action Plan on AI/ML-based Software as a Medical Device emphasizes continuous monitoring as a core requirement. Models that do not monitor performance drift pose patient safety risks.

Real-world failure: IBM Watson for Oncology was adopted at multiple institutions, and internal company documents reported in 2018 described unsafe and incorrect treatment recommendations.

Types of Drift

Understanding the type of drift helps determine appropriate interventions.

1. Data Drift (Covariate Shift)

Definition: Input feature distributions change, but the relationship between features and outcome remains stable.

Example:

  • Training data (2019): Average patient age = 55, BMI = 27
  • Production data (2024): Average patient age = 62, BMI = 31
  • Relationship unchanged: Diabetes risk per BMI unit = same

Impact: Model may become miscalibrated (predicted probabilities off) even if discrimination (AUC-ROC) stays stable.

Detection: Compare feature distributions between training and production data.

2. Concept Drift

Definition: The relationship between features and outcome changes.

Example:

  • Pre-COVID (2019): Fever + cough + dyspnea → Likely bacterial pneumonia
  • During COVID (2020): Fever + cough + dyspnea → Likely COVID-19
  • Same features, different outcome

Impact: Model discrimination (AUC-ROC) degrades significantly.

Detection: Track performance metrics over time; sudden drops indicate concept drift.

3. Label Drift

Definition: Prevalence of outcome changes.

Example:

  • Training data: 5% sepsis prevalence
  • Production: 12% sepsis prevalence (sicker patient population)

Impact: Predicted probabilities may be systematically too low or too high (calibration degrades).

Detection: Compare outcome rates over time.

Practical Drift Detection Methods

Method 1: Statistical Tests for Feature Distribution Changes

Kolmogorov-Smirnov (KS) Test:

Tests whether two distributions are significantly different.

Hide code 
from scipy.stats import ks_2samp
import numpy as np
import pandas as pd

# Example: Monitoring patient age distribution
# Training data (historical)
age_train = np.random.normal(55, 15, 1000) # Mean=55, SD=15

# Production data (current month)
age_prod = np.random.normal(62, 15, 500) # Mean shifted to 62

# Perform KS test
statistic, p_value = ks_2samp(age_train, age_prod)

print(f"KS statistic: {statistic:.3f}")
print(f"P-value: {p_value:.4f}")

if p_value < 0.01:
 print("[ALERT] Significant distribution shift detected!")
 print("→ Review model calibration and consider retraining")
else:
 print("[OK] No significant drift detected")

# Interpretation:
# p < 0.01: Strong evidence of distribution change
# p < 0.05: Moderate evidence of distribution change
# p ≥ 0.05: No significant change detected

When to use: Continuous features (age, lab values, vital signs)

Chi-Square Test (for categorical features):

Hide code 
from scipy.stats import chi2_contingency

# Example: Monitoring sex distribution
# Training data
train_counts = {"Male": 600, "Female": 400} # 60% male

# Production data (current month)
prod_counts = {"Male": 250, "Female": 250} # 50% male

# Create contingency table
contingency_table = pd.DataFrame({
 'Train': [train_counts['Male'], train_counts['Female']],
 'Production': [prod_counts['Male'], prod_counts['Female']]
}, index=['Male', 'Female'])

chi2, p_value, dof, expected = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.3f}")
print(f"P-value: {p_value:.4f}")

if p_value < 0.01:
 print("[ALERT] Significant distribution shift in sex distribution!")

When to use: Categorical features (sex, race, diagnostic codes)

Method 2: Population Stability Index (PSI)

Widely used in industry for monitoring feature drift.

Formula:

\[PSI = \sum_{i=1}^{n} (P_{prod,i} - P_{train,i}) \times \ln\left(\frac{P_{prod,i}}{P_{train,i}}\right)\]

Where:

  • \(P_{train,i}\) = Proportion in training data bin \(i\)
  • \(P_{prod,i}\) = Proportion in production data bin \(i\)

Interpretation:

  • PSI < 0.1: No significant change (green)
  • PSI 0.1–0.25: Moderate change, investigate (yellow)
  • PSI > 0.25: Significant change, likely requires retraining (red)

Code example:

Hide code 
import numpy as np

def calculate_psi(expected, actual, bins=10):
 """
 Calculate Population Stability Index

 Args:
  expected: Training data feature values
  actual: Production data feature values
  bins: Number of bins for discretization

 Returns:
  PSI value
 """
 # Create bins based on training data quantiles
 breakpoints = np.percentile(expected, np.linspace(0, 100, bins + 1))
 breakpoints[-1] = breakpoints[-1] + 0.001 # Ensure max value included

 # Calculate proportions in each bin
 expected_counts = np.histogram(expected, bins=breakpoints)[0]
 actual_counts = np.histogram(actual, bins=breakpoints)[0]

 expected_props = expected_counts / len(expected)
 actual_props = actual_counts / len(actual)

 # Avoid log(0) by adding small constant
 expected_props = np.where(expected_props == 0, 0.0001, expected_props)
 actual_props = np.where(actual_props == 0, 0.0001, actual_props)

 # Calculate PSI
 psi_values = (actual_props - expected_props) * np.log(actual_props / expected_props)
 psi = np.sum(psi_values)

 return psi

# Example: Monitor glucose values
glucose_train = np.random.gamma(5, 20, 1000) # Training data
glucose_prod = np.random.gamma(6, 22, 500) # Production data (shifted)

psi = calculate_psi(glucose_train, glucose_prod, bins=10)

print(f"PSI: {psi:.3f}")

if psi < 0.1:
 print("[OK] No significant drift")
elif psi < 0.25:
 print("[WARNING] Moderate drift - investigate")
else:
 print("[CRITICAL] Significant drift - retrain model")

Method 3: Model Performance Monitoring

Most direct approach: Track actual model performance over time.

Challenge: Requires ground truth labels, which may have delay (e.g., 30-day readmission cannot be verified for 30 days).

Metrics to monitor:

  • AUC-ROC (discrimination)
  • Brier score (calibration)
  • Sensitivity/specificity at operational threshold
  • PPV/NPV

Implementation:

Hide code 
import matplotlib.pyplot as plt
from sklearn.metrics import roc_auc_score, brier_score_loss
from datetime import datetime, timedelta

# Simulate monthly performance tracking
months = pd.date_range(start='2023-01-01', end='2024-12-01', freq='MS')

# Simulated AUC-ROC over time (degrading model)
np.random.seed(42)
baseline_auc = 0.85
auc_scores = baseline_auc - np.linspace(0, 0.10, len(months)) + np.random.normal(0, 0.02, len(months))

# Create monitoring dashboard data
performance_data = pd.DataFrame({
 'month': months,
 'auc': auc_scores
})

# Set alert thresholds
auc_warning_threshold = 0.80
auc_critical_threshold = 0.75

# Plot performance over time
plt.figure(figsize=(10, 6))
plt.plot(performance_data['month'], performance_data['auc'],
   marker='o', linewidth=2, label='Monthly AUC-ROC')

# Add threshold lines
plt.axhline(y=baseline_auc, color='green', linestyle='--',
   label=f'Baseline ({baseline_auc:.2f})', alpha=0.7)
plt.axhline(y=auc_warning_threshold, color='orange', linestyle='--',
   label=f'Warning threshold ({auc_warning_threshold:.2f})', alpha=0.7)
plt.axhline(y=auc_critical_threshold, color='red', linestyle='--',
   label=f'Critical threshold ({auc_critical_threshold:.2f})', alpha=0.7)

# Highlight months below threshold
alerts = performance_data[performance_data['auc'] < auc_warning_threshold]
if not alerts.empty:
 plt.scatter(alerts['month'], alerts['auc'],
    color='red', s=100, zorder=5, label='Alert triggered')

plt.xlabel('Month')
plt.ylabel('AUC-ROC')
plt.title('Model Performance Monitoring Dashboard')
plt.legend()
plt.grid(True, alpha=0.3)
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Check for alerts
latest_auc = performance_data['auc'].iloc[-1]
if latest_auc < auc_critical_threshold:
 print(f"[CRITICAL] AUC-ROC = {latest_auc:.3f} (below {auc_critical_threshold})")
 print("→ IMMEDIATE ACTION: Suspend model or initiate emergency retraining")
elif latest_auc < auc_warning_threshold:
 print(f"[WARNING] AUC-ROC = {latest_auc:.3f} (below {auc_warning_threshold})")
 print("→ ACTION: Schedule model retraining within 30 days")
else:
 print(f"[OK] Performance acceptable: AUC-ROC = {latest_auc:.3f}")

Method 4: Prediction Distribution Monitoring

Insight: Even without ground truth, you can monitor what the model is predicting.

Red flag patterns:

  • Sudden increase in high-risk predictions (model becoming overly sensitive)
  • Sudden decrease in high-risk predictions (model missing cases)
  • Bimodal distribution shifts (calibration degradation)

Example:

Hide code 
# Monitor distribution of predicted probabilities over time

# Week 1 predictions (well-calibrated)
week1_preds = np.random.beta(2, 10, 1000) # Mostly low risk

# Week 12 predictions (drift - more high-risk predictions)
week12_preds = np.random.beta(3, 7, 1000) # Shifted higher

# Visualize
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.hist(week1_preds, bins=50, alpha=0.7, label='Week 1', density=True)
ax1.hist(week12_preds, bins=50, alpha=0.7, label='Week 12', density=True)
ax1.set_xlabel('Predicted Probability')
ax1.set_ylabel('Density')
ax1.set_title('Prediction Distribution Shift')
ax1.legend()

# KS test to detect shift
ks_stat, p_val = ks_2samp(week1_preds, week12_preds)
ax2.text(0.5, 0.5, f'KS test p-value: {p_val:.4f}\n' +
   ('[WARNING] Significant shift detected' if p_val < 0.01 else '[OK] No significant shift'),
   ha='center', va='center', fontsize=14,
   bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))
ax2.axis('off')

plt.tight_layout()
plt.show()

Automated Retraining Triggers

Goal: Define clear rules for when to retrain the model.

Retraining Decision Framework

IMMEDIATE retraining triggered by:

  • AUC-ROC drops >0.05 below baseline
  • Calibration error (Brier score) increases >0.05
  • Safety event: Model missed critical case (e.g., sepsis death after negative prediction)
  • Feature drift: PSI > 0.25 for ≥3 critical features

SCHEDULED retraining triggered by:

  • 12 months since last training (routine maintenance)
  • AUC-ROC drops 0.02–0.05 below baseline (warning level)
  • PSI 0.1–0.25 for multiple features (moderate drift)
  • New data volume: ≥20% new samples since last training

HOLD retraining if:

  • All metrics within acceptable ranges
  • Recent retraining (< 3 months ago)
  • Insufficient new data (< 5% new samples)

Automated monitoring script:

Hide code 
def evaluate_retraining_need(current_auc, baseline_auc,
        psi_scores, months_since_training,
        new_sample_fraction):
 """
 Automated decision system for model retraining

 Returns:
  - "IMMEDIATE": Retrain immediately
  - "SCHEDULED": Schedule retraining within 30 days
  - "MONITOR": Continue monitoring, no action needed
 """
 # Critical performance degradation
 if current_auc < baseline_auc - 0.05:
  return "IMMEDIATE", "AUC-ROC dropped >0.05"

 # Critical feature drift
 critical_drift_count = sum([psi > 0.25 for psi in psi_scores])
 if critical_drift_count >= 3:
  return "IMMEDIATE", f"{critical_drift_count} features with PSI > 0.25"

 # Routine maintenance schedule
 if months_since_training >= 12:
  return "SCHEDULED", "12-month routine retraining"

 # Moderate performance degradation
 if current_auc < baseline_auc - 0.02:
  return "SCHEDULED", "AUC-ROC dropped 0.02-0.05"

 # Moderate drift
 moderate_drift_count = sum([0.1 < psi < 0.25 for psi in psi_scores])
 if moderate_drift_count >= 4:
  return "SCHEDULED", f"{moderate_drift_count} features with moderate drift"

 # Significant new data
 if new_sample_fraction >= 0.20:
  return "SCHEDULED", f"{new_sample_fraction:.0%} new data available"

 return "MONITOR", "All metrics within acceptable ranges"

# Example usage
decision, reason = evaluate_retraining_need(
 current_auc=0.82,
 baseline_auc=0.85,
 psi_scores=[0.08, 0.15, 0.22, 0.18, 0.05], # PSI for 5 key features
 months_since_training=8,
 new_sample_fraction=0.15
)

print(f"Decision: {decision}")
print(f"Reason: {reason}")

Continuous Learning Strategies

Two approaches:

2. Online Learning (Advanced, requires careful monitoring)

  • Process: Model updates continuously with new data
  • Advantages: Always current
  • Disadvantages:
  • Risk of catastrophic forgetting
  • Harder to validate
  • Vulnerable to adversarial data poisoning

Recommendation for healthcare: Use periodic retraining with trigger-based scheduling (not true online learning).

Real-World Example: External Validation After Adoption

Epic’s sepsis model had already been implemented at hundreds of US hospitals when Wong and colleagues published an independent external validation at one academic health system (Wong et al., 2021).

At the evaluated threshold, the study reported:

  • Sensitivity: 33%
  • Positive predictive value: 12%
  • Alert burden: 18% of hospitalizations would have generated at least one alert

The study showed poor discrimination and calibration in that setting. It did not test model drift, continuous monitoring at other deployment sites, or a retraining protocol.

Lesson: Widespread adoption is not evidence of transportability or clinical value. Independent evaluation should precede broad use, and post-deployment monitoring should address questions that a retrospective external validation cannot answer.

Reference: Wong et al., 2021, JAMA Internal Medicine

Key Takeaways: Model Drift and Monitoring

  1. All models drift: Performance degradation is inevitable, not exceptional

  2. Types of drift matter: Data drift vs. concept drift require different interventions

  3. Multiple detection methods: Use statistical tests (KS, Chi-square), PSI, and performance tracking simultaneously

  4. Automated triggers: Define clear thresholds for retraining (do not wait for catastrophic failure)

  5. Continuous monitoring is mandatory: FDA and EU regulations increasingly require post-deployment monitoring

  6. Periodic retraining > online learning: For healthcare, controlled validation is safer than continuous updates

  7. Monitor before you have ground truth: Prediction distribution shifts can signal problems early

  8. Document everything: Track what triggered retraining, what changed, and validation results

Framework reference: Davis et al., 2017, JAMIA - Clinical prediction models degrade; most need recalibration after 2-3 years.

Explainability and Interpretability (XAI)

See Explainability for Public Health AI.

Regulatory Evaluation Frameworks

The Evolving Regulatory Landscape for Medical AI

Current reality: AI evaluation is no longer only a scientific question. It is also a regulatory requirement.

Medical AI systems face scrutiny from multiple regulatory bodies:

  • FDA (United States): Software as a Medical Device (SaMD) framework
  • European Union: AI Act (2024) classifying medical AI as “high-risk”
  • UK MHRA: Software and AI as a Medical Device framework
  • Health Canada: Medical Device Regulations for AI/ML

Key insight: Understanding regulatory evaluation requirements is essential for deployment, not just compliance.

Why Regulatory Context Matters for Evaluation

Even if you are not directly developing a regulatory-approved device, understanding these frameworks helps you:

  1. Design better evaluations: Regulatory standards define best practices
  2. Anticipate deployment requirements: Many health systems require FDA clearance or equivalent
  3. Benchmark your work: Compare your validation against regulatory expectations
  4. Communicate with stakeholders: Speak the language of hospital legal/compliance teams

Cross-reference: Policy and Governance covers broader policy implications; the requirements below are those specific to evaluation.

FDA Software as a Medical Device (SaMD) Framework

Software as a Medical Device (SaMD) refers to software intended for medical purposes that operates independently of hardware medical devices.

Examples:

  • SaMD: Sepsis prediction algorithm, diabetic retinopathy screening app, radiology CAD software
  • Not SaMD: EHR system (administrative), fitness tracker (wellness, not medical diagnosis)

Risk-Based Classification System

The International Medical Device Regulators Forum (IMDRF) framework, linked under Resources, places SaMD in risk categories I (lowest) to IV (highest) according to the state of the healthcare situation and the significance of the information the software provides. This categorization is separate from FDA device classification (Class I, II, or III), which determines the premarket pathway.

IMDRF SaMD Risk Categorization Matrix

State of healthcare Significance of information
Treat or diagnose Drive clinical management Inform clinical management
Critical IV (highest risk) III II
Serious III II I
Non-serious II I I (lowest risk)

Definitions:

  • Critical: Death or permanent impairment (e.g., ICU monitoring)
  • Serious: Long-term morbidity (e.g., cancer diagnosis)
  • Non-serious: Minor conditions (e.g., acne treatment)

Information significance:

  • Treat/Diagnose: Directly triggers treatment decisions
  • Drive clinical management: Significant influence on treatment path
  • Inform clinical management: One input among many

Evaluation Expectations by Risk Level

The three tiers below are illustrative, rising with risk; they are not regulatory definitions.

Level I (Low Risk):

  • Example: App suggesting lifestyle modifications for mild hypertension
  • Requirements:
  • Basic technical performance validation
  • User studies demonstrating safe use
  • Minimal clinical validation

Level II (Moderate Risk):

  • Example: Algorithm flagging abnormal chest X-rays for radiologist review
  • Requirements:
  • Robust internal validation
  • External validation recommended
  • Clinical utility assessment
  • Usability testing
  • Performance monitoring plan

Level III (High Risk):

  • Example: AI system autonomously diagnosing cancer, guiding ICU interventions
  • Requirements:
  • Extensive multi-site external validation
  • Prospective clinical studies
  • Randomized controlled trials (for novel interventions)
  • Comprehensive fairness audits
  • Predetermined change control plan (for adaptive algorithms)
  • Post-market surveillance

Good Machine Learning Practice (GMLP) Principles

FDA, Health Canada, and UK MHRA jointly published 10 GMLP principles (2021) for ML-based medical devices:

1. Multi-disciplinary expertise throughout ML lifecycle

2. Good software engineering practices

  • Version control, testing, documentation

3. Clinical study participants are representative

  • Training data reflects intended use population

4. Training datasets are independent of test datasets

  • No data leakage

5. Selected reference datasets are based on best available methods

  • Gold-standard labels (expert consensus, biopsy confirmation, etc.)

6. Model design is tailored to available data and reflects intended use

  • Avoid overfitting, ensure clinical relevance

7. Focus on human-AI team performance

  • Evaluate AI + clinician together, not AI in isolation

8. Testing demonstrates device performance in clinically relevant conditions

  • Real-world data, relevant patient populations

9. Users are provided with clear, essential information

  • Transparency about limitations, training data, performance metrics

10. Deployed models are monitored for performance

  • Post-market surveillance, drift detection

Reference: FDA/MHRA/Health Canada, 2021 - GMLP Guiding Principles

Predetermined Change Control Plans (PCCP)

Challenge: Traditional medical devices are static; ML models need updating.

FDA’s current framework (December 2024 final guidance): Predetermined Change Control Plans allow FDA-reviewed, pre-specified modifications to AI-enabled devices without separate marketing submissions for each covered update.

What can be included in a PCCP:

1. Allowable modifications:

  • Retraining on new data (within specified bounds)
  • Algorithm parameter adjustments
  • Performance improvements

2. Modification protocol:

  • Data requirements: Minimum sample size, quality standards
  • Performance thresholds: Must maintain ≥ X sensitivity
  • Validation approach: Test set size, external validation sites

3. Update assessment:

  • Performance monitoring triggers retraining
  • Validation results compared to pre-specified thresholds
  • Automated decision: deploy update or flag for review

4. Transparency and reporting:

  • Change documentation
  • Performance reports to regulators
  • User notifications

Example PCCP:

### Sepsis Prediction Model PCCP

**Allowable Change:** Retrain model quarterly using new institutional data

**Modification Protocol:**
- Minimum 5,000 new patient encounters with ≥200 sepsis cases
- Maintain AUC-ROC ≥ 0.82 (original validation: 0.85)
- Sensitivity at 80% specificity must be ≥ 70%
- External validation on ≥1 additional hospital required

**Assessment:**
- Automated performance monitoring (monthly)
- Retraining triggered if AUC drops below 0.83
- New model validated on hold-out test set (20% of new data)
- Deploy if all thresholds met; otherwise, flag for manual review

**Reporting:**
- Quarterly performance reports to FDA
- User notification of model updates via EHR alert

EU AI Act: High-Risk Classification for Medical AI

Entered into force 1 August 2024, with obligations applying in stages. Under the Regulation as adopted, requirements for high-risk AI systems that are products, or safety components of products, covered by EU harmonization legislation such as the Medical Devices Regulation apply from 2 August 2027 (Article 113).

Key classification: Most medical AI systems are high-risk, requiring:

1. Risk management system:

  • Continuous identification and mitigation of risks

2. Data governance:

  • Training data quality assurance
  • Bias detection and mitigation
  • Data representativeness

3. Technical documentation:

  • Detailed model specifications
  • Training procedures
  • Validation results

4. Transparency:

  • Users informed that AI is involved
  • Clear information on limitations

5. Human oversight:

  • Human-in-the-loop for high-stakes decisions

6. Accuracy, robustness, cybersecurity:

  • Performance standards
  • Adversarial robustness testing
  • Security safeguards

7. Post-market monitoring:

  • Continuous performance tracking
  • Incident reporting

Implications for evaluation:

  • Match validation populations, sites, and time periods to the intended use
  • Evaluate subgroup performance and foreseeable discriminatory effects
  • Test robustness and cybersecurity using methods appropriate to the threat model
  • Establish postmarket monitoring where the applicable legal and risk framework requires it

Reference: EU AI Act, 2024

Practical Implications for Evaluation

Regulatory-Aligned Evaluation Checklist

Even if your system is not currently seeking regulatory approval, aligning with these standards ensures quality:

Before Development:

During Development:

Validation Phase:

Pre-Deployment:

Post-Deployment:

Comparison: Research vs. Regulatory Evaluation Standards

Aspect Research Publication Regulatory Approval
External validation Recommended, often skipped (6% in a 2019 review) Expected for higher-risk devices; evidence requirements vary by pathway
Prospective testing Rare Often required for novel high-risk devices
Fairness audit Increasingly expected Mandatory (EU AI Act)
Post-market monitoring Not required Mandatory
Clinical utility Recommended Pathway-dependent; 510(k) requires substantial equivalence, not demonstrated outcome benefit
Documentation Methods section Extensive technical documentation
Timeline Months 1–3+ years (depending on risk level)

Key insight: Regulatory standards are higher than typical research standards. If you aim for deployment, plan for regulatory-level evaluation from the start.

A 19 August 2026 PLOS Digital Health census of all 1,357 FDA-authorized AI/ML devices through 5 December 2025 bounds that “must demonstrate benefit” cell: only 34 (2.5%) had a registered prospective trial, 12 (0.9%) posted results, and 3 (0.2%) were evaluated on patient-centered outcomes such as mortality, morbidity, or readmissions (Abulibdeh et al., 2026). The 510(k) pathway requires substantial equivalence to a predicate, not an NDA-style outcome trial, so clearance is not demonstrated patient benefit. That 0.2% is a public-registry attrition count, not a care endpoint; the authors note the census may undercount manufacturer-held validation and does not mean FDA has no post-market surveillance.

When to Seek Regulatory Approval

You likely need FDA clearance/approval if:

  • System makes diagnostic or treatment recommendations
  • Used for screening (e.g., diabetic retinopathy, cancer)
  • Influences clinical decision-making significantly
  • Marketed as improving health outcomes

You might NOT need approval if:

  • Administrative tools (scheduling, billing)
  • Wellness apps (general fitness, not medical claims)
  • Clinical decision support providing information only (not recommendations)
  • Gray area: FDA discretion, often depends on risk

When in doubt: Consult FDA’s Digital Health Center of Excellence or regulatory counsel.

Resources for Regulatory Evaluation

FDA:

International:

Academic:

Key Takeaways: Regulatory Evaluation

  1. Regulatory requirements are evaluation requirements: FDA, EU standards define rigorous validation expectations

  2. Risk-based approach: Evidence requirements should reflect intended use, the consequence of error, degree of autonomy, and difference between the evaluation setting and intended setting

  3. Applicability must be demonstrated: Regulatory requirements vary by function and pathway; authorization does not substitute for evidence in the intended population and setting

  4. Continuous monitoring is required: Post-market surveillance, not just pre-deployment validation

  5. Good Machine Learning Practice: 10 principles provide practical framework for development and evaluation

  6. Predetermined change control plans: Enable adaptive models while maintaining regulatory compliance

  7. EU AI Act raises the bar: High-risk systems face requirements for risk management, data governance, transparency, human oversight, accuracy, robustness, cybersecurity, and postmarket monitoring. The evidence methods must fit the system and applicable conformity-assessment obligations.

  8. Plan early: Regulatory evaluation takes longer and costs more than research validation; design for it from the start

  9. Seek expert guidance: Regulatory pathways are complex; consult with regulatory specialists

  10. Standards improve quality: Even if not seeking approval, regulatory frameworks represent best practices

Evaluation Framework

Complete Evaluation Checklist

Use this when evaluating AI systems:

AI System Evaluation Checklist

TECHNICAL PERFORMANCE

VALIDATION RIGOR

FAIRNESS AND EQUITY

CLINICAL UTILITY

TRANSPARENCY AND REPRODUCIBILITY

IMPLEMENTATION PLANNING

LIMITATIONS

Reporting Guidelines

TRIPOD: Transparent Reporting of Prediction Models

Collins et al., 2015, BMJ - TRIPOD statement

22-item checklist for prediction model studies:

Title and Abstract

  1. Title
  2. Abstract

Introduction

  1. Background and objectives

Methods

  1. Source of data
  2. Participants
  3. Outcome
  4. Predictors
  5. Sample size
  6. Missing data
  7. Statistical analysis methods
  8. Risk groups
  9. Development versus validation

Results

  1. Participants
  2. Model development
  3. Model specification
  4. Model performance
  5. Model updating

Discussion

  1. Limitations
  2. Interpretation
  3. Implications

Other Information

  1. Supplementary information
  2. Funding

Current guidance: TRIPOD+AI superseded the 2015 TRIPOD checklist for clinical prediction model studies using regression or machine learning methods (Collins et al., 2024). It strengthens reporting of data sources, model development, evaluation, fairness, and open-science practices. TRIPOD+AI is a reporting guideline, not a method for rating evidence certainty or risk of bias; PROBAST+AI serves the latter appraisal role.

TRIPOD-LLM: Reporting Guidelines for LLM Studies

Traditional TRIPOD was designed for classical prediction models. Large language models require fundamentally different reporting standards. TRIPOD-LLM (Gallifant et al., 2025, Nature Medicine) provides the first LLM-specific extension, endorsed by EQUATOR Network.

Why LLM-specific guidelines are needed:

  • Training data for LLMs is often undisclosed or incompletely characterized
  • Performance varies dramatically with prompt wording
  • Contamination between training and test data is difficult to verify
  • Reproducibility requires specifying API versions, dates, and system prompts

Key TRIPOD-LLM additions:

Category New Requirements
Model specification API version, access date, temperature/sampling parameters
Prompting Full prompt text, development process, any prompt optimization
Input/output How health data was formatted, output parsing methods
Reproducibility Whether results vary across API calls, random seed handling
Contamination Assessment of whether test data appeared in training corpus

The LLM reporting crisis:

A systematic review of LLM chatbot health advice studies (Huo et al., 2025, JAMA Network Open) examined 137 studies and found that 136 studies (99.3%) used closed-source models. Key findings:

  • No study sufficiently described model characteristics such as temperature and token length
  • 136 studies (99.3%) did not describe a prompt engineering phase
  • The date of model querying was reported in only 54 studies (39.4%)

These reporting gaps prevent faithful reproduction of most studies. TRIPOD-LLM directly addresses these gaps.

CHART Statement (Chatbot Assessment Reporting Tool):

Complementing TRIPOD-LLM, the CHART statement (Huo et al., 2025, JAMA Network Open) provides a 12-item, 39-subitem checklist specifically for evaluating chatbot and conversational AI systems in healthcare settings.

STARD-AI: Standards for Reporting Diagnostic Accuracy Using AI

Extension of STARD guidelines for diagnostic AI.

Additional items:

  • Model architecture details
  • Training procedure (epochs, batch size, optimization)
  • Validation strategy
  • External validation results
  • Subgroup analyses
  • Calibration assessment
  • Comparison to human performance (if applicable)

Critical Appraisal of Published Studies

Systematic Evaluation Framework

When reading AI studies:

1. Study Design and Data Quality

Questions:

  • Representative sample of target population?
  • External validation performed?
  • Test set truly independent?
  • Outcome objectively defined and consistently measured?
  • Potential for data leakage?

Red flags:

  • No external validation
  • Small sample size (<500 events)
  • Convenience sampling
  • Vague outcome definitions
  • Feature engineering on entire dataset before splitting

2. Model Development and Reporting

Questions:

  • Multiple models compared?
  • Simple baseline included (logistic regression)?
  • Hyperparameters tuned on separate validation set?
  • Feature selection appropriate?
  • Model clearly described?

Red flags:

  • No baseline comparison
  • Hyperparameter tuning on test set
  • Inadequate model description
  • No cross-validation

3. Performance Evaluation

Questions:

  • Appropriate metrics for task?
  • Confidence intervals provided?
  • Calibration assessed?
  • Multiple metrics reported?
  • Statistical testing appropriate?

Red flags:

  • Only accuracy reported (especially for imbalanced data)
  • No calibration assessment
  • No confidence intervals
  • Cherry-picked metrics

4. Fairness and Generalizability

Questions:

  • Performance stratified by subgroups?
  • Diverse populations included?
  • Generalizability limitations discussed?
  • Potential biases identified?

Red flags:

  • No subgroup analysis
  • Homogeneous study population
  • Claims of broad generalizability without external validation
  • Dismissal of fairness concerns

5. Clinical Utility

Questions:

  • Clinical utility assessed (beyond accuracy)?
  • Compared to current practice?
  • Implementation considerations discussed?
  • Cost-effectiveness assessed?

Red flags:

  • Only technical metrics
  • No comparison to existing approaches
  • No implementation discussion
  • Overstated clinical claims

6. Transparency and Reproducibility

Questions:

  • Code and data available?
  • Reporting guidelines followed?
  • Sufficient detail to reproduce?
  • Limitations clearly stated?
  • Conflicts of interest disclosed?

Red flags:

  • No code/data availability
  • Insufficient methodological detail
  • Overstated conclusions
  • Undisclosed industry funding

Key Takeaways

Essential Principles

  1. Evaluation is multidimensional: Technical performance, clinical utility, fairness, and implementation outcomes all matter

  2. Internal validation is insufficient: External validation on independent data is essential to assess generalizability

  3. Calibration is critical: Predicted probabilities must be meaningful for clinical decisions, not just discriminative

  4. Assess fairness proactively: Stratify performance by demographic subgroups; disparities invisible otherwise

  5. Clinical utility ≠ statistical performance: A model can be statistically accurate but clinically useless without improving outcomes

  6. Prospective evaluation answers operational questions: Live testing can assess data pipelines, workflow, timeliness, and human factors; comparative evidence is needed for causal claims about impact

  7. Common pitfalls are avoidable: Data leakage, improper CV, threshold optimization on test set lead to overoptimistic estimates

  8. Implementation determines success: Even well-performing models fail if workflow integration ignored

  9. Transparency enables trust: Follow reporting guidelines (TRIPOD, STARD-AI); share code and data when possible

  10. Continuous monitoring is essential: Model performance drifts over time; plan for ongoing evaluation and updating

Check Your Understanding

See Public Health AI Evaluation Exercises.

What is external validation in health AI?

External validation evaluates a locked model on data that differ meaningfully from development data, such as a new institution, geography, device, population, or time period. The test set should represent the intended transfer and should not be used repeatedly for model selection. External validation addresses transportability for the measured endpoint. It does not by itself establish workflow benefit or patient outcomes.

What is calibration, and why does it matter?

Calibration describes whether predicted probabilities correspond to observed frequencies in the target setting. A well-discriminating model can still systematically overestimate or underestimate risk, especially when prevalence or practice changes. Calibration should be examined overall and in relevant subgroups, with uncertainty. Recalibration may improve probabilities but does not repair a wrong target, missing predictors, or an ineffective intervention.

How should evidence be matched to a public health AI claim?

Technical performance claims require a suitable reference standard and independent test data. Transportability claims require external validation. Workflow claims require prospective use in the intended setting. Outcome-benefit claims require a comparative design measuring the relevant outcome. Economic claims require explicit costs, perspective, time horizon, and comparator. No single study design answers every question, so the claim should be narrowed to what the design and endpoint measured.

What are vendor-evaluation red flags?

Red flags include an undisclosed model version, no independent test set, performance reported only as accuracy or AUC, missing confidence intervals, absent subgroup results, an inappropriate comparator, results from a different workflow, and no monitoring or update plan. Regulatory authorization, publication, or deployment elsewhere should not substitute for local evidence. A vendor should be able to provide a connected record from the claim to the source and exact product version.

Discussion Questions

  1. Claim-specific evidence plan: You’ve developed a hospital-acquired infection prediction model. Which evidence would you require for technical performance, transportability, operational utility, population impact, equity, and lifecycle safety? Which studies would you conduct before deployment, and why?

  2. Fairness trade-offs: Your sepsis model has AUC-ROC = 0.85 overall, but sensitivity is 0.90 for White patients vs. 0.75 for Black patients. What would you do? What are the trade-offs of different mitigation approaches?

  3. Calibration vs. discrimination: Model A: AUC-ROC = 0.85, Brier score = 0.30 (poor calibration). Model B: AUC-ROC = 0.80, Brier score = 0.15 (excellent calibration). Which would you deploy, and why?

  4. External validation failure: Your model achieves AUC-ROC = 0.82 in internal validation but 0.68 in external validation at a different hospital. What explains this? What are the next steps?

  5. Clinical utility skepticism: Model predicts 30-day mortality with AUC-ROC = 0.88. Does this mean it’s clinically useful? What additional evaluations are needed?

  6. Prospective study design: A hospital readmission model needs prospective evaluation. Would you choose an RCT, a stepped-wedge design, or silent mode? What are the trade-offs?

  7. Alert threshold selection: A clinical decision support tool can alert at >10%, >20%, or >30% predicted risk. How would you choose? What factors matter?

  8. Model drift: COVID-19 forecasting model trained on 2020 data; it is now three years later, with new variants. How would you assess whether it is still valid? What should trigger retraining?

Further Resources

Books

Essential Papers

Validation:

Fairness:

Clinical Utility:

Implementation:

Tools

Metrics and Validation:

Fairness:

Explainability:

  • SHAP - Feature importance
  • LIME - Local explanations
Continue Reading
  • AI Deployment in Healthcare: How to move validated models into production, monitor for drift, and manage the organizational change that determines real-world success
  • AI Safety in Healthcare: Risk frameworks, failure mode analysis, and when to pull a deployed model out of production
  • AI Ethics, Bias, and Equity: Deeper coverage of fairness metrics, bias mitigation strategies, and equity considerations across populations

Next: Performance Metrics for Public Health AI →

The Public Health AI Handbook is free to read and cite, with no advertising. If this chapter informed your work, you can support the handbook.