Public Health AI Evaluation Exercises

Applied exercises for evaluating performance, generalization, equity, robustness, and implementation outcomes. The material is maintained separately so each operational question has a stable, focused reference.

Learning Objectives
  • Identify the evidence and controls relevant to this decision area
  • Distinguish technical performance from operational and population impact
  • Apply the included framework without extending claims beyond the cited evidence

Use explicit targets, populations, thresholds, and decision consequences. Require external evidence and local monitoring where deployment can affect people or programs. Preserve uncertainty and document limits.

Introduction

This focused reference is part of the broader Public Health AI Evaluation Exercises overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.

Check Your Understanding

Test your knowledge of the key concepts from this chapter. Click “Show Answer” to reveal the correct response and explanation.

Question 1: Cross-Validation Strategy

You’re building a model to predict hospital readmissions using data from 2018-2023. Which cross-validation strategy is MOST appropriate?

  1. 10-fold random cross-validation
  2. Leave-one-out cross-validation
  3. Stratified K-fold cross-validation
  4. Time-based forward-chaining cross-validation

Answer: d) Time-based forward-chaining cross-validation

Explanation: Time-based (temporal) cross-validation is essential for healthcare data with temporal dependencies:

Why temporal CV is critical:

Fold 1: Train 2018-2019 → Test 2020
Fold 2: Train 2018-2020 → Test 2021
Fold 3: Train 2018-2021 → Test 2022
Fold 4: Train 2018-2022 → Test 2023

What this tests: - Model performance as deployed (using past to predict future) - Robustness to temporal drift (treatment changes, policy updates) - Realistic performance estimates

Why not random K-fold (a)? Creates data leakage:

Train: [2019, 2021, 2023]
Test: [2018, 2020, 2022]

You’re using 2023 data to predict 2018! Inflates performance artificially.

Why not leave-one-out (b)? - Computationally expensive - Still has temporal leakage problem - High variance in estimates

Why not stratified K-fold (c)? - Useful for class imbalance - But still allows temporal leakage - Doesn’t test temporal robustness

Real-world impact: Models validated with random CV often show 10-20% performance drops when deployed because they never faced forward-looking prediction during validation.

Lesson: Healthcare data has temporal structure. Always validate as you’ll deploy, using past to predict future, never the reverse.

Question 2: Calibration Assessment

A cancer risk model predicts 20% risk for 1,000 patients. In reality, 300 of these patients develop cancer. What does this indicate?

  1. The model is well-calibrated
  2. The model is overconfident (underestimates risk)
  3. The model is underconfident (overestimates risk)
  4. The model has good discrimination but poor calibration

Answer: b) The model is overconfident (underestimates risk)

Explanation: Calibration compares predicted probabilities to observed outcomes:

Analysis: - Predicted: 20% of 1,000 patients = 200 patients expected to develop cancer - Observed: 300 patients actually developed cancer - Gap: Predicted 200, observed 300 → Underestimating risk

Calibration terminology: - Well-calibrated: Predicted ≈ Observed (20% predicted → 20% observed) - Overconfident/Underestimate: Predicted < Observed (20% predicted → 30% observed) - This case - Underconfident/Overestimate: Predicted > Observed (20% predicted → 10% observed)

Why it matters:

# Clinical decision: Treat if risk > 25%
model.predict_proba(patient) = 0.20 # Below threshold → No treatment

# Reality: True risk was 0.30
# Patient should have been treated!

How to detect: 1. Calibration plot: Predicted vs observed by risk bin 2. Brier score: Mean squared error of probabilities 3. Expected Calibration Error (ECE): Average absolute calibration error

Lesson: High AUC doesn’t guarantee calibration. When predictions inform decisions with probability thresholds, calibration is critical. Always check calibration plots, not just discrimination metrics.

Question 3: External Validation

Your sepsis model achieves AUC 0.88 on internal test set. You test on external hospitals and get AUC 0.72-0.82 (varying by site). What does this variability indicate?

  1. The model is overfitting
  2. External sites have poor data quality
  3. There is substantial site-specific heterogeneity
  4. The model should not be used

Answer: c) There is substantial site-specific heterogeneity

Explanation: Performance variability across sites reveals important heterogeneity:

What varies between hospitals:

  1. Patient populations:
  • Demographics (age, race, socioeconomic status)
  • Disease severity (tertiary referral vs community hospital)
  • Comorbidity profiles
  1. Clinical practices:
  • Sepsis protocols (early vs delayed antibiotics)
  • ICU admission criteria
  • Documentation practices
  1. Infrastructure:
  • EHR systems (Epic vs Cerner vs homegrown)
  • Lab equipment (different reference ranges)
  • Staffing models (nurse-to-patient ratios)
  1. Data capture:
  • Missing data patterns
  • Measurement frequency
  • Feature definitions

Why not overfitting (a)? Overfitting shows as gap between training and test within same dataset. Here, internal test was fine (0.88). It’s external generalization that varies.

Why not poor data quality (b)? Could contribute, but more likely reflects legitimate differences in populations and practices.

Why not unusable (d)? AUC 0.72-0.82 is still useful! But indicates need for: - Site-specific calibration - Understanding what drives differences - Possibly site-specific models or adjustments

Best practice: External validation almost always shows performance drops. Variability across sites is normal and informative, reveals where model struggles and needs adaptation.

Question 4: Statistical Significance vs Clinical Significance

True or False: If a model improvement is statistically significant (p < 0.05), it is clinically meaningful and should be deployed.

Answer: False

Explanation: Statistical significance ≠ clinical significance. Both are necessary but neither alone is sufficient:

Statistical significance: - Tests if difference is unlikely due to chance - Depends on sample size (large N → small differences become significant) - Answers: “Is there an effect?”

Clinical significance: - Tests if difference matters for patient care - Independent of sample size - Answers: “Is the effect large enough to care?”

Example:

# New model vs baseline
results = {
 'baseline_auc': 0.820,
 'new_model_auc': 0.825,
 'difference': 0.005,
 'p_value': 0.03, # Statistically significant
 'sample_size': 50000 # Large sample
}

Analysis: - Statistically significant: p=0.03 < 0.05 - Clinically insignificant: 0.5% AUC improvement negligible - Why significant? Large sample detects tiny differences - Should deploy? No, not worth the cost/disruption

Lesson: Always evaluate both statistical and clinical significance. With large samples, trivial differences become statistically significant. Ask: “Is this improvement large enough to change practice?” Consider effect sizes, confidence intervals, and practical impact, not just p-values.

Question 5: Confidence Intervals

Two models have been evaluated: - Model A: AUC 0.85 (95% CI: 0.83-0.87) - Model B: AUC 0.86 (95% CI: 0.79-0.93)

Which statement is correct?

  1. Model B is definitely better because it has higher AUC
  2. Model A is more reliable because it has a narrower confidence interval
  3. The models cannot be compared without more information
  4. Model B is better if you’re willing to accept more uncertainty

Answer: b) Model A is more reliable because it has a narrower confidence interval

Explanation: Confidence intervals reveal precision/uncertainty, not just point estimates:

Model A: - AUC: 0.85 - 95% CI: 0.83-0.87 - Width: 0.04 (narrow) - Interpretation: We’re 95% confident true AUC is between 0.83-0.87 (precise estimate)

Model B: - AUC: 0.86 - 95% CI: 0.79-0.93 - Width: 0.14 (wide) - Interpretation: We’re 95% confident true AUC is between 0.79-0.93 (imprecise estimate)

Key insight: CIs overlap substantially (0.83-0.87 vs 0.79-0.93). Cannot conclude Model B is actually better, difference might be due to chance.

In practice: Most organizations prefer Model A: - Predictable performance for planning - Lower risk of underperformance - Easier to set appropriate thresholds - Small gain (0.01 AUC) not worth the uncertainty

Lesson: Always report and consider confidence intervals, not just point estimates. Narrow CIs indicate reliable performance. Wide CIs indicate uncertainty, might get much worse (or better) than point estimate suggests.

One more wrinkle: the comparison above treats Model A and Model B as independent estimates, each with its own confidence interval. If both models were actually scored on the identical set of cases, that independence assumption is wrong, and a paired-difference analysis (using the correlation between the two models’ scores on each shared case) is the statistically correct comparison rather than checking whether two separate confidence intervals overlap. A paired analysis on the same question set is typically more powerful than the unpaired comparison shown here, meaning it could resolve a small gap that overlapping CIs alone leave undecided (Miller, 2024, preprint).

Question 6: Subgroup Analysis

You evaluate a diagnostic model and find: - Overall AUC: 0.84 - Men: AUC 0.88 - Women: AUC 0.78

What should you do?

  1. Report only overall performance (0.84)
  2. Report overall performance but note subgroup differences exist
  3. Investigate why women’s performance is lower and consider separate models or adjustments
  4. Conclude the model is biased and should not be used

Answer: c) Investigate why women’s performance is lower and consider separate models or adjustments

Explanation: Subgroup performance disparities require investigation and action, not just reporting:

Why performance differs: Possible reasons

  1. Biological differences:
  • Disease presents differently (atypical symptoms in women)
  • Different physiological reference ranges
  • Example: Heart attack symptoms differ by sex
  1. Data representation:
  • Fewer women in training data → model learns men’s patterns better
  • Women may be underdiagnosed historically → labels biased
  1. Feature appropriateness:
  • Features optimized for men
  • Missing features relevant for women
  • Example: Pregnancy-related factors not included
  1. Measurement bias:
  • Tests/measurements less accurate for women
  • Different documentation patterns

Potential solutions:

  1. Collect more women’s data (if sample size issue)

  2. Add sex-specific features:

# Include pregnancy status, hormonal factors
features += ['pregnant', 'menopause_status', 'hormone_therapy']
  1. Stratified modeling:
# Separate models for men/women
if patient.sex == 'M':
 prediction = model_men.predict(patient)
else:
 prediction = model_women.predict(patient)
  1. Weighted loss function:
# Penalize errors on women more heavily during training
sample_weights = [2.0 if sex=='F' else 1.0 for sex in data['sex']]
model.fit(X, y, sample_weight=sample_weights)

Lesson: Subgroup analysis is mandatory, not optional. When disparities found, investigate root causes and take corrective action. Don’t hide disparities in overall metrics.

Question 7: LLM Evaluation Strategy

You’re deploying an LLM to summarize public health literature for practitioners. Which evaluation approach is MOST appropriate?

  1. Calculate AUC-ROC on a test set of summaries
  2. Measure perplexity (how surprised the model is by correct summaries)
  3. Use BERTScore to compare generated summaries against expert-written reference summaries + human expert evaluation
  4. Only use BLEU score (n-gram overlap with reference summaries)

Answer: c) Use BERTScore to compare generated summaries against expert-written reference summaries + human expert evaluation

Why:

Correct approach: - BERTScore captures semantic similarity better than simple n-gram matching (BLEU) - Human expert evaluation is essential for: - Factual accuracy (automated metrics can’t verify facts) - Clinical relevance (is the right information prioritized?) - Safety (are there dangerous omissions or errors?) - Multiple metrics needed: BERTScore + factual accuracy + completeness + safety rating

Why other options are wrong:

  • a) AUC-ROC: Not applicable, LLMs generate text, not binary classifications or probabilities
  • b) Perplexity alone: Measures fluency, not accuracy/relevance (fluent nonsense scores well)
  • d) BLEU only: Too limited, high BLEU doesn’t guarantee accurate or clinically appropriate summaries

Best practice for LLM summarization: 1. Automated metrics (BERTScore, ROUGE) for efficiency 2. Human expert review on sample (100+ summaries) 3. Safety audit (check for hallucinations, dangerous errors) 4. Prompt robustness testing (consistency across variations)

Lesson: LLM evaluation requires fundamentally different approaches than traditional ML. Automated metrics alone are insufficient, human expert evaluation is mandatory, especially for clinical applications.

Question 8: Model Drift Detection

You’re monitoring a deployed readmission prediction model. Over 6 months, you observe: - AUC-ROC: Stable at 0.82 (was 0.83 at deployment) - Brier score: Increased from 0.15 to 0.21 - Patient age distribution: Mean shifted from 58 to 65 years - PSI for age feature: 0.28

What does this indicate, and what should you do?

  1. Model is fine, AUC is stable; continue monitoring
  2. Significant data drift occurred; retrain immediately
  3. Concept drift occurred; model is failing and needs urgent retraining
  4. Calibration degraded but discrimination stable; recalibrate or retrain

Answer: d) Calibration degraded but discrimination stable; recalibrate or retrain

Analysis:

What happened: - AUC-ROC stable: Model can still distinguish high-risk from low-risk patients (discrimination intact) - Brier score increased: Predicted probabilities are inaccurate (calibration degraded) - Age distribution shifted: PSI = 0.28 indicates significant data drift (threshold: PSI > 0.25) - Likely cause: Data drift (patient population aging) → calibration degrades even if model’s relative ranking ability (AUC) persists

Why other options are wrong:

  • a) Model is fine: WRONG, Brier score degradation and high PSI require action
  • b) Data drift, retrain immediately: Partially correct but oversimplified, could recalibrate first (faster, simpler)
  • c) Concept drift: UNLIKELY, AUC would degrade if relationship between features and outcome changed; this looks like data drift affecting calibration

What to do:

Immediate (within 1 week): - Recalibrate model on recent data (faster than full retraining) - Test calibration on recent hold-out set - Deploy recalibrated version if performance restored

Scheduled (within 1 month): - Full model retraining recommended (PSI > 0.25 for critical feature) - Validate retrained model on hold-out set - Compare retrained vs. recalibrated performance - Deploy better-performing version

Monitoring: - Track Brier score monthly (calibration early warning) - Track PSI for all critical features - Set alert: PSI > 0.25 for ≥3 features = immediate retraining

Lesson: Different types of drift require different interventions. Data drift may degrade calibration while preserving discrimination. Monitor multiple metrics (not just AUC) to catch drift early.

Question 9: Regulatory Classification

You’re developing an AI system that analyzes electronic health records to flag patients at high risk for cardiovascular disease in the next 5 years. Clinicians review flagged patients and decide whether to prescribe statins. How would the FDA likely classify this system?

  1. Not a medical device (wellness/administrative use)
  2. SaMD Level I (Low Risk)
  3. SaMD Level II (Moderate Risk)
  4. SaMD Level III (High Risk)

Answer: c) SaMD Level II (Moderate Risk)

Classification reasoning:

FDA SaMD Matrix: - State of healthcare situation: Serious (cardiovascular disease can cause long-term morbidity) - Significance of information: Drive clinical management (significantly influences statin prescription decision)

Per FDA matrix: Serious + Drive clinical management = Level II

Why not other levels:

a) Not a medical device: WRONG - System makes medical claims (predicts disease risk) - Influences clinical decisions (statin prescription) - Clearly falls under SaMD definition

b) Level I (Low Risk): WRONG - CVD is not a “non-serious” condition - System does more than just “inform”, it drives treatment decisions

d) Level III (High Risk): WRONG - Not critical (immediately life-threatening) condition, that would be ICU monitoring, acute MI - Not treating/diagnosing directly, clinicians make final decision - If system autonomously prescribed statins → Level III

Evaluation implications for Level II:

Required: - Robust internal validation - External validation recommended (highly encouraged) - Clinical utility assessment (does it actually improve outcomes?) - Usability testing with clinicians - Performance monitoring plan (drift detection)

Not required (Level III would need these): - Randomized controlled trial - Extensive multi-site prospective validation - Predetermined Change Control Plan (optional but recommended)

Borderline considerations:

If the system provided lower-stakes information (e.g., “Consider discussing lifestyle changes”) → Could be Level I

If the condition were critical/life-threatening (e.g., predict sepsis, guide ICU ventilator settings) → Level III

Lesson: FDA classification depends on both severity of condition AND how the information is used. “Driving clinical management” for serious conditions = Level II. Understanding classification early helps you plan appropriate validation rigor.

Question 10: Adversarial Robustness Testing

You’re evaluating a chest X-ray pneumonia detection model. Which robustness test is MOST important for real-world deployment?

  1. Fast Gradient Sign Method (FGSM) adversarial attack testing
  2. Natural perturbation testing (image compression, scanner variations, noise)
  3. Model extraction attack testing (preventing reverse engineering)
  4. Data poisoning resilience testing (can the training set be corrupted?)

Answer: b) Natural perturbation testing (image compression, scanner variations, noise)

Reasoning:

Real-world threat model: - Natural perturbations occur constantly: Different X-ray machines, image compression algorithms, patient positioning variations, image quality differences across sites - Likelihood: 100% of deployed systems encounter natural variation - Impact if not tested: Model may fail on images from different hospitals/equipment, limiting generalizability

Why other options are less critical (though still valuable):

a) FGSM adversarial attacks: - Likelihood: Near zero, no documented malicious adversarial attacks on medical imaging in practice - Value: Academic interest, EU AI Act may require, but not the most pressing real-world concern - When important: High-profile systems, regulatory compliance (EU AI Act)

c) Model extraction: - Risk: Intellectual property theft, but doesn’t directly harm patients - Mitigation: API rate limiting, access control (non-evaluation solutions)

d) Data poisoning: - Risk: Rare unless using crowdsourced/untrusted training data - Prevention: Data provenance, quality control during training (not post-deployment testing)

Practical testing hierarchy:

Essential (all deployed models): 1. Natural perturbation testing: Compression, noise, equipment variation 2. Out-of-distribution detection: Flag unfamiliar images (different scanner types, anatomies) 3. Multi-site external validation: Real-world test of natural robustness

Recommended (high-risk/regulatory): 4. Adversarial attack testing: FGSM, PGD (EU AI Act requirement) 5. Ablation studies: Performance with missing/corrupted inputs

Advanced (specific threats): 6. Data poisoning resilience: If using federated learning or external data 7. Model extraction prevention: If protecting proprietary models

Example test:

# Test robustness to JPEG compression (natural perturbation)
import cv2

compression_qualities = [100, 90, 70, 50, 30]
for quality in compression_qualities:
 # Compress image
 _, compressed = cv2.imencode('.jpg', image,
         [cv2.IMWRITE_JPEG_QUALITY, quality])
 compressed_img = cv2.imdecode(compressed, cv2.IMREAD_COLOR)

 # Test model
 prediction = model.predict(compressed_img)
 print(f"Quality {quality}: Prediction = {prediction:.3f}")

# Acceptable: <10% prediction change across quality 100→70
# Red flag: >20% prediction change (overfitting to high-quality images)

Lesson: Prioritize robustness testing based on real-world threat likelihood. For medical imaging, natural perturbations (equipment variation) are vastly more common than adversarial attacks. Test what will actually break your model in practice.