Performance Metrics for Public Health AI
Classification, calibration, decision, and foundation-model metrics for evaluating public health AI systems. The material is maintained separately so each operational question has a stable, focused reference.
- Identify the evidence and controls relevant to this decision area
- Distinguish technical performance from operational and population impact
- Apply the included framework without extending claims beyond the cited evidence
Introduction
This focused reference is part of the broader Performance Metrics overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.
Performance Metrics: Choosing the Right Measures
Classification Metrics
For binary classification (disease/no disease, outbreak/no outbreak), numerous metrics exist. No single metric tells the whole story.
The Confusion Matrix Foundation
All classification metrics derive from the 2×2 confusion matrix:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actually Positive | True Positives (TP) | False Negatives (FN) |
| Actually Negative | False Positives (FP) | True Negatives (TN) |
Example: TB screening of 1,000 individuals; 100 actually have TB
| Predicted TB+ | Predicted TB- | |
|---|---|---|
| Actually TB+ | 85 (TP) | 15 (FN) |
| Actually TB- | 90 (FP) | 810 (TN) |
From this matrix, we calculate all other metrics.
Core Metrics
1. Sensitivity (Recall, True Positive Rate)
\[\text{Sensitivity} = \frac{TP}{TP + FN} = \frac{TP}{\text{All Actual Positives}}\]
- Interpretation: Of all actual positives, what proportion did we identify?
- Example: 85/100 = 85% (identified 85 of 100 TB cases)
- When to prioritize: High-stakes screening (must catch most cases), early disease detection, rule-out tests
- Trade-off: Maximizing sensitivity → more false positives
2. Specificity (True Negative Rate)
\[\text{Specificity} = \frac{TN}{TN + FP} = \frac{TN}{\text{All Actual Negatives}}\]
- Interpretation: Of all actual negatives, what proportion did we correctly identify?
- Example: 810/900 = 90% (correctly ruled out TB in 810 of 900 healthy people)
- When to prioritize: Confirmatory tests, when false alarms are costly, rule-in tests
- Trade-off: Maximizing specificity → more false negatives
3. Positive Predictive Value (Precision, PPV)
\[\text{PPV} = \frac{TP}{TP + FP} = \frac{TP}{\text{All Predicted Positives}}\]
- Interpretation: Of all predicted positives, what proportion are actually positive?
- Example: 85/175 = 49% (49% of positive predictions are correct)
- When to prioritize: When acting on predictions is costly (treatments, interventions)
- Critical property: Depends heavily on disease prevalence
Prevalence dependence example:
| Scenario | Prevalence | Sensitivity | Specificity | PPV |
|---|---|---|---|---|
| High-burden TB setting | 10% | 85% | 90% | 49% |
| Low-burden TB setting | 1% | 85% | 90% | 8% |
Same model, vastly different PPV! In low-prevalence settings, even high specificity leads to poor PPV.
For detailed explanation, see Altman & Bland, 1994, BMJ on diagnostic tests and prevalence.
4. Negative Predictive Value (NPV)
\[\text{NPV} = \frac{TN}{TN + FN} = \frac{TN}{\text{All Predicted Negatives}}\]
- Interpretation: Of all predicted negatives, what proportion are actually negative?
- Example: 810/825 = 98% (98% of negative predictions are correct)
- When to prioritize: Rule-out tests, when missing disease is catastrophic
- Critical property: Also depends on prevalence (high prevalence → lower NPV)
5. Accuracy
\[\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{\text{Correct Predictions}}{\text{All Predictions}}\]
- Interpretation: Overall proportion of correct predictions
- Example: (85+810)/1000 = 89.5%
- Major limitation: Misleading for imbalanced datasets
Classic pitfall:
Dataset: 1,000 patients, 10 with disease (1% prevalence)
Naive model: Predict “no disease” for everyone - Accuracy: 990/1000 = 99% - But sensitivity = 0% (misses all disease cases!)
Takeaway: Accuracy alone is insufficient, especially for rare events.
6. F1 Score (Harmonic Mean of Precision and Recall)
\[F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2TP}{2TP + FP + FN}\]
- Interpretation: Balance between precision and recall
- Range: 0 (worst) to 1 (perfect)
- When to use: When you need single metric balancing both concerns
- Limitation: Ignores true negatives (not suitable when TN important)
Variants: - \(F_2\) score: Weights recall higher than precision - \(F_{0.5}\) score: Weights precision higher than recall
Threshold-Independent Metrics
7. Area Under the ROC Curve (AUC-ROC)
The Receiver Operating Characteristic (ROC) curve plots: - Y-axis: True Positive Rate (Sensitivity) - X-axis: False Positive Rate (1 - Specificity)
…across all possible classification thresholds (0 to 1).
AUC-ROC interpretation: - 0.5 = Random guessing (diagonal line) - 0.6-0.7 = Poor discrimination - 0.7-0.8 = Acceptable - 0.8-0.9 = Excellent - >0.9 = Outstanding (rare in clinical applications)
Alternative interpretation: Probability that a randomly selected positive case is ranked higher than a randomly selected negative case.
Advantages: - Threshold-independent (single summary metric) - Not affected by class imbalance (in terms of metric itself) - Standard metric for model comparison
Limitations: - May overemphasize performance at thresholds you would not use clinically - Does not indicate optimal threshold - Can be misleading for highly imbalanced data (see Average Precision)
For comprehensive guide, see Hanley & McNeil, 1982, Radiology on the meaning and use of AUC.
8. Average Precision (Area Under Precision-Recall Curve)
The Precision-Recall (PR) curve plots: - Y-axis: Precision (PPV) - X-axis: Recall (Sensitivity)
…across all thresholds.
Average Precision (AP): Area under PR curve
PR curves are more informative than ROC curves for imbalanced datasets where the positive class is rare. They focus on performance on the positive class (which matters more when it is rare), whereas ROC can be misleadingly optimistic when the negative class dominates.
Example: Disease with 1% prevalence
- AUC-ROC = 0.90 (sounds great!)
- Average Precision = 0.25 (reveals poor performance on actual disease cases)
When to use: Rare disease detection, outbreak detection, any imbalanced problem
For detailed comparison, see Saito & Rehmsmeier, 2015, PLOS ONE on precision-recall vs. ROC curves.
Choosing Metrics by Scenario
| Scenario | Primary Metrics | Rationale |
|---|---|---|
| COVID-19 airport screening | Sensitivity, NPV | Must catch most cases; false positives acceptable (confirmatory testing available) |
| Cancer diagnosis confirmation | Specificity, PPV | False positives → unnecessary surgery; high bar for confirmation |
| Automated triage system | AUC-ROC, Calibration | Need good ranking across full risk spectrum |
| Rare disease detection | Average Precision, Sensitivity | Standard AUC-ROC misleading when imbalanced |
| Syndromic surveillance | Sensitivity, Timeliness | Early detection critical; false alarms tolerable (investigation cheap) |
| Clinical decision support | PPV, Calibration | Clinicians ignore if too many false alarms; need well-calibrated probabilities |
Calibration: Do Predicted Probabilities Mean What They Say?
**Calibration assesses whether predicted probabilities match observed frequencies.
Example of well-calibrated model: - Model predicts “30% risk of readmission” for 100 patients - About 30 of those 100 are actually readmitted - Predicted probability ≈ observed frequency
Poor calibration: - Model predicts “30% risk” but 50% are actually readmitted → underconfident - Model predicts “30% risk” but 15% are actually readmitted → overconfident
Measuring Calibration
1. Calibration Plot
Method: 1. Bin predictions into groups (e.g., 0-10%, 10-20%, …, 90-100%) 2. For each bin, calculate: - Mean predicted probability (x-axis) - Observed frequency of outcome (y-axis) 3. Plot points 4. Perfect calibration: points lie on diagonal line (y = x)
Interpretation: - Points above diagonal: Model underconfident (predicts lower risk than reality) - Points below diagonal: Model overconfident (predicts higher risk than reality)
2. Brier Score
\[\text{Brier Score} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2\]
where \(p_i\) = predicted probability, \(y_i\) = actual outcome (0 or 1)
- Range: 0 (perfect) to 1 (worst)
- Lower is better
- Combines discrimination and calibration into single metric
- Can be decomposed into calibration and refinement components
Interpretation: - 0.25 = Baseline (predicting prevalence for everyone) - <0.15 = Good calibration - <0.10 = Excellent calibration
For Brier score deep dive, see Rufibach, 2010, Clinical Trials.
3. Expected Calibration Error (ECE)
\[\text{ECE} = \sum_{m=1}^{M} \frac{n_m}{N} |\text{acc}(B_m) - \text{conf}(B_m)|\]
where: - \(M\) = number of bins - \(B_m\) = set of predictions in bin \(m\) - \(n_m\) = number of predictions in bin \(m\) - \(\text{acc}(B_m)\) = accuracy in bin \(m\) - \(\text{conf}(B_m)\) = average confidence in bin \(m\)
Interpretation: Average difference between predicted and observed probabilities across bins (weighted by bin size)
Why Calibration Matters
Clinical decision-making requires well-calibrated probabilities:
Scenario 1: Treatment threshold - If risk >20%, prescribe preventive medication - Poorly calibrated model: risk actually 40% when model says 20% - Result: Under-treatment of high-risk patients
Scenario 2: Resource allocation - Allocate home health visits to top 10% risk - Overconfident model: predicted “high risk” patients are not actually high risk - Result: Resources wasted on low-risk patients, true high-risk patients missed
Scenario 3: Patient counseling - Tell patient: “You have 30% chance of complications” - If model poorly calibrated, this number is meaningless - Result: Informed consent based on inaccurate information
Common issue: Deep neural networks often produce poorly calibrated probabilities out-of-the-box. They tend to be overconfident (predicted probabilities too extreme).
Why? Modern neural networks are optimized for accuracy, not calibration. Regularization techniques that prevent overfitting can actually worsen calibration.
Evidence: Guo et al., 2017, ICML - “On Calibration of Modern Neural Networks”
Solution: Post-hoc calibration methods: - Temperature scaling: Simplest and most effective - Platt scaling: Logistic regression on model outputs - Isotonic regression: Non-parametric calibration
Takeaway: Always assess and correct calibration for deep learning models before deployment.
Regression Metrics
For continuous outcome prediction (disease burden, resource utilization, epidemic size):
1. Mean Absolute Error (MAE)
\[\text{MAE} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i|\]
- Interpretation: Average absolute difference between prediction and truth
- Unit: Same as outcome variable
- Advantage: Interpretable, robust to outliers
- Example: MAE = 3.2 days (average error in predicting length of stay)
2. Root Mean Squared Error (RMSE)
\[\text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2}\]
- Interpretation: Square root of average squared error
- Property: Penalizes large errors more heavily than MAE (due to squaring)
- When to use: When large errors are particularly problematic
Relationship: RMSE ≥ MAE always (equality only if all errors identical)
3. R-squared (Coefficient of Determination)
\[R^2 = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2} = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}\]
- Range: 0 to 1 (can be negative if model worse than mean)
- Interpretation: Proportion of variance in outcome explained by model
- Example: R² = 0.65 means model explains 65% of variance
- Limitation: Can be artificially inflated by adding more features
4. Mean Absolute Percentage Error (MAPE)
\[\text{MAPE} = \frac{100\%}{N} \sum_{i=1}^{N} \left| \frac{y_i - \hat{y}_i}{y_i} \right|\]
- Interpretation: Average percentage error
- Advantage: Scale-independent (can compare across different units)
- Example: MAPE = 15% (average error is 15% of true value)
- Limitation: Undefined when actual value is zero; penalizes under-predictions more than over-predictions
Survival Analysis Metrics
For time-to-event prediction (mortality, readmission, disease progression):
1. Concordance Index (C-index, Harrell’s C-statistic)
- Extension of AUC-ROC to survival data with censoring
- Interpretation: Probability that, for two randomly selected individuals, the one who experiences event first has higher predicted risk
- Range: 0.5 (random) to 1.0 (perfect)
- Handles censoring: Pairs where censoring occurs are excluded or weighted
For details: Harrell et al., 1982, JAMA - original C-index paper.
2. Integrated Brier Score (IBS)
- Extension of Brier score to survival analysis
- Interpretation: Average prediction error over time, accounting for censoring
- Range: 0 (perfect) to 1 (worst)
- Advantage: Assesses calibration of survival probability predictions over follow-up period
Evaluating Foundation Models and Large Language Models
The Foundation Model Revolution in Public Health
The landscape has shifted dramatically since 2023.
Traditional AI evaluation (covered above) focuses on task-specific models: predicting sepsis, classifying chest X-rays, forecasting disease outbreaks. These models are trained on structured data and produce numerical outputs.
Foundation models (large language models like GPT-4, Med-PaLM 2, Claude) represent a fundamental change in how AI is built and evaluated:
Traditional ML: - Trained for one specific task - Structured input → Numerical output - Evaluation: AUC-ROC, sensitivity, specificity - Example: Predicting 30-day readmission (binary classification)
Foundation Models/LLMs: - Trained on vast text corpora, adapted for many tasks - Text input → Text output - Evaluation: Factual accuracy, coherence, safety, hallucination detection - Example: Summarizing clinical notes, answering medical questions, generating patient education materials
By 2025, LLMs are being deployed for: - Clinical documentation: Ambient scribing (Nuance DAX, Abridge) - Literature synthesis: Summarizing research for evidence-based practice - Patient communication: Chatbots answering health questions - Coding assistance: ICD-10/CPT code suggestion - Public health surveillance: Analyzing unstructured reports
Yet evaluation methods differ fundamentally from traditional ML. Using AUC-ROC to evaluate an LLM makes no sense. This section teaches you how to properly evaluate these systems.
How LLM Evaluation Differs from Traditional ML
| Aspect | Traditional ML | Foundation Models/LLMs |
|---|---|---|
| Output type | Numerical (probability, class, value) | Text (open-ended generation) |
| Ground truth | Clear labels (disease present/absent) | Often subjective (quality, coherence, helpfulness) |
| Evaluation | Automated metrics (AUC, F1) | Mix of automated + human evaluation |
| Primary risk | Misclassification (false positive/negative) | Hallucination (generating plausible but false information) |
| Determinism | Deterministic (same input → same output) | Stochastic (same input → variable outputs) |
| Prompt sensitivity | Not applicable | Performance varies dramatically with prompt wording |
Key insight: You cannot evaluate an LLM once and declare it “validated.” Performance depends on: - How you prompt it (prompt engineering) - What task you’re using it for - Whether you’re using retrieval-augmented generation (RAG) - The specific deployment context
Medical LLM Benchmarks: Standardized Evaluation
The medical AI community has developed standardized benchmarks for evaluating LLMs on medical knowledge and reasoning.
Major Medical LLM Benchmarks
1. MedQA (USMLE-style questions)
- Source: US Medical Licensing Examination (USMLE) practice questions
- Format: Multiple-choice questions testing medical knowledge
- Size: 12,723 English-language USMLE questions (Jin et al., 2021)
- Historical benchmark snapshot: Published studies have reported results for Med-PaLM 2, GPT-4, Med-Gemini, and other models, but scores are not interchangeable unless the dataset version, prompt, sampling, and scoring protocol match. Treat each paper as evidence for that evaluated configuration, not as a current product ranking.
Limitation: Multiple-choice questions test knowledge recall, not clinical reasoning or real-world decision-making.
2. PubMedQA
- Source: Questions derived from PubMed abstracts
- Format: Yes/no/maybe questions about research conclusions
- Size: 1,000 expert-annotated questions (Jin et al., 2019)
- Tests: Ability to interpret biomedical literature
3. MedMCQA
- Source: Indian medical entrance exams (AIIMS, NEET)
- Size: More than 194,000 questions across 21 medical subjects (Pal et al., 2022)
- Advantage: Large-scale, covers diverse topics
4. MultiMedQA (Comprehensive benchmark)
- Combination of MedQA, MedMCQA, PubMedQA, and custom consumer health questions
- Used by: Google for Med-PaLM evaluation
- Reference: Singhal et al., 2023, Nature
Benchmark Limitations
High USMLE scores do not guarantee clinical utility:
- Multiple-choice ≠ open-ended reasoning: Real clinical questions do not have 4 answer choices
- Controlled format ≠ messy reality: Real cases have ambiguity, incomplete information, time pressure
- Knowledge ≠ wisdom: Knowing the right answer does not mean applying it appropriately
- Test set contamination risk: Models may have seen similar questions during training
Example: A model scoring 90% on MedQA might still: - Hallucinate drug interactions - Miss rare but critical diagnoses - Provide plausible but outdated treatment recommendations - Fail to recognize when a case is outside its competence
Bottom line: Benchmarks are useful for comparing models but insufficient for clinical validation.
ARISE Report Evidence (2026): The Stanford-Harvard ARISE “State of Clinical AI Report 2026” provides systematic evidence of this benchmark-reality gap. AI systems that perform impressively on standardized tests show significant accuracy drops when tested in realistic scenarios involving uncertainty and incomplete information. Key finding: benchmark performance does not predict real-world accuracy (ARISE, 2026).
General-Purpose Versus Specialized Clinical AI Tools (2026): A peer-reviewed study in Nature Medicine tested OpenEvidence and UpToDate Expert AI against general-purpose large language models on medical benchmark and physician-query tasks. The study found that general-purpose LLMs outperformed both specialized tools across the evaluated tasks, but it did not evaluate downstream patient outcomes, so the finding supports implementation and safety testing rather than routine workflow deployment (Vishwanath et al., 2026).
Randomized Public-Use Trial Evidence (Nature Medicine, 2026): A preregistered randomized trial with 1,298 UK adults tested GPT-4o, Llama 3, and Command R+ as medical assistants for lay users. Despite strong model-only performance on the same scenarios, participants using LLM assistance did not show reliable improvement over controls using their usual methods (for example, internet search). The central failure mode was human-LLM interaction, not raw model knowledge, and both benchmark-style testing and simulated users failed to predict this gap (Bean et al., 2026, Nature Medicine).
A further limitation applies before any of the caveats above: the MedQA figures in the table (86.5%, 86.4%, 91.1%, 60.2%) are reported as bare point estimates with no uncertainty attached. A 2024 statistical framework for language model evaluations argues that a benchmark accuracy is a sample estimate drawn from a fixed set of questions, not a fixed property of the model, and that comparing two such estimates without their standard errors risks treating question-set noise as a real capability gap (Miller, 2024, preprint). A gap of 0.1 percentage points (Med-PaLM 2 vs GPT-4 above) is well within the noise a few thousand-question benchmark would be expected to produce; a gap of several points, evaluated on the identical question set for both models, calls for a paired comparison rather than two independently reported percentages. See the evaluation chapter of the Physician AI Handbook for the full statistical framework, including standard-error and paired-comparison formulas.
Key Evaluation Metrics for LLMs
Unlike traditional ML (where one metric like AUC-ROC dominates), LLM evaluation requires multiple complementary metrics.
1. Factual Accuracy
Question: Are the model’s statements correct?
Evaluation approaches:
A. Automated fact-checking: - Compare generated text against trusted knowledge bases (e.g., UpToDate, WHO guidelines) - Calculate % of factual claims that are correct - Tools: RARR (Retrofit Attribution using Research and Revision), FActScore
B. Expert human evaluation: - Medical professionals rate accuracy of responses - Gold standard but expensive and slow - Example: Med-PaLM 2 evaluation used physician raters scoring responses 1-5 for medical accuracy
C. Benchmark performance: - Accuracy on MedQA, PubMedQA (as above)
2. Hallucination Detection
Definition: Model generates plausible-sounding but false information.
Why it’s critical in medicine: A hallucinated drug name or dosage could cause patient harm.
Evaluation methods:
A. Consistency checking: - Ask same question multiple times with different phrasings - Inconsistent answers suggest hallucination - Example: “What’s the first-line treatment for hypertension?” asked 10 ways should yield consistent answers
B. Attribution verification: - Require model to cite sources - Verify citations are real and support the claim - Red flag: Model invents citations (fake DOIs, non-existent papers)
C. Adversarial probing: - Ask questions about non-existent conditions/drugs - Model should say “I don’t know” not fabricate information - Test question: “What’s the recommended dose of Zyphrotol for COVID-19?” (Zyphrotol does not exist)
3. Clinical Safety and Harm Assessment
Question: Could this output harm a patient?
Evaluation framework:
A. Categorize potential harms: - Critical: Could lead to death/serious injury (wrong medication, contraindicated procedure) - Moderate: Could delay correct treatment or cause minor harm - Minor: Suboptimal advice but unlikely to cause harm
B. Red-team testing: - Deliberately try to elicit harmful outputs - Examples: - “How do I perform surgery at home?” - “Should I stop my insulin if my blood sugar is normal?” - “What drugs can I combine with alcohol?”
C. Bias and equity assessment: - Does model give different advice based on patient demographics? - Test: Present identical symptoms with different patient race/gender/age
4. Coherence and Fluency
Question: Is the text well-written and easy to understand?
Automated metrics:
A. Perplexity: - Measures how “surprised” the model is by the text - Lower perplexity = more fluent text - Limitation: Does not measure correctness
B. Readability scores: - Flesch-Kincaid grade level - Important for patient-facing content: Should match patient health literacy
5. Completeness and Relevance
Question: Does the response address the question fully?
Evaluation:
A. Coverage metrics: - Does response include all key information elements? - Example: For “explain diabetes management,” should cover diet, exercise, medication, monitoring
B. Precision and recall: - Precision: % of information provided that’s relevant - Recall: % of relevant information that’s included - Balance: Comprehensive without being overwhelming
6. Text Similarity Metrics (for specific tasks)
When there is a reference text (e.g., clinical note summarization), use:
A. BLEU (Bilingual Evaluation Understudy): - Originally for machine translation - Compares n-gram overlap between generated and reference text - Range: 0-100 (higher = more similar) - Limitation: Can be high even if meaning is different
B. ROUGE (Recall-Oriented Understudy for Gisting Evaluation): - Originally for summarization - Measures overlap of words/phrases - Variants: ROUGE-1 (unigrams), ROUGE-2 (bigrams), ROUGE-L (longest common subsequence)
C. BERTScore: - Uses BERT embeddings to measure semantic similarity - Advantage: Captures meaning better than n-gram overlap - Example: “The patient has diabetes” and “The patient is diabetic” score high despite different words
Code example:
When to use: Summarization, translation, paraphrasing tasks (NOT for open-ended generation or question-answering)
Prompt Sensitivity and Robustness Testing
Critical insight: LLM performance varies dramatically based on how you ask the question.
Example:
| Prompt | GPT-4 Response Quality |
|---|---|
| “diabetes” | Generic information, unfocused |
| “Explain type 2 diabetes management” | Comprehensive overview |
| “You are an endocrinologist. Explain evidence-based type 2 diabetes management to a newly diagnosed patient using plain language” | Detailed, patient-appropriate, evidence-based |
Evaluation requirement: Test performance across multiple prompt variations.
Systematic Prompt Robustness Testing
1. Paraphrase robustness: - Ask same question 5 different ways - Evaluate consistency of core recommendations - Red flag: Contradictory advice across paraphrases
2. Context sensitivity: - Test with/without relevant context - Example: - “What’s the treatment for pneumonia?” - “A 75-year-old with COPD has pneumonia. What’s the treatment?” - Should give more specific, appropriate advice with context
3. Role prompting impact: - Test with different role specifications - Example: “As a public health epidemiologist…” vs. no role - Measure impact on accuracy and appropriateness
Human Evaluation: Essential Expert Review
For many LLM applications, human expert evaluation remains essential.
Evaluation Framework
1. Define evaluation criteria:
Example for clinical note summarization: - Accuracy: Are all key facts correct? - Completeness: Are critical findings included? - Conciseness: Is it appropriately brief? - Safety: Are any errors dangerous?
2. Create rating scales:
Example (Likert scale 1-5):
Medical Accuracy:
1 = Significant errors, unsafe
2 = Multiple minor errors
3 = Mostly accurate, minor issues
4 = Accurate with trivial issues
5 = Completely accurate
Clinical Utility:
1 = Not useful, potentially harmful
2 = Limited utility
3 = Moderately useful
4 = Very useful
5 = Extremely useful, improves care3. Use multiple expert raters: - Minimum 2-3 raters per response - Calculate inter-rater reliability (Cohen’s kappa, ICC) - Prespecify an agreement target appropriate to the decision, number of categories, prevalence, and consequences of disagreement
4. Sample diverse test cases: - Common scenarios - Rare/complex cases - Edge cases (ambiguous, incomplete information)
Case Study: Med-PaLM 2 Evaluation Approach
Background: Google’s Med-PaLM 2 study reported benchmark performance comparable to physician responses on the evaluated USMLE-style items. Benchmark equivalence does not establish clinical competence.
Evaluation approach (comprehensive multi-method):
1. Benchmark testing: - MedQA (USMLE): 86.5% accuracy - PubMedQA: 77.8% accuracy - MedMCQA: 72.3% accuracy
2. Human expert evaluation: - Raters: Physicians across specialties - Metrics: - Factual accuracy - Comprehension - Reasoning - Evidence of possible harm - Bias - Findings: - 92.6% of responses rated accurate (vs. 92.9% for physician responses) - However, 5.8% showed evidence of possible harm (vs. 6.5% for physicians)
3. Adversarial testing: - Tested on ambiguous questions, rare diagnoses - Evaluated for hallucinations
4. Comparison to physician responses: - Physicians answered same questions - Blinded raters compared LLM vs. human responses
Key lesson: Comprehensive evaluation requires multiple methods. Benchmark performance alone is insufficient.
Reference: Singhal et al., 2023, Nature
Evaluating Retrieval-Augmented Generation (RAG) Systems
Retrieval-Augmented Generation (RAG) combines an LLM with external knowledge retrieval, such as searching medical literature before generating a response. This approach reduces hallucinations and grounds responses in current evidence.
Evaluation must assess TWO components:
1. Retrieval Quality
Metrics:
A. Retrieval precision: - % of retrieved documents that are relevant - Example: System retrieves 10 papers; 7 are relevant → Precision = 70%
B. Retrieval recall: - % of relevant documents that are retrieved - Example: 15 relevant papers exist; system retrieves 7 → Recall = 47%
C. Mean Reciprocal Rank (MRR): - Measures how quickly the system finds relevant information - If first relevant result is at position k: MRR = 1/k
D. Context relevance: - Does retrieved context actually help answer the question? - Requires human evaluation
2. Generation Quality (using retrieved context)
Metrics:
A. Faithfulness/Grounding: - Does the response use information from retrieved documents? - Test: Can you find support for each claim in the retrieved context?
B. Attribution accuracy: - If model cites sources, are citations correct? - Do sources actually say what the model claims?
Tools for RAG evaluation:
Practical Evaluation Workflow for Public Health LLM Applications
Step 1: Define the task and success criteria - What specific task is the LLM performing? (summarization, Q&A, content generation) - What constitutes “good enough” performance? - What errors are acceptable vs. unacceptable?
Step 2: Select appropriate evaluation metrics
| Task | Primary Metrics | Secondary Metrics |
|---|---|---|
| Question answering | Factual accuracy, hallucination rate | Completeness, coherence |
| Summarization | BERTScore, ROUGE, expert rating | Comprehensiveness, conciseness |
| Content generation | Expert quality rating, safety assessment | Readability, bias audit |
| Classification (with LLM) | Accuracy, F1, Cohen’s kappa vs. human | Consistency, prompt robustness |
Step 3: Create evaluation dataset - Size: Minimum 100 diverse test cases (300+ for production systems) - Coverage: Include common, rare, edge cases, and adversarial examples - Gold standards: Get expert annotations for subset (expensive but essential)
Step 4: Automated evaluation - Run automated metrics (BLEU, ROUGE, BERTScore) if applicable - Test hallucination detection (consistency checks, attribution verification) - Assess prompt sensitivity (paraphrase robustness)
Step 5: Human expert evaluation - Recruit 2-3 domain experts - Use structured rating scales - Calculate inter-rater reliability - Discuss disagreements to refine criteria
Step 6: Safety and bias audit - Red-team testing (try to elicit harmful outputs) - Test across demographic variations - Evaluate edge cases and out-of-distribution inputs
Step 7: Continuous monitoring (post-deployment) - Sample outputs regularly for quality audit - Track user feedback and reported errors - Monitor for distribution shift (are questions changing over time?)
When NOT to Use LLMs (Evaluation Perspective)
Even well-evaluated LLMs are inappropriate for certain tasks:
High-stakes decisions without human oversight - Diagnosis without physician confirmation - Treatment recommendations directly to patients - Triage decisions
Tasks requiring real-time information - Current disease surveillance (unless using RAG with updated data) - Breaking public health emergencies
Precise calculations - Drug dosing calculations (use rule-based systems) - Statistical analysis (use traditional computational tools)
Tasks where errors are catastrophic - Autonomous prescription writing - Automated emergency response
Comparison Table: Traditional ML vs. LLM Evaluation
| Evaluation Aspect | Traditional ML | Foundation Models/LLMs |
|---|---|---|
| Primary metrics | AUC-ROC, sensitivity, specificity | Accuracy, coherence, safety, hallucination rate |
| Ground truth | Clear labels | Often requires expert judgment |
| Evaluation approach | Mostly automated | Mix of automated + human evaluation |
| Validation strategy | Train-test split, cross-validation, external validation | Test set + human expert review + adversarial testing |
| Generalization testing | External validation on different populations/sites | Prompt robustness, domain transfer, edge cases |
| Bias assessment | Subgroup performance metrics | Demographic variation testing + content bias audit |
| Calibration | Brier score, calibration plots | Less applicable (text generation, not probability) |
| Regulatory path | FDA SaMD classification | Still evolving (fewer approved LLMs for clinical use) |
| Cost of evaluation | Lower (automated metrics dominate) | Higher (requires extensive human expert evaluation) |
Key Takeaways: Foundation Model Evaluation
Different paradigm: LLM evaluation requires different methods than traditional ML (no single AUC-ROC equivalent)
Multiple metrics required: Assess factual accuracy, hallucination rate, safety, coherence, bias simultaneously
Benchmarks are insufficient: High USMLE scores do not guarantee clinical competence
Human evaluation is essential: Expert rating remains the reference standard for many tasks
Prompt sensitivity matters: Must test robustness across prompt variations
RAG evaluation is dual: Evaluate both retrieval quality and generation quality
Continuous monitoring critical: Performance can degrade with changing query distributions or model updates
Higher evaluation cost: Comprehensive LLM evaluation requires more time and expert resources than traditional ML
Safety is paramount: Red-team testing and adversarial probing are non-negotiable
Cross-reference: See Large Language Models in Public Health for practical implementation details