Classification Metrics
For binary classification (disease/no disease, outbreak/no outbreak), numerous metrics exist. No single metric tells the whole story.
The Confusion Matrix Foundation
All classification metrics derive from the 2×2 confusion matrix:
| Actually Positive |
True Positives (TP) |
False Negatives (FN) |
| Actually Negative |
False Positives (FP) |
True Negatives (TN) |
Example: TB screening of 1,000 individuals; 100 actually have TB
| Actually TB+ |
85 (TP) |
15 (FN) |
| Actually TB- |
90 (FP) |
810 (TN) |
From this matrix, we calculate all other metrics.
Core Metrics
1. Sensitivity (Recall, True Positive Rate)
\[\text{Sensitivity} = \frac{TP}{TP + FN} = \frac{TP}{\text{All Actual Positives}}\]
- Interpretation: Of all actual positives, what proportion did we identify?
- Example: 85/100 = 85% (identified 85 of 100 TB cases)
- When to prioritize: High-stakes screening (must catch most cases), early disease detection, rule-out tests
- Trade-off: Maximizing sensitivity → more false positives
2. Specificity (True Negative Rate)
\[\text{Specificity} = \frac{TN}{TN + FP} = \frac{TN}{\text{All Actual Negatives}}\]
- Interpretation: Of all actual negatives, what proportion did we correctly identify?
- Example: 810/900 = 90% (correctly ruled out TB in 810 of 900 healthy people)
- When to prioritize: Confirmatory tests, when false alarms are costly, rule-in tests
- Trade-off: Maximizing specificity → more false negatives
3. Positive Predictive Value (Precision, PPV)
\[\text{PPV} = \frac{TP}{TP + FP} = \frac{TP}{\text{All Predicted Positives}}\]
- Interpretation: Of all predicted positives, what proportion are actually positive?
- Example: 85/175 = 49% (49% of positive predictions are correct)
- When to prioritize: When acting on predictions is costly (treatments, interventions)
- Critical property: Depends heavily on disease prevalence
Prevalence dependence example:
| High-burden TB setting |
10% |
85% |
90% |
49% |
| Low-burden TB setting |
1% |
85% |
90% |
8% |
Same model, vastly different PPV! In low-prevalence settings, even high specificity leads to poor PPV.
For detailed explanation, see Altman & Bland, 1994, BMJ on diagnostic tests and prevalence.
4. Negative Predictive Value (NPV)
\[\text{NPV} = \frac{TN}{TN + FN} = \frac{TN}{\text{All Predicted Negatives}}\]
- Interpretation: Of all predicted negatives, what proportion are actually negative?
- Example: 810/825 = 98% (98% of negative predictions are correct)
- When to prioritize: Rule-out tests, when missing disease is catastrophic
- Critical property: Also depends on prevalence (high prevalence → lower NPV)
5. Accuracy
\[\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{\text{Correct Predictions}}{\text{All Predictions}}\]
- Interpretation: Overall proportion of correct predictions
- Example: (85+810)/1000 = 89.5%
- Major limitation: Misleading for imbalanced datasets
Classic pitfall:
Dataset: 1,000 patients, 10 with disease (1% prevalence)
Naive model: Predict “no disease” for everyone - Accuracy: 990/1000 = 99% - But sensitivity = 0% (misses all disease cases!)
Takeaway: Accuracy alone is insufficient, especially for rare events.
6. F1 Score (Harmonic Mean of Precision and Recall)
\[F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2TP}{2TP + FP + FN}\]
- Interpretation: Balance between precision and recall
- Range: 0 (worst) to 1 (perfect)
- When to use: When you need single metric balancing both concerns
- Limitation: Ignores true negatives (not suitable when TN important)
Variants: - \(F_2\) score: Weights recall higher than precision - \(F_{0.5}\) score: Weights precision higher than recall
Threshold-Independent Metrics
7. Area Under the ROC Curve (AUC-ROC)
The Receiver Operating Characteristic (ROC) curve plots: - Y-axis: True Positive Rate (Sensitivity) - X-axis: False Positive Rate (1 - Specificity)
…across all possible classification thresholds (0 to 1).
AUC-ROC interpretation: - 0.5 = Random guessing (diagonal line) - 0.6-0.7 = Poor discrimination - 0.7-0.8 = Acceptable - 0.8-0.9 = Excellent - >0.9 = Outstanding (rare in clinical applications)
Alternative interpretation: Probability that a randomly selected positive case is ranked higher than a randomly selected negative case.
Advantages: - Threshold-independent (single summary metric) - Not affected by class imbalance (in terms of metric itself) - Standard metric for model comparison
Limitations: - May overemphasize performance at thresholds you wouldn’t use clinically - Doesn’t indicate optimal threshold - Can be misleading for highly imbalanced data (see Average Precision)
For comprehensive guide, see Hanley & McNeil, 1982, Radiology on the meaning and use of AUC.
8. Average Precision (Area Under Precision-Recall Curve)
The Precision-Recall (PR) curve plots: - Y-axis: Precision (PPV) - X-axis: Recall (Sensitivity)
…across all thresholds.
Average Precision (AP): Area under PR curve
PR curves are more informative than ROC curves for imbalanced datasets where the positive class is rare. They focus on performance on the positive class (which matters more when it is rare), whereas ROC can be misleadingly optimistic when the negative class dominates.
Example: Disease with 1% prevalence
- AUC-ROC = 0.90 (sounds great!)
- Average Precision = 0.25 (reveals poor performance on actual disease cases)
When to use: Rare disease detection, outbreak detection, any imbalanced problem
For detailed comparison, see Saito & Rehmsmeier, 2015, PLOS ONE on precision-recall vs. ROC curves.
Choosing Metrics by Scenario
| COVID-19 airport screening |
Sensitivity, NPV |
Must catch most cases; false positives acceptable (confirmatory testing available) |
| Cancer diagnosis confirmation |
Specificity, PPV |
False positives → unnecessary surgery; high bar for confirmation |
| Automated triage system |
AUC-ROC, Calibration |
Need good ranking across full risk spectrum |
| Rare disease detection |
Average Precision, Sensitivity |
Standard AUC-ROC misleading when imbalanced |
| Syndromic surveillance |
Sensitivity, Timeliness |
Early detection critical; false alarms tolerable (investigation cheap) |
| Clinical decision support |
PPV, Calibration |
Clinicians ignore if too many false alarms; need well-calibrated probabilities |
Calibration: Do Predicted Probabilities Mean What They Say?
**Calibration assesses whether predicted probabilities match observed frequencies.
Example of well-calibrated model: - Model predicts “30% risk of readmission” for 100 patients - About 30 of those 100 are actually readmitted - Predicted probability ≈ observed frequency
Poor calibration: - Model predicts “30% risk” but 50% are actually readmitted → underconfident - Model predicts “30% risk” but 15% are actually readmitted → overconfident
Measuring Calibration
1. Calibration Plot
Method: 1. Bin predictions into groups (e.g., 0-10%, 10-20%, …, 90-100%) 2. For each bin, calculate: - Mean predicted probability (x-axis) - Observed frequency of outcome (y-axis) 3. Plot points 4. Perfect calibration: points lie on diagonal line (y = x)
Interpretation: - Points above diagonal: Model underconfident (predicts lower risk than reality) - Points below diagonal: Model overconfident (predicts higher risk than reality)
2. Brier Score
\[\text{Brier Score} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2\]
where \(p_i\) = predicted probability, \(y_i\) = actual outcome (0 or 1)
- Range: 0 (perfect) to 1 (worst)
- Lower is better
- Combines discrimination and calibration into single metric
- Can be decomposed into calibration and refinement components
Interpretation: - 0.25 = Baseline (predicting prevalence for everyone) - <0.15 = Good calibration - <0.10 = Excellent calibration
For Brier score deep dive, see Rufibach, 2010, Clinical Trials.
3. Expected Calibration Error (ECE)
\[\text{ECE} = \sum_{m=1}^{M} \frac{n_m}{N} |\text{acc}(B_m) - \text{conf}(B_m)|\]
where: - \(M\) = number of bins - \(B_m\) = set of predictions in bin \(m\) - \(n_m\) = number of predictions in bin \(m\) - \(\text{acc}(B_m)\) = accuracy in bin \(m\) - \(\text{conf}(B_m)\) = average confidence in bin \(m\)
Interpretation: Average difference between predicted and observed probabilities across bins (weighted by bin size)
Why Calibration Matters
Clinical decision-making requires well-calibrated probabilities:
Scenario 1: Treatment threshold - If risk >20%, prescribe preventive medication - Poorly calibrated model: risk actually 40% when model says 20% - Result: Under-treatment of high-risk patients
Scenario 2: Resource allocation - Allocate home health visits to top 10% risk - Overconfident model: predicted “high risk” patients aren’t actually high risk - Result: Resources wasted on low-risk patients, true high-risk patients missed
Scenario 3: Patient counseling - Tell patient: “You have 30% chance of complications” - If model poorly calibrated, this number is meaningless - Result: Informed consent based on inaccurate information
Common issue: Deep neural networks often produce poorly calibrated probabilities out-of-the-box. They tend to be overconfident (predicted probabilities too extreme).
Why? Modern neural networks are optimized for accuracy, not calibration. Regularization techniques that prevent overfitting can actually worsen calibration.
Evidence: Guo et al., 2017, ICML - “On Calibration of Modern Neural Networks”
Solution: Post-hoc calibration methods: - Temperature scaling: Simplest and most effective - Platt scaling: Logistic regression on model outputs - Isotonic regression: Non-parametric calibration
Takeaway: Always assess and correct calibration for deep learning models before deployment.
Regression Metrics
For continuous outcome prediction (disease burden, resource utilization, epidemic size):
1. Mean Absolute Error (MAE)
\[\text{MAE} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i|\]
- Interpretation: Average absolute difference between prediction and truth
- Unit: Same as outcome variable
- Advantage: Interpretable, robust to outliers
- Example: MAE = 3.2 days (average error in predicting length of stay)
2. Root Mean Squared Error (RMSE)
\[\text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2}\]
- Interpretation: Square root of average squared error
- Property: Penalizes large errors more heavily than MAE (due to squaring)
- When to use: When large errors are particularly problematic
Relationship: RMSE ≥ MAE always (equality only if all errors identical)
3. R-squared (Coefficient of Determination)
\[R^2 = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2} = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}\]
- Range: 0 to 1 (can be negative if model worse than mean)
- Interpretation: Proportion of variance in outcome explained by model
- Example: R² = 0.65 means model explains 65% of variance
- Limitation: Can be artificially inflated by adding more features
4. Mean Absolute Percentage Error (MAPE)
\[\text{MAPE} = \frac{100\%}{N} \sum_{i=1}^{N} \left| \frac{y_i - \hat{y}_i}{y_i} \right|\]
- Interpretation: Average percentage error
- Advantage: Scale-independent (can compare across different units)
- Example: MAPE = 15% (average error is 15% of true value)
- Limitation: Undefined when actual value is zero; penalizes under-predictions more than over-predictions
Survival Analysis Metrics
For time-to-event prediction (mortality, readmission, disease progression):
1. Concordance Index (C-index, Harrell’s C-statistic)
- Extension of AUC-ROC to survival data with censoring
- Interpretation: Probability that, for two randomly selected individuals, the one who experiences event first has higher predicted risk
- Range: 0.5 (random) to 1.0 (perfect)
- Handles censoring: Pairs where censoring occurs are excluded or weighted
For details: Harrell et al., 1982, JAMA - original C-index paper.
2. Integrated Brier Score (IBS)
- Extension of Brier score to survival analysis
- Interpretation: Average prediction error over time, accounting for censoring
- Range: 0 (perfect) to 1 (worst)
- Advantage: Assesses calibration of survival probability predictions over follow-up period