Validation, Equity, and Security Testing
External validation, clinical utility, fairness, and adversarial robustness for public health AI evaluation. The material is maintained separately so each operational question has a stable, focused reference.
- Identify the evidence and controls relevant to this decision area
- Distinguish technical performance from operational and population impact
- Apply the included framework without extending claims beyond the cited evidence
Introduction
This focused reference is part of the broader Validation, Equity, and Security Testing overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.
Validation Strategies: Testing Generalization
The validation strategy determines how trustworthy your performance estimates are.
Internal Validation
Purpose: Estimate model performance on new data from the same source.
Critical limitation: Provides no evidence about performance on different populations, institutions, or time periods.
Method 1: Train-Test Split (Hold-Out Validation)
Procedure: 1. Randomly split data into training (70-80%) and test (20-30%) 2. Train model on training set 3. Evaluate on test set (one time only)
Advantages: - Simple and fast - Clear separation between training and testing
Disadvantages: - Single split can be unrepresentative (bad luck in random split) - Wastes data (test set not used for training) - High variance in performance estimate
When to use: Large datasets (>10,000 samples), quick experiments
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2, # 20% for testing
random_state=42, # Reproducible split
stratify=y # Maintain class balance
)Method 2: K-Fold Cross-Validation
Procedure: 1. Divide data into K folds (typically 5 or 10) 2. For each fold: - Train on K-1 folds - Validate on remaining fold 3. Average performance across all K folds
Advantages: - Uses all data for both training and validation - More stable performance estimate (less variance) - Standard practice in machine learning
Disadvantages: - Computationally expensive (train K models) - Still no external validation
When to use: Moderate-sized datasets (1,000-10,000 samples), model selection
from sklearn.model_selection import cross_val_score
scores = cross_val_score(
model, X, y,
cv=5, # 5-fold CV
scoring='roc_auc' # Metric to optimize
)
print(f"AUC-ROC: {scores.mean():.3f} (±{scores.std():.3f})")Method 3: Stratified K-Fold Cross-Validation
Modification: Ensures each fold maintains the same class distribution as the full dataset.
Critical for imbalanced datasets (e.g., 5% disease prevalence).
Why it matters: Without stratification, some folds might have very few positive cases (or none!), leading to unstable estimates.
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring='roc_auc')Method 4: Time-Series Cross-Validation
For temporal data: Never train on future, test on past!
Procedure (expanding window):
Fold 1: Train [1:100] → Test [101:120]
Fold 2: Train [1:120] → Test [121:140]
Fold 3: Train [1:140] → Test [141:160]
...Critical for: Epidemic forecasting, time-series prediction, any data with temporal structure
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# Train and evaluateCritical Considerations for Internal Validation
1. Data Leakage Prevention
Data leakage: Information from test set influencing training process.
Common sources:
WRONG: Feature engineering on entire dataset:
# WRONG: Standardize before splitting
X_scaled = StandardScaler().fit_transform(X) # Uses mean/std from ALL data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
# Test set info leaked into training!CORRECT: Feature engineering within train/test:
# CORRECT: Fit scaler on training only
X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = StandardScaler().fit(X_train) # Learn from training only
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test) # Apply to testWRONG: Feature selection on entire dataset:
# WRONG
selector = SelectKBest(k=10).fit(X, y) # Uses ALL data
X_selected = selector.transform(X)
X_train, X_test = train_test_split(X_selected)CORRECT: Feature selection within training:
# CORRECT
X_train, X_test, y_train, y_test = train_test_split(X, y)
selector = SelectKBest(k=10).fit(X_train, y_train)
X_train_selected = selector.transform(X_train)
X_test_selected = selector.transform(X_test)For comprehensive guide on data leakage, see Kaufman et al., 2012, SIGKDD.
2. Cluster-Aware Splitting
Problem: If data has natural clusters (patients within hospitals, repeated measures within individuals), random splitting can lead to overfitting.
Example: Patient has 5 hospitalizations. Random split → some hospitalizations in training, others in test. Model learns patient-specific patterns → overoptimistic performance.
Solution: Group K-Fold , ensure all samples from same group stay together
from sklearn.model_selection import GroupKFold
# patient_ids: array indicating which patient each sample belongs to
gkf = GroupKFold(n_splits=5)
for train_idx, test_idx in gkf.split(X, y, groups=patient_ids):
# All samples from same patient stay in same fold
X_train, X_test = X[train_idx], X[test_idx]External Validation: Stronger Evidence for Generalizability
External validation: Testing on data from entirely different source, different institution(s), population, time period, or setting.
Why it matters:
Models often learn dataset-specific quirks that do not generalize: - Hospital equipment signatures - Documentation practices - Patient population characteristics - Data collection protocols
Without external validation, you do not know if model learned disease patterns or dataset artifacts.
Types of External Validation
1. Geographic External Validation
Design: - Train: Hospital A (or multiple hospitals in one region) - Test: Hospital B (or hospitals in different region)
What it tests: - Different patient demographics - Different clinical practices - Different data collection protocols - Different equipment (for imaging)
Example: McKinney et al., 2020, Nature - Google breast cancer AI trained on UK data, validated on US data (and vice versa). Performance dropped: UK→US AUC decreased from 0.889 to 0.858.
2. Temporal External Validation
Design: - Train: Data from 2015-2018 - Test: Data from 2019-2021
What it tests: - Temporal stability (concept drift) - Changes in disease patterns - Changes in clinical practice - Changes in data collection
Example: Davis et al., 2017, JAMIA - Clinical prediction models degrade over time; most models need recalibration after 2-3 years.
3. Setting External Validation
Design: - Train: Intensive care unit (ICU) data - Test: General ward data
What it tests: - Performance in different clinical settings - Generalization across disease severity spectra
Example: Sepsis models trained on ICU patients often fail on ward patients (different disease presentation, different monitoring intensity).
External Validation Case Study
Prospective Validation: Real-World Testing
Prospective validation: Model deployed in actual clinical practice, evaluated in real-time.
Why it matters: Retrospective validation cannot capture: - How clinicians actually use (or ignore) model predictions - Workflow integration challenges - Alert fatigue and override patterns - Behavioral changes in response to predictions - Unintended consequences
Study Design 1: Silent Mode Deployment
Design: - Deploy model in background - Generate predictions but do not show to clinicians - Compare predictions to actual outcomes (collected as usual)
Advantages: - Tests real-world data quality and distribution - No risk to patients (clinicians unaware of predictions) - Can assess performance before making decisions based on model
Disadvantages: - Does not test impact on clinical decisions - Does not assess workflow integration
Example: Tomašev et al., 2019, Nature - DeepMind AKI prediction initially deployed silently at VA hospitals to validate real-time performance before clinical integration.
Study Design 2: Randomized Controlled Trial (RCT)
Design: - Randomize: Patients, clinicians, or hospital units to: - Intervention: Model-assisted care - Control: Standard care (no model) - Measure: Clinical outcomes in both groups - Compare: Test if model improves outcomes
Advantages: - Strong causal evidence for the effect of using the intervention - Can estimate effects on outcomes and resource use - Supports causal effectiveness and economic claims when the design and outcomes match those claims
Disadvantages: - Resource-intensive - May require extended follow-up - Requires large sample size - Ethical considerations (withholding potentially beneficial intervention)
Example: Semler et al., 2018, NEJM - SMART trial of balanced crystalloids vs. saline (not AI, but example of rigorous prospective cluster-randomized design)
Study Design 3: Stepped-Wedge Design
Design: - Roll out model sequentially to different units/sites - Each unit serves as its own control (before vs. after) - Eventually all units receive intervention
Advantages: - More feasible than full RCT - All units eventually get intervention (addresses ethical concerns) - Within-unit comparisons reduce confounding
Disadvantages: - Temporal trends can confound results - Less rigorous than RCT (no contemporaneous control group)
Example: Common in health system implementations where full RCT infeasible.
Study Design 4: A/B Testing
Design: - Randomly assign users to model-assisted vs. control in real-time - Continuously measure outcomes - Iterate rapidly based on results
Advantages: - Rapid experimentation - Can test multiple model versions - Common in tech industry
Challenges in healthcare: - Ethical concerns (different care for similar patients) - Regulatory considerations (IRB approval required) - Contamination (clinicians may share information)
Beyond Accuracy: Clinical Utility Assessment
Critical insight: A model can be statistically accurate but clinically useless.
Example: - Model predicts hospital mortality with AUC-ROC = 0.85 - But: If it does not change clinical decisions or improve outcomes, what’s the value? - Moreover: If implementing it disrupts workflow or generates alert fatigue, net impact may be negative.
Before deploying any clinical AI:
- Does it change decisions?
- Do those changed decisions improve outcomes?
- Is the improvement worth the cost (financial, workflow disruption, alert burden)?
If you cannot answer “yes” to all three, do not deploy.
Decision Curve Analysis (DCA)
Developed by: Vickers & Elkin, 2006, Medical Decision Making
Purpose: Assess the clinical net benefit of using a prediction model compared to alternative strategies.
Concept: A model is clinically useful only if using it leads to better decisions than: - Treating everyone - Treating no one - Using clinical judgment alone
How Decision Curve Analysis Works
For each possible risk threshold \(p_t\) (e.g., “treat if risk >10%”):
Calculate Net Benefit (NB):
\[\text{NB}(p_t) = \frac{TP}{N} - \frac{FP}{N} \times \frac{p_t}{1 - p_t}\]
Where: - \(TP/N\) = True positive rate (benefit from correctly treating disease) - \(FP/N \times p_t/(1-p_t)\) = False positive rate, weighted by harm of unnecessary treatment
Interpretation: - If treating disease has high benefit relative to harm of unnecessary treatment → lower \(p_t\) threshold - If treating disease has low benefit relative to harm → higher \(p_t\) threshold
Weight \(p_t/(1-p_t)\): Reflects how much we weight false positives. - At \(p_t\) = 0.10: Weight = 0.10/0.90 ≈ 0.11 (FP weighted 1/9 as much as TP) - At \(p_t\) = 0.50: Weight = 0.50/0.50 = 1.00 (FP and TP equally weighted)
DCA Plot and Interpretation
Create DCA plot: - X-axis: Threshold probability (risk at which you’d intervene) - Y-axis: Net benefit - Plot curves for: - Model: Net benefit using model predictions - Treat all: Net benefit if everyone treated - Treat none: Net benefit if no one treated (= 0)
Interpretation: - Model is useful where its curve is above both “treat all” and “treat none” - Higher net benefit = better clinical value - Range of thresholds where model useful = decision curve clinical range
Example interpretation:
At 15% risk threshold: - Model NB = 0.12 - Treat all NB = 0.05 - Treat none NB = 0.00
Meaning: Using model at 15% threshold is equivalent to correctly treating 12 out of 100 patients with no false positives, compared to only 5 for “treat all” strategy.
Python implementation:
def calculate_net_benefit(y_true, y_pred_proba, thresholds):
"""Calculate net benefit across thresholds for decision curve analysis"""
net_benefits = []
for threshold in thresholds:
# Classify based on threshold
y_pred = (y_pred_proba >= threshold).astype(int)
# Calculate TP, FP, TN, FN
TP = ((y_pred == 1) & (y_true == 1)).sum()
FP = ((y_pred == 1) & (y_true == 0)).sum()
N = len(y_true)
# Net benefit formula
nb = (TP / N) - (FP / N) * (threshold / (1 - threshold))
net_benefits.append(nb)
return np.array(net_benefits)
# Calculate for model, treat all, treat none
thresholds = np.linspace(0.01, 0.99, 100)
nb_model = calculate_net_benefit(y_test, y_pred_proba, thresholds)
nb_treat_all = y_test.mean() - (1 - y_test.mean()) * (thresholds / (1 - thresholds))
nb_treat_none = np.zeros_like(thresholds)
# Plot decision curve
plt.figure(figsize=(10, 6))
plt.plot(thresholds, nb_model, label='Model', linewidth=2)
plt.plot(thresholds, nb_treat_all, label='Treat All', linestyle='--', linewidth=2)
plt.plot(thresholds, nb_treat_none, label='Treat None', linestyle=':', linewidth=2)
plt.xlabel('Threshold Probability', fontsize=12)
plt.ylabel('Net Benefit', fontsize=12)
plt.title('Decision Curve Analysis', fontsize=14, fontweight='bold')
plt.legend()
plt.grid(alpha=0.3)
plt.xlim(0, 0.5) # Focus on clinically relevant range
plt.show()For comprehensive tutorial, see Vickers et al., 2019, Diagnostic and Prognostic Research.
Reclassification Metrics
Purpose: Quantify whether new model improves risk stratification compared to existing approach.
Context: You have an existing risk model (or clinical judgment). New model proposed. Does it reclassify patients into more appropriate risk categories?
Net Reclassification Improvement (NRI)
Concept: Among events (people with disease), what proportion correctly moved to higher risk? Among non-events, what proportion correctly moved to lower risk?
Formula:
\[\text{NRI} = (\text{NRI}_{\text{events}} + \text{NRI}_{\text{non-events}}) / 2\]
Where: - \(\text{NRI}_{\text{events}}\) = P(moved up | event) - P(moved down | event) - \(\text{NRI}_{\text{non-events}}\) = P(moved down | non-event) - P(moved up | non-event)
Interpretation: - NRI > 0: New model improves classification - NRI < 0: New model worsens classification - Typically report with 95% CI
Example:
| Group | Moved Up | Stayed | Moved Down | NRI Component |
|---|---|---|---|---|
| Events (n=100) | 35 | 50 | 15 | (35-15)/100 = 0.20 |
| Non-events (n=900) | 50 | 800 | 50 | (50-50)/900 = 0.00 |
NRI = (0.20 + 0.00) / 2 = 0.10
Interpretation: Net 10% improvement in classification.
For detailed explanation, see Pencina et al., 2008, Statistics in Medicine.
Integrated Discrimination Improvement (IDI)
Concept: Difference in average predicted probabilities between events and non-events.
Formula:
\[\text{IDI} = [\overline{P}_{\text{new}}(\text{events}) - \overline{P}_{\text{old}}(\text{events})] - [\overline{P}_{\text{new}}(\text{non-events}) - \overline{P}_{\text{old}}(\text{non-events})]\]
Interpretation: - How much does new model increase separation between events and non-events? - IDI > 0: Better discrimination - Less sensitive to arbitrary cut-points than NRI
Fairness and Equity in Evaluation
AI systems can exhibit disparate performance across demographic groups, even when overall performance appears strong.
Failure to assess fairness can: - Perpetuate or amplify existing health disparities - Result in differential quality of care based on race, gender, socioeconomic status - Violate ethical principles of justice and equity - Expose organizations to legal liability
Assessing fairness is not optional. It’s essential.
Mathematical Definitions of Fairness
Challenge: Multiple, often conflicting, definitions of fairness exist.
1. Demographic Parity (Statistical Parity)
Definition: Positive prediction rates equal across groups
\[P(\hat{Y}=1 | A=0) = P(\hat{Y}=1 | A=1)\]
where \(A\) = protected attribute (e.g., race, gender)
Example: Model predicts high risk for 20% of White patients and 20% of Black patients
When appropriate: - Resource allocation (equal access to interventions) - Contexts where base rates should be equal
Problem: Ignores actual outcome rates. If disease prevalence differs between groups (due to structural factors), enforcing demographic parity may reduce overall accuracy.
2. Equalized Odds (Equal Opportunity)
Definition: True positive and false positive rates equal across groups
\[P(\hat{Y}=1 | Y=1, A=0) = P(\hat{Y}=1 | Y=1, A=1)\] \[P(\hat{Y}=1 | Y=0, A=0) = P(\hat{Y}=1 | Y=0, A=1)\]
Example: 85% sensitivity for both White and Black patients; 90% specificity for both
When appropriate: - Clinical diagnosis and screening - When both types of errors (false positives and false negatives) matter
More clinically relevant than demographic parity in most healthcare applications.
3. Calibration Fairness
Definition: Predicted probabilities calibrated for all groups
\[P(Y=1 | \hat{Y}=p, A=0) = P(Y=1 | \hat{Y}=p, A=1) = p\]
Example: Among patients predicted 30% risk, ~30% in each group actually experience outcome
When appropriate: - Risk prediction for clinical decision-making - When predicted probabilities guide treatment thresholds
Most important for clinical applications where decisions based on predicted probabilities.
4. Predictive Parity
Definition: Positive predictive values equal across groups
\[P(Y=1 | \hat{Y}=1, A=0) = P(Y=1 | \hat{Y}=1, A=1)\]
Example: Among patients predicted positive, same proportion are true positives in both groups
When appropriate: - When acting on positive predictions (e.g., treatment initiation)
The Impossibility Theorem
Fundamental challenge: Chouldechova, 2017, FAT and Kleinberg et al., 2017, ITCS proved:
If base rates differ between groups, you cannot simultaneously satisfy: 1. Calibration 2. Equalized odds 3. Predictive parity
Implication: Must choose which fairness criterion to prioritize based on context and values.
For healthcare: Calibration typically most important (want predicted probabilities to mean the same thing across groups).
Practical Fairness Assessment
Step-by-Step Fairness Audit
Step 1: Define Protected Attributes
Identify characteristics that should not influence model performance: - Race/ethnicity - Gender/sex - Age - Socioeconomic status (income, insurance, ZIP code) - Language - Disability status
Step 2: Stratify Performance Metrics
Calculate metrics separately for each subgroup:
# Example: Performance by race/ethnicity
groups = data.groupby('race')
fairness_metrics = []
for race, group_data in groups:
y_true = group_data['outcome']
y_pred = group_data['prediction']
metrics = {
'race': race,
'n': len(group_data),
'prevalence': y_true.mean(),
'sensitivity': recall_score(y_true, y_pred > 0.5),
'specificity': recall_score(1 - y_true, 1 - (y_pred > 0.5)),
'PPV': precision_score(y_true, y_pred > 0.5),
'NPV': precision_score(1 - y_true, 1 - (y_pred > 0.5)),
'AUC': roc_auc_score(y_true, y_pred),
'Brier': brier_score_loss(y_true, y_pred)
}
fairness_metrics.append(metrics)
fairness_df = pd.DataFrame(fairness_metrics)
print(fairness_df)Step 3: Assess Calibration by Subgroup
# Calibration plots by race
fig, axes = plt.subplots(1, len(groups), figsize=(15, 5))
for idx, (race, group_data) in enumerate(groups):
y_true = group_data['outcome']
y_pred = group_data['prediction']
prob_true, prob_pred = calibration_curve(y_true, y_pred, n_bins=10)
axes[idx].plot(prob_pred, prob_true, marker='o', label=race)
axes[idx].plot([0, 1], [0, 1], 'k--', label='Perfect calibration')
axes[idx].set_title(f'{race} (n={len(group_data)})')
axes[idx].set_xlabel('Predicted Probability')
axes[idx].set_ylabel('Observed Frequency')
axes[idx].legend()Step 4: Identify Disparities
Calculate disparity metrics:
Absolute disparity: Difference between groups
sens_white = metrics_white['sensitivity']
sens_black = metrics_black['sensitivity']
disparity_abs = sens_white - sens_blackRelative disparity: Ratio between groups
disparity_rel = sens_white / sens_blackExample local review triggers: - A prespecified absolute disparity that could change clinical or program decisions - A prespecified relative disparity interpreted with uncertainty, sample size, prevalence, and outcome severity
These are governance choices, not universal fairness standards.
Step 5: Investigate Root Causes
Potential causes of disparities:
- Data representation
- Underrepresentation in training data
- Different sample sizes → unstable estimates for small groups
- Label bias
- Outcome labels reflect biased processes (e.g., healthcare access disparities)
- Example: Hospitalization rates lower in group with less access, not because they’re healthier
- Feature bias
- Features proxy for protected attributes
- Example: ZIP code strongly correlated with race
- Measurement bias
- Different data quality across groups
- Example: Pulse oximetry less accurate in dark skin (Sjoding et al., 2020, NEJM)
- Prevalence differences
- True differences in disease prevalence
- May be due to structural factors (e.g., environmental exposures)
Step 6: Mitigation Strategies
Pre-processing (adjust training data): - Increase representation of underrepresented groups (oversampling, synthetic data) - Re-weight samples to balance groups - Remove or transform biased features
In-processing (modify algorithm): - Add fairness constraints during training - Adversarial debiasing (penalize predictions that reveal protected attribute) - Multi-objective optimization (accuracy + fairness)
Post-processing (adjust predictions): - Separate thresholds per group to achieve equalized odds - Calibration adjustment per group - Reject option classification (defer to human for uncertain cases)
Structural interventions: - Address root causes (improve data collection for underrepresented groups) - Partner with communities to ensure appropriate representation - Consider whether model should be deployed if disparities cannot be adequately mitigated
For comprehensive fairness toolkit, see Fairlearn by Microsoft.
Landmark Bias Case Study
Adversarial Robustness and Security Evaluation
Why Robustness Matters for Medical AI
Traditional evaluation assumes benign inputs. But deployed models face: - Natural perturbations: Image quality variation, data entry errors, equipment differences - Adversarial attacks: Malicious manipulation to cause misclassification - Out-of-distribution inputs: Cases far from training data
EU context: Article 15 of the EU AI Act requires an appropriate level of accuracy, robustness, and cybersecurity for high-risk systems. It does not prescribe one universal adversarial-testing method (EU AI Act, 2024).
Example scenarios: - Hospital ransomware attack compromises AI model integrity - Malicious actor manipulates medical imaging to hide cancer - Data poisoning during model retraining introduces systematic errors
Unlike traditional software vulnerabilities (which can be patched), ML models can be permanently corrupted or subtly manipulated without obvious signs.
Types of Adversarial Threats
1. Evasion Attacks (Inference-Time)
Goal: Manipulate input to cause misclassification without changing ground truth.
Example: - Add imperceptible noise to chest X-ray → Model misses pneumonia - Modify patient vital signs slightly → Sepsis prediction model fails to alert
Medical relevance: - Natural occurrence: Image compression, scanner differences can mimic adversarial perturbations - Malicious: Rare but theoretically possible (e.g., insurance fraud, medicolegal manipulation)
2. Poisoning Attacks (Training-Time)
Goal: Corrupt training data to degrade model performance or introduce backdoors.
Example: - Insert mislabeled images into training set → Model learns incorrect patterns - Add trigger patterns → Model fails only for specific subgroups
Medical relevance: - Multi-institutional data sharing: If one site’s data is compromised, all participants affected - Crowdsourced labels: If annotations are maliciously manipulated
3. Model Extraction/Stealing
Goal: Query model repeatedly to reverse-engineer its parameters.
Risk: Intellectual property theft, creating surrogate model for further attacks
Evaluating Robustness
Method 1: Input Perturbation Testing
Approach: Systematically perturb inputs and measure performance degradation.
For medical imaging:
For tabular data (clinical variables):
Method 2: Adversarial Attack Testing
Fast Gradient Sign Method (FGSM) - Basic adversarial attack:
Method 3: Out-of-Distribution (OOD) Detection
Goal: Identify when inputs are unlike training data (model should abstain or flag uncertainty).
Robustness Improvement Strategies
1. Data Augmentation: - Train on varied/augmented data (rotations, brightness changes, noise) - Forces model to learn invariant features
2. Adversarial Training: - Include adversarial examples in training set - Trade-off: May slightly reduce clean accuracy
3. Ensemble Methods: - Multiple models often more robust than single model - Harder to fool all models simultaneously
4. Input Validation: - Reject inputs that are outliers (OOD detection) - Flag unusual patterns for human review
5. Certified Defenses: - Provide mathematical guarantees of robustness - Advanced, computationally expensive
Practical Robustness Evaluation Protocol
Minimum requirements (all deployed models): - [ ] Natural perturbation testing: Test with realistic variations (noise, missing data, equipment differences) - [ ] Prediction stability: Measure how often predictions change with small input perturbations and compare the result with a prespecified, clinically justified local threshold - [ ] Out-of-distribution detection: Model should flag or have low confidence on unfamiliar inputs
Recommended (high-risk models): - [ ] Adversarial attack testing: Evaluate vulnerability to FGSM, PGD attacks - [ ] Multi-site robustness: Validate performance across diverse sites/equipment - [ ] Ablation studies: Test performance when features are missing or corrupted
Advanced (critical systems, regulatory requirements): - [ ] Certified robustness: Provide formal guarantees for critical use cases - [ ] Red-team exercise: Security experts attempt to break the model - [ ] Continuous monitoring: Track input distribution shifts, flag anomalies
Security Best Practices
1. Model Access Control: - Limit API access to authenticated users - Rate limiting to prevent model extraction attacks
2. Input Sanitization: - Validate inputs are within expected ranges - Reject clearly anomalous inputs
3. Monitoring and Logging: - Log all predictions and inputs - Monitor for unusual query patterns (potential attacks)
4. Model Versioning and Rollback: - Maintain ability to revert to previous model if compromise detected
5. Regular Security Audits: - Periodic red-team testing - Review access logs for suspicious activity
Landmark Study: Adversarial Perturbations in Medical Imaging
Finlayson et al., 2019, Science
Study: Added imperceptible perturbations to medical images (chest X-rays, fundus photos, dermatology images)
Results: - Successfully fooled leading deep learning classifiers - Adversarial examples transferable across models (attack one model, affects others) - Small perturbations caused dramatic misclassifications
Implications: - Medical AI models are vulnerable to adversarial attacks - Robustness testing should be mandatory for deployed systems - Both accidental (natural variations) and malicious perturbations are risks
Counterpoint: No documented real-world malicious attacks on medical AI systems (yet), but accidental distribution shifts are common (equipment changes, protocol updates).
Key Takeaways: Adversarial Robustness
Robustness must be demonstrated appropriately: EU Article 15 requires appropriate accuracy, robustness, and cybersecurity for high-risk systems. Adversarial testing is one possible evidence method when the threat model warrants it.
Two threat models: Natural perturbations (common) vs. adversarial attacks (rare but possible)
Multiple evaluation methods: Noise robustness, adversarial attacks (FGSM/PGD), OOD detection
Practical importance: Equipment variation, scanner differences, data quality issues are real-world “natural adversarial examples”
Trade-offs exist: Adversarial training improves robustness but may reduce clean accuracy
Prevention strategies: Data augmentation, ensemble methods, input validation, monitoring
Security is patient safety: Model integrity directly affects clinical outcomes
Document robustness: Report perturbation testing results in validation studies
OOD detection is critical: Models must recognize when inputs are outside training distribution
Continuous vigilance: Monitor for distribution shifts and anomalous inputs post-deployment