Explainability for Public Health AI

Interpretability methods, failure modes, and operational use of explanations in public health AI systems. The material is maintained separately so each operational question has a stable, focused reference.

Learning Objectives

• Identify the evidence and controls relevant to this decision area
• Distinguish technical performance from operational and population impact
• Apply the included framework without extending claims beyond the cited evidence

Chapter Summary (TL;DR)

Use explicit targets, populations, thresholds, and decision consequences. Require external evidence and local monitoring where deployment can affect people or programs. Preserve uncertainty and document limits.

Introduction

This focused reference is part of the broader Explainability overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.

Explainability and Interpretability (XAI)

Why Explainability Matters in Public Health AI

The trust problem: Systematic reviews consistently find that clinicians are reluctant to trust or act on predictions from “black box” AI systems they cannot interpret (Antoniadi et al., 2021; Markus et al., 2021).

Why interpretability is critical:

1. Clinical decision-making: Clinicians need to know why before they can decide whether to act
2. Debugging and validation: Explanations reveal spurious correlations and dataset biases
3. Regulatory requirements: FDA and EU AI Act increasingly mandate explainability for high-risk systems
4. Patient autonomy: Patients have a right to understand decisions affecting their health
5. Legal liability: “The algorithm said so” is not a defense in malpractice cases

The Accuracy-Interpretability Trade-off (A False Dichotomy?)

Traditional belief: Deep learning = high accuracy but uninterpretable; simpler models = lower accuracy but interpretable.

2024 reality: Post-hoc explainability methods (SHAP, attention mechanisms) make complex models interpretable without sacrificing accuracy. The choice is no longer binary.

Guideline: Start with the simplest model that meets performance requirements. If you need complex models, invest in robust explainability infrastructure.

---

Levels of Interpretability

Not all interpretability is equal. Different stakeholders need different levels of explanation.

1. Global Interpretability

Definition: Understanding the model’s overall behavior and decision logic.

Questions answered: - What features are most important overall? - How does the model generally make decisions? - Are there unexpected feature relationships?

Methods: - Feature importance rankings - Partial dependence plots - Global SHAP values

Audience: Data scientists, validators, regulators

---

2. Local Interpretability

Definition: Understanding why the model made a specific prediction for a specific patient.

Questions answered: - Why did the model predict this patient is high-risk? - Which patient characteristics drove this prediction? - What would need to change to alter the prediction?

Methods: - LIME (Local Interpretable Model-agnostic Explanations) - SHAP values for individual predictions - Counterfactual explanations

Audience: Clinicians, patients

---

3. Model-Based Interpretability

Definition: Models that are inherently interpretable by design.

Examples: - Linear models: Each coefficient shows feature contribution - Decision trees: Follow the path to understand the decision - Rule-based systems: Explicit IF-THEN logic

When to use: When stakeholder trust is paramount and model performance requirements are modest.

---

Interpretability Methods: Practical Guide

Method 1: SHAP (SHapley Additive exPlanations)

What it is: A unified framework for interpreting model predictions based on game theory (Shapley values).

Why it’s powerful: - Model-agnostic: Works with any ML model (XGBoost, neural networks, etc.) - Theoretically grounded: Satisfies desirable properties (local accuracy, consistency) - Both global and local: Feature importance + individual predictions

Foundational paper: Lundberg & Lee, 2017, NeurIPS

---

SHAP Example: Sepsis Risk Prediction

import shap
import xgboost as xgb
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Simulate patient data
np.random.seed(42)
n_patients = 1000

data = pd.DataFrame({
 'age': np.random.normal(65, 15, n_patients),
 'heart_rate': np.random.normal(90, 20, n_patients),
 'temperature': np.random.normal(37.5, 1.5, n_patients),
 'wbc_count': np.random.lognormal(2.3, 0.5, n_patients), # White blood cell count
 'lactate': np.random.exponential(1.5, n_patients),
 'systolic_bp': np.random.normal(120, 25, n_patients),
})

# Create synthetic sepsis outcome (complex non-linear relationships)
sepsis_risk = (
 0.3 * (data['temperature'] > 38.3) + # Fever
 0.3 * (data['wbc_count'] > 12) +  # Elevated WBC
 0.2 * (data['lactate'] > 2) +   # Elevated lactate
 0.2 * (data['heart_rate'] > 100) + # Tachycardia
 np.random.normal(0, 0.1, n_patients) # Noise
)
data['sepsis'] = (sepsis_risk > 0.6).astype(int)

# Split data
from sklearn.model_selection import train_test_split
X = data.drop('sepsis', axis=1)
y = data['sepsis']
X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.2, random_state=42, stratify=y
)

# Train XGBoost model
model = xgb.XGBClassifier(
 n_estimators=100,
 max_depth=5,
 learning_rate=0.1,
 random_state=42
)
model.fit(X_train, y_train)

# Evaluate
from sklearn.metrics import roc_auc_score
y_pred_proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_pred_proba)
print(f"Model AUC-ROC: {auc:.3f}")

# ===== SHAP ANALYSIS =====

# 1. Create SHAP explainer
explainer = shap.Explainer(model, X_train)

# 2. Calculate SHAP values for test set
shap_values = explainer(X_test)

# 3. GLOBAL INTERPRETABILITY: Feature importance
print("\n=== Global Feature Importance ===")
shap.plots.bar(shap_values, show=False)
plt.title("Global Feature Importance (Mean |SHAP|)")
plt.tight_layout()
plt.savefig("shap_global_importance.png", dpi=150)
plt.show()

# Alternative: Summary plot (beeswarm)
shap.plots.beeswarm(shap_values, show=False)
plt.title("Feature Impact on Model Output")
plt.tight_layout()
plt.savefig("shap_summary.png", dpi=150)
plt.show()

# 4. LOCAL INTERPRETABILITY: Explain specific patient
patient_idx = 0 # First patient in test set
print(f"\n=== Patient {patient_idx} ===")
print(f"Predicted sepsis probability: {y_pred_proba[patient_idx]:.2%}")
print(f"Actual outcome: {'Sepsis' if y_test.iloc[patient_idx] == 1 else 'No sepsis'}")

# Waterfall plot: Show how features contribute to this prediction
shap.plots.waterfall(shap_values[patient_idx], show=False)
plt.title(f"Explanation for Patient {patient_idx}")
plt.tight_layout()
plt.savefig(f"shap_patient_{patient_idx}.png", dpi=150)
plt.show()

# 5. ACTIONABLE INSIGHTS: What drives high-risk predictions?
print("\n=== Feature Values for High-Risk Patient ===")
for feature in X.columns:
 print(f"{feature}: {X_test.iloc[patient_idx][feature]:.2f}")

# 6. Dependence plot: How does lactate affect predictions?
shap.plots.scatter(shap_values[:, "lactate"], color=shap_values, show=False)
plt.title("Lactate Impact on Sepsis Prediction")
plt.tight_layout()
plt.savefig("shap_dependence_lactate.png", dpi=150)
plt.show()

Key outputs:

1. Global importance: Which features matter most across all patients?
2. Waterfall plot: For Patient X, lactate (+0.3) and temperature (+0.2) increased risk; normal BP (-0.1) decreased it
3. Dependence plots: Non-linear relationships (e.g., lactate > 2 mmol/L sharply increases risk)

Clinical translation:

Patient 47: Sepsis Risk = 78%

Main drivers:
+ Lactate 3.2 mmol/L (+0.35 risk contribution) <- **Primary concern**
+ Temperature 39.1°C (+0.22)
+ WBC 15,000/μL (+0.18)
- Normal BP 118/72 (-0.08) <- **Protective factor**

Interpretation: Elevated lactate is the strongest predictor.
Consider serial lactate monitoring and early fluid resuscitation.

---

SHAP Advantages and Limitations

Advantages: - Mathematically principled (satisfies local accuracy, missingness, consistency) - Works with any model architecture - Both global and local explanations - Handles feature interactions

Limitations: - Computational cost: Can be slow for large models/datasets (use TreeSHAP for tree models, faster) - Not causal: High SHAP value ≠ causal relationship (correlation still) - Assumes feature independence: Can give misleading results with highly correlated features

Best practices: - Use TreeSHAP for tree-based models (XGBoost, Random Forest) , 1000x faster - For neural networks, use DeepSHAP or KernelSHAP with background dataset sampling - Always validate explanations with domain experts (do they make clinical sense?)

---

Method 2: LIME (Local Interpretable Model-agnostic Explanations)

What it is: Creates a simple, interpretable model (like linear regression) that approximates the complex model’s behavior locally around a specific prediction.

How it works: 1. Perturb the input (create similar but slightly different patients) 2. Get model predictions for perturbed inputs 3. Fit a simple linear model to these local predictions 4. Linear coefficients = feature importance for this prediction

When to use: - Need quick local explanations - SHAP is too computationally expensive - Want human-readable rules (“If lactate > 2 AND fever, then high risk”)

Foundational paper: Ribeiro et al., 2016, KDD

---

LIME Example: Readmission Risk

import lime
import lime.lime_tabular
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Simulate patient data for hospital readmission
np.random.seed(42)
n_patients = 1000

data = pd.DataFrame({
 'age': np.random.normal(68, 12, n_patients),
 'num_prior_admissions': np.random.poisson(2, n_patients),
 'length_of_stay': np.random.gamma(2, 2, n_patients),
 'num_medications': np.random.poisson(5, n_patients),
 'comorbidity_count': np.random.poisson(3, n_patients),
 'emergency_admission': np.random.binomial(1, 0.3, n_patients),
})

# Create readmission outcome
readmit_risk = (
 0.02 * data['age'] +
 0.15 * data['num_prior_admissions'] +
 0.05 * data['comorbidity_count'] +
 0.1 * data['emergency_admission'] +
 np.random.normal(0, 0.5, n_patients)
)
data['readmitted_30d'] = (readmit_risk > 2).astype(int)

# Train model
X = data.drop('readmitted_30d', axis=1)
y = data['readmitted_30d']
X_train, X_test, y_train, y_test = train_test_split(
 X, y, test_size=0.2, random_state=42, stratify=y
)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# ===== LIME EXPLANATION =====

# 1. Create LIME explainer
explainer = lime.lime_tabular.LimeTabularExplainer(
 training_data=X_train.values,
 feature_names=X_train.columns.tolist(),
 class_names=['No Readmission', 'Readmission'],
 mode='classification',
 random_state=42
)

# 2. Explain a specific patient
patient_idx = 5
patient_data = X_test.iloc[patient_idx].values
predicted_proba = model.predict_proba([patient_data])[0]

print(f"=== Patient {patient_idx} ===")
print(f"Predicted readmission probability: {predicted_proba[1]:.2%}")
print(f"Actual outcome: {'Readmitted' if y_test.iloc[patient_idx] == 1 else 'Not readmitted'}")

# Generate explanation
explanation = explainer.explain_instance(
 data_row=patient_data,
 predict_fn=model.predict_proba,
 num_features=6
)

# 3. Display explanation
print("\n=== LIME Explanation ===")
print("Feature contributions to 'Readmission' class:")
for feature, weight in explanation.as_list():
 print(f" {feature}: {weight:+.3f}")

# 4. Visualize
explanation.show_in_notebook(show_table=True)

# Save as HTML
explanation.save_to_file('lime_explanation_patient5.html')

# 5. Extract feature importance for this patient
feature_importance = dict(explanation.as_list())
print("\n=== Top Risk Factors for This Patient ===")
sorted_features = sorted(feature_importance.items(), key=lambda x: abs(x[1]), reverse=True)
for feature, weight in sorted_features[:3]:
 direction = "↑ Increases" if weight > 0 else "↓ Decreases"
 print(f"{direction} risk: {feature} (impact: {weight:+.3f})")

Example output:

=== Patient 5 ===
Predicted readmission probability: 64%

Feature contributions:
 num_prior_admissions > 3.00: +0.22 ← Major risk factor
 comorbidity_count > 4.00: +0.15
 age > 65.00: +0.08
 emergency_admission = 1: +0.12
 length_of_stay ≤ 3.00: -0.05  ← Protective (longer stays = more stabilization)
 num_medications ≤ 6.00: -0.02

Interpretation: This patient's high readmission risk is driven primarily
by multiple prior admissions (4 in past year) and high comorbidity burden.

---

LIME Advantages and Limitations

Advantages: - Fast: Quicker than SHAP for local explanations - Intuitive: Simple “if-then” rules easy for clinicians to understand - Model-agnostic: Works with any black box model

Limitations: - Instability: Explanations can vary significantly with small input changes - Local only: Doesn’t provide global model understanding - Arbitrary perturbations: Sampling strategy affects explanation quality - No theoretical guarantees: Unlike SHAP, not mathematically principled

When to choose LIME over SHAP: - Real-time explanations needed (speed critical) - Prefer rule-based explanations (“If X > 5 AND Y < 10…”) - SHAP computationally infeasible for your model

---

Method 3: Attention Mechanisms (For Deep Learning)

What it is: Neural network architectures that learn to focus on important input features, making attention weights interpretable.

Where it’s used: - Transformers: BERT, GPT for clinical notes analysis - Vision models: Which parts of chest X-ray drove diagnosis? - Time-series: Which ICU monitoring data points triggered alert?

Example application: Radiology AI highlights suspicious regions in medical images using attention heatmaps.

---

Attention Visualization Example

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# Simple attention-based model for ICU time-series data
class AttentionICU(nn.Module):
 def __init__(self, input_dim, hidden_dim):
  super().__init__()
  self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)
  # Attention mechanism
  self.attention = nn.Linear(hidden_dim, 1)
  self.classifier = nn.Linear(hidden_dim, 1)

 def forward(self, x):
  # x shape: (batch, time_steps, features)
  lstm_out, _ = self.lstm(x) # (batch, time_steps, hidden_dim)

  # Calculate attention scores
  attention_scores = self.attention(lstm_out) # (batch, time_steps, 1)
  attention_weights = torch.softmax(attention_scores, dim=1)

  # Apply attention (weighted sum of LSTM outputs)
  context = torch.sum(attention_weights * lstm_out, dim=1) # (batch, hidden_dim)

  # Final prediction
  output = self.classifier(context)

  return torch.sigmoid(output), attention_weights

# Simulate ICU time-series data
# Features: HR, BP, SpO2, RR over 24 hours (hourly measurements)
torch.manual_seed(42)
n_patients = 100
time_steps = 24
n_features = 4

X = torch.randn(n_patients, time_steps, n_features)
y = torch.randint(0, 2, (n_patients, 1)).float() # Binary outcome

# Train model
model = AttentionICU(input_dim=n_features, hidden_dim=32)
criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# Training loop (simplified)
for epoch in range(50):
 optimizer.zero_grad()
 predictions, attention_weights = model(X)
 loss = criterion(predictions, y)
 loss.backward()
 optimizer.step()

# ===== INTERPRET ATTENTION WEIGHTS =====

# Explain a specific patient
patient_idx = 0
patient_data = X[patient_idx:patient_idx+1]
prediction, attention = model(patient_data)

print(f"Predicted risk: {prediction.item():.2%}")

# Visualize attention over time
attention_np = attention.detach().numpy()[0, :, 0] # Shape: (time_steps,)

plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(range(24), attention_np, marker='o')
plt.xlabel('Hour')
plt.ylabel('Attention Weight')
plt.title('Which Time Points Were Most Important?')
plt.axhline(1/24, color='r', linestyle='--', label='Uniform attention')
plt.legend()

# Identify critical time periods
top_hours = np.argsort(attention_np)[-3:][::-1]
print(f"\nMost important time periods: Hours {top_hours}")
print("Interpretation: Model focused on these specific hours when making prediction")

# Overlay attention on vital signs
plt.subplot(1, 2, 2)
vitals = patient_data.detach().numpy()[0, :, 0] # Heart rate
plt.plot(range(24), vitals, label='Heart Rate', alpha=0.7)
plt.scatter(range(24), vitals, s=attention_np*1000, c='red', alpha=0.5,
   label='Attention (size = importance)')
plt.xlabel('Hour')
plt.ylabel('Heart Rate')
plt.title('Attention-Weighted Vital Signs')
plt.legend()
plt.tight_layout()
plt.savefig('attention_interpretation.png', dpi=150)
plt.show()

print("\nClinical interpretation:")
print(f"The model identified hours {top_hours[0]}, {top_hours[1]}, {top_hours[2]} as critical.")
print("Clinician should review events during these time windows.")

Key insight: Attention mechanisms provide inherent interpretability, the model learns what’s important during training, rather than requiring post-hoc explanation.

Limitations: - Attention ≠ causation - High attention doesn’t guarantee that feature is truly important (attention is correlation) - Requires model architecture modification (can’t apply to existing black boxes)

---

Method 4: Counterfactual Explanations

What it is: “What would need to change for the model to make a different prediction?”

Example: - Prediction: Patient has 75% readmission risk - Counterfactual: “If patient had ≤2 prior admissions (currently 4) OR comorbidity count ≤3 (currently 5), risk would drop to <30%”

Why it’s valuable: - Actionable: Tells clinicians what interventions might help - Patient-friendly: Easy to communicate (“If you lose 10 lbs, your risk decreases…”) - Fair: Reveals whether model relies on unchangeable features (race, gender)

---

Counterfactual Example with DiCE

# Install: pip install dice-ml
import dice_ml
from dice_ml import Dice
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Load data (reuse readmission example from LIME section)
# ... (same data generation code) ...

# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# ===== COUNTERFACTUAL GENERATION =====

# 1. Prepare DiCE
dice_data = dice_ml.Data(
 dataframe=pd.concat([X_train, y_train], axis=1),
 continuous_features=['age', 'num_prior_admissions', 'length_of_stay',
      'num_medications', 'comorbidity_count'],
 outcome_name='readmitted_30d'
)

dice_model = dice_ml.Model(model=model, backend='sklearn')

explainer = Dice(dice_data, dice_model, method='random')

# 2. Generate counterfactuals for high-risk patient
patient_idx = 5
patient_df = X_test.iloc[[patient_idx]]

# Find alternative scenarios where patient would NOT be readmitted
counterfactuals = explainer.generate_counterfactuals(
 query_instances=patient_df,
 total_CFs=3, # Generate 3 alternative scenarios
 desired_class='opposite' # Want opposite prediction
)

# 3. Display results
print("=== Original Patient ===")
print(patient_df.T)
print(f"\nPredicted outcome: Readmission (Risk: {model.predict_proba(patient_df)[0][1]:.2%})")

print("\n=== Counterfactual Scenarios (How to Avoid Readmission) ===")
cf_df = counterfactuals.cf_examples_list[0].final_cfs_df
print(cf_df.T)

# 4. Identify key changes
print("\n=== Key Changes Needed ===")
for col in X_test.columns:
 original = patient_df[col].values[0]
 for i, cf in cf_df.iterrows():
  if abs(cf[col] - original) > 0.01:
   change = cf[col] - original
   print(f" {col}: {original:.1f} → {cf[col]:.1f} (change: {change:+.1f})")

# 5. Clinical translation
print("\n=== Actionable Recommendations ===")
print("To reduce readmission risk below 30%, consider:")
print(" • Reduce medication complexity (consolidate from 8 to ≤6 drugs)")
print(" • Intensive post-discharge follow-up (reduce prior admit pattern)")
print(" • Comorbidity management (focus on top 2-3 conditions)")

Output interpretation:

Original Patient: Readmission Risk = 72%
- Age: 71
- Prior admissions: 4
- Comorbidities: 5
- Medications: 8

Counterfactual Scenario 1: Risk = 18%
- Age: 71 (unchanged)
- Prior admissions: 1 (reduced from 4) ← Major change
- Comorbidities: 5 (unchanged)
- Medications: 6 (reduced from 8)

Interpretation: Model suggests that reducing medication complexity and
preventing repeat admissions are the highest-impact interventions.

Reference: Wachter et al., 2017

---

Method 5: Feature Importance (For Tree-Based Models)

What it is: For models like Random Forest and XGBoost, built-in feature importance scores.

How it works: - Gini importance: How much each feature reduces impurity when splitting - Permutation importance: Performance drop when feature is randomly shuffled

Advantage: Fast, easy to compute Limitation: Can be biased toward high-cardinality features

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# ===== FEATURE IMPORTANCE =====

# Method 1: Built-in feature importance (Gini)
gini_importance = pd.DataFrame({
 'feature': X_train.columns,
 'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

print("=== Gini Importance ===")
print(gini_importance)

# Method 2: Permutation importance (more reliable)
perm_importance = permutation_importance(
 model, X_test, y_test, n_repeats=10, random_state=42
)

perm_df = pd.DataFrame({
 'feature': X_train.columns,
 'importance': perm_importance.importances_mean,
 'std': perm_importance.importances_std
}).sort_values('importance', ascending=False)

print("\n=== Permutation Importance ===")
print(perm_df)

# Visualize
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

axes[0].barh(gini_importance['feature'], gini_importance['importance'])
axes[0].set_xlabel('Gini Importance')
axes[0].set_title('Feature Importance (Gini)')

axes[1].barh(perm_df['feature'], perm_df['importance'])
axes[1].errorbar(perm_df['importance'], perm_df['feature'],
     xerr=perm_df['std'], fmt='none', color='black', alpha=0.5)
axes[1].set_xlabel('Permutation Importance')
axes[1].set_title('Feature Importance (Permutation)')

plt.tight_layout()
plt.savefig('feature_importance_comparison.png', dpi=150)
plt.show()

---

Choosing the Right Explainability Method

Method	Global or Local?	Model-Agnostic?	Speed	Best For
SHAP	Both	Yes	Medium-Slow	Most robust, theoretically grounded explanations
LIME	Local only	Yes	Fast	Quick local explanations, rule-based output
Attention	Local only	No (DL only)	Fast	Deep learning models (transformers, CNNs)
Counterfactual	Local only	Yes	Medium	Actionable recommendations, fairness audits
Feature Importance	Global only	No (tree models)	Very Fast	Tree-based models, quick initial analysis

Decision flowchart:

1. Need global understanding? → SHAP (global) or Feature Importance (trees only)
2. Need local explanation for specific patient? → SHAP (most robust) or LIME (faster)
3. Need actionable recommendations? → Counterfactuals
4. Using deep learning? → Attention mechanisms or SHAP
5. Real-time constraint? → LIME or Feature Importance
6. Regulatory submission? → SHAP (theoretically grounded)

---

Evaluating Explainability: Does Your XAI Actually Work?

Critical question: How do you know if your explanations are good?

Explainability Evaluation Criteria

1. Fidelity: Does the explanation accurately reflect the model’s behavior?

Test: - Remove high-importance features → prediction should change significantly - Flip low-importance features → prediction should stay similar

# Fidelity test
original_pred = model.predict_proba(patient_data)[0][1]

# Remove most important feature (set to mean)
modified_data = patient_data.copy()
modified_data['lactate'] = X_train['lactate'].mean()
modified_pred = model.predict_proba(modified_data)[0][1]

print(f"Original prediction: {original_pred:.2%}")
print(f"After removing 'lactate': {modified_pred:.2%}")
print(f"Change: {abs(original_pred - modified_pred):.2%}")

if abs(original_pred - modified_pred) > 0.1: # >10% change
 print("[OK] High fidelity: Explanation correctly identified important feature")
else:
 print("[FAIL] Low fidelity: Feature removal didn't change prediction as expected")

2. Consistency: Do similar patients get similar explanations?

Test: Generate explanations for similar patients; feature importance rankings should be similar

3. Stability: Do explanations change drastically with small input perturbations?

Problem with LIME: Small changes to patient data can yield very different explanations

4. Clinical validity: Do domain experts agree the explanations make sense?

Gold standard: Clinician review - Do identified features align with medical knowledge? - Are there unexpected/spurious correlations?

---

Regulatory Perspectives on Explainability

FDA AI/ML SaMD Action Plan (2021)

The FDA’s AI/ML SaMD Action Plan emphasizes transparency to users and real-world performance monitoring as priorities for AI/ML-based SaMD (FDA, 2021).

Common transparency elements include: - Explanation of key features driving predictions - Model limitations and failure modes - Performance across demographic subgroups

---

EU AI Act (2024)

Transparency obligations for high-risk AI (includes medical AI):

Article 13 - Transparency: - Users must be informed that they are interacting with AI - Information on the logic involved in decision-making - Information on significance and consequences of predictions

Practical implication: “Black box” systems without explanations will face regulatory barriers in EU.

Reference: EU AI Act, 2024

---

Implementing Explainability in Production Systems

Best Practices for Deployed AI

1. Multi-level explanations for different users:

User	Explanation Level	Method
Patient	Why this prediction affects me?	Simplified counterfactual (“If X, then Y”)
Clinician	What factors drive this prediction?	SHAP/LIME with top 3-5 features
Data Scientist	How does the model work globally?	SHAP global importance, partial dependence
Regulator	Is the model fair and robust?	Subgroup analysis, fairness metrics

2. Explanation caching: Pre-compute SHAP values during batch prediction to avoid real-time latency

3. Explanation documentation: Log explanations alongside predictions for audit trails

4. Explanation monitoring: Track whether explanations remain consistent over time (if not, indicates model drift)

---

Example: Explainable Sepsis Alert System

## Explainability Architecture for Sepsis Early Warning System

**User-facing interface:**

┌─────────────────────────────────────────────┐
│ SEPSIS ALERT: High Risk (82%)    │
├─────────────────────────────────────────────┤
│ Primary Risk Factors:      │
│ [CRITICAL] Lactate: 3.8 mmol/L (Critical: >2.0) │
│ [CRITICAL] Temp: 39.2°C (Elevated: >38.3)   │
│ [ELEVATED] WBC: 13,500 (Elevated: >12,000)  │
│            │
│ Protective Factors:      │
│ [NORMAL] Blood Pressure: Normal (118/76)  │
│            │
│ [View Detailed Explanation]    │
│ [Similar Cases] [Dismiss Alert]   │
└─────────────────────────────────────────────┘

**Backend logging (for audit):**
{
 "patient_id": "47291",
 "timestamp": "2025-10-30T14:23:11Z",
 "prediction": 0.82,
 "model_version": "sepsis_v3.2.1",
 "shap_values": {
 "lactate": 0.35,
 "temperature": 0.22,
 "wbc_count": 0.18,
 "systolic_bp": -0.08
 },
 "explanation_method": "SHAP_TreeExplainer",
 "explanation_fidelity_score": 0.94
}

---

Common Pitfalls and How to Avoid Them

Pitfall 1: Confusing Correlation with Causation

Problem: SHAP/LIME identify correlations, not causal relationships.

Example: - Model assigns high importance to “hospital length of stay” for mortality prediction - Interpretation error: “Longer stays cause death” - Reality: Sicker patients stay longer; length of stay is a proxy for severity

Solution: Always validate explanations with clinical domain knowledge

---

Pitfall 2: Over-relying on Feature Importance

Problem: Global feature importance hides subgroup differences.

Example: - “Age” is most important feature globally (average across all patients) - But for young patients (<40), “comorbidities” might be more important

Solution: Examine SHAP dependence plots and subgroup-specific explanations

---

Pitfall 3: Ignoring Explanation Instability

Problem: LIME explanations can vary substantially between similar patients.

Test:

# Generate 10 explanations for same patient (with different LIME seeds)
explanations = []
for seed in range(10):
 exp = explainer.explain_instance(patient, model.predict_proba, random_state=seed)
 explanations.append(exp.as_list())

# Check consistency
# If feature rankings vary significantly → unstable explanations

Solution: Use SHAP for high-stakes decisions (more stable)

---

Pitfall 4: Explaining the Wrong Model

Problem: Explain a simplified “surrogate” model instead of the actual production model.

Example: - Production: Complex ensemble of 50 models - Explanation: Generated from single decision tree approximation - Risk: Explanations don’t reflect actual system behavior

Solution: Always explain the actual deployed model (even if slower)

---

Key Takeaways: Explainability

1. Trust requires transparency: Clinicians won’t act on predictions they don’t understand

2. Multiple methods, multiple purposes: SHAP for robustness, LIME for speed, counterfactuals for action

3. Evaluate your explanations: Fidelity, consistency, clinical validity

4. Regulatory trend: Explainability moving from “nice-to-have” to mandatory (FDA, EU)

5. Layer explanations by user: Patients need simple “why me?”; regulators need comprehensive validation

6. Correlation ≠ causation: Explanations show what model uses, not necessarily what’s clinically causal

7. Explainability is not a fix for bad models: If your model is biased or poorly validated, explanations just make the problems more visible (which is actually good for debugging)

Essential resources:

• Christoph Molnar, Interpretable Machine Learning (2025): Free online book, comprehensive guide
• SHAP documentation: https://shap.readthedocs.io/
• LIME GitHub: https://github.com/marcotcr/lime
• DiCE (Counterfactuals): https://interpret.ml/DiCE/
• Google’s Explainable AI whitepaper: Exploratory guide to XAI (2019)

Mechanistic Interpretability for Sequential Decision-Making

SHAP and LIME explain individual predictions, but public health AI increasingly involves sequential decision-making where current actions influence future states. Reinforcement learning (RL) systems for population health management require interpretability methods that expose reasoning pathways, not just feature importance.

Case study: Medicaid care coordination. A SARSA reinforcement learning system for Medicaid care management across two U.S. states (Virginia and Washington; 3,175 beneficiaries, 2023–2024) used a mixed-methods approach combining quantitative RL optimization with qualitative clinical validation. In counterfactual analysis, the system was estimated to reduce acute care events by 12 percentage points (NNT 8.3; 20.7% relative reduction) compared to standard practice, while also reducing race/ethnicity equalized odds disparity from 8.9% to 5.6% and gender disparity from 5.3% to 3.8% (Basu et al., 2025).

Implications for public health AI evaluation:

• Sequential decision-making systems require interpretability methods beyond single-prediction explainers like SHAP and LIME
• Mixed-methods validation (combining quantitative metrics with clinical expert review) provides stronger evidence than either approach alone
• Fairness constraints can be integrated into RL optimization without large accuracy trade-offs, suggesting baseline disparities often stem from suboptimal calibration rather than fundamental accuracy-fairness tension
• Tiered oversight (automated decisions for low-risk cases, human review for high-risk cases) is an emerging approach for balancing efficiency with safety

---