Explainability for Public Health AI

Interpretability methods, failure modes, and operational use of explanations in public health AI systems. The material is maintained separately so each operational question has a stable, focused reference.

Learning Objectives
  • Identify the evidence and controls relevant to this decision area
  • Distinguish technical performance from operational and population impact
  • Apply the included framework without extending claims beyond the cited evidence

Use explicit targets, populations, thresholds, and decision consequences. Require external evidence and local monitoring where deployment can affect people or programs. Preserve uncertainty and document limits.

Introduction

This focused reference is part of the broader Explainability overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.

Explainability and Interpretability (XAI)

Why Explainability Matters in Public Health AI

The trust problem: Systematic reviews consistently find that clinicians are reluctant to trust or act on predictions from “black box” AI systems they cannot interpret (Antoniadi et al., 2021; Markus et al., 2021).

Why interpretability is critical:

  1. Clinical decision-making: Clinicians need to know why before they can decide whether to act
  2. Debugging and validation: Explanations reveal spurious correlations and dataset biases
  3. Regulatory requirements: FDA emphasizes transparency to users, while EU Article 13 requires instructions sufficient for deployers to interpret and use high-risk system outputs appropriately
  4. Patient autonomy: Patients have a right to understand decisions affecting their health
  5. Legal liability: “The algorithm said so” is not a defense in malpractice cases
The Accuracy-Interpretability Trade-off (A False Dichotomy?)

Traditional belief: Deep learning = high accuracy but uninterpretable; simpler models = lower accuracy but interpretable.

Current reality: Post-hoc methods can describe associations with model outputs, but they do not make a model intrinsically interpretable. Their fidelity, stability, and usefulness must be evaluated for the intended user and decision.

Guideline: Start with the simplest model that meets performance requirements. If you need complex models, invest in robust explainability infrastructure.

Levels of Interpretability

Not all interpretability is equal. Different stakeholders need different levels of explanation.

1. Global Interpretability

Definition: Understanding the model’s overall behavior and decision logic.

Questions answered: - What features are most important overall? - How does the model generally make decisions? - Are there unexpected feature relationships?

Methods: - Feature importance rankings - Partial dependence plots - Global SHAP values

Audience: Data scientists, validators, regulators

2. Local Interpretability

Definition: Understanding why the model made a specific prediction for a specific patient.

Questions answered: - Why did the model predict this patient is high-risk? - Which patient characteristics drove this prediction? - What would need to change to alter the prediction?

Methods: - LIME (Local Interpretable Model-agnostic Explanations) - SHAP values for individual predictions - Counterfactual explanations

Audience: Clinicians, patients

3. Model-Based Interpretability

Definition: Models that are inherently interpretable by design.

Examples: - Linear models: Each coefficient shows feature contribution - Decision trees: Follow the path to understand the decision - Rule-based systems: Explicit IF-THEN logic

When to use: When stakeholder trust is paramount and model performance requirements are modest.

Interpretability Methods: Practical Guide

Method 1: SHAP (SHapley Additive exPlanations)

What it is: A unified framework for interpreting model predictions based on game theory (Shapley values).

Why it’s powerful: - Model-agnostic: Works with any ML model (XGBoost, neural networks, etc.) - Theoretically grounded: Satisfies desirable properties (local accuracy, consistency) - Both global and local: Feature importance + individual predictions

Foundational paper: Lundberg & Lee, 2017, NeurIPS

SHAP Example: Sepsis Risk Prediction

Key outputs:

  1. Global importance: Which features matter most across all patients?
  2. Waterfall plot: For Patient X, lactate (+0.3) and temperature (+0.2) increased risk; normal BP (-0.1) decreased it
  3. Dependence plots: Non-linear relationships (e.g., lactate > 2 mmol/L sharply increases risk)

Clinical translation:

Patient 47: Sepsis Risk = 78%

Main drivers:
+ Lactate 3.2 mmol/L (+0.35 risk contribution) <- **Primary concern**
+ Temperature 39.1°C (+0.22)
+ WBC 15,000/μL (+0.18)
- Normal BP 118/72 (-0.08) <- **Protective factor**

Interpretation: Elevated lactate is the strongest predictor.
Consider serial lactate monitoring and early fluid resuscitation.
SHAP Advantages and Limitations

Advantages: - Mathematically principled (satisfies local accuracy, missingness, consistency) - Works with any model architecture - Both global and local explanations - Handles feature interactions

Limitations: - Computational cost: Can be slow for large models/datasets (use TreeSHAP for tree models, faster) - Not causal: High SHAP value ≠ causal relationship (correlation still) - Assumes feature independence: Can give misleading results with highly correlated features

Best practices: - Use TreeSHAP for tree-based models (XGBoost, Random Forest) , 1000x faster - For neural networks, use DeepSHAP or KernelSHAP with background dataset sampling - Always validate explanations with domain experts (do they make clinical sense?)

Method 2: LIME (Local Interpretable Model-agnostic Explanations)

What it is: Creates a simple, interpretable model (like linear regression) that approximates the complex model’s behavior locally around a specific prediction.

How it works: 1. Perturb the input (create similar but slightly different patients) 2. Get model predictions for perturbed inputs 3. Fit a simple linear model to these local predictions 4. Linear coefficients = feature importance for this prediction

When to use: - Need quick local explanations - SHAP is too computationally expensive - Want human-readable rules (“If lactate > 2 AND fever, then high risk”)

Foundational paper: Ribeiro et al., 2016, KDD

LIME Example: Readmission Risk

Example output:

=== Patient 5 ===
Predicted readmission probability: 64%

Feature contributions:
 num_prior_admissions > 3.00: +0.22 ← Major risk factor
 comorbidity_count > 4.00: +0.15
 age > 65.00: +0.08
 emergency_admission = 1: +0.12
 length_of_stay ≤ 3.00: -0.05  ← Protective (longer stays = more stabilization)
 num_medications ≤ 6.00: -0.02

Interpretation: This patient's high readmission risk is driven primarily
by multiple prior admissions (4 in past year) and high comorbidity burden.
LIME Advantages and Limitations

Advantages: - Fast: Quicker than SHAP for local explanations - Intuitive: Simple “if-then” rules easy for clinicians to understand - Model-agnostic: Works with any black box model

Limitations: - Instability: Explanations can vary significantly with small input changes - Local only: Does not provide global model understanding - Arbitrary perturbations: Sampling strategy affects explanation quality - No theoretical guarantees: Unlike SHAP, not mathematically principled

When to choose LIME over SHAP: - Real-time explanations needed (speed critical) - Prefer rule-based explanations (“If X > 5 AND Y < 10…”) - SHAP computationally infeasible for your model

Method 3: Attention Mechanisms (For Deep Learning)

What it is: Neural network architectures that learn to focus on important input features, making attention weights interpretable.

Where it’s used: - Transformers: BERT, GPT for clinical notes analysis - Vision models: Which parts of chest X-ray drove diagnosis? - Time-series: Which ICU monitoring data points triggered alert?

Example application: Radiology AI highlights suspicious regions in medical images using attention heatmaps.

Attention Visualization Example

Key insight: Attention mechanisms provide inherent interpretability, the model learns what’s important during training, rather than requiring post-hoc explanation.

Limitations: - Attention ≠ causation - High attention does not guarantee that feature is truly important (attention is correlation) - Requires model architecture modification (cannot apply to existing black boxes)

Method 4: Counterfactual Explanations

What it is: “What would need to change for the model to make a different prediction?”

Example: - Prediction: Patient has 75% readmission risk - Counterfactual: “If patient had ≤2 prior admissions (currently 4) OR comorbidity count ≤3 (currently 5), risk would drop to <30%”

Why it’s valuable: - Actionable: Tells clinicians what interventions might help - Patient-friendly: Easy to communicate (“If you lose 10 lbs, your risk decreases…”) - Fair: Reveals whether model relies on unchangeable features (race, gender)

Counterfactual Example with DiCE

Output interpretation:

Original Patient: Readmission Risk = 72%
- Age: 71
- Prior admissions: 4
- Comorbidities: 5
- Medications: 8

Counterfactual Scenario 1: Risk = 18%
- Age: 71 (unchanged)
- Prior admissions: 1 (reduced from 4) ← Major change
- Comorbidities: 5 (unchanged)
- Medications: 6 (reduced from 8)

Interpretation: Model suggests that reducing medication complexity and
preventing repeat admissions are the highest-impact interventions.

Reference: Wachter et al., 2017

Method 5: Feature Importance (For Tree-Based Models)

What it is: For models like Random Forest and XGBoost, built-in feature importance scores.

How it works: - Gini importance: How much each feature reduces impurity when splitting - Permutation importance: Performance drop when feature is randomly shuffled

Advantage: Fast, easy to compute Limitation: Can be biased toward high-cardinality features

Choosing the Right Explainability Method

Method Global or Local? Model-Agnostic? Speed Best For
SHAP Both Yes Medium-Slow Most robust, theoretically grounded explanations
LIME Local only Yes Fast Quick local explanations, rule-based output
Attention Local only No (DL only) Fast Deep learning models (transformers, CNNs)
Counterfactual Local only Yes Medium Actionable recommendations, fairness audits
Feature Importance Global only No (tree models) Very Fast Tree-based models, quick initial analysis

Decision flowchart:

  1. Need global understanding? → SHAP (global) or Feature Importance (trees only)
  2. Need local explanation for specific patient? → SHAP (most robust) or LIME (faster)
  3. Need actionable recommendations? → Counterfactuals
  4. Using deep learning? → Attention mechanisms or SHAP
  5. Real-time constraint? → LIME or Feature Importance
  6. Regulatory submission? → SHAP (theoretically grounded)

Evaluating Explainability: Does Your XAI Actually Work?

Critical question: How do you know if your explanations are good?

Explainability Evaluation Criteria

1. Fidelity: Does the explanation accurately reflect the model’s behavior?

Test: - Remove high-importance features → prediction should change significantly - Flip low-importance features → prediction should stay similar

2. Consistency: Do similar patients get similar explanations?

Test: Generate explanations for similar patients; feature importance rankings should be similar

3. Stability: Do explanations change drastically with small input perturbations?

Problem with LIME: Small changes to patient data can yield very different explanations

4. Clinical validity: Do domain experts agree the explanations make sense?

Gold standard: Clinician review - Do identified features align with medical knowledge? - Are there unexpected/spurious correlations?

Regulatory Perspectives on Explainability

FDA AI/ML SaMD Action Plan (2021)

The FDA’s AI/ML SaMD Action Plan emphasizes transparency to users and real-world performance monitoring as priorities for AI/ML-based SaMD (FDA, 2021).

Common transparency elements include: - Explanation of key features driving predictions - Model limitations and failure modes - Performance across demographic subgroups

EU AI Act (2024)

Transparency obligations for high-risk AI (includes medical AI):

Article 13 - Transparency: - High-risk systems must be sufficiently transparent for deployers to interpret and use outputs appropriately - Instructions must describe capabilities, limitations, expected accuracy, foreseeable misuse, and the information needed for human oversight

Practical implication: Article 13 does not prescribe SHAP, LIME, attention weights, or another universal explanation method. Documentation and user-facing information must support the specific system and oversight task.

Reference: EU AI Act, 2024

Implementing Explainability in Production Systems

Best Practices for Deployed AI

1. Multi-level explanations for different users:

User Explanation Level Method
Patient Why this prediction affects me? Simplified counterfactual (“If X, then Y”)
Clinician What factors drive this prediction? SHAP/LIME with top 3-5 features
Data Scientist How does the model work globally? SHAP global importance, partial dependence
Regulator Is the model fair and robust? Subgroup analysis, fairness metrics

2. Explanation caching: Pre-compute SHAP values during batch prediction to avoid real-time latency

3. Explanation documentation: Log explanations alongside predictions for audit trails

4. Explanation monitoring: Track whether explanations remain consistent over time (if not, indicates model drift)

Example: Explainable Sepsis Alert System

## Explainability Architecture for Sepsis Early Warning System

**User-facing interface:**

┌─────────────────────────────────────────────┐
│ SEPSIS ALERT: High Risk (82%)    │
├─────────────────────────────────────────────┤
│ Primary Risk Factors:      │
│ [CRITICAL] Lactate: 3.8 mmol/L (Critical: >2.0) │
│ [CRITICAL] Temp: 39.2°C (Elevated: >38.3)   │
│ [ELEVATED] WBC: 13,500 (Elevated: >12,000)  │
│            │
│ Protective Factors:      │
│ [NORMAL] Blood Pressure: Normal (118/76)  │
│            │
│ [View Detailed Explanation]    │
│ [Similar Cases] [Dismiss Alert]   │
└─────────────────────────────────────────────┘

**Backend logging (for audit):**
{
 "patient_id": "47291",
 "timestamp": "2025-10-30T14:23:11Z",
 "prediction": 0.82,
 "model_version": "sepsis_v3.2.1",
 "shap_values": {
 "lactate": 0.35,
 "temperature": 0.22,
 "wbc_count": 0.18,
 "systolic_bp": -0.08
 },
 "explanation_method": "SHAP_TreeExplainer",
 "explanation_fidelity_score": 0.94
}

Common Pitfalls and How to Avoid Them

Pitfall 1: Confusing Correlation with Causation

Problem: SHAP/LIME identify correlations, not causal relationships.

Example: - Model assigns high importance to “hospital length of stay” for mortality prediction - Interpretation error: “Longer stays cause death” - Reality: Sicker patients stay longer; length of stay is a proxy for severity

Solution: Always validate explanations with clinical domain knowledge

Pitfall 2: Over-relying on Feature Importance

Problem: Global feature importance hides subgroup differences.

Example: - “Age” is most important feature globally (average across all patients) - But for young patients (<40), “comorbidities” might be more important

Solution: Examine SHAP dependence plots and subgroup-specific explanations

Pitfall 3: Ignoring Explanation Instability

Problem: LIME explanations can vary substantially between similar patients.

Test:

# Generate 10 explanations for same patient (with different LIME seeds)
explanations = []
for seed in range(10):
 exp = explainer.explain_instance(patient, model.predict_proba, random_state=seed)
 explanations.append(exp.as_list())

# Check consistency
# If feature rankings vary significantly → unstable explanations

Solution: Use SHAP for high-stakes decisions (more stable)

Pitfall 4: Explaining the Wrong Model

Problem: Explain a simplified “surrogate” model instead of the actual production model.

Example: - Production: Complex ensemble of 50 models - Explanation: Generated from single decision tree approximation - Risk: Explanations do not reflect actual system behavior

Solution: Always explain the actual deployed model (even if slower)

Key Takeaways: Explainability

  1. Trust requires transparency: Clinicians will not act on predictions they do not understand

  2. Multiple methods, multiple purposes: SHAP for robustness, LIME for speed, counterfactuals for action

  3. Evaluate your explanations: Fidelity, consistency, clinical validity

  4. Regulatory trend: Transparency and interpretable use are increasingly important, but no universal rule requires a post-hoc feature attribution for every prediction

  5. Layer explanations by user: Patients need simple “why me?”; regulators need comprehensive validation

  6. Correlation ≠ causation: Explanations show what model uses, not necessarily what’s clinically causal

  7. Explainability is not a fix for bad models: If your model is biased or poorly validated, explanations just make the problems more visible (which is actually good for debugging)

Essential resources:

Mechanistic Interpretability for Sequential Decision-Making

SHAP and LIME explain individual predictions, but public health AI increasingly involves sequential decision-making where current actions influence future states. Reinforcement learning (RL) systems for population health management require interpretability methods that expose reasoning pathways, not just feature importance.

Case study: Medicaid care coordination. A SARSA reinforcement learning system for Medicaid care management across two U.S. states (Virginia and Washington; 3,175 beneficiaries, 2023–2024) used a mixed-methods approach combining quantitative RL optimization with qualitative clinical validation. In counterfactual analysis, the system was estimated to reduce acute care events by 12 percentage points (NNT 8.3; 20.7% relative reduction) compared to standard practice, while also reducing race/ethnicity equalized odds disparity from 8.9% to 5.6% and gender disparity from 5.3% to 3.8% (Basu et al., 2025).

Implications for public health AI evaluation:

  • Sequential decision-making systems require interpretability methods beyond single-prediction explainers like SHAP and LIME
  • Mixed-methods validation (combining quantitative metrics with clinical expert review) provides stronger evidence than either approach alone
  • Fairness constraints can be integrated into RL optimization without large accuracy trade-offs, suggesting baseline disparities often stem from suboptimal calibration rather than fundamental accuracy-fairness tension
  • Tiered oversight (automated decisions for low-risk cases, human review for high-risk cases) is an emerging approach for balancing efficiency with safety