The Public Health AI Handbook: Evaluating AI Tools for Public Health Practice

Tegomoh, Bryan

doi:10.5281/zenodo.18263442

AI Safety in Healthcare: Protecting Patients and Populations

AI safety differs from security, privacy, and quality assurance. A technically valid model can still create harm through a poorly chosen threshold, missed cases, excessive alerts, weak workflow fit, automation bias, or silent drift. Healthcare AI therefore requires systematic hazard identification, human-system testing, layered safeguards, fallback, monitoring, and incident learning across the lifecycle.

Learning Objectives

This chapter addresses AI safety in healthcare and public health. You will learn to:

Distinguish AI safety from security, privacy, and quality assurance
Apply safety-critical systems frameworks (IEC 62304, ISO 14971) to AI
Identify failure modes using FMEA and hazard analysis
Implement safety validation beyond standard ML evaluation
Interpret regulatory safety requirements (FDA, EU MDR, MHRA)
Design safeguards: human-in-the-loop, fallback systems, monitoring
Develop incident response protocols for AI-caused harm
Understand why technical excellence ≠ safe deployment

Prerequisites: Evaluating AI Systems, Ethics, Bias, and Equity, Privacy, Security, and Governance.

Chapter Summary (TL;DR)

The Big Picture: AI safety is distinct from security, privacy, and quality. Safety asks how a model, people, workflow, data, and organization can jointly create harm, then builds controls before deployment and monitors them afterward.

Safety vs. Related Concepts (Distinct But Complementary):

Safety: Preventing harm from system failures (false negatives miss disease, false positives cause unnecessary treatment)
Security: Protecting from adversarial attacks (poisoning training data, model theft, adversarial examples)
Privacy: Protecting individual data (re-identification, inference attacks, data breaches)
Quality: Meeting specifications (accuracy, precision, data quality)

All necessary; none sufficient alone.

Why Healthcare AI Has a Safety Problem:

Epic Sepsis: External evaluation found 33% sensitivity and 12% positive predictive value at the evaluated threshold. Deployment did not establish safe workflow performance.
Distribution Shift: Models degrade silently when training and deployment populations diverge (e.g., Epic Sepsis Model: vendor-reported AUC 0.76–0.83 vs externally validated 0.63)
Alert Fatigue: 95% false positive rate means warnings ignored, defeating purpose
Concept Drift: Clinical practice evolves, invalidating training assumptions
Black Box Opacity: Cannot inspect neural network decision logic to verify safety

Safety-Critical Systems Frameworks for AI:

IEC 62304 (Medical Device Software Lifecycle):

Class A: No injury/harm possible
Class B: Non-serious injury possible
Class C: Death or serious injury possible

Most clinical AI is Class B or C, requiring rigorous safety documentation, testing, change control.

ISO 14971 (Risk Management):

Risk Analysis: Identify hazards systematically (FMEA - Failure Modes and Effects Analysis)
Risk Evaluation: Assess severity × probability
Risk Control: Mitigation strategies (reduce likelihood, reduce severity, add safeguards)
Post-Market Surveillance: Continuous monitoring for emerging risks

Good Machine Learning Practice (GMLP): FDA, Health Canada, and MHRA’s 10 guiding principles (2021) stress representative data, independent test sets, human-AI team performance, testing in clinically relevant conditions, and post-deployment monitoring.

Predetermined Change Control Plans (PCCP): FDA innovation allowing model updates within pre-specified boundaries without new clearance (e.g., retrain on new data using same architecture).

Failure Modes and Effects Analysis (FMEA):

Systematic hazard identification:

For Sepsis Prediction Model:

False Negative (Miss Sepsis): Patient deteriorates untreated. Severity: Critical. Mitigation: High sensitivity threshold, human oversight
False Positive (False Alarm): Alert fatigue, clinicians ignore. Severity: Moderate. Mitigation: High specificity, contextual alerts
Data Pipeline Failure: Missing labs → wrong prediction. Severity: High. Mitigation: Input validation, fallback to clinical judgment
Distribution Shift: New patient population, model degrades. Severity: High. Mitigation: Continuous monitoring, automated alerts
Integration Failure: Predictions delayed, arrive after clinical decision. Severity: Moderate. Mitigation: Real-time processing, latency SLAs

Each failure mode needs: Severity rating, probability estimate, detectability assessment, mitigation strategy.

Safety Safeguards (Defense in Depth):

Human-in-the-Loop: AI recommends, human decides. High-stakes decisions require human confirmation
Circuit Breakers: Automatic fallback when model confidence low or input unusual (out-of-distribution detection)
Monitoring Dashboards: Real-time performance tracking (sensitivity, specificity, alert rate, override rate)
Graceful Degradation: System continues functioning safely when AI component fails
Audit Logging: Record all predictions, decisions, overrides for post-hoc analysis

Adversarial Robustness (Security Overlaps Safety):

Medical imaging AI vulnerable to adversarial attacks:

Pixel-level perturbations (invisible to humans) change diagnosis
Natural perturbations (image brightness, contrast) can flip predictions
Out-of-distribution inputs: Model confident on images unlike training data

Testing: FGSM, PGD attacks assess vulnerability. EU AI Act requires robustness testing for high-risk systems.

Post-Market Surveillance (Continuous Monitoring):

Model performance can degrade after deployment:

Monitor for:

Data Drift: Input distributions change (demographics, disease prevalence, diagnostic protocols)
Concept Drift: Relationship between features and outcome changes
Performance Degradation: AUC, sensitivity, specificity decline
Fairness Drift: Disparities emerge in subgroups
Alert Metrics: False alarm rate, override rate, time-to-action

Automated Alerts: Trigger investigations when performance drops below thresholds.

Retraining Decisions: When to retrain? How to validate? FDA PCCP pre-specifies retraining protocols.

Incident Response Protocols:

When AI causes harm:

Immediate: Disable/rollback problematic model, mitigate patient harm
Investigation: Root cause analysis: was it data, model, integration, or workflow?
Reporting: FDA MedWatch (serious adverse events), institutional review boards
Corrective Action: Fix root cause, prevent recurrence
Documentation: Transparent reporting builds trust, enables learning
Communication: Notify stakeholders, affected patients (if applicable), regulatory bodies

Blame-free culture: Encourage reporting, not punishment; hidden failures are more dangerous than disclosed ones.

Why Technical Excellence ≠ Safe Deployment:

Epic Sepsis Example:

90% accuracy retrospectively
Published research
Deployed at 100+ hospitals
33% sensitivity prospectively (67% misses)
12% PPV (88% false alarms)
Alert fatigue, clinicians overriding
No external validation before deployment

Lesson: Safety requires prospective validation, clinical workflow integration, continuous monitoring, not just retrospective metrics.

The Takeaway for Public Health Practitioners:

Healthcare AI has a safety problem: Epic sepsis shows that technical validation ≠ safe deployment. Safety requires systematic frameworks (IEC 62304, ISO 14971, FDA GMLP), not just accuracy metrics. FMEA identifies failure modes before deployment. Defense in depth: human-in-the-loop, circuit breakers, monitoring dashboards, audit logging. Models degrade silently, so post-market surveillance is essential. Adversarial robustness matters (EU AI Act requires testing). Incident response protocols needed for when AI harms patients. Blame-free culture encourages reporting. Most importantly: 90% accuracy can still cause catastrophic harm if 10% errors concentrate in high-stakes scenarios. Safety is a systematic practice requiring continuous vigilance, honest reporting, and humility about AI limitations. When in doubt, fail safe (false alarm better than missed disease in screening; opposite for confirmatory tests).

Introduction: When AI Fails, People Die

Epic Sepsis: Safety Lesson

The external evaluation of a widely deployed proprietary sepsis model showed 33% sensitivity and 12% positive predictive value at the evaluated threshold (Wong et al., 2021). For safety engineering, the lesson is not the product name. It is that a validated score can still create missed cases, false-alert burden, delayed recognition, and unsafe reliance when the threshold and workflow do not match the clinical hazard.

The full case is maintained in The AI Morgue: Epic Sepsis Model. That evidence informs the hazard analysis, monitoring, fallback, and incident-response controls below.

Why Safety Is Different from Security, Privacy, and Quality

These concepts are distinct but complementary:

Dimension	What It Protects	Primary Threats	Example Failure
Safety	Patients/populations from harm caused by system behavior	Design flaws, operational failures, edge cases	Epic sepsis: missed 67% of cases
Security	Systems/data from malicious actors	Hackers, unauthorized access, tampering	Anthem breach: 78M records stolen
Privacy	Sensitive data from unauthorized disclosure	Re-identification, inference attacks	Cambridge Analytica: 87M users exploited
Quality	Meeting specified requirements	Implementation bugs, spec violations	Software crashes, incorrect calculations

Why the distinction matters:

Secure but unsafe: A perfectly secure system can still harm patients if it makes wrong predictions
High-quality but unsafe: Bug-free code with no specification errors can still fail in edge cases
Private but unsafe: Perfect data protection does not prevent clinical harm from model errors

Safety depends on operational behavior in the real world, not only on technical correctness in controlled environments.

The Safety Crisis in Healthcare AI

The Epic case is not isolated. Healthcare AI has a safety problem:

FDA recall example:

Caption Health ultrasound software (May 2021): FDA recorded a Class II recall involving unintended frames that could produce incorrect AutoEF and image-quality scores (FDA Recall Z-1734-2021). A recall record establishes the identified product problem; it does not establish a general failure rate for imaging AI.

High-profile failures are documented in the AI Morgue: IBM Watson for Oncology produced unsafe treatment recommendations, and Google’s diabetic retinopathy screening in Thailand rejected 21% of field images as ungradable (Beede et al., 2020).

The deployment gap:

Most healthcare AI models that show promise in research never reach routine deployment. Safety concerns are a major barrier, alongside infrastructure, workflow integration, and validation gaps documented in real-world implementation studies (e.g., the Duke sepsis deep learning rollout; Sendak et al., 2020, JMIR Medical Informatics).

Why traditional medical device regulation falls short:

Traditional software controls often emphasize:

Static devices - Performance does not change after approval
Transparent logic - Decision rules can be inspected
Predictable behavior - Same input always yields same output
Controlled environment - Operates within specified conditions

Adaptive, generative, and data-dependent systems can challenge some of these assumptions:

Changeable behavior - Some systems are updated, while many regulated models remain locked between controlled releases
Limited inspectability - Some architectures are difficult to interpret, while others are transparent
Stochastic output - Generative systems may vary across runs, while many discriminative models are deterministic for a fixed configuration
Distribution shift - Performance can change when populations, workflows, or inputs differ from validation data

Example: Transportability gap in sepsis prediction

Epic’s proprietary Sepsis Model showed substantial gaps between vendor-reported and externally validated performance (Wong et al., 2021, JAMA Intern Med):

Vendor-reported AUC: 0.76–0.83
External validation at one academic health system: AUROC 0.63
Causes: differences in clinical practice, patient demographics, and documentation patterns between vendor training data and deployment sites, the same dataset-shift mechanisms framed in Finlayson et al., 2021, NEJM

This was a cross-site external validation, not a longitudinal study of gradual degradation. The result demonstrates transportability risk when a model is used outside the setting represented by its development evidence.

Scope

AI safety practice spans six areas:

Safety frameworks and standards - IEC 62304, ISO 14971, FDA guidance
Hazard analysis - Identifying failure modes before they cause harm
Safety validation - Testing beyond standard ML metrics
Safety-critical design - Building safety into system architecture
Post-market surveillance - Detecting and responding to safety incidents
Organizational culture - Creating a culture of safety

Outside scope:

General software quality assurance (see IEC 62304)
Clinical trial safety (see ICH E6 guidance)
Occupational safety (see OSHA standards)

Focus: AI-specific safety challenges in healthcare and public health applications.

Safety Frameworks and Standards

Medical Device Software Standards

IEC 62304: Medical Device Software Lifecycle

Overview: International standard for software development lifecycle of medical devices.

Safety classification (determines rigor of development process):

Class	Definition	Examples	Requirements
A	No injury or damage to health possible	Administrative tools, non-diagnostic displays	Basic documentation
B	Non-serious injury possible	Decision support for non-critical conditions	Moderate rigor
C	Death or serious injury possible	Diagnostic tools for critical conditions, treatment recommendations	Maximum rigor

Most healthcare AI falls into Class B or C.

Key requirements for Class C software:

Risk management process (ISO 14971 integration)
Software safety requirements derived from risk analysis
Architecture designed for safety
Unit, integration, and system testing with traceability
Problem resolution process for post-market issues
Version control and configuration management

IEC 62304 and AI

Traditional IEC 62304 assumes deterministic software with reviewable code. AI systems challenge this:

Black box models: Cannot review decision logic
Data-dependent behavior: Changes with training data
Probabilistic outputs: Scores rather than explicit rules; generative outputs may vary across runs

FDA and EU regulators are updating guidance to address AI-specific concerns, but a fundamental tension remains: safety frameworks designed for transparency are applied to often opaque systems.

ISO 14971: Risk Management for Medical Devices

Overview: Systematic process for identifying and controlling risks throughout device lifecycle.

Five-step risk management process:

graph LR
 A[1. Risk Analysis] --> B[2. Risk Evaluation]
 B --> C[3. Risk Control]
 C --> D[4. Residual Risk Evaluation]
 D --> E[5. Risk Management Report]
 E --> F[Post-Market Surveillance]
 F --> A

Figure 20.1: ISO 14971 five-step risk management process for AI medical devices. The process is cyclical, with post-market surveillance feeding back into ongoing risk analysis to ensure continuous improvement.

Step 1: Risk Analysis

Identify hazards, hazardous situations, and harms:

Hazard: Potential source of harm (e.g., incorrect prediction)
Hazardous situation: Circumstance where hazard could lead to harm (e.g., clinician acts on false negative)
Harm: Injury or damage to health (e.g., patient dies from undetected sepsis)

Step 2: Risk Evaluation

Estimate risk using:

Severity: How bad is the harm? (1=negligible, 5=catastrophic)
Probability: How likely is it? (1=remote, 5=frequent)
Risk = Severity × Probability

Risk matrix example:

Probability ↓ Severity →	Negligible (1)	Minor (2)	Moderate (3)	Major (4)	Catastrophic (5)
Frequent (5)	5	10	15	20	25
Probable (4)	4	8	12	16	20
Occasional (3)	3	6	9	12	15
Remote (2)	2	4	6	8	10
Improbable (1)	1	2	3	4	5

Risk acceptability:

1–5: Acceptable (monitor)
6–12: ALARP (As Low As Reasonably Practicable) - reduce if feasible
15–25: Unacceptable - must reduce before deployment

Step 3: Risk Control

Hierarchy of control measures (in order of preference):

Inherent safety by design - Eliminate hazard (best)
Protective measures - Add safeguards (e.g., human-in-the-loop)
Information for safety - Warnings, training (weakest)

Step 4: Residual Risk Evaluation

After controls, re-evaluate:

Is residual risk acceptable?
Do control measures introduce new risks?
Is overall benefit-risk ratio favorable?

Step 5: Risk Management Report

Document:

All identified hazards
Risk estimates before/after controls
Rationale for acceptability decisions
Post-market surveillance plan

Common ISO 14971 Mistakes in AI Projects

Performing risk analysis only once - Must update as model changes
Using only accuracy metrics - Severity and probability require clinical context
Treating all errors equally - False negative in cancer screening ≠ false positive
Insufficient hazard identification - Missing operational and systemic hazards
No post-market surveillance plan - Required by standard but often omitted

Regulatory Safety Guidance

FDA: AI/ML-Based Software as a Medical Device (SaMD)

FDA’s AI/ML Action Plan (2021) and the joint GMLP guiding principles (FDA, Health Canada, MHRA, 2021) address AI-specific challenges:

1. Good Machine Learning Practice (GMLP)

Multidisciplinary expertise is leveraged throughout the total product life cycle
Good software engineering and security practices are implemented
Clinical study participants and data sets are representative of the intended patient population
Training data sets are independent of test sets
Selected reference datasets are based upon best available methods
Model design is tailored to the available data and reflects the intended use of the device
Focus is placed on the performance of the human-AI team
Testing demonstrates device performance during clinically relevant conditions
Users are provided clear, essential information
Deployed models are monitored for performance, and retraining risks are managed

2. Predetermined Change Control Plans (PCCP)

Allows approved AI/ML models to evolve without a new marketing submission for each change (FDA final PCCP guidance, issued December 2024 and updated August 2025, covering 510(k), De Novo, and PMA), if:

Description of modifications: the planned changes (called SaMD Pre-Specifications, SPS, in the 2019 discussion paper)
Modification protocol: methods to develop, validate, and implement the changes (formerly the Algorithm Change Protocol, ACP)
Impact assessment: benefits and risks of the planned changes and how risks are mitigated

Example description of modifications for diabetic retinopathy screening:

Retrain with new data quarterly
Add image preprocessing techniques
Update decision threshold based on population
Not allowed: Change intended use, add new conditions

3. Real-World Performance Monitoring

FDA expects:

Continuous monitoring of safety and effectiveness
Detection of performance degradation
Timely response to safety signals

FDA risk categorization for SaMD:

State of Healthcare / Significance of Information Provided	Treat/Diagnose	Drive Clinical Management	Inform Clinical Management
Critical	IV	III	II
Serious	III	II	I
Non-Serious	II	I	I

FDA device classes and pathways:

Class I (Low risk): General controls, usually exempt from premarket review
Class II (Moderate risk): 510(k) clearance (substantial equivalence) - most AI devices
Class III (High risk): Premarket Approval (PMA) - most rigorous

FDA 510(k) Shortcut

90% of AI medical devices clear through 510(k) by claiming “substantial equivalence” to existing devices.

Problem: New devices are often compared to predicate devices from the 1990s–2000s that were never rigorously validated. This creates a “grandfather paradox”: new AI compared to old non-AI tools that never proved clinical utility.

Recent FDA tightening: Requiring more clinical evidence, especially for high-risk applications.

EU: Medical Device Regulation (MDR) and AI

EU Medical Device Regulation (2017/745) took full effect May 2021, replacing older directives.

Key requirements for AI SaMD:

1. Clinical Evaluation

Much more rigorous than FDA 510(k):

Clinical investigation often required (not just analytical validation)
Post-market clinical follow-up (PMCF) mandatory
Equivalence harder to claim for novel AI

2. Risk Classification

Rule	AI Application	Class
Rule 11	Software providing information for diagnostic or therapeutic decisions	IIa (moderate-low risk)
Rule 11	Such decisions could cause serious deterioration of health or surgical intervention	IIb (moderate-high risk)
Rule 11	Such decisions could cause death or irreversible deterioration of health	III (high risk)
Rule 11	Software monitoring vital physiological parameters where variations could result in immediate danger	IIb

Most diagnostic AI = Class IIa or IIb (requiring Notified Body involvement).

3. Transparency and Explainability

MDR requires (Annex I, Chapter III):

“Information supplied with the device” describing intended purpose, performance characteristics, and residual risks
Limitations, contraindications, and warnings

4. Post-Market Surveillance

Manufacturers must:

Have post-market surveillance system
Report serious incidents and field safety corrective actions, and report trends in non-serious incidents
Update clinical evaluation with post-market data
Periodic Safety Update Reports (PSUR)

UK: MHRA Software and AI as Medical Device Change Programme

Post-Brexit, the UK is developing its own framework:

Software and AI as a Medical Device Change Programme emphasizes:

Proportionate regulation based on risk and autonomy level
Lifecycle regulation - not just pre-market
Real-world evidence for approval
Transparency - explainability where feasible

International Divergence

FDA, EU, and UK increasingly diverging on AI regulation:

FDA: Risk-based, allows “substantial equivalence,” PCCP for adaptive algorithms
EU MDR: Stricter clinical evidence, harder to claim equivalence, mandatory PMCF
UK: Still defining post-Brexit framework, emphasizing innovation

Implication: Medical AI companies need three different validation strategies for US, EU, UK markets.

Safety-Critical Systems Theory from Other Industries

Healthcare can learn from industries with mature safety cultures:

Commercial Aviation Safety

Swiss Cheese Model of Accidents (James Reason)

[Hazard] → | Hole | → | Hole | → | Hole | → | Hole | → [Accident]
   Layer 1 Layer 2 Layer 3 Layer 4
   (Design) (Training) (Procedure) (Defense)

Key insight: Accidents require multiple failures aligning. Safety requires defense in depth, multiple independent layers.

Application to AI:

Layer 1 (Design): Robust model architecture, diverse training data
Layer 2 (Validation): Comprehensive testing including edge cases
Layer 3 (Deployment): Human-in-the-loop, fallback systems
Layer 4 (Monitoring): Real-time performance tracking, circuit breakers

Just Culture: Aviation learned blame-free reporting increases safety. Healthcare AI needs a similar culture. Punishing failures drives them underground.

Nuclear Safety

Principles from nuclear industry:

Defense in depth: Multiple independent protection layers
Redundancy: Backup systems for critical functions
Diversity: Different technologies for same function (prevents common-mode failures)
Conservative decision-making: Err on side of caution when uncertain
Continuous oversight: Independent safety monitoring

Application to AI:

Defense in depth: Do not rely solely on model accuracy; add monitoring, human oversight, and fallbacks
Redundancy: Multiple models or rule-based backup for critical decisions
Diversity: Ensemble methods, different model architectures
Conservative: Higher confidence thresholds for critical decisions
Oversight: Independent evaluation before deployment, continuous monitoring after

Fail-Safe vs. Fail-Secure

Fail-safe: System defaults to safe state on failure (e.g., railway signals default to red)

Fail-secure: System maintains security on failure (e.g., door lock stays locked)

For medical AI: Usually want fail-safe (defer to human) not fail-secure (block access).

Example: Sepsis prediction model fails → Alert clinician to manual assessment (fail-safe) NOT continue with last prediction (fail-secure).

Hazard Analysis and Failure Modes

AI-Specific Failure Modes

Traditional software has well-understood failure modes (bugs, crashes, security vulnerabilities). AI introduces novel failure modes:

Category 1: Technical Failures

1. Data Drift and Distribution Shift

Definition: Training data distribution differs from deployment distribution.

Types:

Covariate shift: P(X) changes but P(Y|X) stays same
- Example: COVID-19 changes patient demographics (age, comorbidities)
Prior probability shift: P(Y) changes but P(X|Y) stays same
- Example: Disease prevalence changes seasonally
Concept drift: P(Y|X) changes
- Example: Treatment protocols evolve, changing disease outcomes

Why it’s dangerous: Model performance silently degrades without obvious errors or crashes.

Real-world example: Epic Sepsis Model dropped to AUROC 0.63 in external validation despite vendor-reported 0.76–0.83 (Wong et al., 2021, JAMA Intern Med); the dataset-shift framework that explains how this happens is laid out in Finlayson et al., 2021, NEJM:

Changes in clinical documentation practices
Updates to EHR systems
Shifts in patient population

2. Adversarial Examples

Definition: Inputs deliberately crafted to fool model, often imperceptible to humans.

Medical imaging example:

# Simplified adversarial example
import numpy as np

def generate_adversarial_example(model, image, true_label, epsilon=0.01):
 """
 Generate adversarial perturbation using Fast Gradient Sign Method

 Parameters:
 - model: Trained classifier
 - image: Original medical image (e.g., X-ray)
 - true_label: Correct diagnosis
 - epsilon: Perturbation magnitude (small = imperceptible)

 Returns:
 - Adversarial image that model misclassifies
 """
 # Get model's gradient with respect to input
 loss_gradient = model.get_loss_gradient(image, true_label)

 # Create perturbation in direction that increases loss
 perturbation = epsilon * np.sign(loss_gradient)

 # Add imperceptible noise
 adversarial_image = image + perturbation

 # Model misclassifies, but human sees no difference
 return adversarial_image

# Example: Chest X-ray
original_prediction = model.predict(xray) # "Pneumonia: 95% confidence"
adversarial_xray = generate_adversarial_example(model, xray, "pneumonia")
adversarial_prediction = model.predict(adversarial_xray) # "Normal: 92% confidence"

# WARNING: Radiologist sees identical images, but model flips diagnosis

Why it matters:

Accidental adversarial examples: Real medical images might accidentally fall into adversarial regions
Malicious attacks: Bad actors could manipulate images to change diagnoses
Fragility indicator: Sensitivity to tiny perturbations suggests model relies on spurious features

Defense strategies:

Adversarial training (include adversarial examples in training)
Input sanitization and validation
Ensemble methods (adversarial examples often do not transfer)
Certified defenses (formal guarantees of robustness)

3. Spurious Correlations

Definition: Model learns from confounders or artifacts rather than true causal features.

Classic example: Zech et al., 2018, PLoS Medicine

Pneumonia detection model achieved high accuracy
CNNs identified the hospital system of a radiograph with more than 99% accuracy, and pneumonia prevalence differed sharply between hospitals, so hospital identity could confound predictions
Model learned: “Hospital A uses portable X-rays for ICU patients → Portable X-ray marker → Pneumonia”
Confounding: Sicker patients → ICU → Portable X-rays

Safety hazard: Models performed significantly worse on external data than on internal data in 3 of 5 comparisons.

Another example: COVID-19 detection from chest X-rays

Many models learned:

Patients with COVID → Admitted to hospital → Supine position X-rays
Healthy people → Outpatient imaging → Standing position X-rays
Model detected position (supine vs. standing) rather than COVID

4. Training Data Poisoning

Definition: Malicious or erroneous data corrupts training set.

Scenarios:

Accidental: Mislabeled data, data entry errors, EHR errors
Malicious: Adversary injects poisoned samples during training

Example: Model trained on EHR data with documentation errors

Diagnosis codes frequently misapplied
Billing codes used inconsistently
Copy-paste errors in clinical notes

Consequence: Model learns incorrect patterns from noisy labels.

5. Out-of-Distribution (OOD) Detection Failure

Definition: Model makes confident predictions on data unlike anything in training set.

Illustrative example: Diabetic retinopathy screening AI deployed in a rural clinic

Training data: High-quality retinal images from academic centers
Deployment: Poor lighting, cataracts, lower-quality cameras
Model made confident (wrong) predictions on low-quality images it should have rejected

Safety requirement: Model must detect OOD inputs and defer to human.

def safe_prediction_with_ood_detection(model, ood_detector, input_data):
 """
 Make prediction only if input is in-distribution

 Returns:
 - (prediction, confidence) if in-distribution
 - (None, "OOD detected") if out-of-distribution
 """
 # Check if input similar to training distribution
 ood_score = ood_detector.score(input_data)

 if ood_score > THRESHOLD:
  # Out of distribution - defer to human
  log_warning(f"OOD input detected: {ood_score}")
  return None, "Input outside model's training distribution - manual review required"

 # In distribution - proceed with prediction
 prediction = model.predict(input_data)
 return prediction, model.confidence

Category 2: Operational Failures

6. Misuse and Off-Label Use

Definition: Using AI for purposes beyond validated scope.

Examples:

Diabetic retinopathy AI used for other eye diseases
COVID-19 detection AI used for other respiratory conditions
Sepsis prediction AI used in populations different from training

Why it happens:

Intended use not clearly communicated
Clinicians assume broader applicability
Pressure to “do something” with AI investment

Safety requirement: Clear labeling of validated use cases and populations.

7. Automation Bias

Definition: Tendency to favor automated decisions over human judgment, even when human is correct.

Study: Goddard et al., 2012, JAMIA

Systematic review of 74 studies of automation bias, the tendency to over-rely on automation
Mediators included trust and confidence in the system, task experience, workload, task complexity, and time constraint
Mitigators included training, emphasizing user accountability, the position of advice on screen, updated confidence levels, and providing information rather than recommendations

Application to AI: High-confidence predictions can override clinical judgment, even when AI is wrong.

Mitigation:

Present AI as “decision support” not “decision maker”
Display uncertainty/confidence intervals
Encourage critical evaluation
Training on AI limitations

8. Alert Fatigue

Definition: Desensitization to warnings due to high false positive rate.

Epic sepsis model: At the evaluated threshold, the external study reported a positive predictive value of 12%, implying substantial false-alert burden. The study did not establish that clinicians learned to ignore alerts or define a universal acceptable threshold (Wong et al., 2021).

Threshold: Measure alert volume, positive predictive value, missed-event rate, time to review, override behavior, and downstream action in the intended workflow. No single false-positive percentage is acceptable across all alert types.

Trade-off: Sensitivity vs. specificity

Increase sensitivity → Catch more cases but more false alarms
Increase specificity → Fewer false alarms but miss more cases

Safety-critical design must account for alert burden in the operational context, not only optimize AUROC.

9. Workflow Integration Failures

Definition: AI disrupts clinical workflow, leading to workarounds or abandonment.

Example: AI diagnostic tool requires:

Separate login
Data re-entry
5-minute wait for prediction
Results in different system from EHR

Consequence: Clinicians skip using tool or use incorrectly.

Safety requirement: AI must fit smoothly into existing workflow.

Category 3: Systemic Failures

10. Cascading Failures

Definition: AI failure triggers failures in dependent systems.

Example: Hospital resource allocation AI

AI predicts low emergency admissions
Hospital reduces staffing
Unpredicted surge occurs
Insufficient staff → Delayed care → Patient harm

System view: AI is embedded in complex sociotechnical system. Local optimization can create global failure.

11. Common-Cause Failures

Definition: Single event causes multiple AI systems to fail simultaneously.

Example: EHR system update changes data format

Multiple AI models expecting old format fail
All deployed models affected at once
No backup since all rely on same infrastructure

Mitigation: Diversity in system architecture, independent fallback systems.

12. Latent Failures

Definition: Errors or vulnerabilities that exist but do not cause harm until triggered by specific conditions.

Example: Model trained predominantly on data from one demographic

Performs well on similar populations (latent failure undetected)
Deployed to different demographic → Poor performance → Patient harm

Swiss cheese model: Latent failures are “holes” waiting for alignment.

Failure Mode and Effects Analysis (FMEA) for AI

FMEA: Systematic method for identifying failure modes before they cause harm.

Process:

Identify potential failure modes (what could go wrong?)
Determine effects (what happens if it fails?)
Assess severity (how bad?)
Assess occurrence (how likely?)
Assess detection (can we catch it before harm?)
Calculate Risk Priority Number (RPN) = Severity × Occurrence × Detection
Prioritize mitigation (address highest RPNs first)

FMEA Template for AI Systems

Scenario: Sepsis prediction model

Failure Mode	Potential Cause	Effect	Severity (1–5)	Occurrence (1–5)	Detection (1–5)	RPN	Mitigation
Model misses sepsis case (false negative)	Data drift, edge case, training bias	Patient does not receive timely treatment → Death/severe harm	5	3	4	60	1) Human review for borderline cases 2) Ensemble with rule-based backup 3) Lower confidence threshold
False alarm (false positive)	Low specificity, noisy input data	Alert fatigue → Ignore future alerts	4	5	2	40	1) Increase specificity threshold 2) Two-stage alerting 3) Monitor alert burden
Model unavailable (system crash)	Server failure, network outage	No predictions → Revert to standard care	3	2	1	6	1) Redundant systems 2) Graceful degradation 3) Automatic failover
Data drift (silent degradation)	Population changes, practice changes	Gradually increasing errors	5	4	5	100	1) Continuous monitoring 2) Automated drift detection 3) Retraining protocol
OOD input (unfamiliar case)	Patient outside training distribution	Confident wrong prediction	5	3	4	60	1) OOD detection 2) Uncertainty quantification 3) Defer to human
Adversarial attack	Malicious manipulation of input	Incorrect diagnosis	5	1	5	25	1) Input validation 2) Adversarial training 3) Anomaly detection

Severity scale:

1: No harm
2: Minor harm (temporary, reversible)
3: Moderate harm (prolonged recovery)
4: Major harm (permanent injury)
5: Catastrophic (death or multiple major harms)

Occurrence scale:

1: Very rare (< 1 in 10,000)
2: Rare (1 in 1,000)
3: Occasional (1 in 100)
4: Frequent (1 in 10)
5: Very frequent (> 1 in 10)

Detection scale:

1: Almost certain to detect before harm
2: High chance of detection
3: Moderate chance
4: Low chance
5: Almost certain NOT to detect

Risk Priority Number (RPN):

1–30: Low risk (monitor)
31–100: Moderate risk (reduce if feasible)
101–125: High risk (must reduce before deployment)

Post-mitigation: Re-calculate RPN after implementing controls. Goal: All RPN < 100, ideally < 50.

FMEA Workshop

Conduct FMEA with multidisciplinary team:

Data scientists: Technical failure modes
Clinicians: Clinical context and effects
Safety officers: Severity assessment
Quality assurance: Detection methods
Patients: Real-world impact

Timing: Before deployment, then updated:

When model changes
When deployment context changes
After incidents or near-misses
Annually at minimum

Automated FMEA Template

import pandas as pd

class AISystemFMEA:
 """
 Systematic FMEA for AI healthcare systems
 """

 def __init__(self, system_name):
  self.system_name = system_name
  self.failure_modes = []

 def add_failure_mode(self, mode, cause, effect, severity, occurrence, detection):
  """
  Add failure mode to FMEA

  Parameters:
  - mode: Description of failure
  - cause: What causes this failure
  - effect: Consequence of failure
  - severity: 1-5 (1=negligible, 5=catastrophic)
  - occurrence: 1-5 (1=rare, 5=frequent)
  - detection: 1-5 (1=certain to detect, 5=unlikely to detect)
  """
  rpn = severity * occurrence * detection

  self.failure_modes.append({
   'Failure Mode': mode,
   'Cause': cause,
   'Effect': effect,
   'Severity': severity,
   'Occurrence': occurrence,
   'Detection': detection,
   'RPN': rpn,
   'Risk Level': self._classify_risk(rpn)
  })

 def _classify_risk(self, rpn):
  if rpn <= 30:
   return 'Low'
  elif rpn <= 100:
   return 'Moderate'
  else:
   return 'High'

 def generate_report(self):
  """Generate FMEA report sorted by RPN"""
  df = pd.DataFrame(self.failure_modes)
  df = df.sort_values('RPN', ascending=False)

  print(f"\n{'='*80}")
  print(f"FMEA Report: {self.system_name}")
  print(f"{'='*80}\n")

  print(df.to_string(index=False))

  # Risk summary
  risk_counts = df['Risk Level'].value_counts()
  print(f"\n{'='*80}")
  print("Risk Summary:")
  print(f" High Risk:  {risk_counts.get('High', 0)} failure modes (RPN > 100)")
  print(f" Moderate Risk: {risk_counts.get('Moderate', 0)} failure modes (RPN 31-100)")
  print(f" Low Risk:  {risk_counts.get('Low', 0)} failure modes (RPN ≤ 30)")

  if risk_counts.get('High', 0) > 0:
   print(f"\n[WARNING] {risk_counts.get('High', 0)} HIGH RISK failure modes must be mitigated before deployment")

  return df

# Example usage
fmea = AISystemFMEA("Sepsis Prediction Model")

# Add failure modes from clinical team
fmea.add_failure_mode(
 mode="False negative (missed sepsis case)",
 cause="Data drift, patient outside training distribution",
 effect="Delayed treatment → Patient death",
 severity=5,
 occurrence=3,
 detection=4
)

fmea.add_failure_mode(
 mode="False alarm (false positive)",
 cause="Low model specificity",
 effect="Alert fatigue → Future alerts ignored",
 severity=4,
 occurrence=5,
 detection=2
)

fmea.add_failure_mode(
 mode="Silent performance degradation",
 cause="Concept drift over time",
 effect="Gradually increasing error rate",
 severity=5,
 occurrence=4,
 detection=5
)

# Generate report
report = fmea.generate_report()

Output:

================================================================================
FMEA Report: Sepsis Prediction Model
================================================================================

     Failure Mode        Cause          Effect Severity Occurrence Detection RPN Risk Level
Silent performance degradation     Concept drift over time    Gradually increasing error rate   5   4   5 100 Moderate
False negative (missed sepsis case) Data drift, patient outside training...   Delayed treatment → Patient death   5   3   4 60 Moderate
 False alarm (false positive)      Low model specificity Alert fatigue → Future alerts ignored   4   5   2 40 Moderate

================================================================================
Risk Summary:
 High Risk:  0 failure modes (RPN > 100)
 Moderate Risk: 3 failure modes (RPN 31-100)
 Low Risk:  0 failure modes (RPN ≤ 30)

FMEA Limitations for AI

Traditional FMEA assumes:

Known failure modes
Quantifiable probabilities
Independent failures

AI challenges:

Unknown unknowns: Failure modes we have not thought of
Uncertain probabilities: Hard to estimate occurrence of data drift
Correlated failures: Adversarial examples affect whole model class

Solution: Combine FMEA with:

Stress testing for unknown failure modes
Continuous monitoring for early detection
Red team exercises to find vulnerabilities

Other Hazard Analysis Methods

Fault Tree Analysis (FTA)

Top-down approach: Start with harm, work backward to causes.

     Patient Death from Sepsis
       |
    +------------+------------+
    |       |
  Sepsis Not Detected  Treatment Delayed
    |       |
  +--------+--------+  +--------+--------+
  |     |  |     |
Model Missed Case Clinician Alert System
     Didn't  Not  Down
     Notice  Seen

Event Tree Analysis (ETA)

Forward approach: Start with event, trace possible outcomes.

Sepsis Develops → Model Alerts? → Clinician Reviews? → Treatment Given? → Outcome
      ↓     ↓     ↓
     Yes/No    Yes/No    Yes/No  Survive/Die

HAZOP (Hazard and Operability Study)

Use guide words to identify hazards:

Guide Word	Application to AI	Example Hazard
No/Not	No prediction available	System downtime → Manual workflow
More	More false positives	Alert fatigue
Less	Less sensitivity	Missed cases
As Well As	Model captures spurious features	Detects hospital system, not disease
Part Of	Model trained on subset	Bias against underrepresented groups
Reverse	Opposite of intended	Model predicts low risk for high-risk patient
Other Than	Different output	Model provides prediction for wrong patient

Safety Validation Beyond ML Evaluation

Standard ML evaluation (accuracy, AUROC, precision/recall on a test set) is necessary but insufficient for safety validation.

Why Standard Metrics Do Not Ensure Safety

Problem 1: Average performance masks critical failures

95% accuracy sounds great
But 5% errors might all be in high-severity cases

Illustrative example: Cancer screening AI

98% accuracy overall
But 30% false negative rate on early-stage cancers (most treatable)
Model performs well on late-stage (obvious) and no-cancer cases
The model fails exactly where it would be most valuable

Problem 2: Test set does not represent edge cases

IID (independent, identically distributed) assumption
Real world has outliers, distribution shift, adversarial conditions

Problem 3: Metrics do not account for operational context

High sensitivity with low specificity → Alert fatigue → System abandonment
Metrics evaluated in isolation, not system-level impact

Problem 4: No assessment of failure modes

Which errors are acceptable?
Which are catastrophic?
Standard metrics treat all errors equally

Safety-Specific Validation Methods

1. Worst-Case Analysis

Goal: Evaluate performance on hardest cases, not average cases.

def worst_case_validation(model, X, y, protected_groups):
 """
 Evaluate model on worst-performing subgroups

 Returns performance on:
 - Hardest examples (lowest confidence correct predictions)
 - Underrepresented groups
 - Edge cases
 """
 results = {}

 # 1. Evaluate on low-confidence predictions
 predictions = model.predict_proba(X)
 confidence = np.max(predictions, axis=1)

 # Bottom quartile confidence
 low_conf_mask = confidence < np.percentile(confidence, 25)
 results['low_confidence_accuracy'] = accuracy_score(
  y[low_conf_mask],
  model.predict(X[low_conf_mask])
 )

 # 2. Evaluate on underrepresented groups
 for group in protected_groups:
  group_mask = (X['demographic_group'] == group)
  results[f'{group}_accuracy'] = accuracy_score(
   y[group_mask],
   model.predict(X[group_mask])
  )
  results[f'{group}_sample_size'] = group_mask.sum()

 # 3. Evaluate on edge cases (outliers)
 # Use isolation forest or other outlier detection
 from sklearn.ensemble import IsolationForest
 outlier_detector = IsolationForest()
 outlier_scores = outlier_detector.fit_predict(X)
 outlier_mask = (outlier_scores == -1)

 results['outlier_accuracy'] = accuracy_score(
  y[outlier_mask],
  model.predict(X[outlier_mask])
 )

 return results

# Example
worst_case_results = worst_case_validation(
 model=sepsis_model,
 X=test_data,
 y=test_labels,
 protected_groups=['Black', 'Hispanic', 'Asian', 'White']
)

print("Worst-Case Validation Results:")
print(f" Overall accuracy: 92%")
print(f" Low-confidence accuracy: {worst_case_results['low_confidence_accuracy']:.1%}")
print(f" Outlier accuracy: {worst_case_results['outlier_accuracy']:.1%}")
print(f" Black patient accuracy: {worst_case_results['Black_accuracy']:.1%}")
print(f" Hispanic patient accuracy: {worst_case_results['Hispanic_accuracy']:.1%}")

Safety threshold: Worst-case performance must meet minimum safety requirements, not just average.

2. Boundary Condition Testing

Test inputs at extremes:

Input Type	Example	Why It Matters
Out of range	Age = 150, Heart rate = 0	How does model handle impossible values?
Missing data	50% of features missing	Graceful degradation or catastrophic failure?
Corrupted inputs	Image with noise, text with typos	Real-world data is messy
Adversarial examples	Small perturbations designed to fool model	Robustness to attacks
OOD inputs	Data from different population/site	Detects distribution shift?

def boundary_condition_testing(model):
 """
 Test model behavior on edge cases
 """
 test_results = {}

 # Test 1: Out-of-range values
 invalid_input = {
  'age': 200, # Impossible
  'heart_rate': -50, # Invalid
  'temperature': 150 # Lethal
 }

 try:
  prediction = model.predict(invalid_input)
  test_results['invalid_input'] = f"[FAIL] Model accepted invalid input (predicted: {prediction})"
 except ValueError as e:
  test_results['invalid_input'] = f"[PASS] Model rejected invalid input ({e})"

 # Test 2: Missing data
 for missing_pct in [0.1, 0.3, 0.5, 0.7, 0.9]:
  partial_input = create_missing_data(test_data, missing_pct)
  accuracy = evaluate(model, partial_input)
  test_results[f'missing_{int(missing_pct*100)}pct'] = accuracy

 # Test 3: Adversarial robustness
 adversarial_accuracy = evaluate_adversarial_robustness(model, test_data)
 test_results['adversarial_robustness'] = adversarial_accuracy

 # Test 4: OOD detection
 ood_data = load_ood_dataset() # Data from different hospital
 ood_detected = model.detect_ood(ood_data)
 test_results['ood_detection_rate'] = ood_detected.mean()

 return test_results

3. Adversarial Robustness Evaluation

Attacks to test:

Fast Gradient Sign Method (FGSM):

def fgsm_attack(model, image, label, epsilon=0.03):
 """
 Generate adversarial example using FGSM

 Adds small perturbation in direction of gradient
 """
 # Compute gradient of loss with respect to input
 with tf.GradientTape() as tape:
  tape.watch(image)
  prediction = model(image)
  loss = loss_function(label, prediction)

 gradient = tape.gradient(loss, image)

 # Create adversarial image
 adversarial_image = image + epsilon * tf.sign(gradient)

 # Clip to valid range [0, 1]
 adversarial_image = tf.clip_by_value(adversarial_image, 0, 1)

 return adversarial_image

# Evaluate robustness
original_accuracy = evaluate(model, test_images, test_labels)
adversarial_images = [fgsm_attack(model, img, lbl) for img, lbl in zip(test_images, test_labels)]
adversarial_accuracy = evaluate(model, adversarial_images, test_labels)

print(f"Original accuracy: {original_accuracy:.1%}")
print(f"Adversarial accuracy: {adversarial_accuracy:.1%}")
print(f"Robustness gap: {original_accuracy - adversarial_accuracy:.1%}")

if adversarial_accuracy < 0.5:
 print("[WARNING] SAFETY CONCERN: Model highly vulnerable to adversarial attacks")

Projected Gradient Descent (PGD): Stronger iterative attack

Certified defenses: Formal guarantees of robustness within epsilon-ball

4. Stress Testing

Test system under adverse conditions:

class SafetyStressTest:
 """
 Comprehensive stress testing for AI safety
 """

 def test_high_load(self, model, num_requests=10000):
  """Can system handle peak load?"""
  start = time.time()
  predictions = model.batch_predict(num_requests)
  latency = (time.time() - start) / num_requests

  if latency > 1.0: # > 1 second per prediction
   return f"[FAIL] Latency {latency:.2f}s exceeds 1s requirement"
  return f"[PASS] Latency {latency:.2f}s"

 def test_degraded_input_quality(self, model, test_data):
  """Performance with poor-quality inputs"""
  results = {}

  # Add increasing amounts of noise
  for noise_level in [0.1, 0.3, 0.5]:
   noisy_data = test_data + noise_level * np.random.randn(*test_data.shape)
   accuracy = evaluate(model, noisy_data)
   results[f'noise_{noise_level}'] = accuracy

   if accuracy < 0.7: # Arbitrary safety threshold
    results[f'noise_{noise_level}_status'] = "[WARNING] UNSAFE"

  return results

 def test_concurrent_failures(self, model):
  """What happens when multiple things go wrong?"""
  # Simulate database connection failure during high load
  # Simulate network latency while processing corrupted input
  # etc.
  pass

 def test_recovery(self, model):
  """Can system recover from failures?"""
  # Simulate crash and restart
  # Verify state restoration
  # Check for data loss
  pass

5. Subgroup Analysis

Adapted from Ethics, Bias, and Equity in Healthcare AI, but safety-focused:

def safety_subgroup_analysis(model, X, y, subgroups):
 """
 Evaluate safety metrics across demographic subgroups

 Focus on clinical safety, not just fairness
 """
 safety_metrics = {}

 for group, mask in subgroups.items():
  # Standard metrics
  accuracy = accuracy_score(y[mask], model.predict(X[mask]))

  # Safety-critical metrics
  # False negatives in high-severity cases
  high_severity_mask = (y[mask] == 'high_risk')
  false_negative_rate = 1 - recall_score(
   y[mask][high_severity_mask],
   model.predict(X[mask][high_severity_mask])
  )

  safety_metrics[group] = {
   'accuracy': accuracy,
   'false_negative_rate': false_negative_rate,
   'sample_size': mask.sum()
  }

 # Check for subgroup safety disparities
 fnr_values = [m['false_negative_rate'] for m in safety_metrics.values()]
 max_fnr_disparity = max(fnr_values) - min(fnr_values)

 if max_fnr_disparity > 0.1: # 10% disparity threshold
  print(f"[WARNING] SAFETY ALERT: {max_fnr_disparity:.1%} disparity in false negative rates across subgroups")

 return safety_metrics

Safety-Critical Design Patterns

Safety must be designed in, not tested in. Architectural patterns that build safety into AI systems include:

Human-in-the-Loop (HITL) Systems

Levels of automation (adapted from aviation):

Level	Name	Description	AI Role	Human Role	Healthcare Example
1	Manual	Human decides, no AI	None	Decision maker	Traditional diagnosis
2	Decision support	AI suggests, human decides	Advisor	Decision maker	Diagnostic suggestions that clinician reviews
3	Conditional automation	AI decides, human approves before action	Recommender	Approver	Treatment plan requires clinician sign-off
4	High automation	AI acts, human monitors and can intervene	Autonomous actor	Supervisor	Automated insulin pump with override
5	Full automation	AI acts, no human involvement	Fully autonomous	None	(Rare in healthcare)

Safety principle: Higher automation requires higher reliability.

When to use each level:

Level 2 (Decision support): Most current healthcare AI
- Diagnostic aids
- Risk prediction
- Treatment suggestions
Level 3 (Human approval): When AI mistakes would be catastrophic but human can catch
- Chemotherapy dosing
- Surgical planning
- Critical medication changes
Level 4 (Human monitoring): When immediate action required, human response too slow
- Ventilator management
- Insulin pumps
- Anesthesia control
Level 5 (Full automation): Almost never appropriate in healthcare (except routine tasks)

The Ironies of Automation

Bainbridge, 1983 identified paradoxes:

Automation reduces human engagement → When automation fails, humans are out of the loop and cannot respond
Automation handles routine cases → Humans only see hard cases, losing routine expertise
More automation = more complex failures → When AI fails, failures are harder for humans to diagnose

Application to healthcare AI:

Do not automate routine cases and leave only hard cases to humans
Maintain human expertise through regular engagement
Design for graceful degradation, not sudden failure

Fallback Systems and Graceful Degradation

Defense in depth: Multiple independent safety layers.

Primary-Backup Architecture

class SafeAISystem:
 """
 AI system with fallback to rule-based or human decision
 """

 def __init__(self, ai_model, rule_based_backup, confidence_threshold=0.8):
  self.ai_model = ai_model
  self.rule_based_backup = rule_based_backup
  self.confidence_threshold = confidence_threshold
  self.system_health = "operational"

 def predict(self, input_data):
  """
  Make prediction with safety fallbacks
  """
  # Health check
  if self.system_health != "operational":
   return self.safe_mode_prediction(input_data)

  # Try AI prediction
  try:
   prediction = self.ai_model.predict(input_data)
   confidence = prediction.confidence

   # Check confidence threshold
   if confidence >= self.confidence_threshold:
    return {'prediction': prediction, 'source': 'ai'}
   else:
    # Low confidence → Fallback
    logging.warning(f"Low confidence ({confidence:.2f}) → Falling back to rule-based")
    return {'prediction': self.rule_based_backup.predict(input_data), 'source': 'rule_based'}

  except Exception as e:
   # AI failure → Fallback
   logging.error(f"AI prediction failed: {e} → Falling back to rule-based")
   return {'prediction': self.rule_based_backup.predict(input_data), 'source': 'rule_based'}

 def safe_mode_prediction(self, input_data):
  """
  Conservative predictions when system degraded
  """
  # Use most conservative/safe default
  return {
   'prediction': 'manual_review_required',
   'confidence': 0.0,
   'source': 'manual',
   'reason': 'System in safe mode - human evaluation required'
  }

 def health_check(self):
  """
  Continuous system health monitoring
  """
  try:
   # Check model availability
   test_prediction = self.ai_model.predict(self.get_test_input())

   # Check latency
   import time
   start = time.time()
   _ = self.ai_model.predict(self.get_test_input())
   latency = time.time() - start

   if latency > 5.0: # 5 second threshold
    self.system_health = "degraded"
    return "degraded"

   self.system_health = "operational"
   return "operational"

  except Exception as e:
   self.system_health = "failed"
   return "failed"

# Example usage
ai_system = SafeAISystem(
 ai_model=neural_network_model,
 rule_based_backup=clinical_decision_rules,
 confidence_threshold=0.8
)

# Prediction with automatic fallback
result = ai_system.predict(patient_data)

if result['source'] == 'rule_based':
 print("[WARNING] AI fallback activated - using rule-based system")
elif result['source'] == 'manual':
 print("[WARNING] Safe mode - manual review required")

Ensemble Methods for Safety

Diverse models reduce common-mode failures:

class SafetyEnsemble:
 """
 Ensemble with safety-focused disagreement handling
 """

 def __init__(self, models, min_agreement=0.75):
  self.models = models
  self.min_agreement = min_agreement

 def predict(self, input_data):
  """
  Predict with ensemble agreement checking
  """
  predictions = [model.predict(input_data) for model in self.models]

  # Check agreement
  agreement = self.calculate_agreement(predictions)

  if agreement >= self.min_agreement:
   # High agreement → Use ensemble prediction
   return {
    'prediction': self.aggregate(predictions),
    'confidence': agreement,
    'source': 'ensemble'
   }
  else:
   # Low agreement → Flag for human review
   return {
    'prediction': 'uncertain',
    'confidence': agreement,
    'source': 'manual_review_required',
    'reason': f'Low model agreement ({agreement:.2%})'
   }

 def calculate_agreement(self, predictions):
  """
  Fraction of models that agree with majority
  """
  from collections import Counter
  counts = Counter(predictions)
  majority_count = counts.most_common(1)[0][1]
  return majority_count / len(predictions)

 def aggregate(self, predictions):
  """
  Aggregate predictions (majority vote, averaging, etc.)
  """
  from collections import Counter
  return Counter(predictions).most_common(1)[0][0]

Safety Monitoring and Circuit Breakers

Continuous monitoring for safety signals:

class SafetyMonitor:
 """
 Real-time safety monitoring with automatic circuit breakers
 """

 def __init__(self, model, safety_thresholds):
  self.model = model
  self.thresholds = safety_thresholds
  self.metrics_history = []
  self.circuit_breaker_active = False

 def monitor_prediction(self, input_data, prediction, ground_truth=None):
  """
  Monitor individual prediction for safety concerns
  """
  alerts = []

  # Check 1: Confidence too low
  if prediction['confidence'] < self.thresholds['min_confidence']:
   alerts.append(f"Low confidence: {prediction['confidence']:.2f}")

  # Check 2: OOD detection
  ood_score = self.model.ood_detector.score(input_data)
  if ood_score > self.thresholds['max_ood_score']:
   alerts.append(f"Out-of-distribution input: {ood_score:.2f}")

  # Check 3: If ground truth available, check error
  if ground_truth is not None:
   error = (prediction['value'] != ground_truth)
   self.metrics_history.append({'error': error, 'timestamp': time.time()})

   # Check recent error rate
   recent_error_rate = self.calculate_recent_error_rate()
   if recent_error_rate > self.thresholds['max_error_rate']:
    alerts.append(f"Elevated error rate: {recent_error_rate:.1%}")
    self.activate_circuit_breaker()

  if alerts:
   self.log_safety_alert(alerts, input_data, prediction)

  return alerts

 def calculate_recent_error_rate(self, window_minutes=60):
  """
  Calculate error rate in recent time window
  """
  now = time.time()
  window_start = now - (window_minutes * 60)

  recent_errors = [
   m['error'] for m in self.metrics_history
   if m['timestamp'] > window_start
  ]

  if len(recent_errors) == 0:
   return 0.0

  return sum(recent_errors) / len(recent_errors)

 def activate_circuit_breaker(self):
  """
  Stop using AI model when safety threshold exceeded
  """
  self.circuit_breaker_active = True
  logging.critical("CIRCUIT BREAKER ACTIVATED - AI model disabled")
  logging.critical("System falling back to safe mode until manual review")

  # Alert on-call team
  self.send_alert_to_oncall()

 def send_alert_to_oncall(self):
  """Send urgent alert to on-call safety team"""
  # Integration with paging system
  pass

# Example usage
monitor = SafetyMonitor(
 model=sepsis_model,
 safety_thresholds={
  'min_confidence': 0.7,
  'max_ood_score': 0.5,
  'max_error_rate': 0.2 # 20% error rate
 }
)

# Monitor each prediction
prediction = model.predict(patient_data)
alerts = monitor.monitor_prediction(patient_data, prediction, ground_truth)

if monitor.circuit_breaker_active:
 # Use fallback system
 prediction = fallback_system.predict(patient_data)

Safety Interlocks and Guardrails

Prevent unsafe actions before they occur:

class SafetyInterlocks:
 """
 Hard constraints preventing unsafe model behavior
 """

 def validate_input(self, input_data):
  """
  Validate input before allowing prediction
  """
  errors = []

  # Range checks
  if input_data['age'] < 0 or input_data['age'] > 120:
   errors.append(f"Invalid age: {input_data['age']}")

  if input_data['heart_rate'] < 20 or input_data['heart_rate'] > 300:
   errors.append(f"Invalid heart rate: {input_data['heart_rate']}")

  # Completeness checks
  required_fields = ['age', 'heart_rate', 'blood_pressure', 'temperature']
  missing = [f for f in required_fields if f not in input_data or input_data[f] is None]
  if missing:
   errors.append(f"Missing required fields: {missing}")

  # Consistency checks
  if input_data['systolic_bp'] < input_data['diastolic_bp']:
   errors.append("Systolic BP < Diastolic BP (impossible)")

  if errors:
   raise ValueError(f"Input validation failed: {errors}")

 def validate_output(self, prediction, input_data):
  """
  Validate prediction before returning to clinician
  """
  warnings = []

  # Confidence check
  if prediction['confidence'] < 0.5:
   warnings.append("Low confidence prediction")

  # Plausibility check
  # Example: Risk score shouldn't increase dramatically for stable patient
  if 'previous_risk' in input_data:
   change = abs(prediction['risk'] - input_data['previous_risk'])
   if change > 0.5: # >50% change
    warnings.append(f"Large risk change: {input_data['previous_risk']:.2f} → {prediction['risk']:.2f}")

  # Output bounds check
  if prediction['risk'] < 0 or prediction['risk'] > 1:
   raise ValueError(f"Risk score out of bounds: {prediction['risk']}")

  if warnings:
   prediction['warnings'] = warnings

  return prediction

 def cross_check_with_rules(self, prediction, input_data):
  """
  Cross-check AI prediction against clinical decision rules
  """
  # Example: qSOFA score for sepsis
  qsofa = self.calculate_qsofa(input_data)

  if qsofa >= 2 and prediction['sepsis_risk'] < 0.5:
   # qSOFA suggests sepsis but AI says low risk
   return {
    **prediction,
    'cross_check_alert': f"AI predicts low risk but qSOFA={qsofa} suggests high risk - manual review recommended"
   }

  return prediction

Safety Design Principles Summary

Fail-safe: Default to safe state on failure (defer to human)
Defense in depth: Multiple independent safety layers
Diversity: Different approaches prevent common-mode failures
Conservative: When uncertain, err on side of caution
Monitoring: Continuous surveillance for safety signals
Circuit breakers: Automatic shutoff when thresholds exceeded
Graceful degradation: Reduce functionality, do not fail completely
Human-in-the-loop: Appropriate level of automation for risk level

Deferral as a Safety Mechanism

Perfect accuracy is unachievable in healthcare AI, but fortunately, it is also not required. Safety emerges not from flawless performance but from knowing when not to act. A well-designed AI system should recognize its limitations and explicitly defer to human judgment when confidence is low or the clinical scenario falls outside its validated scope (Azad et al., Nature Medicine, 2026).

The deferral principle:

Each healthcare AI task requires an explicit harm budget: the maximum acceptable error rate that still yields net benefit. Critical metrics must align with real-world consequences:

Under-triage rates for emergency systems that might misclassify urgent cases as low-acuity
Omission rates for documentation tools that could miss medications and allergies
Unsafe non-deferral rates for patient messaging systems that provide confident responses when escalation was warranted

The goal is appropriate caution paired with measurable benefit. An AI system that defers appropriately on 15% of cases while performing well on the remaining 85% may be safer and more useful than one that attempts to answer every query with false confidence.

Evaluating deferral capability:

When assessing AI systems for public health deployment, ask: Can this system recognize its limitations and appropriately request human intervention? Systems that fail this test lack readiness for population-level deployment. Deferral awareness, the ability to know what the system does not know, is a first-class safety requirement, not a limitation to hide.

Post-Market Safety Surveillance

Safety validation continues after deployment through continuous safety monitoring.

Why Post-Market Surveillance Is Critical for AI

Traditional medical devices are usually static. AI systems can change, and their environment changes around them:

Data drift: Population changes over time
Concept drift: Relationship between features and outcome changes
Software updates: Model retraining, architecture changes
Integration changes: New EHR versions, workflow updates

Result: Performance validated pre-deployment can degrade silently post-deployment.

Incident Detection and Response

Incident Severity Classification

Severity tiers for internal escalation (regulatory deadlines appear under Regulatory Reporting Requirements):

Severity	Definition	Examples	Internal escalation target
Critical	Death or serious injury	AI missed cancer diagnosis → Patient died	Immediate (within 24h)
Major	Significant harm, prolonged recovery	AI-guided surgery error → Complications requiring additional procedures	Within 30 days
Minor	Temporary harm, full recovery	False alarm causing unnecessary procedure	Annual summary
Near miss	Potential for harm, none occurred	AI error caught by clinician before action	Track for trending

Incident Response Protocol

class IncidentResponse:
 """
 Systematic incident response for AI safety events
 """

 def __init__(self, system_name):
  self.system_name = system_name
  self.incidents = []

 def report_incident(self, severity, description, patient_impact, root_cause_preliminary):
  """
  Report and initiate response to safety incident
  """
  incident = {
   'timestamp': datetime.now(),
   'severity': severity,
   'description': description,
   'patient_impact': patient_impact,
   'root_cause_preliminary': root_cause_preliminary,
   'status': 'reported'
  }

  self.incidents.append(incident)

  # Immediate actions based on severity
  if severity == 'critical':
   self.critical_incident_response(incident)
  elif severity == 'major':
   self.major_incident_response(incident)

  # All incidents trigger investigation
  self.initiate_investigation(incident)

 def critical_incident_response(self, incident):
  """
  Immediate response to critical safety event
  """
  # 1. Activate circuit breaker if systematic issue
  if self.is_systematic_failure(incident):
   self.activate_circuit_breaker()

  # 2. Alert on-call safety team
  self.page_safety_team(incident)

  # 3. Start regulatory notification (internal 24h target; statutory deadlines vary)
  self.notify_regulators(incident, deadline_hours=24)

  # 4. Quarantine affected predictions
  self.quarantine_predictions(incident)

  # 5. Initiate root cause analysis
  self.initiate_rca(incident)

 def is_systematic_failure(self, incident):
  """
  Determine if incident indicates systematic problem
  """
  # Check for similar recent incidents
  recent_similar = [
   i for i in self.incidents[-10:]
   if i['root_cause_preliminary'] == incident['root_cause_preliminary']
  ]

  if len(recent_similar) >= 3:
   return True # Trend suggests systematic issue

  return False

 def initiate_investigation(self, incident):
  """
  Start formal investigation
  """
  investigation = {
   'incident_id': len(self.incidents),
   'team': ['data_scientist', 'clinician', 'safety_officer', 'quality_assurance'],
   'timeline': '30 days for major/critical, 90 days for minor',
   'deliverables': ['root_cause_analysis', 'corrective_actions', 'preventive_actions']
  }

  return investigation

# Example
incident_response = IncidentResponse(system_name="Sepsis Prediction Model")

# Critical incident reported
incident_response.report_incident(
 severity='critical',
 description="Model failed to alert for patient who developed septic shock",
 patient_impact="Patient required ICU admission, prolonged recovery",
 root_cause_preliminary="Data drift - patient demographics outside training distribution"
)

Root Cause Analysis (RCA) for AI Failures

Five Whys method:

Incident: AI missed sepsis case

Why #1: Why did AI miss sepsis?
→ Patient's early sepsis presented atypically

Why #2: Why didn't AI detect atypical presentation?
→ Training data had few atypical cases

Why #3: Why did training data lack atypical cases?
→ Training data from single academic center with specific patient population

Why #4: Why wasn't data diversity validated?
→ Validation focused on overall metrics, not subgroup coverage

Why #5: Why wasn't subgroup validation required?
→ Validation protocol didn't include diversity assessment

ROOT CAUSE: Inadequate validation protocol lacking diversity requirements

Corrective Action: Update validation to require minimum sample sizes across demographic and clinical subgroups

Preventive Action: Implement ongoing diversity monitoring in post-market surveillance

Fishbone Diagram (Ishikawa) for AI Failures

         AI Failure (Missed Sepsis)
           |
   People    Process  |  Technology  Data
   |     |   |   |    |
 Alert fatigue  No FMEA done  | Model too simple Data drift
 Insufficient   Inadequate  | No OOD detection Unrepresentative
  training   validation  | No monitoring  training set

Continuous Performance Monitoring

class PostMarketSurveillance:
 """
 Continuous monitoring of AI safety and performance
 """

 def __init__(self, model, baseline_metrics):
  self.model = model
  self.baseline_metrics = baseline_metrics
  self.current_metrics = {}
  self.degradation_threshold = 0.1 # 10% degradation triggers alert

 def monitor_performance(self, predictions, ground_truth):
  """
  Track performance metrics over time
  """
  from sklearn.metrics import roc_auc_score, accuracy_score

  # Calculate current metrics
  self.current_metrics = {
   'auroc': roc_auc_score(ground_truth, predictions),
   'accuracy': accuracy_score(ground_truth, (predictions > 0.5)),
   'sample_size': len(predictions),
   'timestamp': datetime.now()
  }

  # Check for degradation
  for metric in ['auroc', 'accuracy']:
   baseline = self.baseline_metrics[metric]
   current = self.current_metrics[metric]
   degradation = (baseline - current) / baseline

   if degradation > self.degradation_threshold:
    self.alert_degradation(metric, baseline, current, degradation)

 def monitor_data_drift(self, new_data, reference_data):
  """
  Detect distribution shift using statistical tests
  """
  from scipy.stats import ks_2samp

  drift_detected = {}

  for feature in new_data.columns:
   # Kolmogorov-Smirnov test
   statistic, p_value = ks_2samp(
    reference_data[feature],
    new_data[feature]
   )

   if p_value < 0.05: # Significant drift
    drift_detected[feature] = {
     'statistic': statistic,
     'p_value': p_value
    }

  if drift_detected:
   self.alert_data_drift(drift_detected)

  return drift_detected

 def monitor_subgroup_performance(self, predictions, ground_truth, subgroups):
  """
  Ensure no subgroup experiencing degradation
  """
  subgroup_metrics = {}

  for group, mask in subgroups.items():
   if mask.sum() > 30: # Minimum sample size
    subgroup_metrics[group] = {
     'auroc': roc_auc_score(ground_truth[mask], predictions[mask]),
     'sample_size': mask.sum()
    }

  # Check for subgroup disparities
  aurocs = [m['auroc'] for m in subgroup_metrics.values()]
  disparity = max(aurocs) - min(aurocs)

  if disparity > 0.1: # 10% disparity threshold
   self.alert_subgroup_disparity(subgroup_metrics, disparity)

  return subgroup_metrics

 def generate_periodic_safety_update(self, period='quarterly'):
  """
  Periodic Safety Update Report (PSUR) as required by EU MDR
  """
  report = {
   'period': period,
   'timestamp': datetime.now(),
   'system': self.model.name,
   'deployment_stats': {
    'total_predictions': self.get_total_predictions(),
    'active_sites': self.get_active_sites(),
    'user_count': self.get_user_count()
   },
   'performance_summary': {
    'baseline_auroc': self.baseline_metrics['auroc'],
    'current_auroc': self.current_metrics['auroc'],
    'change': self.current_metrics['auroc'] - self.baseline_metrics['auroc']
   },
   'safety_events': {
    'critical': self.count_incidents('critical'),
    'major': self.count_incidents('major'),
    'minor': self.count_incidents('minor'),
    'near_miss': self.count_incidents('near_miss')
   },
   'corrective_actions': self.get_corrective_actions(),
   'recommendations': self.generate_recommendations()
  }

  return report

Regulatory Reporting Requirements

FDA: MedWatch and MAUDE

When to report:

Death or serious injury caused by or associated with device: Within 30 days (5 days if unexpected death)
Malfunction that could cause death/serious injury: Within 30 days

What to report:

Device identification (model, version, serial number for AI: model hash)
Patient outcomes
Description of event
Suspected cause
Corrective actions taken

EU: Vigilance System

Serious incidents must be reported to competent authority:

2 days for a serious public health threat
10 days for death or unanticipated serious deterioration in health
15 days for other serious incidents

Periodic Safety Update Reports (PSUR):

Class IIb and III devices: at least annually
Class IIa devices: at least every two years

What Constitutes “Serious Incident” for AI

Patient death or serious injury attributed to or contributed by AI system
Patient death/injury where AI could have prevented but failed to (false negative)
Patient harm from unnecessary intervention due to false positive
Systematic error affecting multiple patients
Performance degradation below safety threshold

Gray area: When AI is “decision support” (clinician makes final call), attribution is unclear. Regulators increasingly expect reporting even when AI is indirect contributor.

Organizational Safety Culture

Technology alone cannot ensure safety. Organizations need culture, processes, and governance supporting safety.

Safety Management Systems

Components of effective safety management:

Safety policy and objectives
- Leadership commitment
- Clear safety priorities
- Resources allocated to safety
Safety roles and responsibilities
- Safety officer role
- AI safety review board
- Incident response team
Safety risk management
- FMEA and hazard analysis
- Risk registers
- Periodic risk reviews
Safety assurance
- Post-market surveillance
- Audits and inspections
- Continuous improvement
Safety promotion
- Training and education
- Communication of safety information
- Just culture and reporting

Training and Competency

Clinicians must understand:

How AI works (conceptually)
What AI can and cannot do
Limitations and failure modes
How to use AI safely
What to do when AI fails or gives unexpected results

Developers must understand:

Clinical context and workflows
Consequences of errors
Safety-critical systems engineering
Regulatory requirements
Ethical implications

Example training curriculum for clinicians:

Module 1: AI Basics (30 min)
- What is machine learning?
- How does our model work?
- What data was it trained on?

Module 2: Using the AI System (45 min)
- Workflow integration
- Interpreting predictions
- Understanding confidence scores
- Hands-on practice

Module 3: Limitations and Failure Modes (30 min)
- Known failure modes
- When to override AI
- Reporting errors and near-misses

Module 4: Case Studies (30 min)
- Real examples of AI failures
- Lessons learned
- Discussion and Q&A

Assessment:
- Written quiz (80% passing)
- Practical simulation (demonstrate safe use)
- Annual refresher required

Just Culture vs. Blame Culture

Aviation safety lesson: Punishing errors drives them underground. Just culture encourages reporting.

Just Culture principles:

Distinguish errors from violations
- Honest mistakes → Learning opportunity
- Reckless behavior → Disciplinary action
Encourage reporting
- Blame-free for genuine errors
- Confidential reporting systems
- Recognition for reporting near-misses
System improvements over individual blame
- Why did system allow error to occur?
- What barriers failed?
- How can we prevent recurrence?

Application to AI safety:

BLAME CULTURE        JUST CULTURE
"Dr. Smith ignored the AI alert" →  "Why was alert rate so high that ignoring became norm?"
"Data scientist didn't test edge cases" → "Why didn't our validation protocol require edge case testing?"
"Hospital didn't monitor performance" → "What barriers prevented effective monitoring?"

Building Safety Culture

Leadership commitment: Senior leadership visibly prioritizes safety
Psychological safety: Team members feel safe reporting concerns
Learning mindset: View failures as learning opportunities
Resources: Adequate time and budget for safety activities
Recognition: Celebrate safety successes, recognize reporting
Transparency: Share safety data openly (within organization)
Accountability: Clear roles and responsibilities for safety
Continuous improvement: Regular safety reviews and updates

Key Takeaways

Safety ≠ Security ≠ Quality: Distinct dimensions requiring different frameworks
Technical excellence ≠ Safe deployment: High accuracy does not guarantee safety
Fail-safe by design: Build safety into architecture (HITL, fallbacks, monitoring)
Multiple barriers: Defense in depth prevents single point of failure
Proactive hazard analysis: FMEA and risk assessment before deployment, not after harm
Validation beyond ML metrics: Worst-case analysis, boundary testing, adversarial robustness
Continuous monitoring: Safety validation does not end at deployment; it requires ongoing surveillance
Regulatory compliance is minimum: FDA/EU requirements are floor, not ceiling
Learn from failures: Both yours and others’ (see The AI Morgue)
Organizational culture matters: Technology cannot overcome culture that does not prioritize safety
Edge cases matter more than averages: Safety depends on worst-case performance
Human-centered design: Appropriate autonomy level, interpretability, workflow integration

Hands-On Exercise: Conduct Safety FMEA

Scenario: Your public health department is deploying an AI model to predict COVID-19 hospitalization risk. The model will be used to prioritize antiviral treatments during supply shortages.

Context:

Input: Patient demographics, comorbidities, vaccination status, symptoms
Output: Risk score (0–1) for hospitalization within 14 days
Action: High-risk patients (score > 0.7) offered monoclonal antibodies
Setting: Outpatient clinics, emergency departments
Users: Physicians, nurse practitioners, physician assistants

Task 1: Identify Failure Modes (20 minutes)

For each category, identify at least 2 potential failure modes:

Technical failures:

Data drift
OOD inputs
Adversarial examples
Training data issues

Operational failures:

Misuse/off-label use
Automation bias
Alert fatigue
Workflow integration

Systemic failures:

Cascading failures
Common-cause failures
Equity issues

Task 2: Complete FMEA Table (30 minutes)

For each failure mode, assess:

Failure Mode	Cause	Effect	Severity (1–5)	Occurrence (1–5)	Detection (1–5)	RPN
[Your failure mode]

Task 3: Design Mitigations (20 minutes)

For failure modes with RPN > 50, design specific mitigation strategies:

Design changes
Protective measures
Monitoring and detection
Response protocols

Task 4: Create Monitoring Plan (20 minutes)

Design post-deployment safety monitoring:

What metrics to track:

Performance metrics (AUROC, calibration)
Safety metrics (false negative rate in high-risk groups)
Operational metrics (alert burden, override rate)
Equity metrics (performance across demographics)

How to detect degradation:

Automated alerts
Statistical process control
Manual review frequency

Response protocol:

Degradation thresholds
Escalation procedures
Circuit breaker criteria

Task 5: Draft Incident Response Protocol (10 minutes)

Create incident response workflow:

How are incidents detected and reported?
Who is notified?
What immediate actions are taken?
How is investigation conducted?
When are regulators notified?
How is learning disseminated?

Check Your Understanding

Test your knowledge of the key concepts from this chapter. Click “Show Answer” to reveal the correct response and explanation.

Question 1: AI Accuracy and Safety Validation

A COVID-19 diagnostic AI achieves 97% accuracy in validation. Why might this be insufficient for safety clearance?

Answer: Multiple safety concerns beyond accuracy:

Accuracy does not reveal failure mode distribution - 3% errors might all be false negatives (missing COVID cases), which is more dangerous than false positives
Aggregate metric masks subgroup performance - Model might be 99% accurate on typical presentations but 70% accurate on atypical cases (elderly, immunocompromised)
Test set may not represent edge cases - Validation data might not include poor image quality, operator errors, or equipment variations seen in real clinics
No assessment of operational impact - High accuracy with 30% false positive rate could cause alert fatigue and system abandonment
Missing safety-specific validation - No adversarial robustness testing, OOD detection, worst-case analysis, or boundary condition testing
No consideration of context - In low-prevalence settings, even 97% accuracy can have very low positive predictive value

Safety clearance requires: A documented FMEA, worst-case analysis, subgroup validation, operational testing, and safety-critical design (fallbacks, monitoring, HITL).

Question 2: Autonomous AI Safety Requirements

A hospital wants to deploy fully autonomous AI (Level 5 automation) for sepsis prediction, no human review, automatic treatment orders. What safety requirements must be met?

Answer: The proposal cannot be approved from an autonomy label alone. Requirements depend on intended use, the treatment action, device classification, applicable FDA pathway, evidence, workflow, and local law.

Technical requirements:

Prespecified performance and subgroup acceptance criteria tied to clinical consequences
Robustness and out-of-distribution evaluation for foreseeable conditions
Continuous monitoring, circuit breakers, and validated fallback procedures
Verification of safety properties where feasible

Regulatory requirements:

Determine device status, classification, and premarket pathway from the intended use and product-specific regulation
Generate evidence sufficient for safety and effectiveness in the intended workflow; no universal rule requires superiority for every product
Meet applicable postmarket and change-control obligations

Operational requirements:

A documented hazard analysis with locally justified acceptance criteria; RPN is a prioritization aid, not a universal regulatory threshold
Incident response protocol with immediate escalation
Human monitoring capability (contradiction: “autonomous” yet monitored)
Clear liability assignment

Ethical requirements:

Determine disclosure, consent, and opt-out requirements from the intervention, jurisdiction, and institutional policy
Demonstration that benefits significantly outweigh risks

Realistic recommendation: Do not automate treatment orders until product-specific evidence, regulatory review, human-factors testing, escalation authority, and failure controls support the intended use. A human-approval design is the safer starting architecture for this scenario.

Question 3: Incident Response for Model Degradation

Your AI model’s performance begins degrading after 6 months in production (AUROC dropped from 0.85 to 0.78). Walk through the incident response steps.

Answer: Systematic incident response:

Step 1: Detect and confirm (24–48 hours)

Automated monitoring alerts to degradation
Confirm with manual validation on recent data
Calculate magnitude and statistical significance of degradation
Assess impact: How many patients affected? Any harm occurred?

Step 2: Immediate response (Day 1)

Classify severity: Major (significant degradation) or Critical (if harm occurred)
Notify safety officer, clinical leadership, development team
DO NOT immediately disable system (could disrupt care) unless severe degradation
Increase human oversight temporarily (lower automation level)
Consider lowering confidence threshold (increase sensitivity at cost of specificity)

Step 3: Root cause investigation (Days 2–7)

Data drift analysis: Has population changed? New demographics, comorbidities?
Concept drift analysis: Have clinical practices changed? New protocols, medications?
System changes: EHR updates, integration changes, data pipeline issues?
Ground truth verification: Is degradation real or measurement artifact?

Step 4: Develop corrective actions (Days 7–14)

If data drift: Retrain model on recent data
If concept drift: Retrain with updated labels
If system issue: Fix technical problem
If measurement artifact: Correct monitoring system

Step 5: Validate correction (Days 14–21)

Test updated model on held-out recent data
Confirm performance restored
Perform safety validation (not just performance)
Subgroup analysis to ensure no equity issues

Step 6: Deploy correction and monitor closely (Day 21+)

Deploy updated model
Increase monitoring frequency temporarily
Verify performance in production
Document lessons learned

Step 7: Regulatory reporting

If harm occurred: FDA medical device report within 30 calendar days; EU vigilance report within 2–15 days depending on severity
If no harm: Include in periodic safety update report (PSUR)

Step 8: Preventive actions

Update monitoring to detect this drift pattern earlier
Implement automated retraining protocol
Update FMEA with new failure mode
Train team on lessons learned

Question 4: FMEA vs. Traditional ML Evaluation

Compare FMEA vs. traditional ML evaluation. When is each appropriate?

Answer:

Dimension	Traditional ML Evaluation	FMEA (Failure Mode and Effects Analysis)
Goal	Assess average performance	Identify potential failure modes and harms
Timing	After model training	Before deployment (proactive)
Scope	Model performance only	End-to-end system including operational context
Metrics	Accuracy, AUROC, precision, recall	Risk Priority Number (Severity × Occurrence × Detection)
Errors treated	All errors weighted equally	Errors weighted by clinical severity
Team	Data scientists	Multidisciplinary (clinicians, safety, QA, patients)
Output	Performance report	Risk mitigation plan

When to use ML evaluation:

Iterative model development
Comparing model architectures
Hyperparameter tuning
Model selection

When to use FMEA:

Before clinical deployment
After major system changes
Following incidents or near-misses
Annual safety reviews
Regulatory submissions (ISO 14971 requirement)

Why both are needed:

ML evaluation asks: “How accurate is the model?”

FMEA asks: “What can go wrong, how bad would it be, and how do we prevent it?”

Example:

ML evaluation: “Model has 90% sensitivity and 85% specificity”
FMEA: “5 failure modes identified, including ‘false negative in early-stage cancer’ with RPN=100 (upper end of moderate risk). Mitigations: lower threshold for high-risk demographics, ensemble with rule-based screening, manual review for borderline cases.”

Safe deployment requires more than ML evaluation.

Question 5: FDA vs. EU MDR Regulatory Requirements

Regulatory perspective: FDA vs. EU MDR safety requirements for AI. What are the key differences?

Answer:

Aspect	FDA (US)	EU MDR
Primary pathway	510(k) - Substantial Equivalence (most AI)	Conformity assessment with Notified Body
Clinical evidence	Can rely on equivalence to predicate device; less clinical data required	Requires robust clinical evaluation; equivalence harder to claim
Adaptive algorithms	Predetermined Change Control Plan (PCCP) allows updates without new submission	More restrictive; significant changes require new assessment
Post-market	Mandatory adverse event reporting (21 CFR Part 803); postmarket surveillance studies when FDA orders them (Section 522)	Mandatory Post-Market Clinical Follow-up (PMCF); Periodic Safety Update Reports (PSUR)
Transparency	Less stringent explainability requirements	MDR requires information on intended purpose, performance, and residual risks
Timeline	Faster (510(k) review goal: 90 FDA days, excluding time on hold)	Slower (12–18+ months with Notified Body)
Validation rigor	Analytical + clinical validation (clinical often retrospective)	Strong emphasis on prospective clinical evidence
Updates	PCCP allows planned updates pre-approved	Each significant change reassessed

Key philosophical difference:

FDA approach: Innovation-friendly, risk-based, allows incremental validation, trusts substantial equivalence

EU MDR approach: Precautionary, requires stronger evidence upfront, less reliance on equivalence, more stringent post-market requirements

Practical implications:

Development strategy: EU market requires more upfront clinical data; plan prospective studies early
Adaptive AI: If model will update frequently, FDA PCCP pathway more viable than EU
Cost: EU MDR compliance generally more expensive (Notified Body fees, extensive clinical evaluation)
Timeline: Plan 12-18 months for EU vs. 6-9 months for FDA 510(k)
Post-market: EU requires ongoing clinical follow-up and regular PSUR; budget for this

Divergence increasing: US, EU, and UK (post-Brexit) regulatory approaches diverging. AI companies may need separate validation strategies per market.

Question 6: Safety Interlocks for Chemotherapy AI

Design safety interlocks for chemotherapy dose recommendation AI. What checks must occur before allowing dosing?

Answer: Multi-layered safety interlocks:

Layer 1: Input Validation

def validate_chemotherapy_inputs(patient_data):
 checks_failed = []

 # Range checks
 if patient_data['weight'] < 20 or patient_data['weight'] > 200:
  checks_failed.append("Weight out of plausible range")

 if patient_data['bsa'] < 0.5 or patient_data['bsa'] > 3.0:
  checks_failed.append("Body surface area out of range")

 # Required labs present and recent
 required_labs = ['creatinine', 'bilirubin', 'wbc', 'platelets', 'hemoglobin']
 for lab in required_labs:
  if lab not in patient_data:
   checks_failed.append(f"Missing required lab: {lab}")
  elif patient_data[f'{lab}_date'] > 7: # days old
   checks_failed.append(f"Lab too old: {lab} ({patient_data[f'{lab}_date']} days)")

 # Organ function adequate for chemotherapy
 if patient_data['creatinine'] > 2.0:
  checks_failed.append("Renal function inadequate (Cr > 2.0)")

 if patient_data['bilirubin'] > 2.0:
  checks_failed.append("Hepatic function inadequate (Bili > 2.0)")

 if checks_failed:
  raise ValueError(f"Input validation failed: {checks_failed}")

 return True

Layer 2: Dose Bounds Checking

def validate_dose_recommendation(drug, dose, patient_bsa):
 # Check against standard dosing ranges
 standard_doses = {
  'doxorubicin': {'min': 40, 'max': 75, 'unit': 'mg/m2'},
  'cyclophosphamide': {'min': 500, 'max': 1500, 'unit': 'mg/m2'},
  # etc.
 }

 dose_per_bsa = dose / patient_bsa

 if dose_per_bsa < standard_doses[drug]['min']:
  raise ValueError(f"Dose below standard range: {dose_per_bsa} < {standard_doses[drug]['min']} {standard_doses[drug]['unit']}")

 if dose_per_bsa > standard_doses[drug]['max']:
  raise ValueError(f"[CRITICAL] Dose above standard range: {dose_per_bsa} > {standard_doses[drug]['max']} {standard_doses[drug]['unit']}")

 return True

Layer 3: Cross-Check with Clinical Guidelines

def cross_check_with_guidelines(diagnosis, regimen, patient):
 # Verify regimen appropriate for diagnosis
 approved_regimens = get_nccn_guidelines(diagnosis)

 if regimen not in approved_regimens:
  return {
   'alert': 'WARNING',
   'message': f'Regimen {regimen} not in NCCN guidelines for {diagnosis}',
   'action': 'require_pharmacist_approval'
  }

 # Check for contraindications
 if 'doxorubicin' in regimen and patient['cardiac_ejection_fraction'] < 50:
  return {
   'alert': 'CONTRAINDICATION',
   'message': 'Doxorubicin contraindicated with EF < 50%',
   'action': 'block_order'
  }

 return {'alert': 'PASS'}

Layer 4: Drug Interaction Check

def check_drug_interactions(chemotherapy_drugs, current_medications):
 critical_interactions = []

 interaction_db = load_interaction_database()

 for chemo_drug in chemotherapy_drugs:
  for med in current_medications:
   interaction = interaction_db.check(chemo_drug, med)
   if interaction['severity'] == 'severe':
    critical_interactions.append({
     'drug1': chemo_drug,
     'drug2': med,
     'interaction': interaction['description'],
     'recommendation': interaction['recommendation']
    })

 if critical_interactions:
  return {
   'alert': 'CRITICAL INTERACTION',
   'interactions': critical_interactions,
   'action': 'require_oncologist_review'
  }

 return {'alert': 'PASS'}

Layer 5: Mandatory Human Review

class ChemotherapyAI:
 def recommend_dose(self, patient_data):
  # AI makes recommendation
  recommendation = self.model.predict(patient_data)

  # All safety interlocks MUST pass
  validate_chemotherapy_inputs(patient_data)
  validate_dose_recommendation(
   recommendation['drug'],
   recommendation['dose'],
   patient_data['bsa']
  )
  guideline_check = cross_check_with_guidelines(
   patient_data['diagnosis'],
   recommendation['regimen'],
   patient_data
  )
  interaction_check = check_drug_interactions(
   recommendation['drugs'],
   patient_data['current_meds']
  )

  # Add mandatory review flags
  recommendation['require_oncologist_approval'] = True
  recommendation['require_pharmacist_verification'] = True
  recommendation['safety_checks'] = {
   'input_validation': 'PASS',
   'dose_bounds': 'PASS',
   'guidelines': guideline_check,
   'interactions': interaction_check
  }

  # NEVER allow fully autonomous chemotherapy ordering
  recommendation['autonomous_ordering'] = False

  return recommendation

Key safety principles:

Multiple independent checks: Input validation, dose bounds, guidelines, interactions
Fail-safe: Any check failure blocks recommendation
Mandatory human approval: Oncologist AND pharmacist must review
Audit trail: Document all checks and approvals
Hard limits: Some checks (contraindications) block entirely, not just warn
Conservative defaults: When uncertain, require human review

Chemotherapy is Level 3 automation at most (AI recommends, human approves), never Level 4–5.

Discussion Questions

Ethics vs. Safety: When is it ethical to deploy AI with known failure modes? What risk level is acceptable?
Liability: When a safely designed AI still causes harm (no negligence, just residual risk), who should be liable: developer, hospital, clinician, or patient assumes risk?
Safety standards: Should healthcare AI face higher safety standards than commercial aviation? Why or why not? What about compared to pharmaceuticals?
Innovation vs. Safety: How do we balance innovation velocity (rapid deployment of beneficial AI) with safety rigor (extensive validation)? Is FDA too slow or too fast?
Proof of safety: Can we ever prove AI is “safe enough”? What is the evidentiary standard? How does this compare to drugs (RCTs) or procedures (surgical training)?
Automation levels: As AI becomes more capable, should we allow higher automation (Level 4–5), or should healthcare always require human decision-maker? What about non-critical decisions?
Patient agency: Should patients have right to opt out of AI-assisted care? Right to know when AI was used? Right to appeal AI decisions?
Failure transparency: Should AI failures be publicly disclosed (like aviation accidents)? Or does this create liability disincentives and chill innovation?

Further Resources

Essential Books

Safety-Critical Computer Systems by Neil Storey - Foundational text on safety-critical systems engineering
Engineering a Safer World by Nancy Leveson - Systems-theoretic approach to safety (STAMP methodology)
Medical Device Software Verification, Validation, and Compliance by David A. Vogel - IEC 62304 compliance and software validation
Normal Accidents by Charles Perrow - Why complex systems fail
Introduction to AI Safety, Ethics, and Society by Dan Hendrycks (2024, CRC Press) - Open-access textbook covering catastrophic AI risks, safety engineering, complex systems, and governance. Includes frameworks for tail events and black swans directly applicable to health AI safety

Key Papers

Regulatory Guidance

FDA

European Union

UK (MHRA)

Software and AI as a Medical Device Change Programme

Standards

IEC 62304:2006+AMD1:2015 - Medical device software lifecycle
ISO 14971:2019 - Risk management for medical devices
ISO 13485:2016 - Quality management systems for medical devices

Tools and Frameworks

Safety Validation

Foolbox - Python library for adversarial robustness testing
CleverHans - Library for adversarial examples
AI Verify - Toolkit for validating AI systems

Risk Management

IHI FMEA Tool - Failure mode analysis templates (adaptable for AI systems)
Risk Register Templates - ISO 14971 risk management

Monitoring

Evidently AI - Open-source ML monitoring (drift detection)
Whylabs - Data and ML monitoring platform
Fiddler - AI observability and monitoring

Online Courses

MIT 6.S191: Introduction to Deep Learning - Safety Lecture - AI safety fundamentals
Stanford CS329S: Machine Learning Systems Design - Deployment and monitoring
Fast.ai: Practical Deep Learning - Ethics Module - Practical AI ethics and safety

Next: AI Deployment in Healthcare → covers how to deploy AI that has passed safety validation.