flowchart LR
subgraph Development
A[Foundation Model Creation]
B[Training Data Curation]
C[Safety Testing]
end
subgraph Provision
D[Fine-tuning for Health]
E[Integration into Apps]
F[Plug-in Development]
end
subgraph Deployment
G[Clinical Implementation]
H[Patient Interaction]
I[Ongoing Monitoring]
end
Development --> Provision --> Deployment
AI Governance in Public Health: Regulation and Oversight
AI regulation and organizational governance answer different questions. Regulation determines which legal duties apply to a particular system. Governance determines who may approve, monitor, constrain, and stop its use. In public health, both analyses must begin with the intended use, affected population, operating workflow, and jurisdiction.
Learning Objectives
This chapter equips readers to:
- Classify the exact software function under the applicable regulatory framework rather than infer status from an “AI” or “healthcare” label
- Distinguish regulatory status, regulatory authorization, local validation, and organizational approval
- Compare FDA, European Union, United Kingdom, and international approaches without assuming that their classifications or evidence requirements are interchangeable
- Map responsibilities across AI development, provision, and deployment
- Specify governance controls for procurement, validation, monitoring, change control, incident response, and suspension
- Analyze transparency and liability boundaries without presenting unsettled legal questions as settled law
- Apply a repeatable regulatory and governance assessment to a proposed public health use
Introduction: Regulation and Governance Are Separate Controls
Regulatory classification is a system-specific legal analysis, not a label assigned because a product uses AI or operates in health. In the United States, FDA’s 2026 guidance interprets the statutory criteria that distinguish certain non-device clinical decision support functions from software functions that remain devices. In the European Union, classification follows the AI Act’s risk and product pathways. The exact function, intended purpose, users, claims, setting, and jurisdiction matter in both analyses.
Organizational governance begins with a separate question: whether the evidence, authority, and controls are sufficient for use in the intended public health setting. The NIST AI Risk Management Framework treats risk management as continuous across the AI lifecycle. WHO’s health-specific guidance addresses uses, actors, and risks associated with large multi-modal models, while CDC’s AI Strategy calls for risk-based oversight, transparency, accountability, privacy, and security as public health agencies expand AI use.
Keeping these questions separate prevents two errors: treating regulatory authorization as proof of local effectiveness, and treating the absence of product regulation as permission to deploy. The practical sequence is classification, evidence review, assignment of authority, controlled deployment, and ongoing monitoring.
Regulatory Landscape
United States: FDA Framework
January 2026 Clinical Decision Support Software Guidance
In January 2026, FDA finalized its Clinical Decision Support Software guidance. The document clarifies which software functions are excluded from the device definition under section 520(o)(1)(E) of the FD&C Act and states that FDA’s existing digital health policies still apply to software functions that remain devices (FDA CDS Guidance, January 2026).
Key clarifications from FDA:
| Topic | FDA clarification |
|---|---|
| Non-device CDS threshold | Software must satisfy all four statutory criteria to qualify as non-device CDS |
| Recommendation type | Non-device CDS provides recommendations or options, not a specific output or directive |
| Transparency | Clinicians must be able to review the basis of the recommendations rather than rely primarily on the software |
| Device examples | Risk scores, probability outputs, time-critical outputs, and functions that analyze medical images, signals, or patterns remain device examples |
Practical implications:
Public health teams should not treat the January 2026 guidance as a blanket retreat from oversight. Functions that analyze medical images, signals, or patterns, produce risk scores or time-critical outputs, or fail to explain the basis of their recommendations remain within FDA’s device framework (FDA CDS FAQs, 2026).
What this means for public health practitioners:
For surveillance, triage, or clinical decision support, the practical task is precision rather than panic: first determine whether the software function actually meets all four non-device CDS criteria, then check the rest of FDA’s digital health policy framework before assuming the tool is either fully exempt or fully reviewed.
A September 2026 Nature editorial urges risk-proportionate real-world testing for generative-AI medical products that inform clinical decisions, and points researchers to FDA’s open comment window on generative-AI-enabled devices (Nature, 2026). On 18 August 2026, FDA’s Digital Health Center of Excellence published Considerations for the Regulation of Generative AI-Enabled Medical Devices: Discussion Paper and Request for Feedback (FDA landing; PDF). Comments are due under docket FDA-2026-N-7874 by 19 October 2026. This is a discussion paper seeking input, not draft or final guidance.
Trial-start latency and population access
Population health access to new therapies also depends on how quickly early-phase trials can start, not only on discovery outputs or downstream device clearance. In a September 2026 New York Times opinion, Teslo argues that trial-start latency, rather than discovery science alone, is a binding bottleneck as AI-designed and personalized therapeutic candidates proliferate (Teslo, New York Times, September 2026). Registry-derived metrics such as ClinicalTrials.gov activation intervals quantify trial-start delays separately from this editorial framing; see the Life Sciences AI Handbook section on clinical trial latency in the AI era for measured timelines. This drug IND and trial-activation policy question is distinct from the SaMD and CDS device-classification analysis above; HHS announced Operation TrialBlazer in June 2026 as a department-wide clinical-trials reform effort.
Autonomous Clinical AI: licensure-like oversight gap
A 2026 JAMA Perspective proposes a licensure-based approach for autonomous clinical AI (Bergman et al., 2026).
Federally, ARPA-H’s ADVOCATE awards (September 2026) fund agentic clinical AI for heart failure plus a supervisory safety agent and multi-system deployment plans aimed at FDA authorization pathways, including sites that serve counties without a cardiologist (ARPA-H, 2026; ADVOCATE program). See the Physician AI Handbook policy chapter for performer teams and the evaluation fence.
For public health agencies and safety-net systems, the distinction matters when AI is proposed as a workforce or access intervention. An autonomous primary-care triage system is not only a procurement question. It is a clinical practice authorization question. Before supporting pilots, define a scope of practice, set clinician escalation boundaries, require supervised deployment evidence in the intended setting, specify outcome and subgroup monitoring, and assign authority to restrict, suspend, or revoke operational use after errors.
December 2025 Executive Order: Federal Challenges to State AI Laws
On December 11, 2025, President Trump signed Executive Order 14365, Ensuring a National Policy Framework for Artificial Intelligence. The order directs federal agencies to challenge specified state AI laws and develop legislative recommendations for a federal framework. The order does not itself invalidate all state AI laws; preemption depends on existing federal law, litigation, or congressional action.
Key provisions affecting healthcare:
| Mechanism | Details |
|---|---|
| DOJ Litigation Task Force | The Attorney General was directed to establish the task force within 30 days to challenge state laws the federal government considers unlawful |
| Commerce Department evaluation | The Secretary of Commerce was directed to identify state laws that conflict with the order’s policy within 90 days |
| Funding conditions | The order addresses non-deployment BEAD funding and directs agencies to consider conditions for certain discretionary grants, subject to applicable law |
State healthcare laws under federal scrutiny include:
- California: Prohibits AI systems from using terms that imply they hold a health care license (AB 489, effective January 1, 2026)
- Colorado: Mandatory bias audits for healthcare AI systems
- Texas, New York, Utah: Healthcare-specific AI oversight measures
Gap in Patient Safety Protections
The order’s stated exceptions do not create a general healthcare or patient-safety safe harbor. The legal effect on any healthcare law remains law-specific and may turn on federal statutory authority and litigation (Executive Order 14365, 2025; Ropes & Gray analysis, 2026).
Practical implications for public health practitioners:
State-level requirements that practitioners had relied on (bias audits, algorithmic transparency disclosures, consent frameworks) face legal uncertainty. Health departments operating in affected states should monitor litigation outcomes and prioritize institutional governance frameworks that meet federal standards while preserving patient safety commitments. See Organizational Governance Frameworks for guidance that applies regardless of regulatory jurisdiction.
European Union: AI Act and Medical Device Regulation
EU AI Act (2024)
The official EU AI Act text establishes risk-based obligations and phased application dates.
Risk-Based Classification:
| Risk Level | Healthcare Examples | Requirements |
|---|---|---|
| Unacceptable | Social scoring and specified biometric practices prohibited under Article 5 | PROHIBITED |
| High | AI covered by Article 6, including a product or safety component subject to third-party conformity assessment, plus specified Annex III uses | Risk management, data governance, transparency, human oversight, conformity assessment, post-market monitoring |
| Limited | Direct-interaction chatbots not otherwise classified as high-risk | Transparency obligations, disclose AI use |
| Minimal | Administrative tools | No specific obligations |
High-risk AI obligations are phased, not simultaneous. Following the AI Omnibus, Annex III high-risk rules apply from 2 December 2027, while rules for high-risk AI embedded in regulated products apply from 2 August 2028 (European Commission, 2026). Public health teams should determine classification from the exact intended purpose, product status, and conformity-assessment pathway rather than assuming that every healthcare use is automatically high-risk.
The September 2026 EASAC and FEAM report recommends clearer guidance on which healthcare AI deployments fall under Annex III of the EU AI Act and should therefore be treated as high-risk systems, so that classification is consistent across Member States (Cole-Hunter et al., 2026).
Key requirements for high-risk healthcare AI:
class EUAIActCompliance:
"""Check compliance with EU AI Act for healthcare AI"""
def assess_risk_level(self, ai_system):
"""Classify AI system under EU AI Act"""
# A healthcare domain label alone is insufficient for classification.
if ai_system.get('article_6_1_product') or ai_system.get('annex_iii_high_risk'):
return {
'risk_level': 'High',
'requirements': [
'Risk management system',
'High-quality training data',
'Technical documentation',
'Transparency and user information',
'Human oversight',
'Accuracy, robustness, cybersecurity',
'Conformity assessment',
'Post-market monitoring'
],
'application_date': 'Confirm under Article 113',
'penalties': 'Article 99 penalties depend on the violated obligation and entity type'
}
elif ai_system.get('patient_facing'):
return {
'risk_level': 'Limited',
'requirements': [
'Inform users of AI interaction',
'Detect and disclose deepfakes',
'Label AI-generated content'
]
}
return {
'risk_level': 'Minimal',
'requirements': []
}
def generate_compliance_checklist(self, ai_system):
"""Generate compliance checklist for high-risk system"""
classification = self.assess_risk_level(ai_system)
if classification['risk_level'] != 'High':
return classification
checklist = {
'Article 9: Risk Management': {
'required': [
'Identify and analyze known/foreseeable risks',
'Estimate and evaluate risks',
'Evaluate other possible risks from misuse',
'Adopt risk management measures'
],
'documentation': 'Risk management plan'
},
'Article 10: Data Governance': {
'required': [
'Training data relevant, representative, free of errors',
'Examine for possible biases',
'Data governance and management practices',
'Ensure appropriate statistical properties'
],
'documentation': 'Data quality report'
},
'Article 13: Transparency': {
'required': [
'Instructions for use understandable to users',
'Information on intended purpose',
'Level of accuracy, robustness, cybersecurity',
'Known limitations and circumstances for malfunction',
'Information to enable human oversight'
],
'documentation': 'User manual, model card'
},
'Article 14: Human Oversight': {
'required': [
'Designed for effective oversight by humans',
'Users can interpret outputs',
'Users can decide when not to use',
'Users can interrupt or stop the system'
],
'documentation': 'Human oversight procedures'
},
'Article 15: Accuracy, Robustness, Cybersecurity': {
'required': [
'Achieve appropriate accuracy',
'Robust against errors, faults, inconsistencies',
'Resilient to attempts to alter use/performance',
'Cybersecurity measures'
],
'documentation': 'Technical validation report'
}
}
return {
'classification': classification,
'compliance_checklist': checklist
}
# Example: EU compliance for sepsis AI
sepsis_system_eu = {
'name': 'SepsisPredict AI',
'domain': 'healthcare',
'purpose': 'Diagnostic support',
'patient_facing': False
}
eu_compliance = EUAIActCompliance()
compliance_check = eu_compliance.generate_compliance_checklist(sepsis_system_eu)
print(f"EU AI Act Risk Level: {compliance_check['classification']['risk_level']}")
print(f"\nCompliance Requirements:")
for article, details in compliance_check['compliance_checklist'].items():
print(f"\n{article}:")
for req in details['required']:
print(f" • {req}")
print(f" Documentation: {details['documentation']}")EU Medical Device Regulation (MDR/IVDR)
In Vitro Diagnostic Medical Devices Regulation (IVDR) applies to AI that is, or is part of, an in vitro diagnostic medical device.
Key changes from previous directives:
| Aspect | Previous (MDD/IVDD) | New (MDR/IVDR) |
|---|---|---|
| Clinical evidence | Limited requirements | Extensive clinical evaluation required |
| Post-market surveillance | Basic | Continuous, structured monitoring |
| Documentation | Moderate | Extensive technical documentation |
| Notified body | Some devices | More devices require third-party assessment |
| Transparency | Limited | Public database (EUDAMED) |
Implementation challenges: Sorenson & Drummond, 2014, Milbank Quarterly documented:
- Shortage of notified bodies
- Compliance costs vary by device, evidence package, conformity-assessment route, and notified body
- Extensive documentation burden
- Delayed timelines
United Kingdom: Post-Brexit Approach
MHRA (Medicines and Healthcare products Regulatory Agency) strategy:
MHRA, 2022: Software and AI as a Medical Device Change Programme
Key features:
- Pragmatic regulation - Risk-proportionate approach
- Innovation-friendly - Fast-track pathway for breakthrough devices
- Real-world evidence - Emphasis on post-market data
- International alignment - Mutual recognition with FDA, EU
UKCA marking - UK conformity assessment (replacing CE marking for GB market)
UK National Commission recommendations and government response (2026)
On 10 September 2026, the MHRA-hosted National Commission into the Regulation of AI in Healthcare published 44 recommendations for a future UK regulatory framework (GOV.UK recommendations; GOV.UK news). The Commission reports engagement with more than 12,000 patients, carers, clinicians, and technologists.
The core of the recommendations is lifecycle regulation. Its elements are staged authorisation for new AI models under tight guardrails (described in government communications as similar to learner-driver “L-plates”), continuous real-world monitoring after deployment, stronger traceability and version control, proportionate transparency when AI is used in care, public access to safety information, and enhanced MHRA enforcement tools. Recommendations also address dependencies on general-purpose AI models (including proposed master-file style transparency mechanisms).
On 6 October 2026, the UK government accepted all 44 recommendations and named ten priority areas for the following year (GOV.UK, October 2026). Accepting the recommendations does not change UK law by itself. For 42 of the 44, the response says further details will follow by Spring 2027, when an implementation roadmap is also due. For population-level safety monitoring, the MHRA committed to work toward a public, searchable database of adverse-incident reports for medical devices, including software and AI-enabled devices (Recommendation 19), and opened a third phase of its AI Airlock sandbox focused on post-market surveillance (GOV.UK, October 2026). The recommendations do not change U.S. law.
International Harmonization: IMDRF
International Medical Device Regulators Forum (IMDRF) is working toward global standards.
IMDRF, 2021: AI/ML-Based Software as Medical Device
Goals:
- Harmonized definitions and terminology
- Common risk classification framework
- Shared validation standards
- Mutual recognition agreements
Challenge: Harmonization requires balancing local sovereignty with global interoperability.
AI Value Chain Governance
The WHO’s 2024 guidance on LMMs introduces a governance framework based on the AI “value chain,” recognizing that different actors bear distinct responsibilities at each stage of AI development and deployment (WHO, 2024).
The Three-Phase Model
Phase 1: Development (Foundation Models)
Primary actors: Large technology companies, research institutions, public-private consortia
Key responsibilities:
| Responsibility | Implementation |
|---|---|
| Data governance | Ensure training data collected legally, with appropriate consent, representative of diverse populations |
| Bias mitigation | Involve diverse stakeholders in design; audit for bias during development |
| Transparency | Disclose training data sources and known limitations |
| Safety design | Design for accuracy, predictability, and corrigibility |
| Environmental impact | Address carbon and water footprints in development choices |
Government role: Enact data protection laws, mandate pre-certification programs, invest in public AI infrastructure, require early-stage algorithm registration.
Phase 2: Provision (Adaptation for Healthcare)
Primary actors: Health technology companies, healthcare systems, academic institutions
Key responsibilities:
| Responsibility | Implementation |
|---|---|
| Fitness for purpose | Validate that foundation model is appropriate for intended health use |
| Regulatory compliance | Ensure adapted system meets medical device requirements |
| Performance validation | Conduct clinical validation in target population |
| Documentation | Provide technical documentation including known limitations |
| Privacy safeguards | Implement data protection for user inputs |
Government role: Assign regulatory agencies to assess LMMs for health; enforce medical device regulations; require impact assessments audited by third parties.
Phase 3: Deployment (Clinical Use)
Primary actors: Ministries of health, health systems, hospitals, clinicians
Key responsibilities:
| Responsibility | Implementation |
|---|---|
| Appropriate use | Avoid deploying LMMs in settings they were not designed for |
| Staff training | Train healthcare workers on LMM limitations, bias risks, appropriate use |
| Human oversight | Maintain human decision-making authority for clinical decisions |
| Monitoring | Track performance, identify errors, report harms |
| Feedback loops | Communicate issues to providers and developers |
Government role: Mandate independent post-release audits; use procurement authority to require transparency; engage public on acceptable LMM uses.
Vertical Integration Considerations
When a single entity controls multiple phases (e.g., a technology company that develops, adapts, and deploys an LMM), additional governance safeguards are needed:
- Internal separation: Distinct teams with separate accountability for each phase
- External audit: Third-party review at phase transitions
- Regulatory scrutiny: Enhanced oversight for vertically integrated systems
- Transparency requirements: Public disclosure of internal governance structures
Key Insight from WHO Guidance
“Certain risks can be addressed at each phase of the AI value chain, and certain actors are likely to play more important roles in mitigating each risk and upholding ethical values. While there is likely to be disagreement and tension about where responsibility rests between developers, providers and deployers, there are clear areas in which each actor is best placed or is the only entity with the capacity to address a potential or actual risk.”
The framework helps identify which actor should have prevented a given failure.
Organizational Governance Frameworks
CDC AI Strategy for Public Health Agencies
For U.S. public health agencies, CDC’s FY 2026–2030 AI Strategy makes governance an operating requirement, not a downstream compliance task. The strategy’s governance pillar emphasizes transparency, accountability, enterprise technology and data governance, risk-based oversight, and communication with partners and the public. It also calls for actionable resources to support state, tribal, local, and territorial (STLT) AI adoption (CDC, March 2026).
CDC’s companion GenAI considerations translate that strategy into local practice. The guidance describes practical steps for STLT agencies adopting GenAI tools and building internal policies: define appropriate tasks, protect sensitive data, manage risk, document transparent use, and keep human review in workflows where outputs affect public health action (CDC, March 2026). CDC’s 2026 Public Health Data Strategy milestones make the connection explicit by listing publication of GenAI guidance and resources for STLT partners as an AI integration milestone (CDC, April 2026).
CDC’s companion Considerations for Agentic Research in Public Health (12 Mar 2026) covers “deep research” tools that plan multi-step web work and draft citation-based reports. Drawing on CDC’s 2025 internal exploratory evaluation, the guidance favors well-scoped, information-rich tasks with public sources and qualified human verification (policy scans, early synthesis, first-draft communications) and advises avoiding restricted or sensitive data, statistically rigorous analytics, legal or clinical judgment calls, comprehensive literature reviews, vague prompts, and timelines too short for SME review (CDC; listed with GenAI resources on CDC AI partner resources). Effective-use principles named are mission alignment, human oversight, scientific integrity, disclosure, and security/privacy. This is informational agency guidance from exploratory use, not a vendor accuracy validation and not permission to place PHI in external agents..
In June 2026, ASPPH released Artificial Intelligence Framework: Harnessing Innovation for Academic Public Health, a task-force roadmap for schools and programs across four pillars: Education; Teaching and Learning; Practice and Research; and Policy, Regulatory, and Architectural Frameworks (ASPPH framework report; press, 17 Jun 2026; strategic recommendations summary). The report urges treating AI literacy as core public health professionalism rather than waiting for CEPH mandates, and it elevates “bridge professionals” who can translate across technical teams, leadership, and communities. Supporting work included a late-2025 scan of generative AI policies across 155 member institutions (only about one in eight had formal public policies, mostly classroom-focused) and a job-posting analysis of AI-related skills demand. The framework is an association roadmap for academic public health, not a state or local agency SOP and not proof of population-health outcome gains..
AI for public-benefits eligibility (Medicaid work requirements)
Under the 2025 budget reconciliation law (HR 1), states must redetermine Medicaid eligibility with new community-engagement (work) rules. A JAMA Forum argues for phased AI: start with structured, non-subjective checks and beneficiary outreach, and keep medical-frailty and attestation-credibility judgments with humans (Mello & Nagireddy, 2026). States, not vendors, should set decision thresholds that minimize wrongful disenrollment, with vendor dashboards tracking error rates, appeals, and procedural terminations.
The “Tip of the Iceberg” Problem in Health Systems
Public health and clinical institutions often focus governance exclusively on enterprise-procured, officially integrated AI tools. However, as Ötleş and colleagues highlight in NEJM AI (2026), “Health Systems Govern Only the Tip of the AI Iceberg” (Ötleş et al., 2026). They argue that banning generative AI or heavily restricting enterprise tools does not reduce risk; it merely drives it underground. With many practitioners using “shadow AI” (consumer-grade tools on personal devices) for their work, effective organizational governance must acknowledge and manage this broader, unseen use of AI rather than just policing the visible “tip” of officially approved systems.
The Three Lines of Defense Model
class AIGovernanceFramework:
"""
Three lines of defense for AI governance in healthcare
Line 1: Operational management (owns and manages risk)
Line 2: Oversight functions (monitors and advises on risk)
Line 3: Independent assurance (provides objective assurance)
"""
def define_governance_structure(self):
"""Define three lines of defense for AI governance"""
return {
'Line 1: Operational Management': {
'roles': [
'Data Scientists/ML Engineers',
'Clinical Champions',
'IT Operations'
],
'responsibilities': [
'Develop AI models following organizational standards',
'Implement technical controls and safeguards',
'Monitor model performance continuously',
'Report incidents and issues to Line 2',
'Maintain model documentation'
],
'controls': [
'Code review processes',
'Model validation before deployment',
'Performance dashboards',
'Incident response procedures',
'Version control and change logs'
]
},
'Line 2: Oversight Functions': {
'roles': [
'AI Ethics Committee',
'Clinical Safety Officer',
'Data Governance Board',
'Risk Management',
'Compliance Officer'
],
'responsibilities': [
'Define AI policies, standards, and procedures',
'Review and approve high-risk AI projects',
'Monitor compliance with regulations and policies',
'Investigate AI-related incidents',
'Escalate issues to Line 3 and leadership'
],
'controls': [
'Pre-deployment ethics and safety review',
'Quarterly model performance audits',
'Fairness and bias assessments',
'Clinical validation requirements',
'Policy compliance checks'
]
},
'Line 3: Independent Assurance': {
'roles': [
'Internal Audit',
'External Auditors',
'Clinical Safety Review Board'
],
'responsibilities': [
'Independent assessment of Lines 1 & 2 effectiveness',
'Audit AI governance processes and controls',
'Report findings to Board and Executive Leadership',
'Recommend improvements to governance framework',
'Validate compliance with regulations'
],
'controls': [
'Annual AI governance audits',
'Model recertification reviews',
'Third-party validation studies',
'Board reporting and presentations',
'Regulatory compliance assessments'
]
}
}
def create_ai_ethics_committee(self):
"""
Establish AI Ethics Committee (Line 2)
Based on WHO (2021) guidance and NIH AI Governance Framework
"""
return {
'name': 'AI Ethics and Governance Committee',
'composition': {
'clinical': {
'roles': ['2 Physicians', '1 Nurse', '1 Patient Advocate'],
'rationale': 'Ensure clinical validity and patient-centered perspective'
},
'technical': {
'roles': ['1 Data Scientist', '1 ML Engineer', '1 IT Security'],
'rationale': 'Evaluate technical feasibility and security'
},
'oversight': {
'roles': ['1 Ethicist', '1 Legal Counsel', '1 Risk Manager'],
'rationale': 'Ensure ethical, legal, and risk compliance'
},
'total_members': 10,
'term_length': '2 years (staggered)',
'chair': 'Senior clinician with AI expertise'
},
'charter': {
'mission': 'Ensure responsible development and deployment of AI in healthcare',
'authority': [
'Approve or reject high-risk AI projects',
'Define AI development and deployment standards',
'Investigate AI-related adverse events',
'Recommend policy changes to leadership',
'Mandate corrective actions for non-compliance'
],
'meeting_frequency': 'Monthly (more frequent for urgent reviews)',
'quorum': '60% including at least 1 clinical and 1 technical member',
'voting': 'Majority for approval, unanimous for prohibition'
},
'review_process': {
'triggers': [
'New AI project involving patient care',
'Major model update (>10% parameter change)',
'AI-related adverse event or near-miss',
'Significant performance degradation',
'Fairness or bias concerns raised',
'Expansion to new patient populations'
],
'review_criteria': [
'Clinical validity and utility',
'Fairness across demographic groups',
'Transparency and explainability',
'Privacy and security measures',
'Integration with clinical workflow',
'Liability and accountability clarity',
'Regulatory compliance',
'Resource requirements and cost-effectiveness'
],
'decision_types': {
'Approve': 'Proceed with deployment',
'Approve with Conditions': 'Deploy with specific requirements',
'Defer': 'Additional information needed',
'Reject': 'Do not proceed'
},
'appeal_process': 'Project team can appeal to Executive Leadership'
},
'documentation': {
'required_submissions': [
'Project proposal with clinical rationale',
'Technical specifications and architecture',
'Validation results and performance metrics',
'Fairness assessment across subgroups',
'Risk assessment and mitigation plan',
'Implementation and monitoring plan',
'User training and support plan'
],
'committee_records': [
'Meeting minutes',
'Review decisions with rationale',
'Conditions and monitoring requirements',
'Follow-up actions and timelines'
]
}
}
# Example: Implement AI governance for hospital
hospital_governance = AIGovernanceFramework()
# Define structure
structure = hospital_governance.define_governance_structure()
print("AI Governance Structure: Three Lines of Defense\n")
for line, details in structure.items():
print(f"{line}:")
print(f" Roles: {', '.join(details['roles'])}")
print(f" Key responsibilities: {len(details['responsibilities'])}")
print(f" Controls: {len(details['controls'])}\n")
# Create ethics committee
ethics_committee = hospital_governance.create_ai_ethics_committee()
print("\nAI Ethics Committee:")
print(f" Total members: {ethics_committee['composition']['total_members']}")
print(f" Meeting frequency: {ethics_committee['charter']['meeting_frequency']}")
print(f" Review triggers: {len(ethics_committee['review_process']['triggers'])}")
print(f" Decision types: {len(ethics_committee['review_process']['decision_types'])}")Model Risk Management Framework
Based on SR 11-7 (Federal Reserve guidance for banking, adapted for healthcare):
import pandas as pd
class ModelRiskManagement:
"""
Complete model risk management for healthcare AI
Adapted from OCC/Federal Reserve SR 11-7
"""
def assess_model_risk(self, model_info):
"""
Assess inherent and residual risk of AI model
Returns risk tier: High, Medium, Low
"""
# Inherent risk factors
inherent_risk_score = 0
# Clinical impact
impact_scores = {
'critical': 4, # Death or serious harm possible
'high': 3, # Significant morbidity
'medium': 2, # Minor morbidity
'low': 1 # No direct patient impact
}
inherent_risk_score += impact_scores.get(model_info.get('clinical_impact'), 2)
# Autonomy level
if model_info.get('autonomy') == 'autonomous':
inherent_risk_score += 3
elif model_info.get('autonomy') == 'semi_autonomous':
inherent_risk_score += 2
else:
inherent_risk_score += 1
# Population size
if model_info.get('population_size', 0) > 10000:
inherent_risk_score += 2
elif model_info.get('population_size', 0) > 1000:
inherent_risk_score += 1
# Model complexity
if model_info.get('interpretability') == 'black_box':
inherent_risk_score += 2
# Mitigation factors (reduce residual risk)
mitigation_score = 0
if model_info.get('clinical_validation'):
mitigation_score += 2
if model_info.get('continuous_monitoring'):
mitigation_score += 2
if model_info.get('human_oversight'):
mitigation_score += 2
if model_info.get('explainability_features'):
mitigation_score += 1
if model_info.get('fallback_mechanisms'):
mitigation_score += 1
# Calculate residual risk
residual_risk_score = max(0, inherent_risk_score - mitigation_score)
# Determine risk tier
if residual_risk_score >= 8:
tier = 'High'
elif residual_risk_score >= 5:
tier = 'Medium'
else:
tier = 'Low'
return {
'inherent_risk_score': inherent_risk_score,
'mitigation_score': mitigation_score,
'residual_risk_score': residual_risk_score,
'risk_tier': tier,
'validation_requirements': self.get_validation_requirements(tier)
}
def get_validation_requirements(self, risk_tier):
"""Define validation requirements based on risk tier"""
requirements = {
'High': {
'development_validation': [
'Thorough data quality assessment',
'Feature engineering rationale and sensitivity analysis',
'Model selection justification with alternatives considered',
'Hyperparameter tuning with cross-validation',
'Adversarial testing'
],
'deployment_validation': [
'Independent clinical validation study',
'Multi-site validation',
'Fairness assessment across all demographic groups',
'Prospective validation on live data',
'User acceptance testing with clinical staff'
],
'ongoing_validation': [
'Real-time performance monitoring',
'Weekly performance reports',
'Monthly fairness audits',
'Quarterly model recertification',
'Immediate investigation of performance degradation'
],
'documentation': [
'Detailed model card',
'Technical specification document',
'Validation report',
'Risk assessment and mitigation plan',
'Clinical use protocols'
]
},
'Medium': {
'development_validation': [
'Data quality assessment',
'Model selection justification',
'Cross-validation results'
],
'deployment_validation': [
'Clinical validation study',
'Fairness assessment',
'User acceptance testing'
],
'ongoing_validation': [
'Monthly performance monitoring',
'Quarterly fairness audits',
'Annual recertification'
],
'documentation': [
'Model card',
'Validation summary',
'Use protocols'
]
},
'Low': {
'development_validation': [
'Basic data quality check',
'Cross-validation'
],
'deployment_validation': [
'Pilot testing',
'User feedback'
],
'ongoing_validation': [
'Quarterly performance check',
'Annual review'
],
'documentation': [
'Basic model documentation',
'Use instructions'
]
}
}
return requirements.get(risk_tier, requirements['Medium'])
def create_model_inventory(self, models_list):
"""
Create and maintain model inventory
Critical for governance and compliance
"""
inventory = []
for model in models_list:
risk_assessment = self.assess_model_risk(model)
inventory_entry = {
'model_id': model.get('id'),
'model_name': model.get('name'),
'purpose': model.get('purpose'),
'owner': model.get('owner'),
'status': model.get('status'), # Development, Deployed, Retired
'deployment_date': model.get('deployment_date'),
'risk_tier': risk_assessment['risk_tier'],
'last_validation': model.get('last_validation_date'),
'next_review': model.get('next_review_date'),
'regulatory_status': model.get('regulatory_status'),
'documentation_location': model.get('docs_url')
}
inventory.append(inventory_entry)
return pd.DataFrame(inventory)
# Example: Assess model risk for sepsis predictor
sepsis_model_info = {
'name': 'SepsisPredict AI v2.0',
'clinical_impact': 'high', # Sepsis is life-threatening
'autonomy': 'decision_support', # Not autonomous
'population_size': 15000, # Annual patient volume
'interpretability': 'black_box', # Deep learning
'clinical_validation': True,
'continuous_monitoring': False, # (missing)
'human_oversight': True,
'explainability_features': False, # (missing)
'fallback_mechanisms': True
}
mrm = ModelRiskManagement()
risk_assessment = mrm.assess_model_risk(sepsis_model_info)
print(f"Model Risk Assessment: SepsisPredict AI")
print(f" Inherent Risk Score: {risk_assessment['inherent_risk_score']}")
print(f" Mitigation Score: {risk_assessment['mitigation_score']}")
print(f" Residual Risk Score: {risk_assessment['residual_risk_score']}")
print(f" Risk Tier: {risk_assessment['risk_tier']}\n")
print(f"Validation Requirements for {risk_assessment['risk_tier']} Risk:")
requirements = risk_assessment['validation_requirements']
print(f" Development validation: {len(requirements['development_validation'])} requirements")
print(f" Deployment validation: {len(requirements['deployment_validation'])} requirements")
print(f" Ongoing validation: {len(requirements['ongoing_validation'])} requirements")Assessing Organizational Governance Readiness
The Three Lines of Defense and Model Risk Management frameworks above assume substantial institutional infrastructure. A systematic review of 35 healthcare AI governance frameworks found that most target large academic medical centers with dedicated data science teams and enterprise data warehouses, creating barriers for community health systems and public health agencies (Hussein et al., 2026). The resulting HAIRA maturity model provides a five-level assessment (from Ad Hoc to Leading) across seven governance domains, enabling organizations to identify realistic governance targets based on available resources. For the complete HAIRA framework and its application to public health settings, see Governance Maturity Assessment.
Accountability and Liability
The Liability Challenge
Liability for AI errors involves multiple actors, and existing legal frameworks are still evolving.
When AI contributes to patient harm, potentially liable parties include the AI developer, the data provider, the clinician, the hospital or health system, and, rarely, the regulator.
Price, 2017, Michigan Law Review analyzes medical AI regulation, including liability frameworks for black-box medicine.
Liability Models
1. Product Liability (AI Developer)
Legal basis: Strict liability: no need to prove negligence
Requirements to establish:
- Product was defective
- Defect caused injury
- Product was used as intended
Challenge: Defining “defect” for AI
- Performance below promised accuracy?
- Below human expert performance?
- Below peer AI systems?
Example case:
- Radiologist uses FDA-cleared AI for lung nodule detection
- AI misses obvious cancer
- Patient sues
Potential liability:
- AI developer: Liable if model defective (failed validation standards)
- Radiologist: May still be liable for not catching obvious error (standard of care)
2. Medical Malpractice (Clinician)
Legal basis: Negligence
Must prove:
- Duty of care existed
- Duty was breached
- Breach caused harm
- Damages resulted
Balkin, 2017, Ohio State Law Journal argues clinicians must:
- Understand AI limitations - Know when to override
- Maintain competence - Do not blindly follow AI
- Use clinical judgment - AI is decision support, not replacement
Landmark example: Caruana et al., 2015, KDD
Pneumonia risk model paradox:
- Model learned: Asthma history → Lower mortality risk
- Reality: Asthma patients go straight to ICU → Aggressive treatment → Better outcomes
- If deployed blindly: Asthma patients sent home → Worse outcomes
- Liability: Clinician potentially liable for not recognizing illogical recommendation
3. Institutional Liability (Hospital/Health System)
Legal basis: Corporate negligence doctrine
Hospital must ensure:
- Proper credentialing (approved safe AI)
- Adequate oversight (monitoring in place)
- Sufficient training (staff know how to use AI)
4. Regulatory Liability (Rare)
In rare cases, a regulator may face claims of a negligent approval process.
Liability Risk Assessment
class LiabilityAssessment:
"""
Assess and mitigate liability exposure for AI systems
"""
def assess_developer_liability(self, ai_system):
"""Product liability exposure"""
risks = []
if not ai_system.get('clinical_validation'):
risks.append({
'risk': 'Inadequate validation',
'severity': 'High',
'legal_basis': 'Defective product (strict liability)',
'mitigation': 'Conduct prospective clinical validation study'
})
if not ai_system.get('performance_monitoring'):
risks.append({
'risk': 'No post-market surveillance',
'severity': 'High',
'legal_basis': 'Failure to warn of known defects',
'mitigation': 'Implement continuous performance monitoring with alerts'
})
if not ai_system.get('clear_limitations'):
risks.append({
'risk': 'Inadequate limitations disclosure',
'severity': 'Medium',
'legal_basis': 'Failure to warn',
'mitigation': 'Provide detailed limitations documentation'
})
return {
'liability_type': 'Product Liability (Strict)',
'risks': risks,
'exposure_level': 'High' if len(risks) >= 2 else 'Medium',
'insurance': 'Product liability insurance scaled to deployment exposure',
'recommended_actions': [r['mitigation'] for r in risks]
}
def assess_clinician_liability(self, ai_system):
"""Medical malpractice exposure"""
risks = []
if ai_system.get('autonomy') == 'autonomous':
risks.append({
'risk': 'Over-reliance on autonomous AI',
'severity': 'High',
'legal_basis': 'Failure to exercise clinical judgment',
'mitigation': 'Require mandatory human review and documentation of rationale'
})
if not ai_system.get('training_program'):
risks.append({
'risk': 'Inadequate clinician training',
'severity': 'High',
'legal_basis': 'Incompetent use of medical device',
'mitigation': 'Implement certification program before AI use'
})
if not ai_system.get('uncertainty_display'):
risks.append({
'risk': 'No confidence intervals shown',
'severity': 'Medium',
'legal_basis': 'Lack of informed decision-making',
'mitigation': 'Display prediction uncertainty and confidence scores'
})
return {
'liability_type': 'Medical Malpractice (Negligence)',
'risks': risks,
'exposure_level': 'High' if len(risks) >= 2 else 'Medium',
'insurance': 'Professional malpractice insurance',
'recommended_actions': [r['mitigation'] for r in risks]
}
def assess_institutional_liability(self, ai_system):
"""Corporate negligence exposure"""
risks = []
if not ai_system.get('governance_approval'):
risks.append({
'risk': 'No governance oversight',
'severity': 'High',
'legal_basis': 'Failure to ensure safe practices',
'mitigation': 'Require AI ethics committee approval'
})
if not ai_system.get('incident_response'):
risks.append({
'risk': 'No incident response plan',
'severity': 'High',
'legal_basis': 'Inadequate risk management',
'mitigation': 'Develop AI-specific incident response procedures'
})
if not ai_system.get('credentialing'):
risks.append({
'risk': 'No AI credentialing process',
'severity': 'Medium',
'legal_basis': 'Negligent credentialing',
'mitigation': 'Implement AI credentialing checklist'
})
return {
'liability_type': 'Corporate Negligence',
'risks': risks,
'exposure_level': 'High' if len(risks) >= 2 else 'Medium',
'insurance': 'General liability + Cyber liability insurance',
'recommended_actions': [r['mitigation'] for r in risks]
}
def generate_full_report(self, ai_system):
"""Generate complete liability assessment"""
developer = self.assess_developer_liability(ai_system)
clinician = self.assess_clinician_liability(ai_system)
institutional = self.assess_institutional_liability(ai_system)
report = "=== FULL LIABILITY ASSESSMENT ===\n\n"
for stakeholder, assessment in [
('AI DEVELOPER', developer),
('CLINICIAN', clinician),
('INSTITUTION', institutional)
]:
report += f"{stakeholder}\n"
report += f" Liability Type: {assessment['liability_type']}\n"
report += f" Exposure Level: {assessment['exposure_level']}\n"
report += f" Insurance: {assessment['insurance']}\n"
if assessment['risks']:
report += f"\n Risks Identified:\n"
for risk in assessment['risks']:
report += f" • {risk['risk']} (Severity: {risk['severity']})\n"
report += f" Legal basis: {risk['legal_basis']}\n"
report += f" Mitigation: {risk['mitigation']}\n"
report += "\n"
# Summary recommendations
all_actions = (developer['recommended_actions'] +
clinician['recommended_actions'] +
institutional['recommended_actions'])
if all_actions:
report += "PRIORITY ACTIONS:\n"
for i, action in enumerate(all_actions, 1):
report += f"{i}. {action}\n"
return report
# Example: Full liability assessment
sepsis_system_liability = {
'name': 'SepsisPredict AI',
'clinical_validation': True,
'performance_monitoring': False, # (missing)
'clear_limitations': True,
'autonomy': 'decision_support',
'training_program': False, # (missing)
'uncertainty_display': False, # (missing)
'governance_approval': True,
'incident_response': False, # (missing)
'credentialing': True
}
liability_assessment = LiabilityAssessment()
liability_report = liability_assessment.generate_full_report(sepsis_system_liability)
print(liability_report)Emerging LMM Liability Frameworks
Traditional liability frameworks were designed for static medical devices. Large multi-modal models present novel challenges that legal systems are beginning to address.
The LMM Liability Problem
LMMs complicate traditional liability analysis:
| Traditional Device | LMM Challenge |
|---|---|
| Fixed behavior after approval | Outputs vary unpredictably |
| Clear manufacturer identity | Multiple actors in value chain |
| Defined use cases | General-purpose, unanticipated uses |
| Testable before release | Emergent behaviors post-deployment |
| Traceable decision logic | Black-box reasoning |
Liability Mechanisms for Policy Analysis
The following mechanisms are policy options for legal analysis, not recommendations attributed to the cited WHO guidance. Their availability and design depend on governing law:
1. Presumption of Causality
Shift burden of proof from patients to developers/providers:
- If an LMM was involved in care and harm occurred, presume the LMM contributed to harm
- Defendant must prove the LMM did not cause or contribute to harm
- Rationale: Patients cannot inspect black-box systems to prove causation
2. Strict Liability for High-Risk LMM Uses
Apply strict liability (no need to prove negligence) for:
- Autonomous diagnostic decisions
- Treatment recommendations without human oversight
- High-stakes triage or resource allocation
3. No-Fault Compensation Funds
Establish compensation mechanisms that:
- Provide redress without requiring litigation
- Fund through levies on LMM developers/providers
- Reduce barriers to compensation for harmed patients
- Similar to vaccine injury compensation programs
Withdrawn EU AI Liability Directive Proposal
The European Commission listed the proposed AI Liability Directive for withdrawal in its 2025 work programme. Its proposed disclosure rules and rebuttable presumptions therefore are not current EU law. Organizations should assess liability under applicable national law and enacted EU instruments, including the Product Liability Directive where relevant (European Commission Work Programme 2025).
Practical Implications
For organizations deploying LMMs in healthcare:
- Document everything: Maintain detailed logs of LMM outputs and clinical decisions
- Establish override protocols: Clear procedures for when clinicians should disregard LMM recommendations
- Insurance review: Ensure professional liability coverage explicitly addresses AI/LMM use
- Contractual protections: Negotiate indemnification clauses with LMM providers
- Incident response: Prepare protocols for LMM-related adverse events
Transparency and Explainability
Regulatory Requirements
FDA Guidance: Clinical Decision Support Software, January 2026
Transparency requirements:
- Intended use - Clear description
- Limitations - Known failure modes, validated populations
- Performance metrics - Accuracy, sensitivity, specificity
- Training data - Dataset characteristics, potential biases
EU AI Act Article 13: High-risk AI systems must provide:
- Instructions for use understandable to users
- Information on capabilities and limitations
- Level of accuracy, robustness, cybersecurity
- Circumstances that may lead to risks
Model Cards for Transparency
Mitchell et al., 2019: Model Cards for Model Reporting
class TransparencyFramework:
"""
Ensure AI transparency through model cards and explanations
"""
def create_model_card(self, model_info):
"""
Generate detailed model card
Based on Mitchell et al., 2019
"""
card = f"""
# MODEL CARD: {model_info['name']}
## Model Details
- **Developer:** {model_info['developer']}
- **Model date:** {model_info['date']}
- **Model version:** {model_info['version']}
- **Model type:** {model_info['model_type']}
- **Intended use:** {model_info['intended_use']}
- **Out-of-scope use:** {model_info.get('out_of_scope', 'Not specified')}
## Training Data
- **Dataset:** {model_info['dataset_name']}
- **Sample size:** {model_info['n_samples']:,} patients
- **Time period:** {model_info['time_period']}
- **Demographics:** {model_info['demographics']}
- **Data sources:** {', '.join(model_info['data_sources'])}
- **Exclusion criteria:** {model_info.get('exclusions', 'None')}
## Performance
### Overall Performance (Test Set, n={model_info.get('test_n', 'N/A')})
- **AUC-ROC:** {model_info['performance']['auc']:.3f} (95% CI: {model_info['performance'].get('auc_ci', 'N/A')})
- **Sensitivity:** {model_info['performance']['sensitivity']:.1%}
- **Specificity:** {model_info['performance']['specificity']:.1%}
- **PPV:** {model_info['performance']['ppv']:.1%}
- **NPV:** {model_info['performance']['npv']:.1%}
### Subgroup Performance
"""
# Add subgroup performance table
if model_info.get('subgroup_performance'):
card += "\n| Subgroup | n | AUC | Sensitivity | Specificity |\n"
card += "|----------|---|-----|-------------|-------------|\n"
for subgroup, metrics in model_info['subgroup_performance'].items():
card += f"| {subgroup} | {metrics.get('n', 'N/A')} | {metrics['auc']:.3f} | {metrics['sensitivity']:.1%} | {metrics['specificity']:.1%} |\n"
card += f"""
## Limitations
{chr(10).join(['- ' + lim for lim in model_info['limitations']])}
## Ethical Considerations
{chr(10).join(['- ' + eth for eth in model_info['ethical_considerations']])}
## Recommendations for Use
{chr(10).join(['- ' + rec for rec in model_info.get('recommendations', [])])}
## Regulatory Status
- **FDA:** {model_info.get('fda_status', 'Not FDA-cleared')}
- **EU:** {model_info.get('eu_status', 'No CE marking')}
- **Other:** {model_info.get('other_regulatory', 'N/A')}
## Citation
If you use this model in research, please cite:{model_info.get(‘citation’, ‘No citation provided’)}
## Contact
- **Technical support:** {model_info.get('support_email', 'N/A')}
- **Website:** {model_info.get('website', 'N/A')}
- **Documentation:** {model_info.get('docs_url', 'N/A')}
## Version History
{chr(10).join([f"- **v{v['version']}** ({v['date']}): {v['changes']}" for v in model_info.get('version_history', [])])}
"""
return card
def generate_patient_explanation(self, prediction_info):
"""
Patient-friendly explanation (GDPR Articles 13–14 compliance)
"""
confidence_interpretation = (
"High confidence - The AI is quite certain about this prediction"
if prediction_info['confidence'] > 0.80
else "Moderate confidence - Additional testing is recommended"
if prediction_info['confidence'] > 0.60
else "Low confidence - This prediction has high uncertainty"
)
explanation = f"""
╔══════════════════════════════════════════╗
║ YOUR HEALTHCARE AI PREDICTION ║
╚══════════════════════════════════════════╝
PREDICTION
{prediction_info['prediction_text']}
CONFIDENCE LEVEL
{prediction_info['confidence']:.0%} confidence
{confidence_interpretation}
TOP FACTORS INFLUENCING THIS PREDICTION
"""
for i, factor in enumerate(prediction_info['top_factors'][:3], 1):
explanation += f" {i}. {factor['name']}\n"
explanation += f" {factor['description']}\n"
explanation += f"""
WHAT THIS MEANS FOR YOU
{prediction_info['clinical_interpretation']}
IMPORTANT TO KNOW
• This AI assists your doctor but does NOT replace their judgment
• Your doctor considers this along with other information about you
• You have the right to ask questions or seek a second opinion
• AI predictions are probabilities, not certainties; they can be wrong
YOUR RIGHTS
• You can request an explanation of how this prediction was made
• You can request your doctor make decisions without using this AI
• You can file a complaint if you believe the AI made an error
QUESTIONS OR CONCERNS?
Contact: {prediction_info['contact_info']}
Privacy concerns: {prediction_info.get('privacy_contact', 'N/A')}
"""
return explanation
# Example: Create model card and patient explanation
sepsis_model_card_info = {
'name': 'SepsisPredict AI v2.0',
'developer': 'Example Health AI Lab',
'date': '2024-01-15',
'version': '2.0',
'model_type': 'XGBoost ensemble',
'intended_use': 'Early prediction of sepsis in adult ICU patients to enable timely intervention',
'out_of_scope': 'NOT validated for: pediatric patients, emergency department, outpatient settings',
'dataset_name': 'Multi-Center ICU Database',
'n_samples': 50000,
'test_n': 10000,
'time_period': '2018-2023',
'demographics': 'Adults 18+, 52% female, 48% male, racially diverse (35% White, 28% Black, 22% Hispanic, 15% Asian/Other)',
'data_sources': ['EHR vital signs', 'Laboratory results', 'Medications', 'Nursing assessments'],
'exclusions': 'Patients with <6 hours ICU data, missing key vitals',
'performance': {
'auc': 0.82,
'auc_ci': '0.80-0.84',
'sensitivity': 0.78,
'specificity': 0.75,
'ppv': 0.42,
'npv': 0.94
},
'subgroup_performance': {
'Age 18-50': {'n': 2500, 'auc': 0.84, 'sensitivity': 0.80, 'specificity': 0.77},
'Age 51-70': {'n': 4200, 'auc': 0.82, 'sensitivity': 0.78, 'specificity': 0.75},
'Age 71+': {'n': 3300, 'auc': 0.80, 'sensitivity': 0.76, 'specificity': 0.73},
'Male': {'n': 4800, 'auc': 0.82, 'sensitivity': 0.78, 'specificity': 0.75},
'Female': {'n': 5200, 'auc': 0.82, 'sensitivity': 0.78, 'specificity': 0.75}
},
'limitations': [
'Trained on ICU patients only, not validated for ED or outpatient',
'Performance may degrade with EHR system changes or clinical practice updates',
'Lower PPV (42%) means many alerts are false positives; clinical judgment essential',
'Not validated in pediatric populations or pregnancy',
'Requires minimum 6 hours of ICU data for reliable predictions'
],
'ethical_considerations': [
'Alert fatigue risk; use appropriate thresholds to minimize false positives',
'Ensure equitable performance monitored across all demographic groups',
'Requires prospective clinical validation before deployment in new settings',
'May reflect biases in historical data; continuous monitoring required'
],
'recommendations': [
'Use as clinical decision support, not autonomous decision-making',
'Always combine with clinical assessment',
'Monitor for alert fatigue among clinicians',
'Review false positives regularly to adjust thresholds',
'Revalidate model if EHR system or clinical protocols change'
],
'fda_status': '510(k) cleared (K123456)',
'eu_status': 'CE marked (Class IIb)',
'citation': 'Smith et al. (2024). SepsisPredict AI: Early Sepsis Prediction in ICU Patients. Journal of Critical Care Medicine.',
'support_email': 'ai-support@examplehealth.org',
'website': 'https://examplehealth.org/ai/sepsis',
'docs_url': 'https://docs.examplehealth.org/sepsis-ai',
'version_history': [
{'version': '1.0', 'date': '2022-06-01', 'changes': 'Initial release'},
{'version': '1.5', 'date': '2023-03-15', 'changes': 'Improved sensitivity, added SHAP explanations'},
{'version': '2.0', 'date': '2024-01-15', 'changes': 'Retrained on expanded dataset, added subgroup monitoring'}
]
}
transparency = TransparencyFramework()
# Generate model card
model_card = transparency.create_model_card(sepsis_model_card_info)
print(model_card)
print("\n" + "="*80 + "\n")
# Generate patient explanation
patient_prediction = {
'prediction_text': 'Elevated risk of sepsis within 24 hours',
'confidence': 0.82,
'top_factors': [
{
'name': 'Elevated lactate (4.2 mmol/L)',
'description': 'High lactate levels suggest tissues are not getting enough oxygen'
},
{
'name': 'Low blood pressure (85/50 mmHg)',
'description': 'Hypotension may indicate poor circulation or infection'
},
{
'name': 'Elevated temperature (38.9°C)',
'description': 'Fever suggests your body is fighting an infection'
}
],
'clinical_interpretation': 'Your doctor has been alerted to this prediction. They will evaluate you for possible infection and may order additional tests (like blood cultures) or start antibiotics if appropriate.',
'contact_info': 'patient-services@examplehealth.org or call 1-800-XXX-XXXX',
'privacy_contact': 'privacy@examplehealth.org'
}
patient_explanation = transparency.generate_patient_explanation(patient_prediction)
print(patient_explanation)
International Governance of AI for Health
AI systems, particularly LMMs developed by multinational technology companies, operate across borders. Effective governance requires international coordination.
The Case for International Governance
The WHO’s 2024 guidance emphasizes that “governments must work together to build new institutional structures and rules to ensure that international governance keeps pace with globalization of these technologies” (WHO, 2024).
Why national regulation alone is insufficient:
| Challenge | International Dimension |
|---|---|
| Market concentration | A few companies dominate global LMM development; no single country can regulate effectively |
| Data flows | Training data crosses borders; data protection requires international coordination |
| Regulatory arbitrage | Companies may locate in jurisdictions with weakest oversight |
| Capacity gaps | Low- and middle-income countries lack regulatory infrastructure |
| Standards fragmentation | Incompatible national standards impede beneficial AI deployment |
Current International Initiatives
WHO Role:
AI Across the Evidence-Informed Policy Cycle
WHO’s April 2026 Artificial intelligence and evidence-informed policy: emerging challenges and opportunities maps AI use across problem identification, policy design, implementation, monitoring, and adaptation. It recommends practical safeguards including impact assessment, readiness review, living-evidence workflows that pair automated retrieval with human verification, human decision gates, and multidisciplinary oversight. The document is a discussion paper, not an adopted WHO guideline or binding standard. Its central boundary is directly applicable to public health governance: AI should augment rather than replace human judgment, contextual interpretation, ethical deliberation, and accountability (WHO, 2026, reference B09667).
In October 2026 the OECD and Spain’s Ministry of Health published the proceedings of their Madrid conference on the responsible scale of AI in health, setting out an Expert-Led Madrid Action Plan (EL-MAP): ten priority actions spanning co-design of use cases, capacity building, partnerships, leadership and oversight, baseline policy, full-lifecycle approvals, data and digital infrastructure, adaptability, and economic-return and financing models, plus a cross-cutting action to measure AI readiness, use, and impact (OECD, 2026). The report reframes the task from whether to adopt AI to how to scale it safely, citing an estimate that more than 90% of executives report no measurable productivity impact from AI so far, and positions EL-MAP as a way to reduce fragmented, duplicative guidance. It is a conference proceedings report and action-plan proposal, not an adopted guideline or binding standard.
Other International Bodies:
| Organization | Focus Area |
|---|---|
| OECD | AI Principles (2019), policy recommendations |
| UNESCO | Recommendation on the Ethics of AI (2021) |
| ITU/WHO Focus Group | AI for health standards |
| IMDRF | Medical device regulatory harmonization |
| G7/G20 | High-level AI governance frameworks |
Comparing AI Governance Frameworks in Public Health Policy
Health departments need to know which framework answers which governance problem. The major frameworks are complementary rather than interchangeable.
| Framework | What it is | What it contributes for public health organizations | Main limitation |
|---|---|---|---|
| WHO, 2021 | Health-specific ethics and governance guidance | Gives six principles tailored to health use cases, including surveillance, equity, accountability, and protection of autonomy | Normative guidance; not an operational control checklist |
| WHO, 2024 | Governance guidance for large multimodal models in health | Clarifies responsibilities across the AI value chain: developers, providers, and deployers | Focused on LMMs rather than every AI system used by health departments |
| NIST AI RMF 1.0 | Voluntary risk management framework | Translates governance into concrete functions (govern, map, measure, manage), useful for inventories, procurement review, incident response, and ongoing monitoring. NIST also published an April 7, 2026 concept note for a critical infrastructure profile (NIST, 2026). | Not health-specific and does not itself create legal duties |
| OECD AI Principles | Intergovernmental policy baseline, updated in 2024 | Supports cross-border policy interoperability and national strategy design; especially helpful when public health agencies work across jurisdictions | High-level principles; limited implementation detail for frontline teams |
| UNESCO Recommendation on the Ethics of AI | Global normative standard adopted in 2021 | Strongest on human rights, inclusion, environmental stewardship, and ethical impact assessment, with explicit relevance for LMICs | Broader than health operations, so institutions still need local controls and workflows |
| EU AI Act | Binding risk-based regulation in the European Union | Provides enforceable obligations for high-risk systems, useful as a benchmark even outside Europe when organizations want a stricter compliance floor | Jurisdiction-specific and more legalistic than operational |
A practical way to combine them:
- Use WHO to define the public-health-specific values and actor responsibilities.
- Use NIST AI RMF to operationalize governance inside the organization.
- Use OECD and UNESCO to align national policy with interoperability, human rights, and equity goals.
- Use the EU AI Act as the strongest reference point when building requirements for high-risk use cases.
Most public health institutions are best served by a stacked governance model. WHO supplies sector context, NIST supports execution, OECD and UNESCO support policy alignment, and binding law such as the EU AI Act applies where it governs.
The European Academies Science Advisory Council and the Federation of European Academies of Medicine recommend that European Union policymakers mandate transparent evaluation of an AI system’s net benefit, covering clinical outcomes, patient safety, security, equity, quality of life, and cost-effectiveness, before and after deployment (Cole-Hunter et al., 2026).
Networked Multilateralism
The WHO guidance endorses the UN Secretary-General’s concept of “networked multilateralism,” bringing together:
- United Nations agencies
- International financial institutions
- Regional organizations
- Civil society
- Private sector
- Local authorities
The approach reflects a judgment that traditional intergovernmental processes may be too slow for rapidly evolving AI technologies.
Principles for International AI Governance
Based on WHO guidance, international governance should ensure:
- Inclusive participation: Rules shaped by all countries, not only high-income nations and technology companies
- Accountability: Governments accountable for investments in and deployment of AI systems
- Regulatory convergence: Avoid competitive advantages/disadvantages from regulatory divergence
- Capacity building: Support for countries lacking AI governance infrastructure
- Ethical alignment: International standards upholding human rights and ethical principles
The Governance Gap
International AI governance remains nascent. Current initiatives are largely voluntary, non-binding, and lack enforcement mechanisms. As LMMs become more capable and more widely deployed in healthcare, the gap between technological capability and governance capacity continues to widen.
Public health practitioners should advocate for stronger international coordination and participate in governance discussions when possible.
Policy Recommendations
Evidence-Based Framework
Based on Char et al., 2018, NEJM, Reddy et al., 2020, Journal of the American Medical Informatics Association, and WHO, 2021.
Key Policy Recommendations
1. Adopt Risk-Based Regulatory Framework
- Rationale: Balance innovation with safety proportionate to risk
- Implementation: Classify AI by clinical impact and autonomy level
- Examples: EU AI Act, FDA SaMD framework
- Priority: High | Timeline: Immediate
2. Enable Adaptive AI Regulation
- Rationale: Traditional one-time approval insufficient for learning systems
- Implementation:
- Predetermined Change Control Plans (PCCP)
- Continuous performance monitoring mandates
- Real-world evidence requirements
- Post-market surveillance obligations
- Priority: High | Timeline: 1–2 years
3. Require Prospective Clinical Validation
- Rationale: Retrospective analysis insufficient for clinical deployment
- Implementation:
- Real-world clinical studies (not just algorithm validation)
- Diverse patient populations
- Multiple sites
- Comparison to standard of care
- Clinical outcome measures (not just algorithm metrics)
- Priority: High | Timeline: Immediate
4. Mandate Fairness Audits
- Rationale: Prevent algorithmic bias and health inequities
- Implementation:
- Report performance by age, sex, race/ethnicity
- Context-specific disparity criteria tied to clinical and public health consequences
- Mitigation strategies for identified disparities
- Ongoing fairness monitoring
- Priority: High | Timeline: Immediate
5. Require Model Cards and Transparency
- Rationale: Enable informed use and accountability
- Implementation:
- Standardized model card template
- Public registry of approved AI systems
- Performance metrics by subgroup
- Known limitations and failure modes
- Priority: High | Timeline: 1 year
6. Establish Clear Liability Framework
- Rationale: Clarify accountability when AI causes harm
- Implementation:
- Define “reasonable care” for AI use
- Insurance requirements for high-risk AI
- Incident reporting obligations
- Compensation mechanisms for AI-related harm
- Priority: Medium | Timeline: 2–3 years
7. Support International Harmonization
- Rationale: Reduce duplicative effort, enable global innovation
- Implementation:
- Participate in IMDRF standards development
- Mutual recognition agreements
- Shared validation datasets
- Common terminology and definitions
- Priority: Medium | Timeline: 3–5 years
8. Invest in AI Capacity Building
- Rationale: Ensure workforce readiness and equitable access
- Implementation:
- Training programs for clinicians
- Data science education for health professionals
- Support for LMIC AI development
- Public-private partnerships
- Priority: High | Timeline: Ongoing
Key Takeaways
Risk-based regulation is emerging as a global standard: higher-risk AI requires more stringent oversight
Traditional one-time approval is insufficient for continuously learning AI: adaptive regulatory frameworks are needed
Three lines of defense model provides a robust organizational governance structure
Liability is complex: multiple actors share responsibility when AI causes harm
Transparency is non-negotiable: model cards and explainability are becoming requirements
Clinical validation must be prospective: retrospective analysis alone is insufficient
Fairness audits should be mandatory: performance must be assessed across demographic groups
International harmonization is progressing but remains incomplete: global deployment requires mapping to multiple frameworks
Governance maturity matters: organizations need a structured approach to responsible AI
Policy is evolving rapidly: practitioners should stay informed and engaged in policy development
Hands-On Exercise: Policy Compliance Assessment
Objective: Assess your AI system’s compliance with regulatory and governance requirements.
Part 1: Regulatory Assessment
class RegulatoryComplianceAssessment:
"""Full regulatory compliance checker"""
def assess_compliance(self, ai_system, target_market):
"""
Assess compliance with relevant regulations
Args:
ai_system: Dictionary with system details
target_market: 'US', 'EU', 'UK', 'Global'
"""
results = {}
if target_market in ['US', 'Global']:
results['FDA'] = self.check_fda_compliance(ai_system)
if target_market in ['EU', 'Global']:
results['EU_AI_Act'] = self.check_eu_ai_act_compliance(ai_system)
results['MDR_IVDR'] = self.check_mdr_compliance(ai_system)
if target_market in ['UK', 'Global']:
results['MHRA'] = self.check_mhra_compliance(ai_system)
return results
def check_fda_compliance(self, ai_system):
"""Check FDA compliance"""
requirements = {
'Device classification determined': ai_system.get('fda_class'),
'Appropriate pathway identified': ai_system.get('fda_pathway'),
'Clinical validation completed': ai_system.get('clinical_validation'),
'Labeling includes limitations': ai_system.get('labeling_complete'),
'Performance metrics documented': ai_system.get('performance_documented'),
'Change control plan': ai_system.get('change_control_plan')
}
met = sum(1 for v in requirements.values() if v)
total = len(requirements)
return {
'score': met / total,
'requirements': requirements,
'status': 'Compliant' if met / total >= 0.80 else 'Non-compliant',
'missing': [k for k, v in requirements.items() if not v]
}
def check_eu_ai_act_compliance(self, ai_system):
"""Check EU AI Act compliance"""
requirements = {
'Risk level assessed': ai_system.get('eu_risk_level'),
'Risk management system': ai_system.get('risk_management_system'),
'Data governance': ai_system.get('data_governance'),
'Technical documentation': ai_system.get('technical_docs'),
'Transparency obligations': ai_system.get('transparency_docs'),
'Human oversight measures': ai_system.get('human_oversight'),
'Accuracy/robustness validated': ai_system.get('accuracy_validated'),
'Conformity assessment': ai_system.get('conformity_assessment')
}
met = sum(1 for v in requirements.values() if v)
total = len(requirements)
return {
'score': met / total,
'requirements': requirements,
'status': 'Compliant' if met / total >= 0.80 else 'Non-compliant',
'missing': [k for k, v in requirements.items() if not v]
}
# Example: Assess your AI system
my_ai_system = {
'name': 'My AI System',
'fda_class': 'Class II',
'fda_pathway': '510(k)',
'clinical_validation': True,
'labeling_complete': True,
'performance_documented': True,
'change_control_plan': False, # (missing)
'eu_risk_level': 'High',
'risk_management_system': True,
'data_governance': True,
'technical_docs': True,
'transparency_docs': False, # (missing)
'human_oversight': True,
'accuracy_validated': True,
'conformity_assessment': False # (missing)
}
assessor = RegulatoryComplianceAssessment()
compliance = assessor.assess_compliance(my_ai_system, target_market='Global')
print("REGULATORY COMPLIANCE ASSESSMENT")
print("="*50)
for jurisdiction, results in compliance.items():
print(f"\n{jurisdiction}:")
print(f" Compliance Score: {results['score']:.0%}")
print(f" Status: {results['status']}")
if results['missing']:
print(f" Missing Requirements:")
for req in results['missing']:
print(f" • {req}")Part 2: Governance Assessment
Assess your organization’s AI governance maturity:
class GovernanceMaturityAssessment:
"""Assess organizational AI governance maturity"""
def assess_maturity(self, organization):
"""
Five maturity levels:
1. Initial (Ad hoc, reactive)
2. Developing (Some processes)
3. Defined (Documented processes)
4. Managed (Measured and controlled)
5. Optimizing (Continuous improvement)
"""
criteria = {
'Policy & Strategy': [
'AI strategy defined',
'AI policies documented',
'Board oversight established',
'Regulatory compliance tracked'
],
'Governance Structure': [
'AI ethics committee exists',
'Clear roles and responsibilities',
'Three lines of defense implemented',
'Escalation procedures defined'
],
'Risk Management': [
'Model risk management framework',
'Model inventory maintained',
'Risk assessment for all models',
'Incident response plan'
],
'Validation & Monitoring': [
'Validation standards defined',
'Independent validation required',
'Continuous monitoring implemented',
'Performance reporting automated'
],
'Training & Culture': [
'Staff training programs',
'Ethical AI awareness',
'Clinical engagement',
'Culture of accountability'
]
}
scores = {}
for category, items in criteria.items():
category_score = sum(organization.get(item.lower().replace(' ', '_'), False)
for item in items) / len(items)
scores[category] = category_score
overall_score = sum(scores.values()) / len(scores)
if overall_score >= 0.80:
maturity_level = 5
level_name = 'Optimizing'
elif overall_score >= 0.60:
maturity_level = 4
level_name = 'Managed'
elif overall_score >= 0.40:
maturity_level = 3
level_name = 'Defined'
elif overall_score >= 0.20:
maturity_level = 2
level_name = 'Developing'
else:
maturity_level = 1
level_name = 'Initial'
return {
'maturity_level': maturity_level,
'level_name': level_name,
'overall_score': overall_score,
'category_scores': scores,
'recommendations': self.get_recommendations(maturity_level)
}
def get_recommendations(self, level):
"""Recommendations by maturity level"""
recommendations = {
1: [
'Establish AI ethics committee',
'Draft initial AI policy',
'Create model inventory',
'Identify high-risk AI systems'
],
2: [
'Document AI governance framework',
'Implement pre-deployment review process',
'Establish validation standards',
'Create incident response plan'
],
3: [
'Implement continuous monitoring',
'Automate performance reporting',
'Conduct regular fairness audits',
'Establish training programs'
],
4: [
'Optimize monitoring with AI ops',
'Implement predictive risk management',
'Benchmark against industry',
'Pursue regulatory best practices'
],
5: [
'Lead industry standards development',
'Share best practices publicly',
'Continuous innovation in governance',
'Mentor other organizations'
]
}
return recommendations.get(level, recommendations[3])
# Example: Assess your organization
my_organization = {
'ai_strategy_defined': True,
'ai_policies_documented': True,
'board_oversight_established': False, # (missing)
'regulatory_compliance_tracked': True,
'ai_ethics_committee_exists': True,
'clear_roles_and_responsibilities': True,
'three_lines_of_defense_implemented': False, # (missing)
'escalation_procedures_defined': True,
'model_risk_management_framework': True,
'model_inventory_maintained': True,
'risk_assessment_for_all_models': False, # (missing)
'incident_response_plan': True,
'validation_standards_defined': True,
'independent_validation_required': True,
'continuous_monitoring_implemented': False, # (missing)
'performance_reporting_automated': False, # (missing)
'staff_training_programs': True,
'ethical_ai_awareness': True,
'clinical_engagement': True,
'culture_of_accountability': True
}
maturity_assessor = GovernanceMaturityAssessment()
maturity = maturity_assessor.assess_maturity(my_organization)
print("\nGOVERNANCE MATURITY ASSESSMENT")
print("="*50)
print(f"Maturity Level: {maturity['maturity_level']} - {maturity['level_name']}")
print(f"Overall Score: {maturity['overall_score']:.0%}")
print("\nCategory Scores:")
for category, score in maturity['category_scores'].items():
print(f" {category}: {score:.0%}")
print("\nRecommended Next Steps:")
for i, rec in enumerate(maturity['recommendations'], 1):
print(f" {i}. {rec}")Check Your Understanding
Test your knowledge of AI policy and governance. These questions cover regulatory frameworks, organizational governance, liability, and transparency requirements.
Question 1
An AI startup has developed a diagnostic AI for sepsis prediction that provides decision support to ICU clinicians. The system has 82% accuracy on their internal test set but has NOT been tested in real clinical settings. They plan to seek FDA clearance via the 510(k) pathway using an existing sepsis prediction system as a predicate device. According to the chapter’s discussion of regulatory challenges, what is the PRIMARY concern with this approach?
- The 82% accuracy is too low; FDA requires minimum 90% accuracy for diagnostic AI
- Internal test accuracy alone does not establish substantial equivalence, clinically appropriate performance, or safety for the intended use
- Sepsis prediction is too high-risk and must use the PMA pathway regardless of the predicate
- The system needs to be approved as Class III because sepsis is life-threatening
Correct Answer: b) Internal test accuracy alone does not establish substantial equivalence, clinically appropriate performance, or safety for the intended use
This question tests the distinction between an internal algorithm test and the evidence needed for a specific FDA submission and safe deployment.
Historical evidence boundary: Benjamens and colleagues reported that 55 of 64 AI-enabled devices in their historical database had 510(k) clearance. That snapshot does not establish a current pathway share, and its count of radiology devices is not a measure of the proportion lacking clinical validation (Benjamens et al., 2020).
The 510(k) Pathway Weaknesses:
Known 510(k) weaknesses include:
- “Predicate creep”: cumulative divergence from evidence
- Evidence requirements depend on the device, predicate, intended use, claims, and questions raised during review
- Clearance does not replace local validation, implementation evaluation, or post-deployment monitoring
Why This Is Problematic:
Dataset shift: Finlayson and colleagues explain how changes in populations, workflows, measurement, and clinical practice can alter model performance (Finlayson et al., 2021).
The 510(k) Pathway:
Requirements:
- Demonstrate substantial equivalence to existing device (predicate)
- Provide the performance evidence FDA identifies for the device and intended use; clinical data may be necessary
- Meet current submission, review, labeling, and quality-system requirements
Boundary: FDA clearance and real-world deployment evaluation answer different questions. Health systems still need evidence that the cleared device performs safely in their intended population and workflow.
The scenario describes exactly this problem: 82% accuracy on an internal test set (retrospective analysis) but NOT tested in real clinical settings (no prospective validation).
Why Other Options Are Wrong:
Option (a), 82% accuracy too low, need 90%:
This is factually incorrect. The FDA does NOT have a fixed minimum accuracy threshold:
No universal threshold: FDA evidence expectations depend on the device and its intended use, not a single accuracy number.
Context-dependent: In the IDx-DR pivotal trial, the system reached 87.2% sensitivity and 90.7% specificity for detecting more-than-mild diabetic retinopathy, against prespecified endpoints of 85% and 82.5% (Abràmoff et al., 2018). Those endpoints were set for that application and its clinical need, not as universal FDA requirements.
Misses the real issue: The problem is not the accuracy number itself, but the lack of real-world clinical validation. Even 95% accuracy on a retrospective test set does not prove the system works safely in practice.
Option (c), Too high-risk for 510(k), must use PMA:
This misunderstands FDA classification:
The scenario is insufficient for classification: FDA classification depends on the exact intended use, technological characteristics, product code, and legally marketed predicate.
Pathway cannot be inferred from disease name: A 510(k), De Novo request, PMA, or non-device determination requires product-specific analysis.
Clinical decision support is not one regulatory category: Some functions may be non-device CDS; others remain device functions under section 520(o)(1)(E) and FDA guidance.
Option (d), Must be Class III because sepsis is life-threatening:
This confuses disease severity with device classification:
Disease severity does not determine device class by itself: FDA classification is product-specific and considers intended use, risk, controls, and applicable classification regulation.
Decision support vs. autonomous treatment: The scenario states “provides decision support to ICU clinicians”; this is NOT autonomous treatment. Clinicians make the final decision.
Insufficient information: The scenario does not provide a product code, classification regulation, or predicate analysis, so it cannot establish Class II or Class III status.
The Chapter’s Policy Recommendation:
Policy Recommendation #3: “Require Prospective Clinical Validation”
Rationale: “Retrospective analysis insufficient for clinical deployment”
Implementation:
- Real-world clinical studies (not just algorithm validation)
- Diverse patient populations
- Multiple sites
- Comparison to standard of care
- Clinical outcome measures (not just algorithm metrics)
Priority: High | Timeline: Immediate
The chapter explicitly argues that what the scenario describes (internal test set validation without real-world clinical testing) is insufficient for safe deployment.
The Broader Regulatory Challenge:
Regulation must weigh three goals that can pull against each other:
- Innovation: enable rapid development
- Safety: protect patients from harm
- Equity: ensure fair outcomes
Regulatory clearance, clinical validation, and local implementation assurance are related but distinct. None should be treated as a substitute for the others.
The FDA’s Response:
FDA policy includes:
- Predetermined Change Control Plans (PCCP) - For model updates
- Good Machine Learning Practice (GMLP) - Data quality, validation requirements, real-world performance monitoring
- Patient-Centered Approach - Transparent communication, equity considerations
These policies support lifecycle control, but their applicability and required evidence depend on the exact device and submission.
Real-World Implications: A 510(k) determination addresses substantial equivalence for the cleared intended use. It does not by itself establish effectiveness in every local population, workflow, or downstream clinical outcome.
For practitioners:
The chapter’s message is clear: Retrospective algorithmic validation ≠ Prospective clinical validation
A model can have excellent performance on test data but fail in deployment due to:
- Dataset shift (data distribution changes)
- Integration issues (does not fit workflow)
- Unintended consequences (alert fatigue, over-reliance)
- Unforeseen failure modes
The answer (option B) captures the central concern: a single internal accuracy value is not a complete regulatory or deployment evidence package.
The chapter advocates for requiring prospective clinical validation before deployment, exactly what the scenario’s startup has not done.
Question 2
A hospital is implementing organizational governance for AI clinical decision support systems. According to the chapter’s “Three Lines of Defense” model, who should have the authority to approve or reject high-risk AI deployments, and why is this governance structure important?
- Line 1 (Operational Management - Data Scientists/ML Engineers) because they understand the technical details best
- Line 2 (Oversight Functions - AI Ethics Committee) because they provide independent review with diverse expertise (clinical, technical, ethical, legal) and can mandate corrective actions
- Line 3 (Independent Assurance - Internal Audit) because they provide the most objective assessment
- The hospital CEO because they bear ultimate accountability for patient safety
Correct Answer: b) Line 2 (Oversight Functions - AI Ethics Committee) because they provide independent review with diverse expertise (clinical, technical, ethical, legal) and can mandate corrective actions
This question tests understanding of the Three Lines of Defense governance model, a framework the chapter presents as essential for responsible AI deployment in healthcare organizations.
The Chapter’s Three Lines of Defense Model:
The chapter provides a complete AIGovernanceFramework implementation adapted from IIA, 2020, defining three distinct lines with clear roles:
Line 1: Operational Management
- Roles: Data Scientists/ML Engineers, Clinical Champions, IT Operations
- Responsibilities: Develop models, implement controls, monitor performance, report issues
- Authority: Owns and manages day-to-day risk
Line 2: Oversight Functions
- Roles: AI Ethics Committee, Clinical Safety Officer, Data Governance Board, Risk Management, Compliance Officer
- Responsibilities: Define policies, review and approve high-risk AI projects, monitor compliance, investigate incidents
- Authority: Monitors and advises on risk, can mandate corrective actions
Line 3: Independent Assurance
- Roles: Internal Audit, External Auditors, Clinical Safety Review Board
- Responsibilities: Independent assessment of Lines 1 & 2 effectiveness, validate compliance
- Authority: Provides objective assurance, reports to Board
Why Line 2 (AI Ethics Committee) Approves Deployments:
The chapter provides detailed specification of the AI Ethics Committee charter:
Authority:
- “Approve or reject high-risk AI projects”
- Define AI development and deployment standards
- Investigate AI-related adverse events
- Recommend policy changes to leadership
- Mandate corrective actions for non-compliance
Composition (10 members):
- Clinical: 2 Physicians, 1 Nurse, 1 Patient Advocate (ensures clinical validity, patient perspective)
- Technical: 1 Data Scientist, 1 ML Engineer, 1 IT Security (evaluates technical feasibility, security)
- Oversight: 1 Ethicist, 1 Legal Counsel, 1 Risk Manager (ensures ethical, legal, risk compliance)
Rationale for Diverse Composition:
The chapter emphasizes this diversity is intentional:
Clinical representation: “Ensure clinical validity and patient-centered perspective”
Technical representation: “Evaluate technical feasibility and security”
Oversight representation: “Ensure ethical, legal, and risk compliance”
Review Criteria (8 dimensions):
- Clinical validity and utility
- Fairness across demographic groups
- Transparency and explainability
- Privacy and security measures
- Integration with clinical workflow
- Liability and accountability clarity
- Regulatory compliance
- Resource requirements and cost-effectiveness
Why This Matters:
High-risk AI deployment requires balancing multiple dimensions:
- Is it clinically valid? (Clinical expertise)
- Is it technically sound? (Technical expertise)
- Is it ethically appropriate? (Ethics expertise)
- Is it legally compliant? (Legal expertise)
- Does it manage risk appropriately? (Risk management expertise)
No single stakeholder has all necessary expertise. Line 2’s diverse committee structure ensures all dimensions are evaluated.
Review Triggers:
The chapter specifies when Committee review is required:
- New AI project involving patient care
- Major model update (>10% parameter change)
- AI-related adverse event or near-miss
- Significant performance degradation
- Fairness or bias concerns raised
- Expansion to new patient populations
Decision Types:
- Approve: Proceed with deployment
- Approve with Conditions: Deploy with specific requirements
- Defer: Additional information needed
- Reject: Do not proceed
Why Other Options Are Wrong:
Option (a), Line 1 (Data Scientists) approve:
This creates conflicts of interest and lacks necessary expertise:
Conflict of interest: Line 1 develops the models. Having developers approve their own work violates governance principles of separation of duties.
Limited perspective: Data scientists have technical expertise but may lack:
- Clinical judgment (is this clinically appropriate?)
- Ethical reasoning (does this raise ethical concerns?)
- Legal knowledge (does this comply with regulations?)
- Risk management expertise (what could go wrong?)
Violates Three Lines model: The chapter emphasizes Line 1 “owns and manages risk” but Line 2 “monitors and advises on risk.” Approval authority must be independent of development.
Chapter explicitly states: Line 1’s responsibility is “Report incidents and issues to Line 2”, not approve their own deployments.
Option (c), Line 3 (Internal Audit) approves:
This misunderstands Line 3’s role as independent assurance, not operations:
Wrong function: The chapter defines Line 3’s role as “Independent assessment of Lines 1 & 2 effectiveness”: they audit the governance process; they do not run it.
Timing mismatch: Line 3 conducts periodic audits (annual, quarterly) to validate the process works. They’re not involved in day-to-day approval decisions.
Reporting structure: Line 3 reports to the Board and Executive Leadership, not to operational management. Their role is oversight of the oversight.
Chapter’s framework: Line 3’s responsibilities include “Audit AI governance processes and controls,” “Validate compliance with regulations,” “Recommend improvements to governance framework.” This is evaluating the system, not approving individual deployments.
If Line 3 approved deployments, no independent function would audit whether approvals were appropriate. Line 3 must remain independent to provide objective assurance.
Option (d), CEO approves:
This is impractical and defeats the purpose of governance committees:
Scalability: Hospitals may deploy multiple AI systems. CEOs do not have time or expertise to review each deployment in detail.
Lack of expertise: CEOs are generalists. They lack technical, clinical, and ethical expertise to thoroughly evaluate AI systems.
Defeats committee purpose: If the CEO makes the decisions, the AI Ethics Committee adds no decision value. The chapter’s framework explicitly creates the committee to provide expert review.
Governance best practice: The chapter’s framework has Line 2 “Recommend policy changes to leadership” and Line 3 “Report findings to Board and Executive Leadership.” Leadership provides oversight of the process, not approval of individual deployments.
The CEO’s role: Establish the governance framework, hold Lines 1–3 accountable, and receive reports on AI governance effectiveness. The CEO does not approve every AI deployment.
The Chapter’s Governance Philosophy:
The chapter presents governance as distributed responsibility:
- Line 1 (Operational): Day-to-day development and monitoring
- Line 2 (Oversight): Independent review and approval of high-risk decisions
- Line 3 (Assurance): Periodic audits of the entire system
- Leadership: Oversight of governance effectiveness
Each line has distinct, complementary roles. Collapsing these roles (having developers approve, or executives micromanage) undermines the governance structure.
Real-World Application:
High-Risk AI Deployment Workflow (from chapter):
Step 1: Development (Line 1)
- Data scientists develop sepsis prediction model
- Clinical champions validate clinical appropriateness
- IT Operations tests integration
Step 2: Pre-Deployment Review (Line 2)
- Project team submits required documentation to AI Ethics Committee:
- Project proposal with clinical rationale
- Technical specifications
- Validation results
- Fairness assessment
- Risk assessment and mitigation plan
- Implementation and monitoring plan
- User training plan
- Committee reviews (10 members with diverse expertise)
- Committee decision: Approve, Approve with Conditions, Defer, or Reject
Step 3: Deployment (Line 1, if approved)
- Implement with any conditions from Committee
- Monitor performance continuously
- Report to Committee periodically
Step 4: Audit (Line 3)
- Annual audit validates governance process worked
- Reviews whether Committee decisions were appropriate
- Reports findings to Board
This workflow ensures:
- Development expertise (Line 1) builds the system
- Independent oversight (Line 2) approves deployment
- Objective assurance (Line 3) validates process effectiveness
- Leadership (Board) receives accountability reporting
For practitioners:
The chapter’s message is clear: High-risk AI requires independent, multidisciplinary review before deployment.
Line 2’s AI Ethics Committee structure with diverse expertise (clinical, technical, ethical, legal) is specifically designed to provide this review. This prevents:
- Developers deploying inadequately validated systems (technical blind spots)
- Clinicians deploying ethically problematic systems (ethical blind spots)
- Administrators deploying non-compliant systems (legal blind spots)
The Three Lines of Defense model is the governance framework the chapter presents for healthcare AI governance.
Question 3
A radiologist uses an FDA-cleared AI system to detect lung nodules. The AI misses an obvious lung cancer that the radiologist also fails to identify, resulting in delayed treatment and patient harm. According to the chapter’s discussion of liability frameworks, who is MOST likely to be held liable and under what legal theory?
- Only the AI developer under product liability (strict liability) because the AI failed to detect the nodule
- Only the radiologist under medical malpractice (negligence) for not catching an “obvious” cancer
- Both the AI developer (product liability if AI was defective) AND the radiologist (medical malpractice for not exercising independent clinical judgment), with the radiologist potentially liable even if the AI worked as intended
- The hospital under corporate negligence for deploying inadequately validated AI
Correct Answer: c) Both the AI developer (product liability if AI was defective) AND the radiologist (medical malpractice for not exercising independent clinical judgment), with the radiologist potentially liable even if the AI worked as intended
This question tests understanding of the complex, multi-party liability landscape for medical AI, a critical theme in the chapter’s accountability and liability section.
The Chapter’s Central Liability Question:
The chapter presents this exact dilemma: when AI contributes to patient harm, potentially liable parties include the AI developer, the data provider, the clinician, the hospital or health system, and, rarely, the regulator.
The answer: Multiple parties can be liable simultaneously, under different legal theories.
The Chapter’s Liability Framework:
1. Product Liability (AI Developer)
Legal basis: Strict liability, no need to prove negligence
Requirements to establish:
- Product was defective
- Defect caused injury
- Product was used as intended
The scenario’s AI developer liability:
The chapter provides this EXACT scenario:
“Example case:
- Radiologist uses FDA-cleared AI for lung nodule detection
- AI misses obvious cancer
- Patient sues”
Potential developer liability:
- **“AI developer:** Liable if model defective (failed validation standards)”
Challenge: Defining “defect” for AI
The chapter asks:
- Performance below promised accuracy?
- Below human expert performance?
- Below peer AI systems?
If the AI:
- Performed below its stated accuracy specifications → Defective product
- Failed validation standards → Defective product
- Missed an “obvious” nodule that should be detected → Potentially defective
The chapter’s LiabilityAssessment class identifies developer risks:
if not ai_system.get('clinical_validation'):
risks.append({
'risk': 'Inadequate validation',
'severity': 'High',
'legal_basis': 'Defective product (strict liability)',
'mitigation': 'Conduct prospective clinical validation study'
})2. Medical Malpractice (Clinician)
Legal basis: Negligence
Must prove:
- Duty of care existed (doctor-patient relationship)
- Duty was breached (missed “obvious” cancer)
- Breach caused harm (delayed treatment)
- Damages resulted (patient harm)
The chapter, drawing on Balkin, 2017, expects clinicians to:
- Understand AI limitations - Know when to override
- Maintain competence - Do not blindly follow AI
- Use clinical judgment - AI is decision support, not replacement
Key point: “Radiologist: May still be liable for not catching obvious error (standard of care)”
Even if the AI worked as intended, the radiologist may be liable if the cancer was “obvious” to a competent radiologist.
The Pneumonia Model Example:
The chapter provides the Caruana et al., 2015 example to illustrate clinician liability:
Pneumonia risk model paradox:
- Model learned: Asthma history → Lower mortality risk
- Reality: Asthma patients go straight to ICU → Aggressive treatment → Better outcomes
- If deployed blindly: Asthma patients sent home → Worse outcomes
- Liability: Clinician potentially liable for not recognizing illogical recommendation
This illustrates: Clinicians cannot blindly follow AI. They must exercise independent judgment.
Applied to the scenario:
If the cancer was “obvious,” a reasonable radiologist should have detected it regardless of what the AI said. The AI’s failure does not excuse the radiologist’s failure.
The “AI as decision support, not replacement” principle:
The chapter emphasizes throughout: AI provides decision support. Clinicians retain ultimate responsibility.
From the chapter: “Use clinical judgment - AI is decision support, not replacement”
The chapter’s LiabilityAssessment identifies clinician risks:
if ai_system.get('autonomy') == 'autonomous':
risks.append({
'risk': 'Over-reliance on autonomous AI',
'severity': 'High',
'legal_basis': 'Failure to exercise clinical judgment',
'mitigation': 'Require mandatory human review and documentation of rationale'
})Why Both Can Be Liable Simultaneously:
The chapter’s framework shows liability is not mutually exclusive:
Developer liable IF:
- AI performed below specifications → Defective product
- Inadequate validation → Should have known it would miss cancers
- Failure to disclose limitations → Failure to warn
Radiologist liable IF:
- Failed to catch “obvious” cancer → Below standard of care
- Over-relied on AI → Did not exercise independent judgment
- Did not understand AI limitations → Incompetent use of tool
Both conditions can be true simultaneously. The AI can be defective AND the radiologist can be negligent.
Why Other Options Are Wrong:
Option (a), Only AI developer liable:
This ignores the radiologist’s independent duty of care:
Standard of care: Radiologists have a duty to detect obvious cancers. This duty exists independently of what tools they use.
AI as tool, not replacement: If a carpenter’s saw is defective and they cut themselves, the saw manufacturer may be liable, but the carpenter is also responsible for safe tool use.
Chapter’s explicit statement: “Radiologist: May still be liable for not catching obvious error (standard of care)”
Moral hazard: If only developers are liable, clinicians have no incentive to maintain competence. They could blindly follow AI and escape accountability.
Option (b), Only radiologist liable:
This ignores potential product defect:
AI may be defective: If the AI missed an “obvious” cancer that its specifications said it should detect, it’s defective.
Strict liability exists: Product liability applies if the product was defective and caused harm, regardless of clinician negligence.
Developer responsibilities: The chapter’s
LiabilityAssessmentidentifies developer duties:- Adequate validation
- Post-market surveillance
- Clear limitations disclosure
If the developer failed these duties, they’re liable even if the clinician was also negligent.
Multiple causes: Legal principle: Harm can have multiple causes. Both defective product AND negligent use can contribute to harm.
Option (d), Hospital liable (corporate negligence):
While hospitals CAN be liable, the question asks who is MOST likely liable:
Hospital liability requires: The chapter states hospitals must ensure:
- Proper credentialing (approved safe AI)
- Adequate oversight (monitoring in place)
- Sufficient training (staff know how to use AI)
Scenario does not indicate hospital failure: The AI is “FDA-cleared”, and the scenario gives no indication of inadequate credentialing, oversight, or training.
More direct causes exist: The AI’s failure (developer) and radiologist’s failure (clinician) are more direct causes of harm than institutional failures.
Chapter’s framework: Hospital liability is typically additional, not replacement of developer/clinician liability.
The LiabilityAssessment framework identifies ALL THREE potential liabilities:
developer = self.assess_developer_liability(ai_system) # Product liability
clinician = self.assess_clinician_liability(ai_system) # Medical malpractice
institutional = self.assess_institutional_liability(ai_system) # Corporate negligenceThe chapter’s detailed report structure shows: All three can be liable simultaneously, under different legal theories.
The Practical Implication:
From a liability perspective:
AI Developer must:
- Ensure adequate validation (catch obvious cancers in validation)
- Disclose known limitations
- Monitor post-market performance
- Insurance: Product liability insurance scaled to deployment exposure
Radiologist must:
- Understand AI limitations
- Maintain competence (do not deskill)
- Exercise independent judgment (do not blindly follow)
- Insurance: Professional malpractice insurance
Hospital must:
- Credential AI systems (governance approval)
- Train clinicians adequately
- Monitor for incidents
- Insurance: General liability + Cyber liability
For practitioners:
The chapter’s message: Liability in AI-augmented healthcare is complex and multi-party.
Key principle: AI is a tool, not a replacement for clinical judgment. Clinicians cannot escape liability by claiming “the AI told me to” any more than they can escape liability by claiming “the blood test was wrong.”
Developer liability does not absolve clinician liability, and vice versa.
The scenario exemplifies this: Both the AI developer (for potentially defective product) and the radiologist (for not catching obvious cancer) can be held liable under their respective legal frameworks.
Option C correctly captures this complex, multi-party liability reality that the chapter emphasizes throughout its accountability section.
Question 4
According to the EU AI Act discussed in the chapter, a hospital’s AI triage system that allocates ICU resources during a pandemic would be classified as high-risk healthcare AI. Which requirement would be MOST critical for compliance, and why?
- Prohibit the system entirely as “unacceptable risk” because it involves resource allocation that affects access to care
- Require human oversight designed so users can interpret outputs, decide when not to use the system, and interrupt or stop it (Article 14)
- Require only transparency obligations to inform users they’re interacting with AI
- No specific requirements since administrative tools are classified as “minimal risk”
Correct Answer: b) Require human oversight designed so users can interpret outputs, decide when not to use the system, and interrupt or stop it (Article 14)
This question tests understanding of the EU AI Act’s risk-based framework and human oversight requirements for high-risk healthcare AI, a central regulatory approach presented in the chapter.
The EU AI Act Risk-Based Classification:
The EU AI Act establishes explicit risk-based categories and phased obligations:
| Risk Level | Healthcare Examples | Requirements |
|---|---|---|
| Unacceptable | Social scoring and specified biometric practices prohibited under Article 5 | PROHIBITED |
| High | AI covered by Article 6, including a product or safety component subject to third-party conformity assessment, plus specified Annex III uses such as emergency healthcare patient triage systems (Annex III, point 5(d)) | Risk management, quality data, transparency, human oversight, conformity assessment, post-market monitoring |
| Limited | Direct-interaction chatbots not otherwise classified as high-risk | Transparency obligations, disclose AI use |
| Minimal | Administrative tools | No specific obligations |
A triage system falls in the high-risk category when its intended purpose matches an Annex III use, such as emergency healthcare patient triage; classification still requires intended-purpose analysis.
Why Human Oversight (Article 14) is Critical:
The chapter provides the complete EU AI Act Article 14 requirements in the EUAIActCompliance class:
Article 14: Human Oversight
- Designed for effective oversight by humans
- Users can interpret outputs
- Users can decide when not to use
- Users can interrupt or stop the system
Documentation: Human oversight procedures
Why This Matters for Triage:
The High-Stakes Nature of Triage:
Triage systems allocate scarce resources (ICU beds, ventilators) with life-or-death consequences:
- Who gets an ICU bed during overwhelmed capacity?
- Who receives a ventilator when supply is limited?
- Who is prioritized for treatment?
These decisions:
- Cannot be fully automated: Require human judgment for individual circumstances
- Must be accountable: Clinicians/administrators must be able to explain why decisions were made
- May need override: Edge cases require human expertise to override algorithmic recommendations
- Involve ethical trade-offs: Utilitarian calculations (save the most lives) vs. fairness (first-come-first-served, random lottery) require human deliberation
The Four Human Oversight Requirements:
1. “Users can interpret outputs”
For triage, this means:
- Understanding WHY a patient was prioritized or deprioritized
- Knowing WHAT factors the AI considered (age, comorbidities, severity, likelihood of survival)
- Seeing the evidence behind the recommendation
Implementation: Explainable AI showing key factors influencing triage score
2. “Users can decide when not to use”
For triage, this means:
- Clinicians can choose NOT to follow AI recommendation
- Alternative decision-making process exists (e.g., clinical ethics committee)
- No punishment for overriding AI in appropriate circumstances
Implementation: Clear protocols for when to use/not use AI triage, escalation procedures
3. “Users can interrupt or stop the system”
For triage, this means:
- Emergency override capability (if AI behaves erratically during crisis)
- Ability to pause system for investigation if bias/errors detected
- Fallback to manual triage protocols
Implementation: Kill switch, fallback procedures, incident response
4. “Designed for effective oversight by humans”
For triage, this means:
- Interface shows relevant information for human decision-making
- Appropriate response times (not so fast humans cannot evaluate)
- Training for clinicians on how to exercise oversight
Implementation: User-centered design, training programs, decision support (not automation)
Why This Is THE MOST Critical Requirement:
While all EU AI Act requirements are important, human oversight is uniquely critical for triage because:
Ethical necessity: Resource allocation decisions involve ethical trade-offs that algorithms cannot resolve. The chapter emphasizes: “AI is decision support, not replacement.”
Accountability: Without human oversight, accountability for wrong triage decisions becomes unclear. The chapter’s liability section emphasizes accountability requires human involvement.
Trust and legitimacy: Patients and society must trust triage is fair. Fully automated triage without human oversight undermines legitimacy.
Error correction: Triage occurs in chaotic, evolving situations (pandemics). Humans must be able to recognize when AI recommendations are inappropriate for current circumstances.
Contrast with Other Requirements:
- Risk management (Article 9): Important but generic, applies to development process
- Data governance (Article 10): Important but focuses on training data quality
- Transparency (Article 13): Important but focuses on documentation
- Accuracy/robustness (Article 15): Important but focuses on technical performance
Human oversight (Article 14) directly addresses the deployment decision-making process, the moment when AI recommendations translate to actual resource allocation affecting patients.
Why Other Options Are Wrong:
Option (a), Prohibit as unacceptable risk:
This misunderstands the EU AI Act’s risk categories:
Triage is “High” risk, not “Unacceptable”: Article 5 does not list triage among prohibited practices, and emergency healthcare patient triage systems appear among the Annex III high-risk uses.
Unacceptable = Prohibited entirely: Social scoring and specified biometric practices are prohibited under Article 5. Triage systems are not categorically prohibited, but the exact intended purpose determines the applicable requirements.
The distinction: Unacceptable risk harms fundamental rights with no legitimate purpose. Triage has legitimate purpose (save lives during resource scarcity) but requires safeguards.
EU’s approach is risk-based regulation, not prohibition: The chapter emphasizes the EU allows high-risk AI with appropriate safeguards, not blanket prohibition.
Option (c), Only transparency obligations (limited risk):
This underestimates triage system risk:
Limited-risk transparency duties can apply to direct-interaction chatbots not otherwise classified as high-risk. A chatbot’s intended purpose can instead trigger high-risk requirements. Triage systems require intended-purpose analysis and cannot be classified from the interface alone.
Transparency alone is insufficient: Knowing you’re interacting with AI does not protect you if the AI makes bad triage decisions.
Chapter’s framework: High risk requires 8 detailed requirements, not just transparency:
- Risk management system
- High-quality training data
- Technical documentation
- Transparency AND user information
- Human oversight
- Accuracy, robustness, cybersecurity
- Conformity assessment
- Post-market monitoring
Option (d), Minimal risk (administrative tools):
This completely misclassifies triage systems:
Triage is patient care, not administration: Administrative tools (scheduling, billing) have minimal patient safety impact. Triage determines who lives and who dies; this is HIGH risk.
Intended-purpose classification: A triage system whose intended purpose matches Annex III, point 5(d), is high-risk, not minimal-risk.
No requirements for minimal risk: If triage were minimal risk, it would need no special compliance, which is clearly inappropriate for life-or-death decisions.
The Chapter’s Compliance Checklist Example:
The generate_compliance_checklist method classifies a system as High risk only when its Article 6(1) product or Annex III flag is set; a healthcare label alone is not enough. For a high-risk system, its detailed requirements include:
Article 14: Human Oversight: (required list)
- Designed for effective oversight by humans
- Users can interpret outputs
- Users can decide when not to use
- Users can interrupt or stop the system
Broader Context: The Chapter’s Governance Philosophy:
This aligns with multiple chapter themes:
1. Innovation, Safety, and Equity:
- Innovation: AI triage can optimize resource allocation
- Safety: Require human oversight to prevent harm
- Equity: Require fairness assessment to prevent discrimination
Human oversight helps balance all three.
2. The Three Lines of Defense:
- Line 1: Develops triage system with human oversight interface
- Line 2: Ethics committee reviews human oversight procedures before approval
- Line 3: Audits whether human oversight is actually used in practice
3. Accountability and Liability:
Without human oversight:
- Liability for a failed triage decision becomes unclear
- Clinicians lose the opportunity to exercise clinical judgment
- Decisions cannot be explained to patients and families
The chapter’s liability framework requires human involvement for accountability.
Real-World Implementation:
Compliant AI Triage System:
Interface shows:
- Patient severity score with confidence interval
- Key factors: age, comorbidities, vital signs, expected survival probability
- Alternative patients competing for resources
- Historical triage decisions for comparison
Human controls:
- Checkbox: “I have reviewed this recommendation and agree/disagree”
- Override button: “Prioritize this patient for clinical reasons”
- Notes field: “Document rationale for override”
- Emergency stop: “Pause AI and revert to manual triage”
Training:
- How to interpret AI triage scores
- When to override (examples: pregnancy, clinical information the model does not capture, atypical presentations)
- Escalation to ethics committee for difficult decisions
Monitoring:
- Track override rates (too high = AI not useful, too low = over-reliance)
- Investigate outcomes of overrides vs. followed recommendations
- Audit fairness across demographic groups
For practitioners:
The chapter’s message: High-risk healthcare AI requires human oversight to ensure accountability, ethical appropriateness, and ability to handle edge cases.
The EU AI Act Article 14’s human oversight requirements ensure:
- Humans remain in the loop for high-stakes decisions
- Accountability is clear (human made final decision)
- Error correction is possible (human can override)
- Ethical deliberation occurs (human considers factors AI cannot)
For triage systems, where decisions affect who lives and who dies, human oversight is not optional. It’s the most critical requirement for ethical, accountable, and trusted AI deployment.
Option B correctly identifies this as the EU AI Act’s key safeguard for high-risk healthcare AI systems.
Question 5
A sepsis prediction AI is deployed and performs well initially (AUC 0.82) but degrades over 18 months to AUC 0.68 due to changes in EHR documentation practices and COVID-19 altering patient populations. According to the chapter, which regulatory approach would BEST address this “concept drift” challenge?
- Require resubmission for full FDA approval every time any performance drop is detected
- Use the FDA PCCP framework for planned modifications, where applicable, together with continuous performance monitoring and degradation alerts
- Prohibit any model updates after approval to ensure consistency
- Require only annual performance reports with no real-time monitoring
Correct Answer: b) Use the FDA PCCP framework for planned modifications, where applicable, together with continuous performance monitoring and degradation alerts
This question tests understanding of dataset shift, planned device modifications, and ongoing performance monitoring.
The Concept Drift Problem:
Illustrative sepsis scenario: The AUC values in the question are hypothetical. They illustrate the types of dataset shift discussed by Finlayson and colleagues; the paper does not report those values (Finlayson et al., 2021).
Possible causes of performance degradation:
- Changes in clinical practice (COVID-19 protocols)
- Different patient population demographics
- Electronic health record system updates
- New treatment protocols
The chapter’s Policy Recommendation #2 states: “Traditional one-time approval insufficient for learning systems.”
Why Traditional Regulation Falls Short:
Traditional regulatory assumptions contrast with AI behavior:
Traditional regulations assume:
- Static devices - Do not change after approval
- Transparent logic - Decision rules can be inspected
- Predictable performance - Same input → same output
AI systems violate these assumptions:
- Continuous learning - Models update with new data
- Black box decisions - Neural networks lack interpretability
- Distribution shift - Performance degrades when data changes
FDA Lifecycle Policy:
FDA’s final PCCP guidance describes how planned modifications may be reviewed within a marketing submission (FDA, 2025).
Key controls:
1. Predetermined Change Control Plans (PCCP)
- Pre-specify allowed model update types
- Monitor performance without new submission for each update
- Define planned modifications, modification protocols, and impact assessment
2. Good Machine Learning Practice (GMLP)
- Data quality standards
- Model validation requirements
- Real-world performance monitoring
3. Patient-Centered Approach
- Transparent communication about AI limitations
- Patient involvement in development
- Health equity considerations
Why PCCP is the Answer:
What PCCP Does:
Pre-approval of update protocols: At initial FDA review, the developer specifies:
- What types of updates will be made (e.g., retrain on new data, adjust thresholds)
- How updates will be validated (holdout test sets, performance metrics)
- What triggers updates (performance degradation, new data availability)
- What safety rails prevent harmful updates (minimum performance thresholds, rollback procedures)
During deployment:
- Continuous monitoring detects performance degradation (AUC 0.82 → 0.68)
- Automated alerts notify when performance drops below threshold
- Modifications authorized within the PCCP may be implemented without an additional marketing submission for each covered change
- Changes outside the authorized PCCP require separate regulatory assessment
Applied to the Scenario:
With PCCP:
Illustrative submission record, not an FDA template:
- Sepsis prediction model (AUC 0.82 on validation set)
- PCCP specifying:
- Performance monitoring at a clinically justified interval
- A prospectively justified alert threshold
- A defined modification protocol
- Validation criteria tied to intended use and risk
- Rollback: If updated model performs worse, revert to previous version
- Annual report: Summary of updates, performance trends, safety signals
During deployment:
- Monitoring: A prospectively defined alert detects material degradation
- Response: Investigate cause (EHR documentation changes identified)
- Modification: The covered update is developed under the authorized protocol
- Validation: The update meets the pre-specified acceptance criteria
- Deployment: Update deployed under pre-approved PCCP (no full resubmission needed)
- Monitoring continues: Ensure new model maintains performance
Advantages of PCCP:
- Addresses concept drift: Allows updates without full reapproval process
- Maintains safety: Pre-specified validation ensures updates do not harm patients
- Enables learning systems: AI can adapt to changing clinical environment
- Reduces regulatory burden: Updates follow approved protocol, not full resubmission
- Requires monitoring: Continuous performance tracking catches degradation early
- Maintains accountability: Developer responsible for monitoring and safe updates
Why Other Options Fail:
Option (a), Resubmit for full approval for any drop:
This is impractically burdensome and does not match real-world needs:
Too slow: Full FDA submission takes months. By the time approval comes through, performance may have degraded further or the clinical environment changed again.
Too rigid: Small performance fluctuations are normal. Requiring full resubmission for minor drops (0.82 → 0.80) is excessive.
Defeats AI advantage: If you cannot update AI models, you lose the benefit of adaptive systems that improve over time.
Not sustainable: Clinical practice evolves constantly (new EHRs, new treatments, pandemics). Models need to adapt accordingly.
Chapter’s critique of traditional approach: The whole point of PCCP is that traditional “one-time approval” is insufficient. This option doubles down on the failed approach.
Option (c), Prohibit updates after approval:
This ensures safety through stagnation, which is worse than adaptation:
Risks degradation: As clinical practice changes, a locked model can degrade. The scenario’s illustrative AUC drop from 0.82 to 0.68 would be dangerous.
Defeats AI purpose: Machine learning’s strength is learning. Prohibiting learning eliminates AI’s advantage over rule-based systems.
Patient harm: A degraded model (AUC 0.68) provides worse care than an updated model (AUC 0.82 restored). Prohibiting updates harms patients.
Unsustainable: Eventually, the model becomes so degraded it must be retired. Then you need a new model, new approval process, new validation. Better to allow controlled updates.
Contradicts FDA’s direction: The FDA’s AI/ML Action Plan explicitly recognizes the need for adaptive regulation. This option rejects that entirely.
Option (d), Annual reports only, no real-time monitoring:
This detects problems too late:
18-month degradation undetected: The scenario shows degradation over 18 months. Annual reporting might not catch this until significant harm has occurred.
Slow response: Even if the annual report shows degradation, it takes additional time to update the model. Meanwhile, poor performance continues.
No alerts: Without real-time monitoring, nobody knows performance is degrading. Clinicians may trust a model that’s actually unreliable.
Chapter’s recommendations: The chapter’s Policy Recommendation #2 explicitly calls for:
- “Continuous performance monitoring mandates”
- “Real-world evidence requirements”
- “Post-market surveillance obligations”
Annual reporting alone does not meet these requirements.
- The FDA’s GMLP proposal: Includes “Real-world performance monitoring”, not just annual reports.
The Chapter’s Broader Context:
Policy Recommendation #2: Enable Adaptive AI Regulation
Rationale: “Traditional one-time approval insufficient for learning systems”
Implementation:
- Predetermined Change Control Plans (PCCP)
- Continuous performance monitoring mandates
- Real-world evidence requirements
- Post-market surveillance obligations
Priority: High | Timeline: 1–2 years
The chapter explicitly identifies PCCP as high-priority solution to the concept drift problem.
Innovation, Safety, and Equity:
- Innovation: PCCP enables AI to adapt and improve → Supports innovation
- Safety: Pre-specified validation ensures updates are safe → Maintains safety
- Equity: Monitoring can track performance across demographic groups → Supports equity
PCCP balances all three objectives better than rigid traditional approval.
International Alignment:
The chapter notes IMDRF (International Medical Device Regulators Forum) is working toward harmonized approaches. PCCP-like frameworks are emerging in several jurisdictions as one approach to adaptive AI regulation.
Implementation Considerations:
PCCP Development (for developers):
Required components:
- Performance monitoring plan: What metrics, how often, what thresholds
- Update triggers: What circumstances initiate model updates
- Validation protocol: How updates are tested before deployment
- Safety rails: Minimum performance, rollback procedures, human oversight
- Documentation: What records are kept, what’s reported to FDA
- Periodic review: How often FDA reviews the PCCP effectiveness
FDA Review (for regulators):
- Initial approval: Evaluate whether PCCP adequately protects patients
- Periodic audits: Verify developer follows PCCP, updates are safe
- Post-market surveillance: Aggregate data across multiple AI systems to identify trends
For practitioners:
The chapter’s message: AI is different from traditional medical devices. Regulation must evolve.
Traditional approach:
- Approve once, assume device stays static
- Works for pacemakers, surgical instruments
AI reality:
- Performance drifts as world changes
- Models must adapt to maintain safety and efficacy
PCCP solution:
- Pre-approve update protocols
- Require continuous monitoring
- Enable safe, controlled adaptation
This balances innovation (AI can improve), safety (updates are validated), and practicality (not every update requires full resubmission).
Option B correctly identifies PCCP with continuous monitoring as the FDA’s lifecycle mechanism for managing concept drift, the central regulatory challenge the chapter uses to motivate need for adaptive frameworks.
Question 6
A public health AI system will be deployed globally (US, EU, UK). According to the chapter, which strategy would be MOST effective for navigating the different regulatory requirements across jurisdictions while ensuring the system meets high standards?
- Design for the loosest regulatory requirements (to minimize cost and speed deployment), then add compliance features only when regulators demand them
- Build a reusable control and evidence baseline, then map the exact system to each jurisdiction’s classification, submission, and post-market requirements
- Create completely separate AI systems for each jurisdiction to perfectly match local regulations
- Wait for international harmonization to complete before deploying in any market
Correct Answer: b) Build a reusable control and evidence baseline, then map the exact system to each jurisdiction’s classification, submission, and post-market requirements
This question tests understanding of practical multi-jurisdictional regulatory strategy, synthesizing the chapter’s coverage of FDA, EU, and UK regulatory frameworks and international harmonization efforts.
The Chapter’s Regulatory Landscape:
The chapter presents three major regulatory frameworks:
1. United States (FDA):
- Software as a Medical Device (SaMD) framework
- Three pathways: 510(k), De Novo, PMA
- Risk-based classification (Class I, II, III)
- AI/ML Action Plan (PCCP, GMLP, patient-centered approach)
2. European Union (EU AI Act + MDR/IVDR):
- Risk-based classification (Unacceptable, High, Limited, Minimal)
- Detailed requirements for high-risk AI:
- Risk management (Article 9)
- Data governance (Article 10)
- Transparency (Article 13)
- Human oversight (Article 14)
- Accuracy/robustness/cybersecurity (Article 15)
- Conformity assessment
- Post-market monitoring
- Penalties: Article 99 sets different maxima by violation category; the highest tier is EUR 35 million or 7% for prohibited practices
3. United Kingdom (MHRA):
- Post-Brexit pragmatic approach
- Risk-proportionate regulation
- Innovation-friendly fast-track
- International alignment (mutual recognition with FDA, EU)
Comparison boundary: There is no single, transferable ranking of jurisdictional stringency. The EU AI Act, EU medical-device rules, FDA requirements, and UK rules classify systems differently and may require different evidence, quality-system controls, submissions, and post-market duties.
The “Design Up” Strategy:
How a shared baseline helps:
1. Shared Coverage:
EU AI Act controls can contribute to a shared assurance baseline, but they do not automatically satisfy FDA or MHRA requirements:
Article 9 (Risk Management):
- Identified risks
- Risk mitigation measures
- May support risk-management evidence used in other jurisdictions
Article 10 (Data Governance):
- High-quality, representative, bias-examined data
- May support data-governance evidence, subject to jurisdiction-specific requirements
Article 13 (Transparency):
- Instructions for use, limitations, accuracy levels, failure modes
- May support labeling and transparency work, but required content and format differ
Article 14 (Human Oversight):
- Users can interpret, override, stop system
- May support human-oversight evidence, without determining FDA device status
Article 15 (Accuracy/Robustness):
- Validated accuracy
- Robust against errors
- Cybersecurity measures
- May support validation planning, but the required endpoints and evidence remain product-specific
2. Documentation Reusability:
The EU AI Act requires extensive documentation:
- Technical documentation
- Risk management plan
- Data quality report
- Model card
- Validation report
- Human oversight procedures
Parts of this documentation may support other jurisdictions’ applications:
- FDA 510(k) submission: Use technical documentation, validation report, risk assessment
- FDA De Novo: Use clinical validation, performance metrics, intended use documentation
- MHRA UKCA marking: Use conformity assessment, technical documentation, performance data
A controlled core dossier can reduce duplication, but some markets require distinct analyses, evidence, forms, and legal representatives.
3. Future-Proofing:
The chapter notes regulatory convergence:
- IMDRF (International Medical Device Regulators Forum) working toward harmonization
- Common risk classification frameworks emerging
- Mutual recognition agreements developing
IMDRF work can improve alignment, but it does not make one jurisdiction’s compliance package automatically valid in another.
4. Penalties are obligation-specific: Under Article 99, prohibited-practice violations can reach EUR 35 million or 7% of worldwide annual turnover; specified operator and transparency violations can reach EUR 15 million or 3%; and misleading information can reach EUR 7.5 million or 1%, subject to the regulation’s rules for undertakings and SMEs (EU AI Act, Article 99).
The Jurisdiction-Specific Documentation Caveat:
Each jurisdiction has specific documentation formats and submission requirements:
FDA-specific:
- 510(k) premarket notification format
- Predicate device comparison (if using 510(k))
- Specific performance metrics (sensitivity/specificity)
- FDA-mandated labeling format
EU-specific:
- CE marking conformity declaration
- Notified body assessment (for certain devices)
- EUDAMED database registration
- EU-specific adverse event reporting
UK-specific:
- UKCA marking declaration
- MHRA-specific submission format
- UK-specific post-market surveillance reporting
Practical approach: Maintain a controlled core evidence dossier, then complete a jurisdiction-specific classification and gap assessment before preparing each submission or conformity package.
Why Other Options Fail:
Option (a), Design for loosest requirements:
This is a “race to the bottom” that creates multiple problems:
Eventual retrofitting costs: When you try to enter stricter markets (EU), you’ll need extensive redesign and revalidation. Retrofitting is more expensive than designing right initially.
Reputation risk: If your system causes harm in a loosely-regulated market, it damages brand reputation globally. The chapter’s liability section shows this can be catastrophic.
Ethical problems: The chapter emphasizes patient safety and equity. Designing to minimum standards means accepting lower safety/performance, contradicting responsible AI principles.
Regulatory change: Requirements evolve, so change control and periodic legal review are necessary.
Enforcement risk: Penalties and remedies vary by obligation, entity, jurisdiction, and facts. Compliance must be evaluated in each market.
Option (c), Separate systems per jurisdiction:
This is inefficient and unsustainable:
Development costs: Building three entirely separate AI systems triples development costs, technical team, data collection, validation, documentation for each.
Maintenance burden: Three separate systems need three separate update processes, three monitoring systems, three incident response procedures. As the chapter discusses with concept drift, AI requires ongoing maintenance.
Knowledge fragmentation: Learnings from one market do not transfer to others. If you discover a bias in the EU system, you must separately discover and fix it in FDA and MHRA systems.
Scaling problems: Adding Canada, Australia, Japan, or Singapore would require yet more separate systems. This does not scale.
Misses harmonization trend: The chapter discusses IMDRF working toward harmonization. Separate systems do not use converging standards.
The chapter’s discussion of international harmonization (IMDRF section) implies a common system with jurisdiction-specific documentation is the intended future state, not completely separate systems.
Option (d), Wait for complete harmonization:
This is overly cautious and impractical:
Indefinite wait: The chapter notes: “Challenge: Harmonization requires balancing local sovereignty with global interoperability.” Full harmonization may take years or decades (if ever).
Opportunity cost: While waiting, competitors deploy in available markets. Patients in those markets do not benefit from your AI.
No learning: You do not learn from real-world deployment while waiting. The chapter emphasizes real-world evidence and post-market surveillance; you cannot get this while waiting.
Harmonization progress requires participation: IMDRF harmonization happens through industry engagement. Sitting on the sidelines does not advance harmonization.
Chapter’s policy recommendation (#7): “Support International Harmonization” (Priority: Medium | Timeline: 3–5 years). This is long-term, not immediate. Do not wait 5 years to deploy.
The Pragmatic Multi-Jurisdiction Strategy (Option B):
Phase 1: Design a shared control baseline
- Build lifecycle controls for risk, data, validation, human oversight, security, monitoring, and change management
- Map each control to EU, FDA, and UK requirements without assuming equivalence
- Maintain jurisdiction-specific legal and regulatory gap analyses
Phase 2: Validate for each intended use and market
- Define endpoints, comparators, populations, and evidence from the applicable pathway
- Reuse verified core evidence where appropriate, while completing market-specific requirements
- Generate traceable documentation that distinguishes shared evidence from jurisdiction-specific evidence
Phase 3: Regulatory Submissions
- EU: Complete the applicable AI Act and medical-device conformity pathway
- FDA: Determine device status, classification, and the applicable 510(k), De Novo, or PMA pathway
- MHRA: Apply current Great Britain or Northern Ireland requirements and transition rules
Phase 4: Deployment
- Deploy in all three markets
- Single unified system (easier to maintain)
- Jurisdiction-specific labels/documentation
Phase 5: Post-Market
- Single monitoring system tracking performance globally
- Report to each jurisdiction in their required format
- Updates apply globally (with PCCP or equivalent)
The Chapter’s Supporting Evidence:
1. MHRA’s “International alignment”:
The chapter states MHRA seeks “Mutual recognition with FDA, EU.” This does not make EU or FDA compliance automatically sufficient for MHRA.
2. FDA lifecycle controls:
FDA’s final PCCP guidance and GMLP principles support lifecycle planning, but they do not make EU compliance a substitute for FDA requirements.
3. IMDRF harmonization goals:
- Harmonized definitions and terminology
- Common risk classification framework
- Shared validation standards
- Mutual recognition agreements
These efforts can reduce unnecessary divergence, but they do not establish automatic cross-jurisdictional compliance.
For practitioners:
The chapter’s multi-jurisdiction guidance is implicit but clear:
Global regulatory strategy should:
- Use a strong shared control baseline without ranking unlike legal regimes as a single “highest” standard
- Maintain documentation supporting each jurisdiction’s specific submission format
- Use harmonization efforts (IMDRF, mutual recognition) to reduce duplicative work
- Monitor regulatory evolution (FDA’s GMLP, EU AI Act implementation) and adapt
Option B embodies this strategy: maintain shared evidence and controls, then complete jurisdiction-specific classification and compliance work.
This approach supports reuse without erasing differences in intended use, legal classification, or evidence requirements.
Discussion Questions
Innovation vs. Safety: How should regulators balance enabling rapid AI innovation with ensuring patient safety? Where should the line be drawn?
Adaptive Regulation: Should continuously learning AI models be allowed? If so, what safeguards are necessary?
Liability: Who should bear primary liability when AI causes harm: developer, clinician, or hospital? Should AI developers have liability caps?
Transparency: How much transparency is enough? Should all AI models be fully explainable, or is “black box” acceptable with sufficient validation?
International Harmonization: Should AI regulations be harmonized globally, or should countries have different standards based on local values and priorities?
Clinical Validation: What level of clinical validation should be required before AI deployment? Is retrospective analysis sufficient, or should prospective trials be mandatory?
Equity: How can policy ensure AI does not widen health disparities? Should performance across demographic groups be regulated?
Workforce: How should healthcare professionals be trained and credentialed to use AI? Should AI competency be required for licensure?
Further Resources
Key Guidance Documents
Regulatory:
- FDA: AI/ML SaMD Action Plan (Jan 2021)
- FDA: AI-Enabled Medical Device List
- EU AI Act: European Parliament overview
- MHRA: Software and AI as Medical Device
- WHO: Ethics and Governance of AI for Health
Governance:
- IIA: Three Lines Model
- OCC/Federal Reserve: SR 11-7 - Model Risk Management
Essential Papers
Regulation and Policy:
- Gerke et al., 2020, npj Digital Medicine - Regulatory challenges
- Benjamens et al., 2020, npj Digital Medicine - FDA-authorized AI medical-device database
- Char et al., 2018, NEJM - Policy recommendations
- Reddy et al., 2020, Journal of the American Medical Informatics Association - Governance framework
Liability:
- Price, 2017, Michigan Law Review - Regulating black-box medicine
- Balkin, 2017, Ohio State Law Journal - The three laws of robotics in the age of big data
Transparency:
- Mitchell et al., 2019 - Model Cards
- Finlayson et al., 2021, NEJM - Dataset shift
Implementation:
- Abràmoff et al., 2018, npj Digital Medicine - IDx-DR pivotal trial and FDA authorization
- Caruana et al., 2015, KDD - Intelligible models
Tools and Resources
Regulatory Databases:
- FDA 510(k) Database - Cleared medical devices
- EUDAMED - EU medical device database
Governance Tools:
- IMDRF Resources - International harmonization
- Model Card Toolkit - Create model cards
Training and Education
Courses:
- FDA: AI/ML Medical Device Regulation (FDA training programs)
- Coursera: AI in Healthcare Specialization
- edX: Ethics of AI (various universities)
Professional Organizations:
- Healthcare Information and Management Systems Society (HIMSS)
- American Medical Informatics Association (AMIA)
- International Medical Device Regulators Forum (IMDRF)
Looking Ahead
This handbook has covered the full lifecycle of AI in public health:
- Part I: Foundations: AI history, machine learning fundamentals, and public health data
- Part II: Current Applications: surveillance, forecasting, genomics, geospatial analysis, clinical decision support, and overdose prevention
- Part III: Implementation and Evaluation: evaluation, equity, explainability, ethics, privacy, safety, deployment, and operations
- Part IV: Practical Tools and Resources: toolkits, first projects, and AI-assisted coding
- Part V: The Future: emerging technologies, global health equity, policy, misinformation, and large language models
For a ready-to-adapt governance policy you can implement at your organization, see the AI Governance Policy Template.
The Public Health AI Handbook is free to read and cite, with no advertising. If this chapter informed your work, you can support the handbook.