Ethics, Bias, and Equity in Healthcare AI

Algorithms amplify patterns in data. When healthcare cost data encodes decades of structural racism, AI perpetuates that inequity at scale. Obermeyer’s landmark 2019 study revealed a widely deployed risk prediction algorithm systematically disadvantaged Black patients because it used spending as a proxy for need, ignoring that Black patients receive less care when equally sick. Ethical AI in public health requires understanding fairness is contested, bias exists across the entire development lifecycle, and technical solutions alone cannot substitute for representative data, stakeholder engagement, and governance.

Learning Objectives

This chapter examines ethical challenges in public health AI. You will learn to:

  • Identify sources of bias across AI lifecycle (Obermeyer case study)
  • Apply competing fairness definitions and their mathematical trade-offs
  • Distinguish individual fairness from group fairness
  • Implement algorithmic impact assessment frameworks
  • Conduct equity audits across demographic subgroups and regions
  • Apply bias mitigation techniques at each development stage
  • Establish governance structures for accountable AI deployment

Prerequisites: familiarity with data collection methods (The Data Problem) and model evaluation metrics (Evaluating AI Systems).

The Big Picture: Algorithms amplify whatever patterns exist in your data. If your data encodes decades of inequality, your algorithm will perpetuate that inequality efficiently, at scale, with the appearance of objectivity. Obermeyer’s healthcare cost algorithm systematically disadvantaged Black patients because it used spending as a proxy for need, ignoring that structural racism means Black patients receive less care when equally sick.

The Obermeyer Case Study (Why This Matters):

Commercial risk prediction tool deployed at major hospitals to identify high-risk patients for care management programs: - The Problem: At any given risk score, Black patients were significantly sicker than White counterparts - Root Cause: Algorithm used healthcare costs as proxy for healthcare needs - Fatal Assumption: Spending reflects illness severity (false, systemic disparities mean Black patients receive less care = lower costs despite equal/greater illness) - Impact: Black patients had to be considerably sicker to receive same risk score and access to care programs - Lesson: Technical accuracy ≠ ethical fairness. Must evaluate equity explicitly.

Sources of Bias Across AI Lifecycle:

  1. Historical Bias: Training data reflects past inequities (redlining, discriminatory lending, unequal access)
  2. Representation Bias: Underrepresented groups in training data → poor performance for those groups
  3. Measurement Bias: Proxies systematically mismeasure outcomes for certain groups (pulse oximetry less accurate for darker skin)
  4. Aggregation Bias: One-size-fits-all model when subgroups have different patterns (disease progression varies by population)
  5. Evaluation Bias: Testing on non-representative samples, missing disparities
  6. Deployment Bias: Unequal access to AI-enabled interventions, differential impact

Competing Fairness Definitions (You Can’t Satisfy All):

  1. Demographic Parity: Equal selection rates across groups (50% men, 50% women flagged)
  2. Equalized Odds: Equal true/false positive rates across groups
  3. Predictive Parity (Calibration): Equal precision, if flagged, equal probability of true positive
  4. Individual Fairness: Similar individuals treated similarly

The Impossibility Theorem: Except in trivial cases, you cannot satisfy all fairness criteria simultaneously. Trade-offs are inevitable, must make explicit value judgments about which fairness concept matters most for your context.

“Fairness Through Blindness” Fails:

Removing protected attributes does NOT guarantee fairness: - Proxies remain (ZIP code correlates with race, name correlates with ethnicity) - Historical patterns baked into other variables - You must measure disparities to address them, removing race prevents you from detecting discrimination

Bias Mitigation Techniques:

  1. Pre-Processing: Reweight training data, augment underrepresented groups, remove biased features
  2. In-Processing: Constrain model during training (fairness-aware algorithms, adversarial debiasing)
  3. Post-Processing: Adjust predictions/thresholds by group to achieve fairness criteria

No technical fix alone is sufficient, requires stakeholder engagement, governance, continuous monitoring.

Algorithmic Impact Assessment Framework:

Before deployment, systematically evaluate: 1. Who is affected? Identify all stakeholders, especially vulnerable populations 2. What are potential harms? Direct (wrong decisions) and indirect (perpetuating stereotypes) 3. Are benefits/burdens distributed equitably? Who gains, who bears risk? 4. Can harms be mitigated? Technical, procedural, governance interventions 5. Is deployment justified? Do benefits outweigh risks? Are alternatives better?

Equity Audits, Continuous Monitoring:

  • Disaggregate performance metrics by race, ethnicity, sex, age, socioeconomic status, geography
  • Monitor for disparate impact post-deployment
  • Establish feedback mechanisms for affected communities
  • Critical: Disparities can emerge post-deployment even if absent in testing

Governance Structures for Accountable AI:

  • Algorithm Review Boards: Interdisciplinary oversight (clinicians, ethicists, community representatives, patients)
  • Transparency Requirements: Document training data sources, fairness trade-offs, intended use
  • Participatory Design: Involve affected communities in development, not just deployment
  • Contestability Mechanisms: Right to explanation, appeal, human override

Public Health’s Dual Mandate:

  1. Population Health: Maximize health outcomes
  2. Health Equity: Ensure fair distribution of those outcomes

AI must actively advance equity, not just avoid worsening it. Equity-blind AI perpetuates the status quo, which already embeds systematic inequities.

The Takeaway for Public Health Practitioners:

Ethics isn’t a final checklist. It’s embedded throughout the AI lifecycle from problem formulation to ongoing monitoring. Obermeyer proves that technically accurate algorithms can be deeply unjust. Fairness has multiple valid definitions that conflict, you must make explicit value judgments. Removing protected attributes doesn’t guarantee fairness, you must measure disparities to address them. Bias mitigation techniques help but require stakeholder engagement and governance structures. Public health’s equity mandate means AI systems must be held to higher standards than commercial applications. Before deploying, conduct algorithmic impact assessment. After deploying, continuously monitor for emergent disparities. Establish review boards with community representation. Most importantly: when in doubt about fairness, err on the side of caution, an accurate but inequitable algorithm is worse than no algorithm at all.

Introduction: The Stakes of Getting It Wrong

The Obermeyer Case: When Algorithms Amplify Inequality

In 2019, researchers led by Ziad Obermeyer uncovered a troubling discovery embedded in one of healthcare’s most widely used algorithms (Obermeyer et al., 2019, Science). A commercial risk prediction tool, deployed across major hospital systems to identify patients who would benefit from high-risk care management programs, was systematically discriminating against Black patients. The algorithm appeared neutral on its surface, making predictions based purely on data. Yet at any given risk score, Black patients were significantly sicker than their White counterparts, suffering from more chronic conditions and demonstrating greater medical need. The algorithm was missing the patients who needed help most.

The root cause lay in a seemingly reasonable design choice: the algorithm used healthcare costs as a proxy for healthcare need. This assumption, that spending reflects illness severity, failed to account for a fundamental reality of American healthcare: systemic racial disparities in access to care mean that Black patients historically receive less care, and therefore incur lower costs, than equally sick White patients. By training on historical data that encoded decades of structural inequity, the algorithm learned to perpetuate that same inequity at scale. The consequence was stark: Black patients had to be considerably sicker than White patients to receive the same risk score and access to care management programs.

This is AI ethics in one brutal lesson: algorithms amplify whatever patterns exist in your data. If your data encodes decades of inequality, your algorithm will perpetuate that inequality efficiently, at scale, with the appearance of objectivity.

Key findings from the study:

  • Algorithm used healthcare costs as proxy for healthcare needs
  • Systematically disadvantaged Black patients
  • At given risk score, Black patients significantly sicker than White patients
  • Impact: Reduced access to care management programs for those who needed it most
  • Root cause: Historical inequities embedded in training data

Why Ethics Matters in Public Health AI

  • Public health has dual mandate: population health + health equity
  • AI systems can amplify existing inequities at scale
  • Decisions affect vulnerable populations with limited recourse
  • Trust is essential for public health effectiveness
  • Historical context of medical racism and exploitation

The Challenge of Operationalizing Ethics

  • Moving from abstract principles to concrete practices
  • Trade-offs between competing ethical values
  • Technical constraints vs. ethical ideals
  • Balancing innovation with precaution

Ethical Frameworks for AI in Public Health

Traditional Biomedical Ethics

The Four Principles:

  1. Beneficence - Obligation to benefit patients/populations
  • AI should improve health outcomes
  • Evidence of clinical utility required
  • Consider opportunity costs
  1. Non-maleficence - “First, do no harm”
  • Minimize risks from errors, bias, and misuse
  • Consider indirect harms (e.g., opportunity costs, psychological impact)
  • Precautionary principle in high-stakes settings
  1. Autonomy - Respect for individual agency
  • Informed consent challenges with AI
  • Right to explanation of AI decisions
  • Opt-out mechanisms where appropriate
  • Protection of decisional capacity
  1. Justice - Fair distribution of benefits and burdens
  • Equitable access to AI-enabled interventions
  • Fair representation in training data
  • Avoiding discrimination
  • Addressing structural inequities

Public Health Ethics Extensions

Population-Level Considerations:

  • Collective welfare vs. individual rights
  • Acceptable trade-offs for population benefit
  • Special obligations to vulnerable populations
  • Solidarity and social cohesion

Health Equity as Core Principle:

  • Not just equal treatment, but equitable outcomes
  • Targeted universalism: universal goals, targeted strategies
  • Addressing social determinants of health
  • Centering most marginalized populations

AI-Specific Ethical Frameworks

The EU AI Act Approach:

  • Risk-based regulatory framework
  • Prohibited practices (social scoring, exploitation of vulnerabilities)
  • High-risk AI systems (healthcare, critical infrastructure)
  • Transparency requirements

IEEE Ethically Aligned Design:

  • Human rights framework
  • Well-being prioritization
  • Data agency
  • Effectiveness and transparency
  • Accountability

WHO Guidance on AI Ethics:

The World Health Organization established six consensus principles for AI in health, originally published in 2021 and updated in 2024 with specific guidance for large multi-modal models (WHO, 2024):

  1. Protecting human autonomy - Humans remain in control of health systems and medical decisions; patients understand AI’s role in their care
  2. Promoting human well-being and safety - Designers satisfy regulatory requirements; AI is not used if alternatives would prevent avoidable harm
  3. Ensuring transparency, explainability, and intelligibility - Sufficient information published for meaningful public debate; users can make informed choices
  4. Fostering responsibility and accountability - Clear accountability throughout AI lifecycle; mechanisms for redress when harm occurs
  5. Ensuring inclusiveness and equity - AI accessible regardless of characteristics; designed with input from affected communities
  6. Promoting AI that is responsive and sustainable - Continuously assessed for real-world impact; minimizes environmental footprint
LMM-Specific Considerations

The 2024 WHO guidance addresses large multi-modal models (LMMs) like GPT-4 and Claude specifically, noting that these systems pose unique ethical challenges:

  • Epistemic risks: LMMs providing authoritative-seeming but potentially false health information may “undermine human epistemic authority” in healthcare and science
  • Automation bias: Health professionals and patients may over-rely on LMM outputs, overlooking errors
  • Consent complexity: Patients may not understand when LMMs are involved in their care
  • Environmental costs: LMM training and operation have significant carbon and water footprints

See Large Language Models in Public Health for practical LMM guidance and AI Policy and Governance for regulatory frameworks.

The Ethics-Safety Interconnection

Ethical violations often manifest as safety failures, and safety failures typically have ethical dimensions:

How ethics and safety connect:

Ethical Violation Safety Consequence Example
Bias/Inequity Differential harm across groups Obermeyer algorithm denies care to Black patients → Health disparities worsen
Lack of transparency Misuse and over-reliance Clinicians trust opaque model → Fail to catch errors → Patient harm
Inadequate consent Loss of trust → System abandonment Patients discover unconsented AI use → Refuse care → Population health impact
Privacy violation Reduced participation Data breach → Community distrust → Lower surveillance effectiveness

The relationship:

  • Ethics provides the “should”: What outcomes and processes are morally acceptable?
  • Safety provides the “how”: What technical and operational safeguards prevent harm?

This chapter addresses ethical frameworks and bias mitigation. For operational safety validation, failure mode analysis, and safety-critical system design, see AI Safety in Healthcare.

The Bias Landscape: Where Things Go Wrong

Taxonomy of Bias in AI Systems

1. Historical Bias

Definition: Bias existing in the world that gets encoded in data

Examples: - Historical healthcare disparities reflected in EHR data - Underdiagnosis of certain conditions in specific populations - Differential access to care - Systemic discrimination in social determinants

Impact: Models learn and perpetuate existing inequities

2. Representation Bias

Definition: Training data does not reflect target population

Examples: - Underrepresentation of racial/ethnic minorities - Gender imbalance in clinical trials - Geographic concentration (e.g., only urban hospitals) - Age distribution skew

Impact: Poor performance on underrepresented groups

3. Measurement Bias

Definition: Features systematically measured differently across groups

Examples: - Pulse oximetry accuracy varies by skin tone - Different diagnostic thresholds by race/ethnicity - Differential missingness patterns - Proxy variables that encode discrimination

Impact: Biased model predictions even with unbiased algorithms

4. Aggregation Bias

Definition: One-size-fits-all model applied to diverse populations

Examples: - Single diabetes risk model across ethnic groups - Ignoring gene-environment interactions - Universal treatment protocols despite heterogeneity - Population-average effects masking subgroup variation

Impact: Poor performance for minority groups, missed opportunities for personalization

5. Evaluation Bias

Definition: Benchmarks that do not reflect diverse populations

Examples: - Test sets with inadequate minority representation - Performance metrics that hide disparities - Validation in non-representative settings - Ignoring subgroup performance differences

Impact: Biased systems appear accurate, disparities go undetected

6. Deployment Bias

Definition: Mismatch between development and deployment context

Examples: - Model trained in academic center, deployed in community hospital - Different workflows, patient populations, resources - Changed incentive structures - Differential adoption by provider characteristics

Impact: Real-world performance differs from validation, disparities emerge

Fairness: Defining an Elusive Concept

Mathematical Definitions of Fairness

Group Fairness Metrics

1. Demographic Parity (Statistical Parity)

P(Ŷ = 1 | A = a) = P(Ŷ = 1 | A = b)

Equal selection rates across groups

When appropriate: - Loans, hiring, college admissions - When base rates reflect historical discrimination

Limitations: - May require accepting less qualified applicants - Ignores relevant group differences

2. Equalized Odds (Error Rate Parity)

P(Ŷ = 1 | Y = y, A = a) = P(Ŷ = 1 | Y = y, A = b) for all y

Equal true positive rates AND false positive rates

When appropriate: - Healthcare screening - Bail/parole decisions - When ground truth is reliable

Limitations: - Requires labeled outcomes for all groups - May be impossible to achieve if base rates differ

3. Predictive Parity (Outcome Parity)

P(Y = 1 | Ŷ = 1, A = a) = P(Y = 1 | Ŷ = 1, A = b)

Equal positive predictive value across groups

When appropriate: - Resource allocation decisions - When acting on predictions - Diagnostic settings

4. Calibration Fairness

P(Y = 1 | Ŷ = p, A = a) = p for all groups a

Predicted probabilities match observed frequencies in each group

When appropriate: - Risk communication - Clinical decision support - When probabilities inform decisions

The Impossibility Theorem

Chouldechova (2017) and Kleinberg et al. (2017):

Cannot simultaneously achieve equalized odds, predictive parity, and calibration when: - Base rates differ between groups - Model is not perfect

Implications: - Must choose which fairness criterion to prioritize - Trade-offs are inevitable - Context determines appropriate choice - Transparency about choices is essential

Individual Fairness

Definition: Similar individuals should receive similar predictions

d(Ŷ(x₁), Ŷ(x₂)) ≤ L · d(x₁, x₂)

Challenges: - Defining “similarity” without encoding bias - Computational complexity - Requires domain expertise

Fairness Through Awareness vs. Blindness

Fairness Through Blindness: - Remove protected attributes from model - Appeal: Seems to prevent discrimination

Why it fails: - Proxy variables (zip code → race) - Historical bias remains in other features - Prevents monitoring for disparities

Fairness Through Awareness: - Explicitly consider protected attributes - Monitor performance by subgroup - Apply group-specific interventions if needed - Enables accountability

Bias Mitigation: A Lifecycle Approach

Pre-processing: Fixing the Data

Data Collection Strategies

1. Representative Sampling - Oversample underrepresented groups - Stratified sampling by key attributes - Community-engaged data collection - Multi-site studies for diversity

2. Measurement Standardization - Validate instruments across populations - Culturally adapted assessments - Minimize differential missingness - Address measurement bias (e.g., pulse oximetry)

3. Synthetic Data Augmentation - Generate samples for minority groups - Preserve distributional properties - Validate augmented data carefully - Combine with real data

Data Pre-processing Techniques

1. Reweighing - Assign weights to balance groups - Maintains original data - Simple to implement

Code example:

# Reweighing for fairness
from aif360.algorithms.preprocessing import Reweighing

reweigher = Reweighing(
 unprivileged_groups=[{'race': 0}],
 privileged_groups=[{'race': 1}]
)
dataset_transformed = reweigher.fit_transform(dataset)

2. Sampling - Oversample minority class - Undersample majority class - SMOTE for synthetic minority samples

3. Relabeling - Correct biased labels when ground truth available - Use multiple annotators - Calibrate for systematic errors

In-processing: Constraining the Model

Fairness-Constrained Optimization

Adversarial Debiasing:

from aif360.algorithms.inprocessing import AdversarialDebiasing

debiaser = AdversarialDebiasing(
 privileged_groups=[{'sex': 1}],
 unprivileged_groups=[{'sex': 0}],
 scope_name='debiaser',
 debias=True,
 adversary_loss_weight=0.5
)
debiaser.fit(dataset_train)

How it works: - Primary model predicts outcome - Adversary tries to predict protected attribute from predictions - Train to maximize prediction accuracy while minimizing adversary’s accuracy

Fairness Regularization:

Add fairness penalty to loss function:

L_total = L_prediction + λ × L_fairness

Examples: - Demographic parity penalty: |P(Ŷ=1|A=a) - P(Ŷ=1|A=b)| - Equalized odds penalty: difference in TPR and FPR

Multi-objective Optimization: - Pareto frontier of accuracy vs. fairness - Explore trade-off space - Stakeholder input on acceptable trade-offs

Group-Specific Models

Separate Models: - Train distinct models for each subgroup - Maximizes within-group performance - Challenges: sample size, generalizability, explicit group-based treatment

Mixture of Experts: - Learn to route instances to specialized sub-models - Soft assignments - Can discover latent subgroups

Post-processing: Adjusting Predictions

Threshold Optimization

Group-Specific Thresholds:

def optimize_group_thresholds(y_true, y_pred_proba, groups, criterion='equalized_odds'):
 thresholds = {}
 for group in groups.unique():
  mask = (groups == group)
  # Find threshold that optimizes criterion for this group
  thresholds[group] = find_optimal_threshold(
   y_true[mask],
   y_pred_proba[mask],
   criterion
  )
 return thresholds

Calibration: - Platt scaling by group - Isotonic regression - Ensure predicted probabilities match observed frequencies

Reject Option Classification: - Create “uncertain” zone around decision boundary - For instances in this zone, assign label to favor disadvantaged group - Trade-off between fairness and accuracy

Conducting an Algorithmic Impact Assessment

Framework Overview

Systematic evaluation of potential impacts before deployment

Key Questions: 1. What problem are we solving and for whom? 2. Who might be harmed? 3. What are the fairness considerations? 4. What safeguards are in place? 5. How will we monitor ongoing impacts?

Step 1: Stakeholder Mapping

Identify All Affected Parties: - Direct users (clinicians, public health officials) - Affected populations (patients, communities) - Indirect stakeholders (payers, policymakers) - Vulnerable subgroups

Engagement Strategies: - Community advisory boards - Focus groups with affected populations - Expert consultations - Participatory design workshops

Step 2: Problem Formulation Audit

Critical Questions:

  • Framing: How is the problem defined? Who defined it?
  • Objective: What are we optimizing for? Whose values?
  • Scope: What’s included/excluded? Why?
  • Assumptions: What do we assume about the world?

Red Flags: - Problem defined without input from affected communities - Optimization metric misaligned with equity goals - Narrow framing that misses root causes - Unstated assumptions that encode bias

Step 3: Data Assessment

Assessing population representativeness is a critical first step.

def assess_representation(data, population_stats, protected_attr):
 """Compare data demographics to population"""
 data_props = data[protected_attr].value_counts(normalize=True)

 for group in population_stats.keys():
  data_pct = data_props.get(group, 0) * 100
  pop_pct = population_stats[group] * 100
  disparity = data_pct - pop_pct

  print(f"{group}:")
  print(f" Data: {data_pct:.1f}%")
  print(f" Population: {pop_pct:.1f}%")
  print(f" Disparity: {disparity:+.1f}pp")

Check for Measurement Bias: - Differential missingness by group - Proxy variables that encode discrimination - Validated measures across subgroups

Step 4: Model Fairness Analysis

Disaggregated Performance:

def disaggregated_metrics(y_true, y_pred, groups):
 """Calculate metrics by subgroup"""
 results = []
 for group in groups.unique():
  mask = (groups == group)
  metrics = {
   'group': group,
   'n': mask.sum(),
   'prevalence': y_true[mask].mean(),
   'sensitivity': recall_score(y_true[mask], y_pred[mask]),
   'specificity': recall_score(1-y_true[mask], 1-y_pred[mask]),
   'ppv': precision_score(y_true[mask], y_pred[mask]),
   'auc': roc_auc_score(y_true[mask], y_pred[mask])
  }
  results.append(metrics)
 return pd.DataFrame(results)

Fairness Metrics: - Calculate multiple fairness definitions - Visualize disparities - Identify which groups are disadvantaged

Step 5: Deployment Context Analysis

Workflow Integration: - How will AI fit into existing processes? - Who interprets and acts on outputs? - What happens when AI is wrong?

Incentive Structures: - How might AI change behavior? - Could it be gamed or manipulated? - Unintended consequences?

Differential Impact: - Will some groups experience AI differently? - Access barriers to AI-enabled interventions? - Digital divide considerations?

Step 6: Safeguards and Mitigation

Technical Safeguards: - Uncertainty quantification - Anomaly detection - Human oversight mechanisms - Audit trails

Governance Safeguards: - Clear accountability structures - Regular audits - Stakeholder feedback mechanisms - Sunset clauses and re-evaluation triggers

Recourse Mechanisms: - Appeals process - Human review of AI decisions - Right to explanation - Compensation for harms

Health Equity-Centered AI Development

Principles of Equity-Centered Design

1. Center Marginalized Voices - Participatory design with affected communities - Community advisory boards - Compensate community experts - Power-sharing in decision-making

2. Address Root Causes - Don’t just predict inequities, act to reduce them - Target interventions at structural determinants - Avoid “technological solutionism”

3. Targeted Universalism - Universal goals (e.g., reduce diabetes complications) - Targeted strategies for disadvantaged groups - Invest more in groups with greater needs

4. Monitor and Mitigate Continuously - Equity is not a one-time check - Ongoing monitoring by subgroup - Rapid response to emerging disparities - Transparent reporting

Equity Metrics Beyond Fairness

1. Differential Impact

Change in disparities after AI deployment:

Disparity_after - Disparity_before

Goal: Reduce existing gaps, do not widen them

2. Benefit Distribution

Who benefits from AI-enabled improvements?

def benefit_distribution(outcomes_before, outcomes_after, groups):
 """Calculate which groups benefit most"""
 benefits = outcomes_after - outcomes_before

 for group in groups.unique():
  mask = (groups == group)
  avg_benefit = benefits[mask].mean()
  print(f"{group}: {avg_benefit:.3f} improvement")

 # Ratio of benefits between most/least advantaged
 benefit_ratio = benefits[disadvantaged].mean() / benefits[advantaged].mean()
 print(f"Proportional benefit ratio: {benefit_ratio:.2f}")

3. Access Equity

  • Can all who would benefit actually access AI-enabled care?
  • Digital literacy requirements
  • Infrastructure barriers
  • Cost barriers

4. Trust and Acceptability

  • Do vulnerable populations trust the AI system?
  • Cultural acceptability
  • Historical context of medical exploitation
  • Community acceptance

Case Study: Equity-Centered Sepsis Prediction

[Detailed case study showing equity-centered development process]

Context: Sepsis disproportionately affects low-income patients and racial minorities

Traditional Approach: - Train on all EHR data - Optimize for overall accuracy - Deploy universally

Problems: - Underrepresentation of minorities in training data - Different sepsis presentation patterns by group - Lower baseline access to ICU care

Equity-Centered Approach:

  1. Stakeholder Engagement:
  • Community health workers
  • Patient advocates
  • Clinicians serving underserved populations
  1. Problem Reframing:
  • Goal: Reduce sepsis mortality disparities
  • Not just predict sepsis, but ensure equitable access to treatment
  1. Data Strategy:
  • Oversample minority patients
  • Multi-site data including safety-net hospitals
  • Validate measurement equivalence (vital signs, labs)
  1. Fairness Constraints:
  • Equalized false negative rates (do not miss cases in any group)
  • Group-specific calibration
  1. Deployment Design:
  • Automated alerts for high-risk patients
  • Proactive outreach to underserved communities
  • Resource allocation to address capacity constraints
  1. Monitoring:
  • Track sepsis mortality by race/ethnicity/SES
  • Monitor alert response times by patient characteristics
  • Quarterly equity audits

Outcomes: - Reduced sepsis mortality overall - Narrowed mortality gap between groups - High clinician trust and acceptance

Governance and Accountability

Organizational Structures for Responsible AI

AI Ethics Committee

Composition: - Technical experts (data science, AI) - Domain experts (epidemiology, clinical) - Ethicists - Community representatives - Legal/compliance - Patients/public

Responsibilities: - Review proposed AI projects - Approve high-risk deployments - Monitor ongoing systems - Investigate concerns - Update policies

Decision-Making Process: - Risk-based review (low/medium/high) - Structured checklist - Required documentation - Community consultation for high-risk - Transparent decisions with rationale

Roles and Responsibilities

Data Scientists: - Conduct bias audits - Implement fairness constraints - Document limitations - Transparent reporting

Domain Experts: - Define appropriate use cases - Interpret predictions in context - Monitor clinical validity - Engage affected communities

Leadership: - Allocate resources for responsible AI - Set organizational values - Ensure accountability - Model ethical behavior

Community Members: - Provide lived experience expertise - Review from affected population perspective - Hold institutions accountable - Co-design solutions

Accountability Mechanisms

1. Documentation Requirements

Model Cards: - Intended use - Performance across subgroups - Training data characteristics - Fairness metrics - Limitations - Ethical considerations

Datasheets for Datasets: - Motivation and composition - Collection process - Preprocessing - Distribution - Maintenance plan

2. Audit Requirements

Pre-deployment: - Bias audit - Fairness assessment - Impact assessment - Stakeholder review

Ongoing: - Quarterly performance monitoring - Annual comprehensive audit - Triggered by concerns

3. Transparency Measures

Internal: - Documentation accessible to all staff - Training on responsible AI - Clear escalation pathways

External: - Public model cards (when appropriate) - Aggregate performance reporting - Stakeholder engagement - Incident reporting

4. Recourse Mechanisms

  • Clear process to contest AI decisions
  • Human review available
  • Timeline for resolution
  • Compensation for verified harms

Regulatory Landscape

FDA Regulation of AI/ML Medical Devices: - Risk-based framework - Predetermined change control plans - Real-world performance monitoring

EU AI Act: - Prohibited practices - High-risk AI requirements - Transparency obligations - Conformity assessments

State-Level Initiatives: - Algorithmic accountability acts - Impact assessment requirements - Anti-discrimination protections

Practical Tools and Workflows

Bias Audit Toolkit

Step 1: Define Protected Attributes

protected_attributes = ['race', 'ethnicity', 'sex', 'age_group', 'income_level', 'insurance_type']

Step 2: Assess Representation

from bias_toolkit import check_representation

representation_report = check_representation(
 data=training_data,
 protected_attrs=protected_attributes,
 reference_population=census_data
)

representation_report.plot()
representation_report.to_html('representation_audit.html')

Step 3: Calculate Performance by Subgroup

from sklearn.metrics import classification_report
import pandas as pd

def subgroup_performance(y_true, y_pred, y_proba, groups, group_col):
 results = []
 for group in groups[group_col].unique():
  mask = (groups[group_col] == group)

  perf = {
   'group': group,
   'n': mask.sum(),
   'prevalence': y_true[mask].mean(),
   'auc': roc_auc_score(y_true[mask], y_proba[mask]),
   'sensitivity': recall_score(y_true[mask], y_pred[mask]),
   'specificity': recall_score(1-y_true[mask], 1-y_pred[mask]),
   'ppv': precision_score(y_true[mask], y_pred[mask]),
   'npv': precision_score(1-y_true[mask], 1-y_pred[mask])
  }
  results.append(perf)

 return pd.DataFrame(results)

# Generate report for each protected attribute
for attr in protected_attributes:
 report = subgroup_performance(y_true, y_pred, y_proba, test_data, attr)
 report.to_csv(f'performance_by_{attr}.csv')

 # Visualize disparities
 plot_performance_disparities(report, attr)

Step 4: Calculate Fairness Metrics

from aif360.metrics import BinaryLabelDatasetMetric, ClassificationMetric
from aif360.datasets import BinaryLabelDataset

# Prepare data
dataset = BinaryLabelDataset(
 df=test_data,
 label_names=['outcome'],
 protected_attribute_names=['race']
)

predictions = BinaryLabelDataset(
 df=test_data.assign(outcome=y_pred),
 label_names=['outcome'],
 protected_attribute_names=['race']
)

# Calculate fairness metrics
metric = ClassificationMetric(
 dataset,
 predictions,
 unprivileged_groups=[{'race': 0}],
 privileged_groups=[{'race': 1}]
)

fairness_report = {
 'Statistical Parity Difference': metric.statistical_parity_difference(),
 'Disparate Impact': metric.disparate_impact(),
 'Equal Opportunity Difference': metric.equal_opportunity_difference(),
 'Average Odds Difference': metric.average_odds_difference(),
 'Theil Index': metric.theil_index()
}

print(pd.Series(fairness_report))

Step 5: Intersectional Analysis

def intersectional_analysis(y_true, y_pred, groups, attrs):
 """Analyze performance across intersectional groups"""

 # Create intersectional categories
 groups['intersectional'] = groups[attrs].astype(str).agg('_'.join, axis=1)

 results = []
 for group in groups['intersectional'].unique():
  mask = (groups['intersectional'] == group)
  if mask.sum() < 30: # Skip small groups
   continue

  perf = {
   'group': group,
   'n': mask.sum(),
   'auc': roc_auc_score(y_true[mask], y_pred[mask]) if mask.sum() > 0 else None
  }
  results.append(perf)

 return pd.DataFrame(results).sort_values('auc')

# Example: Race × Gender × Age
intersectional_results = intersectional_analysis(
 y_true, y_pred, test_data,
 attrs=['race', 'sex', 'age_group']
)

Equity Assessment Checklist

Problem Formulation - [ ] Problem defined with input from affected communities - [ ] Objective aligned with equity goals - [ ] Considered alternatives to AI - [ ] Identified potential for harm

Data - [ ] Assessed representativeness vs. target population - [ ] Checked for measurement bias across groups - [ ] Investigated differential missingness - [ ] Documented data limitations

Model Development - [ ] Disaggregated performance by subgroup - [ ] Calculated multiple fairness metrics - [ ] Conducted intersectional analysis - [ ] Applied bias mitigation techniques

Validation - [ ] Tested in diverse settings - [ ] Validated with underrepresented groups - [ ] Assessed calibration by subgroup - [ ] Evaluated equity metrics

Deployment Planning - [ ] Analyzed deployment context for differential impacts - [ ] Designed safeguards and human oversight - [ ] Established monitoring plan - [ ] Created recourse mechanisms

Governance - [ ] Ethics committee review - [ ] Stakeholder engagement - [ ] Documented decisions - [ ] Assigned accountability

Model Cards and Datasheets: Documentation Standards

Transparent documentation is essential for responsible AI deployment. Two complementary frameworks have emerged as industry standards.

Model Cards (Mitchell et al., 2019)

Model Cards provide structured documentation for trained machine learning models, similar to nutrition labels for food.

Required sections:

Section Content
Model Details Developer, version, type, training date, license
Intended Use Primary use cases, users, out-of-scope uses
Factors Groups, instrumentation, environment relevant to performance
Metrics Performance measures and why they were chosen
Evaluation Data Datasets used, preprocessing, motivation
Training Data Same as evaluation if different
Quantitative Analyses Performance disaggregated by factors
Ethical Considerations Sensitive uses, risks, mitigations
Caveats and Recommendations Known limitations, guidance

Example for a sepsis prediction model:

## Model Card: Sepsis Early Warning System v2.1

### Model Details
- Developer: [Health System] Data Science Team
- Version: 2.1 (Updated March 2025)
- Type: Gradient Boosted Decision Tree (XGBoost)
- Training Date: January 2025
- License: Internal use only

### Intended Use
- Primary: Alert clinicians to patients at elevated sepsis risk
- Users: ICU nurses, hospitalists, rapid response teams
- Out of Scope: Diagnosis, treatment decisions, pediatric patients

### Performance Metrics
- AUROC: 0.82 (95% CI: 0.79-0.85)
- Sensitivity at 85% specificity: 0.71
- PPV at operating threshold: 0.23

### Disaggregated Performance
| Subgroup | AUROC | Sensitivity | Sample Size |
|----------|-------|-------------|-------------|
| White patients | 0.83 | 0.73 | 12,450 |
| Black patients | 0.79 | 0.68 | 3,210 |
| Hispanic patients | 0.81 | 0.70 | 2,890 |
| Age 65+ | 0.78 | 0.65 | 8,340 |
| Age 18-64 | 0.85 | 0.76 | 10,210 |

### Ethical Considerations
- Lower performance for Black patients requires monitoring
- Alert fatigue risk at current threshold (77% false positive rate)
- Model should augment, not replace, clinical judgment

### Caveats
- Not validated for patients with immunosuppression
- Performance may degrade during viral outbreaks (training data pre-COVID)
- Requires recalibration if EHR system changes

Datasheets for Datasets (Gebru et al., 2021)

Datasheets for Datasets document training and evaluation datasets, enabling informed decisions about appropriate use.

Key questions answered:

  1. Motivation: Why was the dataset created? Who funded it?
  2. Composition: What do instances represent? How many? What data is included?
  3. Collection Process: How was data acquired? Who was involved? Was consent obtained?
  4. Preprocessing: What cleaning, filtering, labeling was done?
  5. Uses: What tasks is this appropriate for? What should it NOT be used for?
  6. Distribution: How is it shared? Are there restrictions?
  7. Maintenance: Who maintains it? How are updates communicated?

Why Documentation Matters for Public Health

For procurement: Require model cards from vendors. If they cannot provide one, reconsider.

For internal development: Create model cards before deployment. The process surfaces issues.

For regulatory compliance: FDA increasingly expects transparency documentation for AI/ML medical devices.

For equity: Disaggregated performance data reveals disparities before deployment.

Resources

Case Studies in Ethics and Equity

Case Study 1: COMPAS Recidivism Risk

Background: Commercial risk assessment tool used in criminal justice

Bias Findings (ProPublica 2016): - Black defendants received higher risk scores than White defendants - False positive rate 45% for Black, 23% for White defendants - False negative rate 28% for White, 18% for Black defendants

Ethical Issues: - Violates equalized odds (error rate parity) - BUT satisfies calibration - Demonstrates impossibility theorem in practice

Public Health Parallels: - Risk stratification tools - Resource allocation algorithms - Surveillance systems

Lessons: - Choice of fairness metric has real consequences - Transparency enables accountability - Context matters: criminal justice stakes very high

Case Study 2: COVID-19 Vaccine Allocation

Challenge: Limited vaccines, how to allocate fairly?

Ethical Principles Applied: - Beneficence: Maximize lives saved - Justice: Equitable distribution - Reciprocity: Reward essential workers - Priority to vulnerable populations

AI Role: - Risk stratification models - Prevalence estimation - Supply chain optimization

Equity Considerations: - Risk models overestimated risk for minorities (protective) - But: Access barriers prevented equitable uptake - Geographic allocation algorithms favored urban areas

Outcomes: - Iterative refinement based on equity data - Targeted outreach to underserved communities - Mobile clinics and community partnerships

Lessons: - Prediction alone insufficient - Must address structural barriers - Ongoing monitoring essential

Case Study 3: Pulse Oximetry and Racial Bias

Discovery: Pulse oximeters overestimate oxygen saturation in patients with darker skin

Impact on COVID-19: - Black and Hispanic patients experienced “silent hypoxia” - Delayed treatment - Contributed to mortality disparities

AI Implications: - SpO2 commonly used as input feature - Measurement bias propagates to model predictions - Models using SpO2 may underestimate severity for minorities

Response: - FDA safety communication - Updated clinical guidelines - Research into bias-corrected sensors

Lessons: - Measurement bias can have severe consequences - Validate instruments across populations - Historical context: dark-skinned patients excluded from early pulse oximetry studies

Moving Forward: Building an Ethical AI Culture

Organizational Culture Change

From: “Move fast and break things” To: “Move thoughtfully and build trust”

Key Elements:

  1. Values Alignment
  • Explicit commitment to health equity
  • Ethics in mission statement
  • Resource allocation reflects priorities
  1. Education and Training
  • Responsible AI training for all staff
  • Domain-specific ethics modules
  • Community engagement skills
  1. Incentive Structures
  • Reward equity-centered work
  • Value thoroughness over speed
  • Recognize ethical leadership
  1. Psychological Safety
  • Encourage raising concerns
  • No retaliation for questioning
  • Learn from mistakes

Individual Practices

For Data Scientists:

  • Ask “should we?” not just “can we?”
  • Seek diverse perspectives
  • Document limitations honestly
  • Stay current on bias mitigation techniques
  • Engage with affected communities

For Public Health Leaders:

  • Prioritize equity in AI strategy
  • Allocate resources for responsible AI
  • Model ethical behavior
  • Hold teams accountable
  • Transparent communication with public

For Clinicians/End Users:

  • Understand AI limitations
  • Advocate for patients
  • Report concerns
  • Resist automation bias
  • Maintain clinical judgment

Community Engagement Best Practices

Authentic Partnership: - Engage early, not after decisions made - Compensate community experts - Share decision-making power - Long-term relationships, not extractive

Accessible Communication: - Avoid jargon - Visual aids and examples - Multiple formats and languages - Two-way dialogue

Addressing Power Imbalances: - Acknowledge historical harms - Build trust incrementally - Respect community priorities - Flexibility in approach

Environmental Sustainability of AI Systems

AI systems, particularly large multi-modal models, have significant environmental footprints that create ethical tensions with public health’s mission to protect population health.

The Carbon and Water Cost of AI

Recent research quantifies AI’s growing environmental impact:

  • Carbon footprint: AI systems may emit 32.6-79.7 million tons of CO2 in 2025, comparable to New York City’s total emissions (Vrije Universiteit Amsterdam, 2024)
  • Water consumption: AI’s water footprint could reach 312-765 billion liters in 2025, equivalent to global annual bottled water consumption (Cornell University, 2025)
  • Data center demand: A single medium-sized data center can consume 110 million gallons of water annually, equivalent to 1,000 households (EESI, 2024)

The WHO’s 2024 guidance explicitly identifies environmental sustainability as an ethical principle, noting that LMMs “require both significant energy and contribute to AI’s growing water footprint” with potential negative impacts on “water-stressed communities” (WHO, 2024).

Ethical Implications for Public Health

The environmental costs of AI create a paradox for public health applications:

Consideration Tension
Climate-health nexus AI tools to address climate-related health threats may themselves contribute to climate change
Environmental justice Data center water consumption may disproportionately affect water-stressed communities, often the same populations facing health inequities
Resource allocation Computing resources for AI could alternatively power healthcare infrastructure in low-resource settings

Mitigation Strategies

When deploying AI systems, consider environmental impact alongside performance:

  1. Model efficiency: Prefer smaller, quantized models when performance is adequate (see Global Health chapter for edge computing approaches)
  2. Compute optimization: Batch processing, caching, and inference optimization reduce energy consumption
  3. Green infrastructure: Prioritize cloud providers with renewable energy commitments and efficient cooling
  4. Proportionality: Match model complexity to task requirements; avoid using frontier models for tasks simpler models can handle
Practical Guidance

Before deploying an AI system, ask:

  • Is AI necessary, or would a simpler approach suffice?
  • Can we use a smaller model with acceptable performance?
  • What is the expected inference volume and associated energy cost?
  • Does the public health benefit clearly outweigh environmental costs?

Key Takeaways

  1. Ethics is Not Optional: In public health, equity is a core mandate. AI systems must actively advance health equity or risk amplifying existing disparities.

  2. Bias is Pervasive: Every stage of AI development introduces potential bias, from problem formulation to deployment. Vigilance required throughout.

  3. Fairness Has Trade-offs: No universal definition of fairness. Must choose appropriate criteria based on context and stakeholder input.

  4. Technical Solutions Alone Insufficient: Bias mitigation algorithms help, but do not substitute for representative data, stakeholder engagement, and structural change.

  5. Transparency Enables Accountability: Document decisions, report disaggregated performance, engage stakeholders openly.

  6. Continuous Monitoring Essential: Bias and disparities can emerge post-deployment. Ongoing equity audits required.

  7. Center Affected Communities: Those impacted by AI systems must have voice in their development and deployment.

  8. Culture Matters: Building responsible AI requires organizational commitment, not just individual effort.

Hands-On Exercise: Conduct a Bias Audit

Scenario: You’ve inherited a readmission risk prediction model used at a large hospital system. Recent concerns have been raised about potential bias.

Task: Conduct a comprehensive bias audit

Dataset: hospital_readmissions.csv (synthetic data provided)

Steps:

  1. Exploratory Analysis
  • Assess representativeness of training data
  • Check for differential missingness
  • Identify potential proxy variables
  1. Performance Evaluation
  • Calculate AUC, sensitivity, specificity overall
  • Disaggregate by race, ethnicity, sex, age, insurance
  • Conduct intersectional analysis
  1. Fairness Assessment
  • Calculate demographic parity difference
  • Calculate equalized odds difference
  • Assess calibration by subgroup
  • Which fairness criteria are violated?
  1. Root Cause Analysis
  • Where does bias originate? (Data, model, or both?)
  • Are there measurement issues?
  • Feature importance differences by group?
  1. Mitigation Strategy
  • Propose at least two mitigation approaches
  • Implement one and evaluate impact
  • What trade-offs emerge?
  1. Reporting
  • Create stakeholder-friendly summary
  • Visualize disparities
  • Make recommendations

Check Your Understanding

Test your knowledge of the key concepts from this chapter. Click “Show Answer” to reveal the correct response and explanation.

Question 1: Informed Consent for AI

A hospital implements an AI-powered diagnostic support system. Which approach best satisfies the ethical principle of patient autonomy?

  1. Using the AI system without informing patients since it is only a decision support tool
  2. Providing a general consent form mentioning “advanced analytics” may be used
  3. Informing patients specifically about AI use, how it works, and allowing them to opt out
  4. Only informing patients if the AI disagrees with the clinician’s judgment

Answer: c) Informing patients specifically about AI use, how it works, and allowing them to opt out

Explanation: The principle of autonomy in healthcare ethics requires that patients have the information necessary to make informed decisions about their care. This includes knowing when AI systems are being used in their diagnosis or treatment planning, understanding in general terms how these systems work, and having the option to decline AI-assisted care. Option (a) fails to respect patient autonomy by not informing patients at all. Option (b) is too vague and doesn’t provide meaningful information for informed consent. Option (d) is inadequate because patients should be informed regardless of whether the AI agrees or disagrees with clinicians. Meaningful informed consent for AI requires transparency about its use, explanation of its role in decision-making, and respect for patient choice.

Question 2: Algorithmic Transparency

A public health department is deciding between two COVID-19 risk prediction models. Model A is a random forest with 85% accuracy but limited interpretability. Model B is a logistic regression model with 80% accuracy but clear feature weights. Which factor is MOST important in choosing between them?

  1. Always choose the higher accuracy model to maximize public health benefit
  2. Always choose the interpretable model to ensure transparency
  3. Consider the stakes of decisions, need for clinician trust, and accountability requirements
  4. Use the model that was published in the highest-impact journal

Answer: c) Consider the stakes of decisions, need for clinician trust, and accountability requirements

Explanation: The transparency vs. performance trade-off must be evaluated contextually rather than applying a blanket rule. Important considerations include: the magnitude of the performance difference (5% in this case), the consequences of prediction errors, whether post-hoc explanations could provide sufficient transparency for Model A, the need for clinician buy-in and trust, and legal/regulatory requirements for explainability. In high-stakes scenarios with significant accountability needs, the interpretable model might be preferred despite lower accuracy. In lower-stakes scenarios with a large performance gap, the black-box model might be acceptable with appropriate safeguards. Option (a) ignores the importance of trust and accountability. Option (b) dismisses performance entirely. Option (d) is irrelevant to the ethical trade-offs at hand.

Question 3: Data Privacy in Surveillance

A state health department wants to use de-identified hospital admission data for real-time disease surveillance. Which approach provides the strongest privacy protection?

  1. De-identification: Remove direct identifiers (name, SSN) but keep detailed demographics and dates
  2. Anonymization: Remove identifiers and use techniques like generalization and noise addition to prevent re-identification
  3. Encryption: Encrypt all data fields so they appear as random strings
  4. Aggregation: Only report county-level totals without any individual records

Answer: b) Anonymization: Remove identifiers and use techniques like generalization and noise addition to prevent re-identification

Explanation: While all options provide some privacy protection, true anonymization using techniques like k-anonymity, differential privacy, or generalization provides the strongest protection against re-identification attacks. De-identification (option a) is vulnerable to re-identification through linking with other datasets, especially when quasi-identifiers like detailed demographics, ZIP codes, and exact dates are retained. Studies have shown that 87% of the U.S. population can be uniquely identified using just ZIP code, birthdate, and sex. Encryption (option c) protects data in transit and storage but someone with the decryption key can still access identifiable information. Aggregation (option d) provides strong privacy protection but sacrifices the granularity needed for effective real-time surveillance. Anonymization strikes the balance between privacy protection and data utility, though it requires careful implementation to resist modern re-identification techniques.

Question 4: Bias Amplification

You discover that your sepsis prediction model has a higher false negative rate for Black patients (misses 30% of cases) compared to White patients (misses 15% of cases). This is an example of which type of bias?

  1. Historical bias - the model learned patterns from biased training data
  2. Representation bias - Black patients were underrepresented in training data
  3. Measurement bias - sepsis indicators measured differently across racial groups
  4. All of the above could contribute to this disparity

Answer: d) All of the above could contribute to this disparity

Explanation: Disparate performance across demographic groups typically results from multiple, interconnecting sources of bias. Historical bias could manifest if Black patients historically received delayed sepsis treatment, resulting in training data that shows different progression patterns. Representation bias would occur if Black patients were underrepresented in the training dataset, leading to a model optimized primarily for the majority group. Measurement bias could arise from tools like pulse oximeters that perform differently across skin tones, or from differential documentation practices across patient populations. In practice, these biases often compound each other - for example, underrepresentation (representation bias) makes it harder to detect measurement problems in minority groups, while historical disparities in care affect what patterns exist in the data. Addressing such disparities requires examining all potential sources of bias throughout the AI development lifecycle, not just focusing on one type.

Question 5: The Dual Use Dilemma

A research team develops an AI model that accurately predicts individual risk of substance use disorder using social media data. The model could be used for early intervention and treatment referral, but could also be used by employers or insurers to discriminate. What is the MOST appropriate ethical response?

  1. Publish the model openly to maximize potential benefits; restrict use cases through licensing
  2. Keep the model confidential to prevent misuse, only deploying it in controlled clinical settings
  3. Publish the methodology but not the trained model weights or code
  4. Only publish after establishing legal protections against discriminatory use

Answer: b) Keep the model confidential to prevent misuse, only deploying it in controlled clinical settings

Explanation: This represents a classic “dual use” dilemma where beneficial technology also has significant potential for harm. In this case, the risks of discrimination and privacy violations are particularly severe because: (1) substance use disorder is a highly stigmatized condition, (2) employment and insurance discrimination could cause major harm, (3) social media data raises additional privacy concerns, and (4) existing anti-discrimination laws have gaps in coverage for algorithmic decision-making. Option (a) is problematic because licensing restrictions are difficult to enforce once a model is publicly released, and the harm from misuse could be severe and irreversible. Option (c) still enables others to reproduce the model and potentially misuse it. Option (d) places too much faith in legal protections that may not yet exist or may be difficult to enforce. The most appropriate approach is to deploy the model only in settings where its use can be monitored and controlled, such as healthcare facilities with ethical oversight, where it can provide benefits for early intervention while minimizing risks of discriminatory misuse. This follows the precautionary principle in high-stakes scenarios.

Question 6: Algorithmic Recourse

A predictive model denies a patient enrollment in a preventive care program for high-risk individuals. Which feature of the system is MOST important for ethical deployment?

  1. High overall accuracy of the model (>90% AUC)
  2. A clear explanation of which factors led to the decision
  3. A process for patients to appeal the decision and receive human review
  4. Regular retraining of the model on new data

Answer: c) A process for patients to appeal the decision and receive human review

Explanation: While all options have value, algorithmic recourse, the ability to contest and potentially reverse automated decisions, is most critical for ethical deployment. Even highly accurate models (option a) make errors, and those errors can have significant consequences for individuals. The ability to appeal ensures that: (1) individual circumstances not captured by the model can be considered, (2) model errors can be identified and corrected, (3) patients maintain some agency in decisions affecting their care, and (4) accountability mechanisms exist when systems fail. Explanations (option b) are valuable but insufficient if patients cannot act on them to challenge incorrect decisions. Regular retraining (option d) improves model performance over time but doesn’t help individuals currently harmed by incorrect predictions. The right to meaningful recourse is a fundamental principle of justice and has been recognized in frameworks like the EU’s GDPR (right to human review of automated decisions) and WHO AI ethics guidelines. Without recourse mechanisms, algorithmic decision-making can become a form of “automated injustice” with no remedy.

Discussion Questions

  1. Consider the impossibility theorem: If you cannot achieve all fairness criteria simultaneously, which should take priority in public health applications? Why?

  2. When, if ever, is it acceptable to use race/ethnicity as a feature in a clinical prediction model? What are the risks and potential benefits?

  3. You’ve developed an AI system that improves overall population health but performs worse for a minority group. The alternative is to use no AI, which results in worse outcomes for everyone. What do you do?

  4. How can we balance the need for transparency in AI systems with proprietary concerns? Should all AI systems used in public health be open-source?

  5. What are the limits of individual consent for AI systems used in population health surveillance? When is it acceptable to use data without explicit consent?

  6. How should we handle cases where community preferences conflict with technical recommendations for bias mitigation?

  7. Who should be held accountable when an AI system causes harm, developers, deployers, end users, or all of the above?

Further Resources

Essential Books

Key Papers

Foundational: - Obermeyer et al., 2019 - Racial bias in healthcare algorithm - Buolamwini & Gebru, 2018 - Gender Shades - Facial recognition bias - ProPublica COMPAS analysis - Recidivism prediction bias

Fairness Theory: - Chouldechova, 2017 - Fair prediction with disparate impact - Kleinberg et al., 2017 - Inherent trade-offs in algorithmic fairness - Dwork et al., 2012 - Fairness through awareness

Health Applications: - Rajkomar et al., 2018 - Ensuring fairness in ML for healthcare - Chen et al., 2019 - Addressing bias in clinical algorithms - Sjoding et al., 2020 - Pulse oximetry racial bias

Mitigation Strategies: - Bellamy et al., 2019 - AI Fairness 360 - Open source toolkit - Agarwal et al., 2018 - Reductions approach to fair classification - Kamiran & Calders, 2012 - Data preprocessing for discrimination prevention

Courses and Tutorials

Tools and Toolkits

Fairness Auditing: - AI Fairness 360 (AIF360) - IBM comprehensive toolkit - Fairlearn - Microsoft fairness assessment and mitigation - Aequitas - University of Chicago bias audit tool - What-If Tool - Google visual debugging

Documentation: - Model Cards - Documentation standard - Datasheets for Datasets - Dataset documentation

Explainability: - SHAP - Feature importance - LIME - Local explanations - InterpretML - Microsoft interpretability toolkit

Guidelines and Frameworks

Regulatory: - EU AI Act - Comprehensive AI regulation - FDA AI/ML Action Plan - Medical device regulation - WHO Ethics and Governance of AI for Health - Six ethical principles

Professional: - ACM Code of Ethics - Computing professional ethics - IEEE Ethically Aligned Design - Comprehensive framework - Montreal Declaration for Responsible AI - Ten principles

Next: Privacy, Security, and Governance for Health AI →