11  Ethics, Bias, and Equity

Learning Objectives

Tip

By the end of this chapter, you will:

  • Understand fundamental ethical principles for AI in public health (beneficence, non-maleficence, autonomy, justice)
  • Identify sources of bias throughout the AI development lifecycle (data, algorithmic, deployment)
  • Apply multiple fairness definitions and recognize their inherent trade-offs
  • Conduct algorithmic impact assessments and equity audits
  • Design bias mitigation strategies at each stage of model development
  • Evaluate AI systems for their impact on health equity and vulnerable populations
  • Navigate ethical dilemmas in AI deployment using structured frameworks
  • Implement governance structures for responsible AI development

Time to complete: 90-120 minutes Prerequisites: Chapter 3: Data, Chapter 9: Evaluating AI Systems

What you’ll build: 💻 Bias audit toolkit, equity assessment framework, ethical review checklist, stakeholder engagement protocol, and algorithmic impact assessment template

11.1 Introduction: The Stakes of Getting It Wrong

11.1.1 The Obermeyer Case: When Algorithms Amplify Inequality

[Opening case study of Obermeyer et al., 2019 racial bias in healthcare algorithm]

  • Algorithm used healthcare costs as proxy for healthcare needs
  • Systematically disadvantaged Black patients
  • At given risk score, Black patients significantly sicker than White patients
  • Impact: Reduced access to care management programs for those who needed it most
  • Root cause: Historical inequities embedded in training data

11.1.2 Why Ethics Matters in Public Health AI

  • Public health has dual mandate: population health + health equity
  • AI systems can amplify existing inequities at scale
  • Decisions affect vulnerable populations with limited recourse
  • Trust is essential for public health effectiveness
  • Historical context of medical racism and exploitation

11.1.3 The Challenge of Operationalizing Ethics

  • Moving from abstract principles to concrete practices
  • Trade-offs between competing ethical values
  • Technical constraints vs. ethical ideals
  • Balancing innovation with precaution

11.2 Ethical Frameworks for AI in Public Health

11.2.1 Traditional Biomedical Ethics

The Four Principles:

  1. Beneficence - Obligation to benefit patients/populations
    • AI should improve health outcomes
    • Evidence of clinical utility required
    • Consider opportunity costs
  2. Non-maleficence - “First, do no harm”
    • Minimize risks from errors, bias, and misuse
    • Consider indirect harms (e.g., opportunity costs, psychological impact)
    • Precautionary principle in high-stakes settings
  3. Autonomy - Respect for individual agency
    • Informed consent challenges with AI
    • Right to explanation of AI decisions
    • Opt-out mechanisms where appropriate
    • Protection of decisional capacity
  4. Justice - Fair distribution of benefits and burdens
    • Equitable access to AI-enabled interventions
    • Fair representation in training data
    • Avoiding discrimination
    • Addressing structural inequities

11.2.2 Public Health Ethics Extensions

Population-Level Considerations:

  • Collective welfare vs. individual rights
  • Acceptable trade-offs for population benefit
  • Special obligations to vulnerable populations
  • Solidarity and social cohesion

Health Equity as Core Principle:

  • Not just equal treatment, but equitable outcomes
  • Targeted universalism: universal goals, targeted strategies
  • Addressing social determinants of health
  • Centering most marginalized populations

11.2.3 AI-Specific Ethical Frameworks

The EU AI Act Approach:

  • Risk-based regulatory framework
  • Prohibited practices (social scoring, exploitation of vulnerabilities)
  • High-risk AI systems (healthcare, critical infrastructure)
  • Transparency requirements

IEEE Ethically Aligned Design:

  • Human rights framework
  • Well-being prioritization
  • Data agency
  • Effectiveness and transparency
  • Accountability

WHO Guidance on AI Ethics:

Six principles for AI in health: 1. Protecting human autonomy 2. Promoting human well-being and safety 3. Ensuring transparency, explainability, and intelligibility 4. Fostering responsibility and accountability 5. Ensuring inclusiveness and equity 6. Promoting AI that is responsive and sustainable

11.3 The Bias Landscape: Where Things Go Wrong

11.3.1 Taxonomy of Bias in AI Systems

11.3.1.1 1. Historical Bias

Definition: Bias existing in the world that gets encoded in data

Examples: - Historical healthcare disparities reflected in EHR data - Underdiagnosis of certain conditions in specific populations - Differential access to care - Systemic discrimination in social determinants

Impact: Models learn and perpetuate existing inequities

11.3.1.2 2. Representation Bias

Definition: Training data doesn’t reflect target population

Examples: - Underrepresentation of racial/ethnic minorities - Gender imbalance in clinical trials - Geographic concentration (e.g., only urban hospitals) - Age distribution skew

Impact: Poor performance on underrepresented groups

11.3.1.3 3. Measurement Bias

Definition: Features systematically measured differently across groups

Examples: - Pulse oximetry accuracy varies by skin tone - Different diagnostic thresholds by race/ethnicity - Differential missingness patterns - Proxy variables that encode discrimination

Impact: Biased model predictions even with unbiased algorithms

11.3.1.4 4. Aggregation Bias

Definition: One-size-fits-all model applied to diverse populations

Examples: - Single diabetes risk model across ethnic groups - Ignoring gene-environment interactions - Universal treatment protocols despite heterogeneity - Population-average effects masking subgroup variation

Impact: Poor performance for minority groups, missed opportunities for personalization

11.3.1.5 5. Evaluation Bias

Definition: Benchmarks that don’t reflect diverse populations

Examples: - Test sets with inadequate minority representation - Performance metrics that hide disparities - Validation in non-representative settings - Ignoring subgroup performance differences

Impact: Biased systems appear accurate, disparities go undetected

11.3.1.6 6. Deployment Bias

Definition: Mismatch between development and deployment context

Examples: - Model trained in academic center, deployed in community hospital - Different workflows, patient populations, resources - Changed incentive structures - Differential adoption by provider characteristics

Impact: Real-world performance differs from validation, disparities emerge

11.4 Fairness: Defining an Elusive Concept

11.4.1 Mathematical Definitions of Fairness

11.4.1.1 Group Fairness Metrics

1. Demographic Parity (Statistical Parity)

P(Ŷ = 1 | A = a) = P(Ŷ = 1 | A = b)

Equal selection rates across groups

When appropriate: - Loans, hiring, college admissions - When base rates reflect historical discrimination

Limitations: - May require accepting less qualified applicants - Ignores relevant group differences

2. Equalized Odds (Error Rate Parity)

P(Ŷ = 1 | Y = y, A = a) = P(Ŷ = 1 | Y = y, A = b) for all y

Equal true positive rates AND false positive rates

When appropriate: - Healthcare screening - Bail/parole decisions - When ground truth is reliable

Limitations: - Requires labeled outcomes for all groups - May be impossible to achieve if base rates differ

3. Predictive Parity (Outcome Parity)

P(Y = 1 | Ŷ = 1, A = a) = P(Y = 1 | Ŷ = 1, A = b)

Equal positive predictive value across groups

When appropriate: - Resource allocation decisions - When acting on predictions - Diagnostic settings

4. Calibration Fairness

P(Y = 1 | Ŷ = p, A = a) = p for all groups a

Predicted probabilities match observed frequencies in each group

When appropriate: - Risk communication - Clinical decision support - When probabilities inform decisions

11.4.2 The Impossibility Theorem

Chouldechova (2017) and Kleinberg et al. (2017):

Cannot simultaneously achieve equalized odds, predictive parity, and calibration when: - Base rates differ between groups - Model is not perfect

Implications: - Must choose which fairness criterion to prioritize - Trade-offs are inevitable - Context determines appropriate choice - Transparency about choices is essential

11.4.3 Individual Fairness

Definition: Similar individuals should receive similar predictions

d(Ŷ(x₁), Ŷ(x₂)) ≤ L · d(x₁, x₂)

Challenges: - Defining “similarity” without encoding bias - Computational complexity - Requires domain expertise

11.4.4 Fairness Through Awareness vs. Blindness

Fairness Through Blindness: - Remove protected attributes from model - Appeal: Seems to prevent discrimination

Why it fails: - Proxy variables (zip code → race) - Historical bias remains in other features - Prevents monitoring for disparities

Fairness Through Awareness: - Explicitly consider protected attributes - Monitor performance by subgroup - Apply group-specific interventions if needed - Enables accountability

11.5 Bias Mitigation: A Lifecycle Approach

11.5.1 Pre-processing: Fixing the Data

11.5.1.1 Data Collection Strategies

1. Representative Sampling - Oversample underrepresented groups - Stratified sampling by key attributes - Community-engaged data collection - Multi-site studies for diversity

2. Measurement Standardization - Validate instruments across populations - Culturally adapted assessments - Minimize differential missingness - Address measurement bias (e.g., pulse oximetry)

3. Synthetic Data Augmentation - Generate samples for minority groups - Preserve distributional properties - Validate augmented data carefully - Combine with real data

11.5.1.2 Data Pre-processing Techniques

1. Reweighing - Assign weights to balance groups - Maintains original data - Simple to implement

Code example:

# Reweighing for fairness
from aif360.algorithms.preprocessing import Reweighing

reweigher = Reweighing(
    unprivileged_groups=[{'race': 0}],
    privileged_groups=[{'race': 1}]
)
dataset_transformed = reweigher.fit_transform(dataset)

2. Sampling - Oversample minority class - Undersample majority class - SMOTE for synthetic minority samples

3. Relabeling - Correct biased labels when ground truth available - Use multiple annotators - Calibrate for systematic errors

11.5.2 In-processing: Constraining the Model

11.5.2.1 Fairness-Constrained Optimization

Adversarial Debiasing:

from aif360.algorithms.inprocessing import AdversarialDebiasing

debiaser = AdversarialDebiasing(
    privileged_groups=[{'sex': 1}],
    unprivileged_groups=[{'sex': 0}],
    scope_name='debiaser',
    debias=True,
    adversary_loss_weight=0.5
)
debiaser.fit(dataset_train)

How it works: - Primary model predicts outcome - Adversary tries to predict protected attribute from predictions - Train to maximize prediction accuracy while minimizing adversary’s accuracy

Fairness Regularization:

Add fairness penalty to loss function:

L_total = L_prediction + λ × L_fairness

Examples: - Demographic parity penalty: |P(Ŷ=1|A=a) - P(Ŷ=1|A=b)| - Equalized odds penalty: difference in TPR and FPR

Multi-objective Optimization: - Pareto frontier of accuracy vs. fairness - Explore trade-off space - Stakeholder input on acceptable trade-offs

11.5.2.2 Group-Specific Models

Separate Models: - Train distinct models for each subgroup - Maximizes within-group performance - Challenges: sample size, generalizability, explicit group-based treatment

Mixture of Experts: - Learn to route instances to specialized sub-models - Soft assignments - Can discover latent subgroups

11.5.3 Post-processing: Adjusting Predictions

11.5.3.1 Threshold Optimization

Group-Specific Thresholds:

def optimize_group_thresholds(y_true, y_pred_proba, groups, criterion='equalized_odds'):
    thresholds = {}
    for group in groups.unique():
        mask = (groups == group)
        # Find threshold that optimizes criterion for this group
        thresholds[group] = find_optimal_threshold(
            y_true[mask],
            y_pred_proba[mask],
            criterion
        )
    return thresholds

Calibration: - Platt scaling by group - Isotonic regression - Ensure predicted probabilities match observed frequencies

Reject Option Classification: - Create “uncertain” zone around decision boundary - For instances in this zone, assign label to favor disadvantaged group - Trade-off between fairness and accuracy

11.6 Conducting an Algorithmic Impact Assessment

11.6.1 Framework Overview

Systematic evaluation of potential impacts before deployment

Key Questions: 1. What problem are we solving and for whom? 2. Who might be harmed? 3. What are the fairness considerations? 4. What safeguards are in place? 5. How will we monitor ongoing impacts?

11.6.2 Step 1: Stakeholder Mapping

Identify All Affected Parties: - Direct users (clinicians, public health officials) - Affected populations (patients, communities) - Indirect stakeholders (payers, policymakers) - Vulnerable subgroups

Engagement Strategies: - Community advisory boards - Focus groups with affected populations - Expert consultations - Participatory design workshops

11.6.3 Step 2: Problem Formulation Audit

Critical Questions:

  • Framing: How is the problem defined? Who defined it?
  • Objective: What are we optimizing for? Whose values?
  • Scope: What’s included/excluded? Why?
  • Assumptions: What do we assume about the world?

Red Flags: - Problem defined without input from affected communities - Optimization metric misaligned with equity goals - Narrow framing that misses root causes - Unstated assumptions that encode bias

11.6.4 Step 3: Data Assessment

Representative of Population?

def assess_representation(data, population_stats, protected_attr):
    """Compare data demographics to population"""
    data_props = data[protected_attr].value_counts(normalize=True)

    for group in population_stats.keys():
        data_pct = data_props.get(group, 0) * 100
        pop_pct = population_stats[group] * 100
        disparity = data_pct - pop_pct

        print(f"{group}:")
        print(f"  Data: {data_pct:.1f}%")
        print(f"  Population: {pop_pct:.1f}%")
        print(f"  Disparity: {disparity:+.1f}pp")

Check for Measurement Bias: - Differential missingness by group - Proxy variables that encode discrimination - Validated measures across subgroups

11.6.5 Step 4: Model Fairness Analysis

Disaggregated Performance:

def disaggregated_metrics(y_true, y_pred, groups):
    """Calculate metrics by subgroup"""
    results = []
    for group in groups.unique():
        mask = (groups == group)
        metrics = {
            'group': group,
            'n': mask.sum(),
            'prevalence': y_true[mask].mean(),
            'sensitivity': recall_score(y_true[mask], y_pred[mask]),
            'specificity': recall_score(1-y_true[mask], 1-y_pred[mask]),
            'ppv': precision_score(y_true[mask], y_pred[mask]),
            'auc': roc_auc_score(y_true[mask], y_pred[mask])
        }
        results.append(metrics)
    return pd.DataFrame(results)

Fairness Metrics: - Calculate multiple fairness definitions - Visualize disparities - Identify which groups are disadvantaged

11.6.6 Step 5: Deployment Context Analysis

Workflow Integration: - How will AI fit into existing processes? - Who interprets and acts on outputs? - What happens when AI is wrong?

Incentive Structures: - How might AI change behavior? - Could it be gamed or manipulated? - Unintended consequences?

Differential Impact: - Will some groups experience AI differently? - Access barriers to AI-enabled interventions? - Digital divide considerations?

11.6.7 Step 6: Safeguards and Mitigation

Technical Safeguards: - Uncertainty quantification - Anomaly detection - Human oversight mechanisms - Audit trails

Governance Safeguards: - Clear accountability structures - Regular audits - Stakeholder feedback mechanisms - Sunset clauses and re-evaluation triggers

Recourse Mechanisms: - Appeals process - Human review of AI decisions - Right to explanation - Compensation for harms

11.7 Health Equity-Centered AI Development

11.7.1 Principles of Equity-Centered Design

1. Center Marginalized Voices - Participatory design with affected communities - Community advisory boards - Compensate community experts - Power-sharing in decision-making

2. Address Root Causes - Don’t just predict inequities, act to reduce them - Target interventions at structural determinants - Avoid “technological solutionism”

3. Targeted Universalism - Universal goals (e.g., reduce diabetes complications) - Targeted strategies for disadvantaged groups - Invest more in groups with greater needs

4. Monitor and Mitigate Continuously - Equity is not a one-time check - Ongoing monitoring by subgroup - Rapid response to emerging disparities - Transparent reporting

11.7.2 Equity Metrics Beyond Fairness

1. Differential Impact

Change in disparities after AI deployment:

Disparity_after - Disparity_before

Goal: Reduce existing gaps, don’t widen them

2. Benefit Distribution

Who benefits from AI-enabled improvements?

def benefit_distribution(outcomes_before, outcomes_after, groups):
    """Calculate which groups benefit most"""
    benefits = outcomes_after - outcomes_before

    for group in groups.unique():
        mask = (groups == group)
        avg_benefit = benefits[mask].mean()
        print(f"{group}: {avg_benefit:.3f} improvement")

    # Ratio of benefits between most/least advantaged
    benefit_ratio = benefits[disadvantaged].mean() / benefits[advantaged].mean()
    print(f"Proportional benefit ratio: {benefit_ratio:.2f}")

3. Access Equity

  • Can all who would benefit actually access AI-enabled care?
  • Digital literacy requirements
  • Infrastructure barriers
  • Cost barriers

4. Trust and Acceptability

  • Do vulnerable populations trust the AI system?
  • Cultural acceptability
  • Historical context of medical exploitation
  • Community acceptance

11.7.3 Case Study: Equity-Centered Sepsis Prediction

[Detailed case study showing equity-centered development process]

Context: Sepsis disproportionately affects low-income patients and racial minorities

Traditional Approach: - Train on all EHR data - Optimize for overall accuracy - Deploy universally

Problems: - Underrepresentation of minorities in training data - Different sepsis presentation patterns by group - Lower baseline access to ICU care

Equity-Centered Approach:

  1. Stakeholder Engagement:
    • Community health workers
    • Patient advocates
    • Clinicians serving underserved populations
  2. Problem Reframing:
    • Goal: Reduce sepsis mortality disparities
    • Not just predict sepsis, but ensure equitable access to treatment
  3. Data Strategy:
    • Oversample minority patients
    • Multi-site data including safety-net hospitals
    • Validate measurement equivalence (vital signs, labs)
  4. Fairness Constraints:
    • Equalized false negative rates (don’t miss cases in any group)
    • Group-specific calibration
  5. Deployment Design:
    • Automated alerts for high-risk patients
    • Proactive outreach to underserved communities
    • Resource allocation to address capacity constraints
  6. Monitoring:
    • Track sepsis mortality by race/ethnicity/SES
    • Monitor alert response times by patient characteristics
    • Quarterly equity audits

Outcomes: - Reduced sepsis mortality overall - Narrowed mortality gap between groups - High clinician trust and acceptance

11.9 Governance and Accountability

11.9.1 Organizational Structures for Responsible AI

11.9.1.1 AI Ethics Committee

Composition: - Technical experts (data science, AI) - Domain experts (epidemiology, clinical) - Ethicists - Community representatives - Legal/compliance - Patients/public

Responsibilities: - Review proposed AI projects - Approve high-risk deployments - Monitor ongoing systems - Investigate concerns - Update policies

Decision-Making Process: - Risk-based review (low/medium/high) - Structured checklist - Required documentation - Community consultation for high-risk - Transparent decisions with rationale

11.9.1.2 Roles and Responsibilities

Data Scientists: - Conduct bias audits - Implement fairness constraints - Document limitations - Transparent reporting

Domain Experts: - Define appropriate use cases - Interpret predictions in context - Monitor clinical validity - Engage affected communities

Leadership: - Allocate resources for responsible AI - Set organizational values - Ensure accountability - Model ethical behavior

Community Members: - Provide lived experience expertise - Review from affected population perspective - Hold institutions accountable - Co-design solutions

11.9.2 Accountability Mechanisms

1. Documentation Requirements

Model Cards: - Intended use - Performance across subgroups - Training data characteristics - Fairness metrics - Limitations - Ethical considerations

Datasheets for Datasets: - Motivation and composition - Collection process - Preprocessing - Distribution - Maintenance plan

2. Audit Requirements

Pre-deployment: - Bias audit - Fairness assessment - Impact assessment - Stakeholder review

Ongoing: - Quarterly performance monitoring - Annual comprehensive audit - Triggered by concerns

3. Transparency Measures

Internal: - Documentation accessible to all staff - Training on responsible AI - Clear escalation pathways

External: - Public model cards (when appropriate) - Aggregate performance reporting - Stakeholder engagement - Incident reporting

4. Recourse Mechanisms

  • Clear process to contest AI decisions
  • Human review available
  • Timeline for resolution
  • Compensation for verified harms

11.9.3 Regulatory Landscape

FDA Regulation of AI/ML Medical Devices: - Risk-based framework - Predetermined change control plans - Real-world performance monitoring

EU AI Act: - Prohibited practices - High-risk AI requirements - Transparency obligations - Conformity assessments

State-Level Initiatives: - Algorithmic accountability acts - Impact assessment requirements - Anti-discrimination protections

11.10 Practical Tools and Workflows

11.10.1 Bias Audit Toolkit

Step 1: Define Protected Attributes

protected_attributes = ['race', 'ethnicity', 'sex', 'age_group', 'income_level', 'insurance_type']

Step 2: Assess Representation

from bias_toolkit import check_representation

representation_report = check_representation(
    data=training_data,
    protected_attrs=protected_attributes,
    reference_population=census_data
)

representation_report.plot()
representation_report.to_html('representation_audit.html')

Step 3: Calculate Performance by Subgroup

from sklearn.metrics import classification_report
import pandas as pd

def subgroup_performance(y_true, y_pred, y_proba, groups, group_col):
    results = []
    for group in groups[group_col].unique():
        mask = (groups[group_col] == group)

        perf = {
            'group': group,
            'n': mask.sum(),
            'prevalence': y_true[mask].mean(),
            'auc': roc_auc_score(y_true[mask], y_proba[mask]),
            'sensitivity': recall_score(y_true[mask], y_pred[mask]),
            'specificity': recall_score(1-y_true[mask], 1-y_pred[mask]),
            'ppv': precision_score(y_true[mask], y_pred[mask]),
            'npv': precision_score(1-y_true[mask], 1-y_pred[mask])
        }
        results.append(perf)

    return pd.DataFrame(results)

# Generate report for each protected attribute
for attr in protected_attributes:
    report = subgroup_performance(y_true, y_pred, y_proba, test_data, attr)
    report.to_csv(f'performance_by_{attr}.csv')

    # Visualize disparities
    plot_performance_disparities(report, attr)

Step 4: Calculate Fairness Metrics

from aif360.metrics import BinaryLabelDatasetMetric, ClassificationMetric
from aif360.datasets import BinaryLabelDataset

# Prepare data
dataset = BinaryLabelDataset(
    df=test_data,
    label_names=['outcome'],
    protected_attribute_names=['race']
)

predictions = BinaryLabelDataset(
    df=test_data.assign(outcome=y_pred),
    label_names=['outcome'],
    protected_attribute_names=['race']
)

# Calculate fairness metrics
metric = ClassificationMetric(
    dataset,
    predictions,
    unprivileged_groups=[{'race': 0}],
    privileged_groups=[{'race': 1}]
)

fairness_report = {
    'Statistical Parity Difference': metric.statistical_parity_difference(),
    'Disparate Impact': metric.disparate_impact(),
    'Equal Opportunity Difference': metric.equal_opportunity_difference(),
    'Average Odds Difference': metric.average_odds_difference(),
    'Theil Index': metric.theil_index()
}

print(pd.Series(fairness_report))

Step 5: Intersectional Analysis

def intersectional_analysis(y_true, y_pred, groups, attrs):
    """Analyze performance across intersectional groups"""

    # Create intersectional categories
    groups['intersectional'] = groups[attrs].astype(str).agg('_'.join, axis=1)

    results = []
    for group in groups['intersectional'].unique():
        mask = (groups['intersectional'] == group)
        if mask.sum() < 30:  # Skip small groups
            continue

        perf = {
            'group': group,
            'n': mask.sum(),
            'auc': roc_auc_score(y_true[mask], y_pred[mask]) if mask.sum() > 0 else None
        }
        results.append(perf)

    return pd.DataFrame(results).sort_values('auc')

# Example: Race × Gender × Age
intersectional_results = intersectional_analysis(
    y_true, y_pred, test_data,
    attrs=['race', 'sex', 'age_group']
)

11.10.2 Equity Assessment Checklist

Problem Formulation - [ ] Problem defined with input from affected communities - [ ] Objective aligned with equity goals - [ ] Considered alternatives to AI - [ ] Identified potential for harm

Data - [ ] Assessed representativeness vs. target population - [ ] Checked for measurement bias across groups - [ ] Investigated differential missingness - [ ] Documented data limitations

Model Development - [ ] Disaggregated performance by subgroup - [ ] Calculated multiple fairness metrics - [ ] Conducted intersectional analysis - [ ] Applied bias mitigation techniques

Validation - [ ] Tested in diverse settings - [ ] Validated with underrepresented groups - [ ] Assessed calibration by subgroup - [ ] Evaluated equity metrics

Deployment Planning - [ ] Analyzed deployment context for differential impacts - [ ] Designed safeguards and human oversight - [ ] Established monitoring plan - [ ] Created recourse mechanisms

Governance - [ ] Ethics committee review - [ ] Stakeholder engagement - [ ] Documented decisions - [ ] Assigned accountability

11.11 Case Studies in Ethics and Equity

11.11.1 Case Study 1: COMPAS Recidivism Risk

Background: Commercial risk assessment tool used in criminal justice

Bias Findings (ProPublica 2016): - Black defendants received higher risk scores than White defendants - False positive rate 45% for Black, 23% for White defendants - False negative rate 28% for White, 18% for Black defendants

Ethical Issues: - Violates equalized odds (error rate parity) - BUT satisfies calibration - Demonstrates impossibility theorem in practice

Public Health Parallels: - Risk stratification tools - Resource allocation algorithms - Surveillance systems

Lessons: - Choice of fairness metric has real consequences - Transparency enables accountability - Context matters: criminal justice stakes very high

11.11.2 Case Study 2: COVID-19 Vaccine Allocation

Challenge: Limited vaccines, how to allocate fairly?

Ethical Principles Applied: - Beneficence: Maximize lives saved - Justice: Equitable distribution - Reciprocity: Reward essential workers - Priority to vulnerable populations

AI Role: - Risk stratification models - Prevalence estimation - Supply chain optimization

Equity Considerations: - Risk models overestimated risk for minorities (protective) - But: Access barriers prevented equitable uptake - Geographic allocation algorithms favored urban areas

Outcomes: - Iterative refinement based on equity data - Targeted outreach to underserved communities - Mobile clinics and community partnerships

Lessons: - Prediction alone insufficient - Must address structural barriers - Ongoing monitoring essential

11.11.3 Case Study 3: Pulse Oximetry and Racial Bias

Discovery: Pulse oximeters overestimate oxygen saturation in patients with darker skin

Impact on COVID-19: - Black and Hispanic patients experienced “silent hypoxia” - Delayed treatment - Contributed to mortality disparities

AI Implications: - SpO2 commonly used as input feature - Measurement bias propagates to model predictions - Models using SpO2 may underestimate severity for minorities

Response: - FDA safety communication - Updated clinical guidelines - Research into bias-corrected sensors

Lessons: - Measurement bias can have severe consequences - Validate instruments across populations - Historical context: dark-skinned patients excluded from early pulse oximetry studies

11.12 Moving Forward: Building an Ethical AI Culture

11.12.1 Organizational Culture Change

From: “Move fast and break things” To: “Move thoughtfully and build trust”

Key Elements:

  1. Values Alignment
    • Explicit commitment to health equity
    • Ethics in mission statement
    • Resource allocation reflects priorities
  2. Education and Training
    • Responsible AI training for all staff
    • Domain-specific ethics modules
    • Community engagement skills
  3. Incentive Structures
    • Reward equity-centered work
    • Value thoroughness over speed
    • Recognize ethical leadership
  4. Psychological Safety
    • Encourage raising concerns
    • No retaliation for questioning
    • Learn from mistakes

11.12.2 Individual Practices

For Data Scientists:

  • Ask “should we?” not just “can we?”
  • Seek diverse perspectives
  • Document limitations honestly
  • Stay current on bias mitigation techniques
  • Engage with affected communities

For Public Health Leaders:

  • Prioritize equity in AI strategy
  • Allocate resources for responsible AI
  • Model ethical behavior
  • Hold teams accountable
  • Transparent communication with public

For Clinicians/End Users:

  • Understand AI limitations
  • Advocate for patients
  • Report concerns
  • Resist automation bias
  • Maintain clinical judgment

11.12.3 Community Engagement Best Practices

Authentic Partnership: - Engage early, not after decisions made - Compensate community experts - Share decision-making power - Long-term relationships, not extractive

Accessible Communication: - Avoid jargon - Visual aids and examples - Multiple formats and languages - Two-way dialogue

Addressing Power Imbalances: - Acknowledge historical harms - Build trust incrementally - Respect community priorities - Flexibility in approach

11.13 Key Takeaways

  1. Ethics is Not Optional: In public health, equity is a core mandate. AI systems must actively advance health equity or risk amplifying existing disparities.

  2. Bias is Pervasive: Every stage of AI development introduces potential bias—from problem formulation to deployment. Vigilance required throughout.

  3. Fairness Has Trade-offs: No universal definition of fairness. Must choose appropriate criteria based on context and stakeholder input.

  4. Technical Solutions Alone Insufficient: Bias mitigation algorithms help, but don’t substitute for representative data, stakeholder engagement, and structural change.

  5. Transparency Enables Accountability: Document decisions, report disaggregated performance, engage stakeholders openly.

  6. Continuous Monitoring Essential: Bias and disparities can emerge post-deployment. Ongoing equity audits required.

  7. Center Affected Communities: Those impacted by AI systems must have voice in their development and deployment.

  8. Culture Matters: Building responsible AI requires organizational commitment, not just individual effort.

11.14 Hands-On Exercise: Conduct a Bias Audit

Scenario: You’ve inherited a readmission risk prediction model used at a large hospital system. Recent concerns have been raised about potential bias.

Task: Conduct a comprehensive bias audit

Dataset: hospital_readmissions.csv (synthetic data provided)

Steps:

  1. Exploratory Analysis
    • Assess representativeness of training data
    • Check for differential missingness
    • Identify potential proxy variables
  2. Performance Evaluation
    • Calculate AUC, sensitivity, specificity overall
    • Disaggregate by race, ethnicity, sex, age, insurance
    • Conduct intersectional analysis
  3. Fairness Assessment
    • Calculate demographic parity difference
    • Calculate equalized odds difference
    • Assess calibration by subgroup
    • Which fairness criteria are violated?
  4. Root Cause Analysis
    • Where does bias originate? (Data, model, or both?)
    • Are there measurement issues?
    • Feature importance differences by group?
  5. Mitigation Strategy
    • Propose at least two mitigation approaches
    • Implement one and evaluate impact
    • What trade-offs emerge?
  6. Reporting
    • Create stakeholder-friendly summary
    • Visualize disparities
    • Make recommendations

Check Your Understanding

Test your knowledge of the key concepts from this chapter. Click “Show Answer” to reveal the correct response and explanation.

NoteQuestion 1: Informed Consent for AI

A hospital implements an AI-powered diagnostic support system. Which approach best satisfies the ethical principle of patient autonomy?

  1. Using the AI system without informing patients since it’s only a decision support tool
  2. Providing a general consent form mentioning “advanced analytics” may be used
  3. Informing patients specifically about AI use, how it works, and allowing them to opt out
  4. Only informing patients if the AI disagrees with the clinician’s judgment

Answer: c) Informing patients specifically about AI use, how it works, and allowing them to opt out

Explanation: The principle of autonomy in healthcare ethics requires that patients have the information necessary to make informed decisions about their care. This includes knowing when AI systems are being used in their diagnosis or treatment planning, understanding in general terms how these systems work, and having the option to decline AI-assisted care. Option (a) fails to respect patient autonomy by not informing patients at all. Option (b) is too vague and doesn’t provide meaningful information for informed consent. Option (d) is inadequate because patients should be informed regardless of whether the AI agrees or disagrees with clinicians. Meaningful informed consent for AI requires transparency about its use, explanation of its role in decision-making, and respect for patient choice.

NoteQuestion 2: Algorithmic Transparency

A public health department is deciding between two COVID-19 risk prediction models. Model A is a random forest with 85% accuracy but limited interpretability. Model B is a logistic regression model with 80% accuracy but clear feature weights. Which factor is MOST important in choosing between them?

  1. Always choose the higher accuracy model to maximize public health benefit
  2. Always choose the interpretable model to ensure transparency
  3. Consider the stakes of decisions, need for clinician trust, and accountability requirements
  4. Use the model that was published in the highest-impact journal

Answer: c) Consider the stakes of decisions, need for clinician trust, and accountability requirements

Explanation: The transparency vs. performance trade-off must be evaluated contextually rather than applying a blanket rule. Important considerations include: the magnitude of the performance difference (5% in this case), the consequences of prediction errors, whether post-hoc explanations could provide sufficient transparency for Model A, the need for clinician buy-in and trust, and legal/regulatory requirements for explainability. In high-stakes scenarios with significant accountability needs, the interpretable model might be preferred despite lower accuracy. In lower-stakes scenarios with a large performance gap, the black-box model might be acceptable with appropriate safeguards. Option (a) ignores the importance of trust and accountability. Option (b) dismisses performance entirely. Option (d) is irrelevant to the ethical trade-offs at hand.

NoteQuestion 3: Data Privacy in Surveillance

A state health department wants to use de-identified hospital admission data for real-time disease surveillance. Which approach provides the strongest privacy protection?

  1. De-identification: Remove direct identifiers (name, SSN) but keep detailed demographics and dates
  2. Anonymization: Remove identifiers and use techniques like generalization and noise addition to prevent re-identification
  3. Encryption: Encrypt all data fields so they appear as random strings
  4. Aggregation: Only report county-level totals without any individual records

Answer: b) Anonymization: Remove identifiers and use techniques like generalization and noise addition to prevent re-identification

Explanation: While all options provide some privacy protection, true anonymization using techniques like k-anonymity, differential privacy, or generalization provides the strongest protection against re-identification attacks. De-identification (option a) is vulnerable to re-identification through linking with other datasets, especially when quasi-identifiers like detailed demographics, ZIP codes, and exact dates are retained. Studies have shown that 87% of the U.S. population can be uniquely identified using just ZIP code, birthdate, and sex. Encryption (option c) protects data in transit and storage but someone with the decryption key can still access identifiable information. Aggregation (option d) provides strong privacy protection but sacrifices the granularity needed for effective real-time surveillance. Anonymization strikes the balance between privacy protection and data utility, though it requires careful implementation to resist modern re-identification techniques.

NoteQuestion 4: Bias Amplification

You discover that your sepsis prediction model has a higher false negative rate for Black patients (misses 30% of cases) compared to White patients (misses 15% of cases). This is an example of which type of bias?

  1. Historical bias - the model learned patterns from biased training data
  2. Representation bias - Black patients were underrepresented in training data
  3. Measurement bias - sepsis indicators measured differently across racial groups
  4. All of the above could contribute to this disparity

Answer: d) All of the above could contribute to this disparity

Explanation: Disparate performance across demographic groups typically results from multiple, interconnecting sources of bias. Historical bias could manifest if Black patients historically received delayed sepsis treatment, resulting in training data that shows different progression patterns. Representation bias would occur if Black patients were underrepresented in the training dataset, leading to a model optimized primarily for the majority group. Measurement bias could arise from tools like pulse oximeters that perform differently across skin tones, or from differential documentation practices across patient populations. In practice, these biases often compound each other - for example, underrepresentation (representation bias) makes it harder to detect measurement problems in minority groups, while historical disparities in care affect what patterns exist in the data. Addressing such disparities requires examining all potential sources of bias throughout the AI development lifecycle, not just focusing on one type.

NoteQuestion 5: The Dual Use Dilemma

A research team develops an AI model that accurately predicts individual risk of substance use disorder using social media data. The model could be used for early intervention and treatment referral, but could also be used by employers or insurers to discriminate. What is the MOST appropriate ethical response?

  1. Publish the model openly to maximize potential benefits; restrict use cases through licensing
  2. Keep the model confidential to prevent misuse, only deploying it in controlled clinical settings
  3. Publish the methodology but not the trained model weights or code
  4. Only publish after establishing legal protections against discriminatory use

Answer: b) Keep the model confidential to prevent misuse, only deploying it in controlled clinical settings

Explanation: This represents a classic “dual use” dilemma where beneficial technology also has significant potential for harm. In this case, the risks of discrimination and privacy violations are particularly severe because: (1) substance use disorder is a highly stigmatized condition, (2) employment and insurance discrimination could cause major harm, (3) social media data raises additional privacy concerns, and (4) existing anti-discrimination laws have gaps in coverage for algorithmic decision-making. Option (a) is problematic because licensing restrictions are difficult to enforce once a model is publicly released, and the harm from misuse could be severe and irreversible. Option (c) still enables others to reproduce the model and potentially misuse it. Option (d) places too much faith in legal protections that may not yet exist or may be difficult to enforce. The most appropriate approach is to deploy the model only in settings where its use can be monitored and controlled, such as healthcare facilities with ethical oversight, where it can provide benefits for early intervention while minimizing risks of discriminatory misuse. This follows the precautionary principle in high-stakes scenarios.

NoteQuestion 6: Algorithmic Recourse

A predictive model denies a patient enrollment in a preventive care program for high-risk individuals. Which feature of the system is MOST important for ethical deployment?

  1. High overall accuracy of the model (>90% AUC)
  2. A clear explanation of which factors led to the decision
  3. A process for patients to appeal the decision and receive human review
  4. Regular retraining of the model on new data

Answer: c) A process for patients to appeal the decision and receive human review

Explanation: While all options have value, algorithmic recourse—the ability to contest and potentially reverse automated decisions—is most critical for ethical deployment. Even highly accurate models (option a) make errors, and those errors can have significant consequences for individuals. The ability to appeal ensures that: (1) individual circumstances not captured by the model can be considered, (2) model errors can be identified and corrected, (3) patients maintain some agency in decisions affecting their care, and (4) accountability mechanisms exist when systems fail. Explanations (option b) are valuable but insufficient if patients cannot act on them to challenge incorrect decisions. Regular retraining (option d) improves model performance over time but doesn’t help individuals currently harmed by incorrect predictions. The right to meaningful recourse is a fundamental principle of justice and has been recognized in frameworks like the EU’s GDPR (right to human review of automated decisions) and WHO AI ethics guidelines. Without recourse mechanisms, algorithmic decision-making can become a form of “automated injustice” with no remedy.

11.15 Discussion Questions

  1. Consider the impossibility theorem: If you can’t achieve all fairness criteria simultaneously, which should take priority in public health applications? Why?

  2. When, if ever, is it acceptable to use race/ethnicity as a feature in a clinical prediction model? What are the risks and potential benefits?

  3. You’ve developed an AI system that improves overall population health but performs worse for a minority group. The alternative is to use no AI, which results in worse outcomes for everyone. What do you do?

  4. How can we balance the need for transparency in AI systems with proprietary concerns? Should all AI systems used in public health be open-source?

  5. What are the limits of individual consent for AI systems used in population health surveillance? When is it acceptable to use data without explicit consent?

  6. How should we handle cases where community preferences conflict with technical recommendations for bias mitigation?

  7. Who should be held accountable when an AI system causes harm—developers, deployers, end users, or all of the above?

11.16 Further Resources

11.16.1 📚 Essential Books

11.16.2 📄 Key Papers

Foundational: - Obermeyer et al., 2019 - Racial bias in healthcare algorithm 🎯 - Buolamwini & Gebru, 2018 - Gender Shades - Facial recognition bias - ProPublica COMPAS analysis - Recidivism prediction bias 🎯

Fairness Theory: - Chouldechova, 2017 - Fair prediction with disparate impact - Kleinberg et al., 2017 - Inherent trade-offs in algorithmic fairness - Dwork et al., 2012 - Fairness through awareness

Health Applications: - Rajkomar et al., 2018 - Ensuring fairness in ML for healthcare - Chen et al., 2019 - Addressing bias in clinical algorithms - Sjoding et al., 2020 - Pulse oximetry racial bias 🎯

Mitigation Strategies: - Bellamy et al., 2019 - AI Fairness 360 - Open source toolkit - Agarwal et al., 2018 - Reductions approach to fair classification - Kamiran & Calders, 2012 - Data preprocessing for discrimination prevention

11.16.3 🎓 Courses and Tutorials

11.16.4 💻 Tools and Toolkits

Fairness Auditing: - AI Fairness 360 (AIF360) - IBM comprehensive toolkit - Fairlearn - Microsoft fairness assessment and mitigation - Aequitas - University of Chicago bias audit tool - What-If Tool - Google visual debugging

Documentation: - Model Cards - Documentation standard - Datasheets for Datasets - Dataset documentation

Explainability: - SHAP - Feature importance - LIME - Local explanations - InterpretML - Microsoft interpretability toolkit

11.16.5 🎯 Guidelines and Frameworks

Regulatory: - EU AI Act - Comprehensive AI regulation - FDA AI/ML Action Plan - Medical device regulation - WHO Ethics and Governance of AI for Health - Six ethical principles

Professional: - ACM Code of Ethics - Computing professional ethics - IEEE Ethically Aligned Design - Comprehensive framework - Montreal Declaration for Responsible AI - Ten principles

Next: Chapter 11: Privacy, Security, and Governance →