12 Privacy, Security, and Governance

Learning Objectives

Tip

By the end of this chapter, you will:

Understand fundamental privacy principles and their application to public health AI
Navigate key data protection regulations (HIPAA, GDPR, CCPA) and their requirements
Implement privacy-preserving machine learning techniques (differential privacy, federated learning, homomorphic encryption)
Design secure AI systems that protect against common threats and vulnerabilities
Apply de-identification and anonymization techniques appropriately
Conduct privacy impact assessments for AI systems
Establish data governance frameworks for responsible AI development
Balance privacy protection with public health utility

Time to complete: 90-120 minutes Prerequisites: Chapter 3: Data, Chapter 10: Ethics

What you’ll build: 💻 Privacy impact assessment template, secure data pipeline, differential privacy implementation, federated learning prototype, and data governance framework

12.1 Introduction: The Privacy Paradox in Public Health

12.1.1 The Cambridge Analytica Wake-Up Call

[Opening case study of Cambridge Analytica and subsequent privacy awakening]

87 million Facebook users’ data harvested without consent
Personal data used for political microtargeting
Revealed how easily data can be misused
Catalyzed GDPR enforcement and global privacy awareness
Public health implications: erosion of trust in data collection

12.1.2 The Public Health Dilemma

Need for Data: - Disease surveillance requires population-level data - AI models need large, diverse datasets - Early detection depends on data sharing - Pandemic response requires rapid data access

Need for Privacy: - Protect individual autonomy and dignity - Prevent discrimination and stigma - Maintain public trust essential for participation - Legal and ethical obligations - Historical context of medical data misuse

12.1.3 High-Profile Health Data Breaches

Examples: - Anthem (2015): 78.8 million patient records - Premera Blue Cross (2015): 11 million members - UCLA Health (2015): 4.5 million patients - Quest Diagnostics (2019): 11.9 million patients - Universal Health Services (2020): Ransomware attack disrupting 400+ facilities

Consequences: - Financial costs (average $9.23M per breach in healthcare) - Identity theft and fraud - Loss of public trust - Regulatory penalties - Chilling effect on data sharing

12.1.4 What Makes Health Data Special?

Sensitive: Reveals intimate information about individuals
Permanent: Health conditions often lifelong
Identifiable: Rich auxiliary information enables re-identification
High value: Attractive target for criminals (sells for 10-50x credit card data)
Regulatory: Subject to strict legal protections
Stigmatizing: Can lead to discrimination in employment, insurance, social relationships

12.2 Privacy Principles and Frameworks

12.2.1 Fair Information Practice Principles (FIPPs)

Core Principles:

Notice/Awareness
- Inform individuals about data collection
- Explain purposes and uses
- Disclose third-party sharing
Choice/Consent
- Provide meaningful opt-in/opt-out
- Granular consent for different uses
- Easy-to-understand consent forms
- Right to withdraw consent
Access/Participation
- Individuals can view their data
- Correct inaccuracies
- Download or transfer data
Integrity/Security
- Accurate and up-to-date data
- Protected against unauthorized access
- Secure storage and transmission
Enforcement/Redress
- Mechanisms to enforce privacy rules
- Remedies for violations
- Accountability

12.2.2 Privacy by Design

Principles (Ann Cavoukian):

Proactive not reactive - Prevent privacy issues before they occur
Privacy as default - No action required by individual
Privacy embedded in design - Core functionality, not add-on
Full functionality - Positive-sum, not zero-sum
End-to-end security - Full lifecycle protection
Visibility and transparency - Keep it open
Respect for user privacy - User-centric

Application to AI: - Privacy considerations from project inception - Minimize data collection (data minimization) - De-identify at earliest possible stage - Encrypt data in transit and at rest - Implement access controls and audit logs - Privacy-preserving algorithms - Regular privacy audits

12.2.3 The Privacy-Utility Trade-off

Tension: - More privacy protection → Less data utility - More data utility → Less privacy protection

Strategies to optimize: - Use privacy-preserving techniques that maintain utility - Collect only necessary data (data minimization) - Aggregate when possible - Apply differential privacy with careful privacy budget selection - Use synthetic data when appropriate - Federated learning to keep data decentralized

12.3 Data Protection Regulations

12.3.1 HIPAA (Health Insurance Portability and Accountability Act)

Scope: US healthcare providers, health plans, healthcare clearinghouses, and business associates

Protected Health Information (PHI): - 18 identifiers that must be removed for de-identification - Includes names, dates, addresses, SSNs, medical records, etc.

Key Requirements:

Privacy Rule: - Minimum necessary standard - Patient access rights - Notice of privacy practices - Business associate agreements

Security Rule: - Administrative safeguards (policies, procedures, training) - Physical safeguards (facility access, workstation security) - Technical safeguards (access controls, encryption, audit logs)

Breach Notification Rule: - Report breaches affecting 500+ individuals to HHS within 60 days - Individual notification - Media notification for large breaches

De-identification Methods:

Safe Harbor Method:
- Remove 18 specific identifiers
- No actual knowledge that remaining data could identify individual
Expert Determination:
- Statistical/scientific analysis by qualified expert
- Very small risk of re-identification

HIPAA and AI: - De-identified data not subject to HIPAA - Business associate agreements required for AI vendors - Minimum necessary applies to AI model training - Access controls for AI systems handling PHI

12.3.3 CCPA/CPRA (California Consumer Privacy Act / California Privacy Rights Act)

Scope: Businesses collecting California residents’ personal information

Consumer Rights:

Right to know - What personal information collected, used, shared, sold
Right to delete - Request deletion
Right to opt-out - Opt out of sale of personal information
Right to non-discrimination - Equal service regardless of privacy choices
CPRA additions:
- Right to correct inaccurate information
- Right to limit use of sensitive personal information
- Opt-out of automated decision-making

CCPA and AI: - Disclosure of data collection and AI use - Opt-out of profiling - Access to data used in AI decisions - Non-discrimination (can’t charge more for opting out)

12.3.4 Other Key Regulations

PIPEDA (Canada): - Similar to GDPR - Consent, purpose limitation, individual access

Data Protection Act 2018 (UK): - UK implementation of GDPR - Post-Brexit, largely mirrors GDPR

LGPD (Brazil): - Brazilian GDPR equivalent - 10 legal bases for processing - National Data Protection Authority

Emerging Regulations: - India: Digital Personal Data Protection Act - China: Personal Information Protection Law (PIPL) - US Federal: American Data Privacy and Protection Act (proposed)

12.4 De-identification and Anonymization

12.4.1 Terminology

Identified Data: - Contains direct identifiers (name, SSN, etc.) - Clearly linked to individual

De-identified Data: - Direct identifiers removed - May still contain indirect identifiers - Re-identification possible with auxiliary information

Anonymized Data: - All identifiers removed (direct and indirect) - Re-identification computationally infeasible - Often not achievable in practice

Pseudonymized Data: - Direct identifiers replaced with pseudonyms - Linkage key stored separately - Can be re-identified by authorized parties

12.4.2 De-identification Techniques

12.4.2.1 1. Suppression

Remove or mask identifying fields

Example:

def suppress_identifiers(df, identifiers):
    """Remove direct identifiers"""
    df_deidentified = df.drop(columns=identifiers)
    return df_deidentified

# Remove names, SSNs, medical record numbers
identifiers = ['name', 'ssn', 'mrn', 'email']
df_clean = suppress_identifiers(df, identifiers)

Limitations: - Doesn’t address quasi-identifiers - May not prevent re-identification

12.4.2.2 2. Generalization

Replace specific values with broader categories

Example:

def generalize_age(age):
    """Generalize age to 5-year bins"""
    return f"{(age // 5) * 5}-{(age // 5) * 5 + 4}"

def generalize_zipcode(zipcode):
    """Keep only first 3 digits"""
    return str(zipcode)[:3] + "**"

df['age_group'] = df['age'].apply(generalize_age)
df['zip3'] = df['zipcode'].apply(generalize_zipcode)

Trade-off: - More generalization → Less re-identification risk, less utility - Less generalization → More utility, higher risk

12.4.2.3 3. Perturbation

Add noise to data

Example:

import numpy as np

def perturb_continuous(values, noise_level=0.1):
    """Add random noise to continuous variables"""
    noise = np.random.normal(0, noise_level * values.std(), size=len(values))
    return values + noise

# Add noise to lab values
df['glucose_perturbed'] = perturb_continuous(df['glucose'], noise_level=0.05)

Limitations: - Can distort statistical relationships - May affect model performance - Doesn’t prevent membership inference

12.4.2.4 4. Aggregation

Report only summary statistics, not individual records

Example:

# Instead of individual records
df_aggregated = df.groupby(['zip3', 'age_group']).agg({
    'diabetes': 'mean',
    'bmi': 'mean',
    'patient_id': 'count'
}).rename(columns={'patient_id': 'count'})

Limitations: - Reduced granularity - Can’t train individual-level models

12.4.3 K-Anonymity

Definition: Each record is indistinguishable from at least k-1 other records with respect to quasi-identifiers

Example: If k=5, every combination of (age group, gender, zip code) appears at least 5 times

Implementation:

def check_k_anonymity(df, quasi_identifiers, k=5):
    """Check if dataset satisfies k-anonymity"""
    group_sizes = df.groupby(quasi_identifiers).size()

    if (group_sizes < k).any():
        print(f"❌ Dataset does not satisfy {k}-anonymity")
        print(f"Smallest group size: {group_sizes.min()}")
        return False
    else:
        print(f"✅ Dataset satisfies {k}-anonymity")
        return True

quasi_identifiers = ['age_group', 'gender', 'zip3']
check_k_anonymity(df, quasi_identifiers, k=5)

Limitations: - Homogeneity attack: If all k records have same sensitive value - Background knowledge attack: Auxiliary information enables re-identification

12.4.4 L-Diversity

Definition: Each equivalence class has at least l “well-represented” values for sensitive attributes

Addresses: Homogeneity attack

Example: If k=5 and l=3, each group of 5 people with same quasi-identifiers has at least 3 different diagnoses

12.4.5 T-Closeness

Definition: Distribution of sensitive attribute in each equivalence class is close to distribution in overall dataset

Addresses: Skewness attack

Measures: Distance between distributions (e.g., Earth Mover’s Distance)

12.4.6 The Re-identification Challenge

Famous Examples:

1. Massachusetts Governor’s Medical Records (Latanya Sweeney, 1997) - Released “anonymized” state employee health data - Linked to voter registration using ZIP, birth date, gender - Re-identified Governor William Weld

2. Netflix Prize Dataset (Narayanan & Shmatikov, 2008) - “Anonymized” movie ratings - Linked to IMDb reviews using timestamps and ratings - Re-identified individuals including their political preferences

3. NYC Taxi Data (2014) - Released taxi trip data with “anonymized” medallion numbers - Poor hashing allowed reversal - Could identify celebrity trips, mosque visits, strip club visits

Lessons: - Quasi-identifiers enable re-identification - Auxiliary data often publicly available - “Anonymization” often insufficient - Need formal privacy guarantees

12.5 Privacy-Preserving Machine Learning

12.5.1 Differential Privacy

Core Idea: Provide plausible deniability - Individual’s data doesn’t significantly affect query results

Formal Definition:

A mechanism M satisfies ε-differential privacy if for all datasets D₁ and D₂ differing in one record, and all possible outputs S:

P(M(D₁) ∈ S) ≤ e^ε × P(M(D₂) ∈ S)

Intuition: - ε (epsilon) is privacy budget - Smaller ε → Stronger privacy - Can’t tell if individual’s data was included based on output

Properties:

Composition: Multiple queries consume privacy budget
Post-processing: Can’t reduce privacy by analyzing DP output
Group privacy: Protects groups, though with weaker guarantee

12.5.1.1 Implementing Differential Privacy

Laplace Mechanism:

Add noise from Laplace distribution to query results

import numpy as np

def laplace_mechanism(true_value, sensitivity, epsilon):
    """Add Laplace noise for differential privacy"""
    scale = sensitivity / epsilon
    noise = np.random.laplace(0, scale)
    return true_value + noise

# Example: Private count query
true_count = 1000
sensitivity = 1  # Removing one person changes count by 1
epsilon = 0.1  # Privacy budget

private_count = laplace_mechanism(true_count, sensitivity, epsilon)
print(f"True count: {true_count}")
print(f"Private count: {private_count:.0f}")

Gaussian Mechanism:

Add Gaussian noise (for (ε, δ)-differential privacy)

def gaussian_mechanism(true_value, sensitivity, epsilon, delta):
    """Add Gaussian noise for differential privacy"""
    sigma = np.sqrt(2 * np.log(1.25 / delta)) * sensitivity / epsilon
    noise = np.random.normal(0, sigma)
    return true_value + noise

12.5.1.2 Differentially Private Machine Learning

DP-SGD (Differentially Private Stochastic Gradient Descent):

import tensorflow as tf
from tensorflow_privacy.privacy.optimizers.dp_optimizer_keras import DPKerasSGDOptimizer

# Standard model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# DP optimizer
optimizer = DPKerasSGDOptimizer(
    l2_norm_clip=1.0,        # Clip gradients
    noise_multiplier=0.5,     # Noise level
    num_microbatches=250,     # Batch processing
    learning_rate=0.01
)

model.compile(
    optimizer=optimizer,
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# Track privacy budget
from tensorflow_privacy.privacy.analysis import compute_dp_sgd_privacy

eps, delta = compute_dp_sgd_privacy.compute_dp_sgd_privacy(
    n=50000,              # Training set size
    batch_size=250,
    noise_multiplier=0.5,
    epochs=10,
    delta=1e-5
)

print(f"Privacy guarantee: (ε={eps:.2f}, δ={delta})")

How DP-SGD Works:

Clip gradients: Limit individual influence
Add noise: Random noise to gradient updates
Track privacy budget: Account for cumulative privacy loss

Trade-offs: - Privacy (small ε) ↔︎ Model accuracy - More training data → Better accuracy with same privacy - Hyperparameter tuning crucial

12.5.1.3 Practical Considerations

Choosing ε: - ε < 1: Strong privacy - ε = 1-10: Moderate privacy (common in practice) - ε > 10: Weak privacy - Context-dependent: Depends on sensitivity of data, threat model

Privacy Budget Management: - Total budget for entire analysis - Allocate across different queries/models - Once exhausted, can’t answer more queries - Strategic allocation to most important analyses

Utility-Privacy Trade-off:

import matplotlib.pyplot as plt

epsilons = [0.1, 0.5, 1.0, 2.0, 5.0, 10.0]
accuracies = []

for eps in epsilons:
    # Train DP model with epsilon
    model = train_dp_model(X_train, y_train, epsilon=eps)
    acc = model.evaluate(X_test, y_test)
    accuracies.append(acc)

plt.plot(epsilons, accuracies, marker='o')
plt.xlabel('Privacy Budget (ε)')
plt.ylabel('Model Accuracy')
plt.title('Privacy-Utility Trade-off')
plt.xscale('log')

12.5.2 Federated Learning

Core Idea: Train models on decentralized data without sharing raw data

Architecture:

Central server initializes model
Clients (hospitals, devices) receive model
Local training on private data
Share updates (gradients, not data) to server
Server aggregates updates
Repeat until convergence

Implementation Example:

# Central server
class FederatedServer:
    def __init__(self, model):
        self.global_model = model

    def aggregate(self, client_updates):
        """Federated averaging"""
        # Average model weights from clients
        avg_weights = []

        for weights_list in zip(*client_updates):
            avg_weights.append(
                np.mean(weights_list, axis=0)
            )

        self.global_model.set_weights(avg_weights)
        return self.global_model

# Client
class FederatedClient:
    def __init__(self, data, labels):
        self.data = data
        self.labels = labels

    def train(self, global_model, epochs=5):
        """Train on local data"""
        local_model = tf.keras.models.clone_model(global_model)
        local_model.set_weights(global_model.get_weights())

        local_model.fit(
            self.data,
            self.labels,
            epochs=epochs,
            verbose=0
        )

        return local_model.get_weights()

# Federated training loop
server = FederatedServer(initial_model)

for round in range(num_rounds):
    # Distribute model to clients
    global_model = server.global_model

    # Each client trains locally
    client_updates = []
    for client in clients:
        updated_weights = client.train(global_model)
        client_updates.append(updated_weights)

    # Server aggregates updates
    server.aggregate(client_updates)

    print(f"Round {round} complete")

Privacy Properties: - Raw data never leaves client - Only model updates shared - Can combine with differential privacy (DP-FL)

Challenges:

Communication costs - Frequent updates expensive
Non-IID data - Data distribution varies across clients
Client availability - Devices may go offline
Byzantine attacks - Malicious clients send bad updates

Secure Aggregation:

Ensure server can’t see individual client updates

# Simplified secure aggregation concept
def secure_aggregate(client_updates):
    """
    Server learns sum of updates but not individual contributions
    Uses secret sharing and cryptographic techniques
    """
    # Clients add secret shares that cancel out in aggregate
    # Server computes sum without seeing individual values
    return sum(client_updates)

Real-World Applications: - Google Gboard: Next-word prediction - Apple: Emoji predictions, QuickType - Healthcare: Multi-hospital model training without data sharing

12.5.3 Homomorphic Encryption

Core Idea: Compute on encrypted data without decrypting

Types:

Partially Homomorphic: Support one operation (+ or ×)
Somewhat Homomorphic: Limited operations
Fully Homomorphic (FHE): Unlimited operations on encrypted data

Example (Conceptual):

#| eval: false
# Simplified example (real implementations much more complex)
from phe import paillier

# Generate keys
public_key, private_key = paillier.generate_paillier_keypair()

# Encrypt data
x1_encrypted = public_key.encrypt(10)
x2_encrypted = public_key.encrypt(5)

# Compute on encrypted data
sum_encrypted = x1_encrypted + x2_encrypted
product_encrypted = x1_encrypted * 3

# Decrypt results
sum_result = private_key.decrypt(sum_encrypted)  # 15
product_result = private_key.decrypt(product_encrypted)  # 30

ML with Homomorphic Encryption:

#| eval: false
# Train model on plaintext
model = train_model(X_train, y_train)

# Client encrypts their data
client_data_encrypted = [public_key.encrypt(x) for x in client_data]

# Server makes predictions on encrypted data
encrypted_prediction = model.predict_encrypted(client_data_encrypted)

# Client decrypts prediction
prediction = private_key.decrypt(encrypted_prediction)

Challenges: - Computational cost: 100-1000x slower than plaintext - Implementation complexity: Requires specialized expertise - Limited operations: Some ML operations difficult to implement - Noise accumulation: Errors compound with operations

Practical Use Cases: - Cloud prediction without revealing inputs - Privacy-preserving medical diagnosis - Encrypted database queries

12.5.4 Secure Multi-Party Computation (MPC)

Core Idea: Multiple parties jointly compute function without revealing inputs

Example: Private Set Intersection

Two hospitals find shared patients without revealing full patient lists

# Simplified concept (real MPC uses cryptographic protocols)
def private_set_intersection(hospital1_patients, hospital2_patients):
    """
    Find intersection without revealing individual lists
    Uses cryptographic protocols in practice
    """
    # Each party encrypts their set
    # Compute intersection on encrypted sets
    # Reveal only intersection

    return shared_patients

# Hospital 1 has patients A, B, C
# Hospital 2 has patients B, C, D
# Output: B, C (without revealing A or D)

Applications: - Multi-institutional studies - Federated cohort identification - Privacy-preserving record linkage

12.5.5 Synthetic Data

Core Idea: Generate artificial data that preserves statistical properties

Methods:

1. Statistical Matching:

from synthpop import Synthpop

# Generate synthetic data matching distributions
synth = Synthpop()
synthetic_data = synth.fit_transform(real_data)

2. Generative Models:

from sdv.tabular import CTGAN

# Train GAN on real data
model = CTGAN()
model.fit(real_data)

# Generate synthetic samples
synthetic_data = model.sample(num_rows=10000)

3. Differentially Private Synthetic Data:

from smartnoise.synthesizers import PytorchDPSynthesizer

synth = PytorchDPSynthesizer(
    epsilon=1.0,
    preprocessor_eps=0.5
)

synth.fit(real_data)
synthetic_data = synth.sample(10000)

Advantages: - Can be shared publicly - No individual privacy concerns - Unlimited access

Limitations: - May not capture rare events - Statistical properties approximate - Validation against real data needed - Can still leak information if not properly generated

12.6 Security for AI Systems

12.6.1 Threat Landscape

Categories of Threats:

Data poisoning - Manipulate training data
Model inversion - Reconstruct training data from model
Membership inference - Determine if individual in training set
Model stealing - Replicate model through queries
Adversarial examples - Inputs crafted to fool model
Backdoor attacks - Trigger model to behave maliciously
Infrastructure attacks - Compromise systems running AI

12.6.2 Adversarial Examples

Definition: Inputs designed to cause model misclassification

Example:

import tensorflow as tf
from cleverhans.tf2.attacks.fast_gradient_method import fast_gradient_method

# Original image correctly classified
prediction = model.predict(original_image)
print(f"Original prediction: {prediction}")

# Generate adversarial example
adversarial_image = fast_gradient_method(
    model,
    original_image,
    eps=0.3,  # Perturbation size
    norm=np.inf
)

# Adversarial image misclassified (but looks identical to humans)
adv_prediction = model.predict(adversarial_image)
print(f"Adversarial prediction: {adv_prediction}")

Real-World Impact: - Medical imaging: Adversarial noise causes misdiagnosis - Autonomous vehicles: Stickers fool object detection - Voice assistants: Inaudible commands

Defenses:

1. Adversarial Training:

# Train on mix of clean and adversarial examples
for epoch in range(num_epochs):
    for batch in dataset:
        # Generate adversarial examples
        adv_batch = generate_adversarial(batch)

        # Train on both
        model.train_on_batch(batch, labels)
        model.train_on_batch(adv_batch, labels)

2. Input Validation:

def detect_adversarial(input_data):
    """Check for adversarial perturbations"""
    # Statistical tests
    # Feature squeezing
    # Ensemble consistency checks

    if is_suspicious(input_data):
        return "Rejected: Possible adversarial input"
    else:
        return model.predict(input_data)

3. Certified Defenses: - Provide mathematical guarantee of robustness - Randomized smoothing - Interval bound propagation

12.6.3 Model Inversion Attacks

Attack: Reconstruct training data from model

Example:

# Attacker has access to face recognition model
# Reconstruct faces from model weights/outputs

def model_inversion_attack(model, target_label):
    """
    Reconstruct training example for target label
    """
    # Start with random input
    reconstructed = np.random.rand(input_shape)

    # Optimize to maximize confidence for target label
    for iteration in range(1000):
        with tf.GradientTape() as tape:
            tape.watch(reconstructed)
            prediction = model(reconstructed)
            loss = -prediction[target_label]  # Maximize target class

        gradient = tape.gradient(loss, reconstructed)
        reconstructed -= learning_rate * gradient

    return reconstructed

Real Example: - Fredrikson et al. (2015): Reconstructed faces from facial recognition models - Revealed sensitive training data

Defenses: - Differential privacy in training - Output perturbation - Limit model access (no confidence scores, only labels)

12.6.4 Membership Inference Attacks

Attack: Determine if specific individual’s data was in training set

Method:

def membership_inference_attack(model, data_point):
    """
    Predict if data_point was in training set
    Overfitting causes different behavior on training vs. test data
    """
    # Get model confidence
    confidence = model.predict_proba(data_point)

    # High confidence suggests training set member
    # Train attack model on confidence scores
    was_in_training = attack_model.predict(confidence)

    return was_in_training

Real Example: - Shokri et al. (2017): Membership inference on medical data - Privacy violation even with “anonymous” data

Defenses: - Differential privacy (most effective) - Regularization to prevent overfitting - Confidence masking

12.6.5 Data Poisoning

Attack: Inject malicious data to corrupt model

Example:

# Attacker adds poisoned samples to training data
poisoned_samples = [
    (malicious_input_1, wrong_label_1),
    (malicious_input_2, wrong_label_2),
    ...
]

# Model trained on poisoned data performs poorly
training_data_poisoned = clean_data + poisoned_samples
model = train(training_data_poisoned)

Types:

Availability attacks - Degrade overall performance
Targeted attacks - Cause specific misclassifications
Backdoor attacks - Trigger malicious behavior with specific input pattern

Real-World Concern: - Crowdsourced data - User-contributed labels - Data scraped from internet

Defenses:

1. Data Validation:

def validate_data(new_samples):
    """Detect outliers and suspicious samples"""
    # Statistical outlier detection
    outliers = detect_outliers(new_samples)

    # Consistency checks
    inconsistent = check_consistency(new_samples)

    # Remove suspicious samples
    clean_samples = remove_flagged(new_samples, outliers, inconsistent)

    return clean_samples

2. Robust Training: - Median instead of mean for aggregation - Trim extreme values - Weight samples by trustworthiness

3. Anomaly Detection: - Monitor training loss for suspicious patterns - Detect samples with unusual influence on model

12.6.6 Security Best Practices

1. Secure Data Storage:

# Encrypt data at rest
from cryptography.fernet import Fernet

key = Fernet.generate_key()
cipher = Fernet(key)

# Encrypt
encrypted_data = cipher.encrypt(sensitive_data.encode())

# Decrypt only when needed
decrypted_data = cipher.decrypt(encrypted_data).decode()

2. Access Controls:

# Role-based access control
class DataAccess:
    def __init__(self, user, role):
        self.user = user
        self.role = role

    def can_access(self, resource):
        """Check if user has permission"""
        permissions = {
            'researcher': ['read'],
            'analyst': ['read', 'aggregate'],
            'admin': ['read', 'write', 'delete']
        }

        required_permission = resource.permission_level
        user_permissions = permissions.get(self.role, [])

        return required_permission in user_permissions

3. Audit Logging:

import logging

def log_access(user, resource, action):
    """Log all data access for audit trail"""
    logging.info(f"User: {user} | Resource: {resource} | Action: {action} | Time: {timestamp}")

# Log every model prediction
def secure_predict(model, input_data, user):
    log_access(user, "model_predictions", "predict")
    return model.predict(input_data)

4. Secure Model Deployment:

API authentication and rate limiting
Input validation and sanitization
Model versioning and rollback capability
Monitoring for anomalous queries

5. Regular Security Audits:

Penetration testing
Code review for vulnerabilities
Dependency scanning (vulnerable libraries)
Red team exercises

12.7 Privacy Impact Assessment

12.7.1 Framework Overview

Systematic evaluation of privacy risks before deployment

Key Questions: 1. What personal data is collected and why? 2. Who has access to the data? 3. How long is data retained? 4. What are privacy risks? 5. What safeguards are in place? 6. How will we monitor compliance?

12.7.2 Step 1: Data Inventory

Map All Personal Data:

data_inventory = {
    'demographics': {
        'fields': ['age', 'sex', 'race', 'ethnicity', 'zip_code'],
        'sensitivity': 'medium',
        'purpose': 'risk stratification',
        'retention': '7 years',
        'source': 'EHR'
    },
    'clinical': {
        'fields': ['diagnoses', 'medications', 'lab_results', 'vital_signs'],
        'sensitivity': 'high',
        'purpose': 'model features',
        'retention': '7 years',
        'source': 'EHR'
    },
    'genetic': {
        'fields': ['snps', 'polygenic_scores'],
        'sensitivity': 'very high',
        'purpose': 'precision medicine',
        'retention': '10 years',
        'source': 'genetic_testing'
    }
}

12.7.3 Step 2: Legal and Regulatory Analysis

Checklist:

HIPAA compliance (if US healthcare data)
GDPR compliance (if EU residents)
State privacy laws (CCPA, etc.)
Institutional Review Board (IRB) approval
Data use agreements
Business associate agreements
Consent forms adequate and up-to-date

12.7.4 Step 3: Risk Assessment

Identify Risks:

risk_assessment = [
    {
        'risk': 'Re-identification of patients',
        'likelihood': 'medium',
        'impact': 'high',
        'overall': 'high',
        'mitigation': 'k-anonymity, differential privacy'
    },
    {
        'risk': 'Data breach (unauthorized access)',
        'likelihood': 'low',
        'impact': 'very high',
        'overall': 'high',
        'mitigation': 'Encryption, access controls, audit logs'
    },
    {
        'risk': 'Discrimination based on AI predictions',
        'likelihood': 'medium',
        'impact': 'high',
        'overall': 'high',
        'mitigation': 'Fairness audits, human oversight'
    },
    {
        'risk': 'Function creep (data used beyond original purpose)',
        'likelihood': 'medium',
        'impact': 'medium',
        'overall': 'medium',
        'mitigation': 'Purpose limitation, governance oversight'
    }
]

Risk Matrix:

Likelihood	Impact	Overall Risk
Low	Low	Low
Low	High	Medium
High	Low	Medium
High	High	Very High

12.7.5 Step 4: Privacy-Preserving Measures

Technical Safeguards:

privacy_measures = {
    'data_minimization': {
        'description': 'Collect only necessary fields',
        'implementation': 'Remove free text, limit date precision',
        'effectiveness': 'high'
    },
    'de_identification': {
        'description': 'Remove direct identifiers',
        'implementation': 'HIPAA Safe Harbor method',
        'effectiveness': 'medium'
    },
    'differential_privacy': {
        'description': 'Add statistical noise',
        'implementation': 'DP-SGD with ε=1.0',
        'effectiveness': 'high'
    },
    'encryption': {
        'description': 'Encrypt data at rest and in transit',
        'implementation': 'AES-256, TLS 1.3',
        'effectiveness': 'high'
    },
    'access_controls': {
        'description': 'Limit who can access data',
        'implementation': 'Role-based access control, MFA',
        'effectiveness': 'high'
    }
}

Organizational Safeguards:

Privacy training for all staff
Data governance committee
Incident response plan
Regular privacy audits
Data retention and deletion policies

12.7.6 Step 5: Stakeholder Consultation

Engage: - Patients/community members - Privacy officer - Legal counsel - Ethics committee - Information security team - Clinical staff (end users)

Document Feedback: - Concerns raised - Modifications made - Remaining tensions

12.7.7 Step 6: Ongoing Monitoring

Metrics to Track:

privacy_metrics = {
    'access_logs': 'Daily review of unusual access patterns',
    'breach_incidents': 'Count and severity of privacy incidents',
    're_identification_attempts': 'Monitor for linkage attacks',
    'consent_withdrawals': 'Track and honor opt-outs',
    'privacy_budget': 'Track cumulative ε for DP queries',
    'audit_findings': 'Issues identified in privacy audits'
}

Triggers for Re-assessment: - New data sources added - Change in purpose or use - New regulations - Privacy incident - Annual review

12.8 Data Governance

12.8.1 Governance Framework

Core Components:

Policies - High-level principles and rules
Procedures - Specific steps to implement policies
Roles - Who is responsible for what
Controls - Mechanisms to enforce policies
Monitoring - Track compliance and effectiveness

12.8.2 Data Governance Committee

Composition: - Chief Data Officer (chair) - Privacy Officer - Security Officer - Legal Counsel - Clinical/Public Health Leadership - Data Scientists - Community Representative

Responsibilities: - Approve data access requests - Review high-risk AI projects - Set data classification standards - Oversee data quality - Manage data lifecycle - Incident response - Policy updates

12.8.3 Data Classification

Levels:

data_classification = {
    'public': {
        'examples': 'Aggregate statistics, published reports',
        'controls': 'None required',
        'encryption': 'Optional'
    },
    'internal': {
        'examples': 'Non-sensitive operational data',
        'controls': 'Authentication required',
        'encryption': 'Recommended'
    },
    'confidential': {
        'examples': 'De-identified patient data',
        'controls': 'Role-based access, audit logging',
        'encryption': 'Required'
    },
    'restricted': {
        'examples': 'Identified PHI, genetic data',
        'controls': 'Approval required, strict access controls, MFA',
        'encryption': 'Required (strong)'
    }
}

12.8.4 Data Access Request Process

Workflow:

def process_data_request(request):
    """
    Systematic review of data access requests
    """
    # 1. Initial screening
    if not request.has_valid_purpose():
        return "Denied: Purpose not aligned with data use agreement"

    # 2. Legal review
    if not request.has_required_approvals():
        return "Pending: Requires IRB approval"

    # 3. Privacy review
    if request.needs_identified_data():
        if not request.has_compelling_justification():
            return "Denied: De-identified data sufficient"

    # 4. Security review
    if not request.has_secure_environment():
        return "Conditional: Implement additional security controls"

    # 5. Data governance committee review
    committee_decision = committee.review(request)

    if committee_decision == "approved":
        # 6. Grant limited access
        access = grant_access(
            user=request.user,
            data=request.data_requested,
            duration=request.duration,
            conditions=request.conditions
        )

        # 7. Monitor usage
        monitor_access(access)

        return f"Approved: Access granted until {access.expiration}"

    return committee_decision

12.8.5 Data Lifecycle Management

Stages:

Collection
- Purpose specification
- Consent obtained
- Minimize collection
- Security during collection
Storage
- Encrypted at rest
- Access controls
- Regular backups
- Geographic restrictions
Use
- Audit all access
- Purpose limitation
- Data quality checks
- Minimal necessary access
Sharing
- Data use agreements
- De-identification
- Limited datasets
- Track recipients
Retention
- Follow retention policies
- Periodic review
- Justify ongoing storage
Disposal
- Secure deletion
- Certificate of destruction
- Update records
- Notify stakeholders if required

12.8.6 Data Quality Governance

Dimensions:

Accuracy: Correct and error-free
Completeness: All required data present
Consistency: Uniform across systems
Timeliness: Up-to-date
Validity: Conforms to schema
Uniqueness: No duplicates

Monitoring:

def assess_data_quality(dataset):
    """
    Comprehensive data quality checks
    """
    quality_report = {}

    # Completeness
    quality_report['completeness'] = {
        'missing_values': dataset.isnull().sum().to_dict(),
        'missing_percentage': (dataset.isnull().sum() / len(dataset) * 100).to_dict()
    }

    # Accuracy (range checks)
    quality_report['accuracy'] = {
        'age_out_of_range': ((dataset['age'] < 0) | (dataset['age'] > 120)).sum(),
        'implausible_values': check_clinical_plausibility(dataset)
    }

    # Consistency
    quality_report['consistency'] = {
        'date_inconsistencies': (dataset['death_date'] < dataset['birth_date']).sum(),
        'conflicting_values': check_logical_consistency(dataset)
    }

    # Uniqueness
    quality_report['uniqueness'] = {
        'duplicate_rows': dataset.duplicated().sum(),
        'duplicate_ids': dataset['patient_id'].duplicated().sum()
    }

    return quality_report

12.9 Case Studies

12.9.1 Case Study 1: COVID-19 Contact Tracing Apps

Challenge: Balance privacy with public health surveillance

Approaches:

Centralized (France’s StopCovid): - Central server stores contact graph - Government can identify contacts - More powerful epidemiological analysis - Higher privacy risk

Decentralized (Germany’s Corona-Warn-App, Apple/Google): - Bluetooth tokens exchanged locally - No central contact graph - Notifications without revealing identity - Uses differential privacy for aggregate statistics

Lessons: - Technical design has major privacy implications - Public trust essential for adoption - Transparency in design builds confidence - Privacy-preserving approaches can still be effective

12.9.3 Case Study 3: Strava Fitness App Heat Map

2018 Incident:

What happened: - Strava published global heat map of user activities - Aggregated, “anonymized” data - Revealed locations of military bases - Could track individual soldiers’ routines

Privacy Failures: - Insufficient aggregation in sparse areas - Didn’t anticipate security implications - User consent for research, not military intelligence

Lessons: - Context matters for privacy - Aggregate data can still reveal sensitive information - Consider adversarial use cases - Privacy review should include domain experts

12.10 Practical Implementation Checklist

12.10.1 Privacy Protection Checklist

Data Collection: - [ ] Collect only necessary data (data minimization) - [ ] Obtain informed consent - [ ] Document purpose and legal basis - [ ] Provide privacy notice - [ ] Implement opt-out mechanism

Data Storage: - [ ] Encrypt data at rest (AES-256 or equivalent) - [ ] Secure key management - [ ] Access controls (role-based) - [ ] Multi-factor authentication - [ ] Regular backups - [ ] Geographic restrictions if required

Data Processing: - [ ] De-identify early in pipeline - [ ] Apply k-anonymity or differential privacy - [ ] Process on secure infrastructure - [ ] Limit access to trained personnel - [ ] Audit all data access - [ ] Data lineage tracking

Model Development: - [ ] Use privacy-preserving ML techniques - [ ] Federated learning if data can’t be centralized - [ ] Differential privacy in training - [ ] Limit model access (APIs, not raw model) - [ ] Monitor for model inversion risks

Deployment: - [ ] Secure API with authentication - [ ] Rate limiting to prevent model stealing - [ ] Input validation - [ ] Anomaly detection - [ ] Logging and monitoring - [ ] Incident response plan

Governance: - [ ] Privacy impact assessment complete - [ ] Data governance committee approval - [ ] Regular privacy audits - [ ] Staff privacy training - [ ] Data retention policy - [ ] Secure disposal procedures

12.10.2 Security Checklist

Infrastructure: - [ ] Network segmentation - [ ] Firewall configuration - [ ] Intrusion detection system - [ ] Patch management - [ ] Vulnerability scanning - [ ] Penetration testing

Application: - [ ] Secure coding practices - [ ] Input validation and sanitization - [ ] Output encoding - [ ] Dependency scanning - [ ] Code review - [ ] Security testing

Data: - [ ] Encryption in transit (TLS 1.3) - [ ] Encryption at rest - [ ] Secure key management - [ ] Data loss prevention - [ ] Backup encryption - [ ] Secure deletion

Access: - [ ] Principle of least privilege - [ ] Role-based access control - [ ] Multi-factor authentication - [ ] Password policies - [ ] Access review and revocation - [ ] Audit logging

Monitoring: - [ ] Security information and event management (SIEM) - [ ] Anomaly detection - [ ] Incident response plan - [ ] Disaster recovery plan - [ ] Regular security audits - [ ] Compliance monitoring

12.11 Key Takeaways

Privacy and Utility Can Coexist: Privacy-preserving techniques enable valuable AI while protecting individuals. It’s not zero-sum.
Multiple Regulations Apply: HIPAA, GDPR, CCPA, and others have overlapping but distinct requirements. Compliance is complex but essential.
De-identification ≠ Anonymization: Removing identifiers helps but doesn’t guarantee privacy. Re-identification is often possible.
Differential Privacy Provides Formal Guarantees: Strongest privacy definition, but requires careful implementation and privacy budget management.
Federated Learning Keeps Data Decentralized: Train models without centralizing sensitive data. Powerful for multi-institutional collaboration.
Security Is Essential: Privacy techniques don’t help if systems are compromised. Defense in depth required.
Governance Enables Responsible Use: Policies, procedures, and oversight ensure privacy protection is ongoing, not one-time.
Transparency Builds Trust: Clear communication about data use and privacy protections essential for public acceptance.

12.12 Hands-On Exercise: Build a Privacy-Preserving Pipeline

Scenario: You’re building a diabetes risk prediction model using multi-hospital data. Each hospital is reluctant to share raw patient data.

Task: Implement a privacy-preserving solution

Dataset: hospital_*.csv (synthetic data for 3 hospitals)

Steps:

Privacy Impact Assessment
- Identify sensitive fields
- Assess re-identification risks
- Document legal requirements
- Plan mitigation strategies
De-identification
- Apply HIPAA Safe Harbor method
- Check for k-anonymity (k=5)
- Remove high-risk fields
Implement Differential Privacy
- Train DP-SGD model (ε=1.0)
- Compare accuracy to non-private baseline
- Visualize privacy-utility trade-off
Federated Learning
- Implement federated averaging
- Train local models at each hospital
- Aggregate without sharing data
- Compare to centralized approach
Security Measures
- Encrypt data at rest
- Implement access controls
- Add audit logging
- Create API with authentication
Governance
- Draft data use agreement
- Define roles and responsibilities
- Create monitoring plan
- Document the system

Deliverable: Privacy-preserving ML pipeline with documentation

Check Your Understanding

Test your knowledge of privacy, security, and governance in public health AI. Each question builds on the key concepts from this chapter.

Question 1

A research team applies HIPAA Safe Harbor de-identification to a dataset by removing all 18 specified identifiers (names, dates, addresses, etc.). The dataset includes quasi-identifiers like 5-digit ZIP code, exact age, and gender. They plan to share this data publicly for a machine learning competition. What is the PRIMARY risk they should be concerned about?

HIPAA violations, since Safe Harbor method doesn’t apply to machine learning competitions
Re-identification through linkage with publicly available voter registration or other auxiliary data containing similar quasi-identifiers
Model inversion attacks that could reconstruct individual records from trained models
Copyright infringement for sharing patient data without explicit consent

Correct Answer: b) Re-identification through linkage with publicly available voter registration or other auxiliary data containing similar quasi-identifiers

This question addresses one of the most critical lessons in privacy: de-identification through identifier removal alone is often insufficient to prevent re-identification. The chapter extensively discusses this through famous examples like Latanya Sweeney’s 1997 re-identification of Massachusetts Governor William Weld’s medical records.

While the team correctly removed the 18 HIPAA identifiers (direct identifiers like names, SSNs, medical record numbers), they left quasi-identifiers intact: 5-digit ZIP code, exact age, and gender. Research has shown that 87% of the US population can be uniquely identified using just these three fields when linked to publicly available databases like voter registration, which contain the same information.

The mechanism of re-identification is straightforward: an attacker obtains the “de-identified” dataset and a public database (voter rolls, real estate records, social media profiles). They join the datasets on quasi-identifiers (ZIP, age, gender), creating links between supposedly anonymous health records and identified individuals. Even if not everyone is uniquely identified, significant proportions of the dataset can be linked, especially for individuals in low-population areas or with unusual combinations of characteristics.

Option (a) is incorrect; HIPAA Safe Harbor can apply to various uses including research, though public release requires additional considerations. The violation concern isn’t about the competition itself but about insufficient privacy protection. Option (c) identifies a real risk but not the primary one—model inversion attacks require access to trained models and are more complex than simple linkage attacks. The immediate and more likely risk is re-identification through direct data linkage. Option (d) is incorrect; copyright isn’t the relevant legal framework for health data privacy (HIPAA, state privacy laws, and institutional policies are).

The chapter emphasizes that effective de-identification requires addressing both direct identifiers AND quasi-identifiers. Solutions include: - K-anonymity (k=5 or greater): ensure each quasi-identifier combination appears at least k times - Generalization: aggregate 5-digit ZIP to 3-digit, exact age to age ranges - Differential privacy: add statistical noise to make linkage attacks computationally infeasible - Limited datasets: provide only aggregate statistics or restricted access with data use agreements

The broader lesson extends to the Netflix Prize and NYC Taxi cases discussed in the chapter. In both, organizations released “anonymized” data that was subsequently re-identified through clever linkage attacks using auxiliary information. The Netflix dataset was linked to IMDb reviews using timestamp patterns, and NYC taxi data revealed celebrity trips and mosque visits despite “anonymized” medallion numbers.

This underscores a fundamental principle in privacy: anonymization is not a binary state but a spectrum of risk. The chapter argues that true anonymization (where re-identification is computationally infeasible for any attacker) is often unachievable, and we should instead focus on formal privacy guarantees like differential privacy that provide mathematically provable protection bounds.

For public health practitioners: never assume identifier removal alone provides sufficient protection. Always assess re-identification risk by considering what auxiliary data might exist, conduct k-anonymity checks, and consider whether differential privacy or restricted access models (data enclaves, remote analysis) are more appropriate than public data release.

Question 2

A multi-hospital consortium wants to train a disease prediction model using patient data from 5 hospitals. Each hospital’s data contains sensitive patient information and cannot be shared due to privacy regulations and institutional policies. Which privacy-preserving approach would BEST allow collaborative model training while keeping data decentralized?

Have each hospital apply k-anonymity (k=10) to their data, then share the anonymized datasets to a central location for training
Implement federated learning where models are trained locally at each hospital and only model updates are shared with a central server
Use homomorphic encryption to encrypt all patient data, share encrypted datasets centrally, and train on encrypted data
Generate synthetic data at each hospital using GANs, share the synthetic datasets, and train on the combined synthetic data

Correct Answer: b) Implement federated learning where models are trained locally at each hospital and only model updates are shared with a central server

This question tests understanding of privacy-preserving machine learning techniques and their appropriate use cases. Federated learning is specifically designed for the scenario described: collaborative machine learning across multiple institutions with decentralized data that cannot be shared.

The chapter details how federated learning works: (1) A central server initializes a global model, (2) Each hospital receives a copy and trains it locally on their private data, (3) Hospitals send only model updates (gradients or weights) back to the central server—NOT raw data, (4) The server aggregates updates using federated averaging, (5) The process repeats until convergence. Critically, raw patient data never leaves each hospital’s secure environment, satisfying both regulatory requirements (HIPAA, institutional policies) and privacy goals.

Real-world implementations include Google’s Gboard (next-word prediction trained on millions of phones without collecting typing data) and multi-hospital clinical AI collaborations. The chapter notes this can be further strengthened with secure aggregation (server can’t see individual hospital updates) and differential privacy (DP-FL, adding noise to updates).

Option (a) has multiple problems. K-anonymity reduces but doesn’t eliminate re-identification risk (as discussed in Question 1). More fundamentally, k-anonymity requires substantial data utility loss through generalization, and once shared centrally, the data is still pooled and vulnerable to various attacks. The question specifies data cannot be shared, which k-anonymity doesn’t solve—it just reduces one type of privacy risk. Option (c)—homomorphic encryption—is theoretically appealing but practically challenging. The chapter notes FHE is 100-1000x slower than plaintext computation, requires specialized expertise, and many ML operations are difficult to implement homomorphically. While promising for specific use cases (cloud prediction without revealing inputs), FHE is not yet practical for training complex models on large medical datasets. Option (d)—synthetic data—can be useful but has limitations: synthetic data may not capture rare events or complex relationships critical for medical prediction, requires validation against real data (creating a chicken-and-egg problem), and doesn’t fundamentally solve the collaboration challenge since generating high-quality synthetic data itself requires significant expertise and may leak information if not done with differential privacy.

The chapter emphasizes federated learning’s advantages for this exact scenario: - Privacy: Raw data never centralized - Compliance: Satisfies data governance restrictions - Scalability: Can include many institutions - Practicality: Implemented successfully in production systems - Flexibility: Compatible with differential privacy for stronger guarantees

Challenges mentioned include communication costs (frequent model updates), non-IID data (heterogeneous patient populations across hospitals), and Byzantine attacks (malicious participants), but these are manageable through techniques like federated averaging variants, stratified sampling, and robust aggregation.

The key principle: federated learning enables “bringing the model to the data” rather than “bringing data to the model”—a fundamental shift that respects data locality requirements while enabling collaborative learning. This approach aligns with Privacy by Design principles (privacy embedded in architecture) and data governance needs (institutional control maintained).

For public health practitioners: federated learning is particularly relevant for multi-institutional research, rare disease studies requiring data from many hospitals, and international collaborations where data transfer restrictions apply. Frameworks like TensorFlow Federated, PySyft, and Flower make implementation increasingly accessible, though proper deployment requires expertise in both ML and distributed systems.

Question 3

You’re implementing differential privacy (DP) for a diabetes prediction model. You train three models with different epsilon (ε) values: Model A (ε=0.1), Model B (ε=1.0), and Model C (ε=10.0). Accuracy on test data: Model A=72%, Model B=81%, Model C=84%. Non-private baseline accuracy=85%. Which statement BEST describes the privacy-utility trade-off and appropriate model choice?

Model C is best because its accuracy is closest to the non-private baseline, and ε=10.0 still provides strong privacy
Model A provides the strongest privacy guarantee and should be chosen unless the accuracy loss is clinically unacceptable
Model B balances privacy and utility, provides moderate privacy protection (ε=1.0 is common in practice), and maintains reasonable accuracy
Differential privacy should not be used for medical models because even small accuracy losses could harm patients

Correct Answer: c) Model B balances privacy and utility, provides moderate privacy protection (ε=1.0 is common in practice), and maintains reasonable accuracy

This question requires understanding differential privacy fundamentals, particularly the meaning of epsilon (ε) and how to make practical trade-offs between privacy and utility. The chapter explains that ε quantifies privacy loss: smaller ε means stronger privacy but typically lower utility.

The chapter provides guidance on interpreting ε values: - ε < 1: Strong privacy - ε = 1-10: Moderate privacy (common in practice) - ε > 10: Weak privacy

Model A (ε=0.1) provides very strong formal privacy guarantees, but the 13 percentage point accuracy loss (72% vs. 85%) is substantial. Whether this is acceptable depends entirely on the clinical context and alternative options, but such significant accuracy degradation often limits practical deployment. Model C (ε=10.0) maintains near-baseline accuracy (84% vs. 85%) but provides quite weak privacy protection. While ε=10.0 is technically “differentially private,” it’s at the upper end of what’s considered meaningful privacy protection. The distinguishability between outputs with/without any individual’s data is relatively high. Model B (ε=1.0) represents the practical middle ground: moderate privacy (within the commonly used range), reasonable accuracy (81%), and only 4 percentage points below baseline. This is often acceptable in practice, especially considering the model provides formal privacy guarantees unavailable with standard training.

Option (a) mischaracterizes ε=10.0 as “strong privacy”—it’s actually weak privacy by the standards discussed in the chapter. While close to baseline accuracy, this comes at significant privacy cost. Option (b) is too absolutist. While strong privacy is desirable, the chapter emphasizes context-dependent decision-making. A model with 72% accuracy might be unacceptable if alternatives exist (like the 81% model), or it might be fine if the application isn’t critical or if no better privacy-preserving alternative is available. The blanket recommendation to “choose unless clinically unacceptable” doesn’t account for the availability of Model B, which offers better accuracy with still-reasonable privacy. Option (d) commits the opposite error—rejecting DP entirely because of accuracy loss. The chapter emphasizes that privacy and utility can coexist (it’s not zero-sum), and formal privacy guarantees are often worth moderate utility loss. The 4-point accuracy reduction for Model B may be entirely acceptable depending on the use case.

The chapter discusses DP-SGD implementation showing exactly this trade-off: privacy budget selection is a critical hyperparameter requiring careful tuning. Factors to consider include: - Sensitivity of data: More sensitive data justifies stronger privacy (lower ε) - Dataset size: Larger datasets allow better utility with same privacy - Use case: Exploratory research might tolerate higher ε than clinical deployment - Alternative options: If non-private models are available, higher standards apply to DP alternatives - Threat model: Who might attack? What’s their motivation and capability?

The chapter provides practical guidance: start with ε around 1.0, measure utility, and adjust based on specific requirements. Organizations like Apple and Google use ε in the range of 1-10 for real-world deployments, suggesting this range represents acceptable privacy-utility trade-offs for many applications.

An additional consideration: DP provides composition guarantees. If multiple queries or models are trained on the same data, privacy budgets add up. Strategic allocation is essential—save budget for most critical analyses.

For public health practitioners: differential privacy is increasingly required by privacy regulations and data use agreements. Understanding this trade-off is essential for practical deployment. Work with privacy experts and domain experts jointly to select appropriate ε values. Document the rationale for privacy budget choices in governance documentation and be prepared to justify decisions to stakeholders who may not understand the technical details but need assurance that privacy and utility are appropriately balanced.

Question 4

During a data breach, attackers gain access to a trained machine learning model used for hospital sepsis prediction. The attackers don’t have access to the training data itself. Which attack vector should the hospital be MOST immediately concerned about?

Data poisoning: attackers could corrupt the model by injecting malicious training examples
Model inversion: attackers could potentially reconstruct sensitive patient information used to train the model
Adversarial examples: attackers could craft inputs to cause misdiagnosis of specific patients
Model stealing: attackers could replicate the model through repeated queries and use it without authorization

Correct Answer: b) Model inversion: attackers could potentially reconstruct sensitive patient information used to train the model

This question tests understanding of AI security threats in the context of a specific breach scenario. The key detail is that attackers have gained access to the trained model but not the training data itself. This eliminates and prioritizes certain attack vectors.

Model inversion attacks attempt to reconstruct training data from model parameters or predictions. The chapter details the Fredrikson et al. (2015) study where researchers reconstructed faces from facial recognition models—revealing sensitive training data. For a medical model, this could mean reconstructing patient records, including diagnoses, lab values, demographic information, and other protected health information.

The mechanism works by optimizing inputs to maximize confidence for specific outputs. If the model memorized aspects of training data (common with overfitting), this optimization can recover patient-specific information. The chapter notes this is particularly concerning for medical data because: (1) Health information is highly sensitive, (2) Reconstruction could identify specific individuals if combined with auxiliary information, (3) The breach could violate HIPAA and other privacy regulations, (4) Public trust in hospital AI systems would be severely damaged.

Option (a)—data poisoning—occurs during training, not after deployment. Since attackers don’t have access to the training pipeline, they cannot inject malicious training examples. This attack is most relevant when data collection is ongoing (crowdsourced labels, user-contributed data). Option (c)—adversarial examples—is a valid concern but not the most immediate one for a privacy/data breach scenario. Adversarial attacks cause operational problems (misdiagnosis) rather than privacy violations. While serious, the question focuses on what happens when a model is “accessed” in a breach, suggesting information disclosure is the primary concern. Moreover, adversarial attacks typically require query access for crafting adversarial inputs, not just model access. Option (d)—model stealing—means replicating model functionality through queries. While this has intellectual property implications and could enable competitive misuse, it doesn’t directly compromise patient privacy or training data. It’s a business concern more than a privacy emergency.

The chapter discusses defenses against model inversion: - Differential privacy during training: Prevents the model from memorizing individual training examples - Output perturbation: Add noise to predictions to make reconstruction harder - Limit model access: Provide only class labels, not confidence scores - Regularization: Prevent overfitting that enables reconstruction

The scenario has broader implications for AI governance. The breach reveals several security failures: 1. Access control failure: How did attackers gain model access? 2. Insufficient defense-in-depth: Models should have multiple protection layers 3. Monitoring gaps: Was the breach detected promptly? 4. Incident response: Is there a plan to mitigate privacy harm?

Post-breach actions should include: - Immediate: Revoke attacker access, assess extent of exposure - Short-term: Notify affected patients if reconstruction risk is high, report to HHS if HIPAA applies - Long-term: Implement stronger access controls, retrain model with DP if possible, audit security architecture

The chapter emphasizes that model deployment creates new attack surfaces. Traditional security focuses on protecting data at rest and in transit, but ML introduces risks through the model itself. Organizations must: - Treat trained models as sensitive assets - Implement model-specific security controls (API authentication, rate limiting, query monitoring) - Consider privacy implications during training (DP can prevent these attacks) - Plan for breach scenarios specific to AI systems

For public health practitioners: this case study illustrates why AI security goes beyond traditional IT security. Privacy-preserving ML techniques like differential privacy aren’t just theoretical niceties—they’re practical defenses against real attacks that could occur during breaches. Security architecture for AI systems must consider the model as a potential privacy liability, not just a functional asset.

Question 5

A health department conducts a privacy impact assessment for an AI-based contact tracing system. The assessment identifies high re-identification risk but concludes deployment is justified because of the pandemic emergency and potential lives saved. A community advisory board objects, citing lack of transparency, unclear data retention policies, and absence of independent oversight. According to the chapter’s governance principles, what is the MOST appropriate next step?

Proceed with deployment since the privacy impact assessment concluded benefits outweigh risks
Pause deployment, address the advisory board’s concerns through revised governance structures, and demonstrate meaningful community engagement before proceeding
Deploy the system but add a warning label about privacy risks and provide an opt-out mechanism
Replace the community advisory board with members more supportive of public health priorities

Correct Answer: b) Pause deployment, address the advisory board’s concerns through revised governance structures, and demonstrate meaningful community engagement before proceeding

This question synthesizes multiple themes from the chapter: privacy impact assessments, governance, stakeholder engagement, and the critical importance of public trust in public health AI systems. The scenario deliberately creates tension between technical assessment (PIA concludes benefits outweigh risks) and community concerns (lack of transparency, unclear policies, no oversight).

The chapter emphasizes throughout that technical compliance and risk-benefit analysis, while necessary, are not sufficient for responsible AI deployment. The UK NHS care.data case study provides a cautionary tale: technically compliant data sharing with promised de-identification was cancelled after public outcry due to lack of transparency, inadequate communication, and perceived violation of trust. The program wasted £27 million and caused lasting trust damage.

Key principles from the chapter that apply here:

Governance requires meaningful stakeholder engagement: The chapter specifies that privacy impact assessments should include consultation with patients/community members, privacy officers, legal counsel, ethics committees, and others. Community advisory board objections represent exactly this stakeholder input process functioning as designed.

Transparency builds trust: The advisory board’s concerns about transparency and unclear retention policies address fundamental requirements. The chapter discusses how COVID-19 contact tracing apps succeeded or failed largely based on transparency about data practices. Germany’s Corona-Warn-App gained adoption through open-source code, published privacy analysis, and clear communication, while France’s centralized approach faced resistance.

Social license, not just legal compliance: The care.data lesson was that legal compliance alone is insufficient—you need social license from the community you serve. Proceeding over community objections risks losing trust that’s essential for public health function generally, not just this specific system.

Privacy by Design includes governance: Privacy-preserving technical measures are necessary but not sufficient. Governance structures (clear policies, independent oversight, accountability mechanisms) are equally essential components of Privacy by Design.

Option (a) treats the PIA as a final decision-making authority, ignoring the governance principle that technical assessment should inform, not replace, broader stakeholder deliberation. Option (c)—adding warnings and opt-outs—addresses symptoms but not root causes. The concerns raised aren’t about individual choice but about system design: transparency, governance, and oversight. Opt-out mechanisms may be appropriate, but they don’t substitute for proper governance. Option (d) is fundamentally inconsistent with responsible governance. If community feedback is inconvenient, the answer isn’t to silence it—that would confirm concerns about lack of accountability and transparency.

The appropriate response follows the chapter’s stakeholder consultation guidance: 1. Acknowledge concerns: Take advisory board feedback seriously 2. Enhance transparency: Publish data practices, algorithms (if possible), and use policies 3. Establish oversight: Create independent review mechanism with community representation 4. Clarify policies: Document data retention, deletion procedures, and sunset provisions 5. Demonstrate accountability: Show how feedback influenced system design 6. Communicate changes: Explain modifications made in response to community input

The pandemic context matters but doesn’t override these principles. The chapter discusses COVID-19 contact tracing, noting that privacy-preserving technical approaches (decentralized architecture, differential privacy for aggregates, local data storage) can maintain public health efficacy while respecting privacy. The emergency doesn’t force a choice between health and privacy—it demands we do both well.

This scenario also connects to the chapter’s discussion of function creep and data retention. Clear sunset provisions were crucial to COVID-19 Data Store acceptance—the system had defined end dates and purposes. Without these safeguards, community concerns about mission creep are reasonable.

For public health practitioners: trust is infrastructure. Proceeding with technically sound but socially contentious AI systems risks damaging relationships essential for all public health functions: disease reporting, outbreak investigation, vaccination campaigns, and more. Taking the time to address governance concerns isn’t just ethically appropriate—it’s strategically necessary for maintaining the social license public health requires.

The better approach: use the delay to strengthen both technical privacy protections AND governance structures, demonstrate that community input meaningfully shapes policy, and build systems that serve communities rather than being imposed on them. This aligns with the chapter’s emphasis that privacy is not just a technical problem with technical solutions—it’s a socio-technical challenge requiring governance, transparency, and community engagement alongside cryptographic techniques and statistical methods.

Question 6

A research institution considers three approaches for sharing a sensitive HIV surveillance dataset with external researchers: (A) Public release with HIPAA Safe Harbor de-identification, (B) Controlled access data enclave where researchers analyze data remotely without downloading, or (C) Synthetic data generation using differentially private GANs. Budget constraints allow only one approach. Which factors should MOST heavily influence the decision?

Cost-effectiveness: choose whichever is cheapest to implement
Ease of use: choose public release because it maximizes researcher convenience
Risk assessment and data sensitivity: evaluate re-identification risk, stigma potential, and regulatory requirements to determine appropriate protection level
Publication impact: choose whichever enables the most high-profile research publications

Correct Answer: c) Risk assessment and data sensitivity: evaluate re-identification risk, stigma potential, and regulatory requirements to determine appropriate protection level

This question synthesizes the chapter’s content on de-identification techniques, data governance, and context-specific risk assessment. HIV surveillance data represents a high-stakes scenario due to significant stigma, discrimination risks, and historical context of medical data misuse affecting vulnerable communities.

The chapter emphasizes that data protection decisions must be risk-based and context-specific. Factors to consider include:

Data sensitivity: HIV status is highly sensitive personal information. Disclosure risks include employment discrimination, insurance denial, relationship impacts, and social stigma. The chapter discusses how health data sensitivity varies—HIV surveillance data is toward the highest end. Historical context matters: communities affected by HIV/AIDS have experienced discrimination, and trust in public health institutions was damaged by incidents like the Tuskegee study. This history makes privacy protection not just a legal requirement but an ethical imperative for maintaining community trust.

Re-identification risk: The chapter’s discussion of Latanya Sweeney’s work and other re-identification cases shows that quasi-identifiers in HIV data (demographics, location, timing of diagnosis, risk factors) could enable linkage attacks. HIV surveillance often includes small populations in some geographic areas, making de-identified individuals more identifiable. Public auxiliary data (obituaries, court records, news articles) might enable re-identification.

Regulatory requirements: HIPAA applies to HIV surveillance data. Beyond HIPAA, many states have specific HIV confidentiality laws with stronger protections than general health privacy regulations. Violating these could result in legal liability and loss of public health authority.

Let’s evaluate each approach through this framework:

Option A (Public release with Safe Harbor): As discussed in Question 1, Safe Harbor alone is often insufficient, especially for sensitive data with high re-identification motivation. For HIV data, this approach presents unacceptable risks: attackers have strong motivation (blackmail, discrimination), small populations in some areas increase uniqueness, and stigma magnifies potential harm. The chapter repeatedly emphasizes Safe Harbor as a starting point, not a sufficient end state.

Option B (Data enclave): This approach provides strong protection through access controls, monitoring, and restricted data movement. Researchers analyze data remotely through secure systems, results are vetted for privacy risks before release. The chapter discusses data enclaves as appropriate for highly sensitive data. Disadvantages include infrastructure costs and researcher friction (less convenient than direct data access), but these are acceptable trade-offs for high-sensitivity data. Benefits include granular access controls, audit logging, prevention of unauthorized downloads, and ability to update protection mechanisms without redistributing data.

Option C (Synthetic data with DP-GANs): Potentially provides formal privacy guarantees while enabling broader sharing. The chapter discusses synthetic data as preserving statistical properties while eliminating individual-level privacy risks. Challenges include validation requirements (how well does synthetic data capture real patterns?), potential loss of rare but important patterns, computational resources for generation, and expertise requirements. For HIV surveillance, rare subpopulations and unusual transmission patterns might be lost in synthetic data generation.

Now addressing the incorrect options:

Option (a)—cost-effectiveness—is relevant but shouldn’t be the primary driver for highly sensitive data. Inadequate protection that saves money but causes privacy breaches, stigma, and discrimination represents false economy. The chapter’s discussion of health data breaches notes average costs of $9.23 million plus immeasurable trust damage. Option (b)—researcher convenience—misunderstands the priority hierarchy. While data utility matters, it cannot override privacy protection for highly sensitive data. The chapter explicitly discusses the privacy-utility trade-off and notes that for sensitive data, privacy must be prioritized even at utility cost. Option (d)—publication impact—represents misaligned incentives. Research impact is important but secondary to protecting vulnerable populations. This option also assumes easier data access leads to better research, which isn’t necessarily true—more careful analysis in protected environments can produce higher-quality research.

The chapter’s governance framework provides guidance for this decision: 1. Data classification: HIV surveillance is “restricted” level—highest sensitivity 2. Privacy impact assessment: Identifies significant risks given stigma and discrimination potential 3. Stakeholder consultation: Community affected by HIV/AIDS should have input 4. Proportional protection: Match protection to sensitivity and risk

Given these factors, Option B (data enclave) likely represents the best balance: strong protection appropriate to sensitivity, enables detailed research, maintains control, and provides audit trails. Option C (synthetic data) might be appropriate if validation demonstrates sufficient utility preservation and proper DP implementation provides strong guarantees.

For public health practitioners: the decision framework for data sharing should prioritize: 1. Assess data sensitivity and re-identification risk 2. Consider historical context and community trust 3. Evaluate regulatory requirements and institutional policies 4. Match protection level to risk (higher sensitivity = stronger protection) 5. Engage affected communities in decision-making 6. Document rationale for choices 7. Monitor and audit access

Cost, convenience, and research productivity are legitimate considerations but must be balanced against core obligations to protect privacy and maintain trust, especially for vulnerable populations. The chapter’s emphasis on Privacy by Design means making architectural choices (like data enclaves) that enable both privacy and utility, rather than treating them as competing goals.

12.13 Discussion Questions

Should there be exceptions to privacy protections for public health emergencies? If so, what safeguards are needed?
Is perfect anonymization achievable, or should we focus on “good enough” privacy with differential privacy?
Who should own health data generated by AI systems - individuals, providers, payers, or AI developers?
How should we balance the right to privacy with the right to benefit from AI systems trained on population data?
Should privacy laws allow deceased individuals’ data to be used more freely for research? What about family members’ privacy?
How can we ensure privacy-preserving techniques don’t create barriers to equity (e.g., differential privacy reducing accuracy for minorities)?
What level of re-identification risk is acceptable for different use cases (research vs. clinical care vs. public health surveillance)?

12.14 Further Resources

12.14.1 📚 Essential Books

The Ethical Algorithm by Kearns & Roth - Accessible intro to algorithmic fairness and privacy
Privacy is Power by Carissa Véliz - Why privacy matters
Practical Data Privacy by Katharine Jarmul - Technical implementation guide

12.14.2 📄 Key Papers

Foundational: - Dwork, 2006 - Differential Privacy - Original DP paper - Sweeney, 2002 - k-Anonymity - Re-identification risks 🎯 - Narayanan & Shmatikov, 2008 - Robust De-anonymization - Netflix re-identification 🎯

Privacy-Preserving ML: - Abadi et al., 2016 - Deep Learning with Differential Privacy - DP-SGD - McMahan et al., 2017 - Federated Learning - Google’s federated learning - Gentry, 2009 - Fully Homomorphic Encryption - FHE breakthrough

Attacks: - Shokri et al., 2017 - Membership Inference 🎯 - Fredrikson et al., 2015 - Model Inversion 🎯 - Carlini et al., 2019 - The Secret Sharer - Training data extraction

Policy and Governance: - Cavoukian, 2009 - Privacy by Design - Nissenbaum, 2004 - Privacy as Contextual Integrity

12.14.3 🎓 Courses and Tutorials

Applied Cryptography - Stanford (Dan Boneh)
The Algorithmic Foundations of Differential Privacy - Free textbook
Programming Differential Privacy - Practical implementations
Privacy in Statistics and Machine Learning - Course materials

12.14.4 💻 Tools and Libraries

Differential Privacy: - Google DP Library - OpenDP - Open source DP platform - Tumult Analytics - DP data analytics - SmartNoise - Microsoft DP toolkit

Federated Learning: - TensorFlow Federated - PySyft - Privacy-preserving ML - Flower - Federated learning framework - FATE - Federated AI Technology Enabler

Homomorphic Encryption: - Microsoft SEAL - HElib - IBM HE library - PySEAL - Python wrapper

De-identification: - ARX Data Anonymization Tool - sdcMicro - R package - Amnesia - Data anonymization

Synthetic Data: - Synthetic Data Vault (SDV) - Synthpop - R package - CTGAN - GAN-based synthesis

12.14.5 🎯 Guidelines and Standards

Regulatory: - HIPAA Privacy Rule - GDPR Full Text - CCPA/CPRA Information

Best Practices: - NIST Privacy Framework - ISO/IEC 27001 - Information security - OWASP Top 10 - Web application security - CIS Controls - Cybersecurity controls

Health-Specific: - HITRUST CSF - Healthcare security framework - FDA Guidance on Cybersecurity

Next: Chapter 12: Deployment, Monitoring, and Maintenance →

Learning Objectives

12.1 Introduction: The Privacy Paradox in Public Health

12.1.1 The Cambridge Analytica Wake-Up Call

12.1.2 The Public Health Dilemma

12.1.3 High-Profile Health Data Breaches

12.1.4 What Makes Health Data Special?

12.2 Privacy Principles and Frameworks

12.2.1 Fair Information Practice Principles (FIPPs)

12.2.2 Privacy by Design

12.2.3 The Privacy-Utility Trade-off

12.3 Data Protection Regulations

12.3.1 HIPAA (Health Insurance Portability and Accountability Act)

12.3.2 GDPR (General Data Protection Regulation)

12.3.3 CCPA/CPRA (California Consumer Privacy Act / California Privacy Rights Act)

12.3.4 Other Key Regulations

12.4 De-identification and Anonymization

12.4.1 Terminology

12.4.2 De-identification Techniques

12.4.2.1 1. Suppression

12.4.2.2 2. Generalization

12.4.2.3 3. Perturbation

12.4.2.4 4. Aggregation

12.4.3 K-Anonymity

12.4.4 L-Diversity

12.4.5 T-Closeness

12.4.6 The Re-identification Challenge

12.5 Privacy-Preserving Machine Learning

12.5.1 Differential Privacy

12.5.1.1 Implementing Differential Privacy

12.5.1.2 Differentially Private Machine Learning

12.5.1.3 Practical Considerations

12.5.2 Federated Learning

12.5.3 Homomorphic Encryption

12.5.4 Secure Multi-Party Computation (MPC)

12.5.5 Synthetic Data

12.6 Security for AI Systems

12.6.1 Threat Landscape

12.6.2 Adversarial Examples

12.6.3 Model Inversion Attacks

12.6.4 Membership Inference Attacks

12.6.5 Data Poisoning

12.6.6 Security Best Practices

12.7 Privacy Impact Assessment

12.7.1 Framework Overview

12.7.2 Step 1: Data Inventory

12.7.3 Step 2: Legal and Regulatory Analysis

12.7.4 Step 3: Risk Assessment

12.7.5 Step 4: Privacy-Preserving Measures

12.7.6 Step 5: Stakeholder Consultation

12.7.7 Step 6: Ongoing Monitoring

12.8 Data Governance

12.8.1 Governance Framework

12.8.2 Data Governance Committee

12.8.3 Data Classification

12.8.4 Data Access Request Process

12.8.5 Data Lifecycle Management

12.8.6 Data Quality Governance

12.9 Case Studies

12.9.1 Case Study 1: COVID-19 Contact Tracing Apps

12.9.2 Case Study 2: UK NHS Data Sharing Controversy

12.9.3 Case Study 3: Strava Fitness App Heat Map

12.10 Practical Implementation Checklist

12.10.1 Privacy Protection Checklist

12.10.2 Security Checklist

12.11 Key Takeaways

12.12 Hands-On Exercise: Build a Privacy-Preserving Pipeline

Check Your Understanding

12.13 Discussion Questions

12.14 Further Resources

12.14.1 📚 Essential Books

12.14.2 📄 Key Papers

12.14.3 🎓 Courses and Tutorials

12.14.4 💻 Tools and Libraries

12.14.5 🎯 Guidelines and Standards