The Data Problem in Public Health AI

Public health data is incomplete, delayed, biased, and inconsistent by nature. Surveillance captures only the tip of the iceberg, reporting lags create temporal misalignment, and testing availability determines who appears in your dataset. Learning to assess data quality across four dimensions (completeness, accuracy, timeliness, representativeness) and engineer features with epidemiological insight matters more than algorithm sophistication.

Learning Objectives

This chapter addresses messy public health data realities. You will learn to:

• Assess data quality dimensions (completeness, accuracy, timeliness, representativeness)
• Distinguish missing data mechanisms (MCAR, MAR, MNAR)
• Recognize how selection bias invalidates predictions (COVID-19 testing)
• Identify when data quality makes AI inappropriate
• Implement practical quality assessments and consistency checks
• Develop courage to recommend against deployment when appropriate

Prerequisites: AI in Context, Machine Learning Fundamentals.

Chapter Summary (TL;DR)

The Big Picture: Most AI tutorials use clean, perfect datasets. Real public health data is incomplete, delayed, biased, inconsistent, messy, and heterogeneous. Learning to handle messy data is more important than learning fancy algorithms. Garbage in = garbage out, the most sophisticated deep learning model cannot overcome fundamentally flawed data.

Why Public Health Data Is Uniquely Challenging:

• The Surveillance Pyramid Problem: You see lab-confirmed cases (tip of iceberg) but want to predict all infections (entire iceberg). Training on hospitalized patients, deploying on community cases = selection bias by design
• Reporting Delays: “Today’s” data is actually a mixture of cases from 2-10 days ago with unknown proportions → right-censoring systematically underestimates recent trends
• Changing Definitions: COVID-19 case definitions changed multiple times. Testing availability evolved. Models learn spurious patterns from surveillance artifacts, not real epidemiology
• The Denominator Problem: 1,000 cases with 50 deaths (5% CFR) vs. 500 cases with 100 deaths (20% CFR), which is truly more dangerous, or which just has less testing?
• Privacy Constraints: Aggregation, suppression (<5 cases), coarsening (age brackets), delayed release → loss of granularity reduces predictive power

The Four Dimensions of Data Quality (Assess Before Building):

1. Completeness: How much is missing, why is it missing, and how does missingness affect the estimand? The percentage alone does not determine usability or method.

• MCAR (Missing Completely At Random): Complete-case analysis may be unbiased under additional assumptions, but simple imputation can still distort variance and uncertainty
• MAR (Missing At Random): Depends on observed variables; model-based imputation, weighting, or sensitivity analysis may be appropriate
• MNAR (Missing Not At Random): Missingness itself is informative (sickest patients too ill to complete surveys)

2. Accuracy: Are values correct? Check for impossible values (age >120), implausible values (BMI <10), unit mixing (temperature in Fahrenheit vs. Celsius), data entry errors (decimal points), age heaping

3. Timeliness: Reporting lags create temporal misalignment. Nowcasting methods adjust for delays

4. Representativeness: Does your sample match the target population? Compare demographics to census data. Critical for population-level inferences, less critical for outbreak detection

COVID-19 Forecasting Lessons:

Despite unprecedented data, forecast error increased with horizon and models varied widely. The U.S. Forecast Hub ensemble outperformed individual models on average, showing both the value and limits of structured forecasting (Cramer et al., 2022). Recurring data problems included: - Ignored reporting delays (treated “reported March 15” as “occurred March 15”) - Confused testing expansion with disease growth - Selection bias (trained on strict lockdown data, applied to different compliance) - Missing confounders (behavioral responses, policy changes) - Result: Even leading models were barely better than a plain autoregressive baseline

The Critical Lesson: Forecast accuracy was limited more by data quality issues than modeling approaches. Simple models with good data outperformed complex models with poor data.

Feature Engineering (Where Domain Expertise Wins):

• Time-based: Day of week, season, days since outbreak start, rolling averages (7-day, 14-day)
• Epidemiological: R0 estimation, attack rate, case fatality rate, test positivity
• Geographic: Distance to healthcare, spatial clustering, cases within 5km
• Lag features: Cases 7 days ago, 14 days ago (but never use future information!)

Critical Pitfall, Temporal Data Leakage:

WRONG: df['cases_rolling'] = df['daily_cases'].rolling(window=7, center=True).mean()
CORRECT: df['cases_rolling'] = df['daily_cases'].rolling(window=7).mean()

Center=True uses future data. For time series, always split by date, never random shuffle.

When Data Problems Make AI Inappropriate (Know When to Stop):

Red flags that should halt the project: 1. Extreme selection bias with no adjustment method 2. Outcome labels are inconsistent or unreliable 3. Critical features have missingness that cannot be addressed for the intended estimand 4. Data-generating process changed mid-dataset (pre/post policy change) 5. Sample size too small for model complexity (100 cases, 50 features, deep learning = disaster) 6. External validity concerns (one hospital’s data, deploying elsewhere)

The Courage to Say “No”: Good data science includes knowing when NOT to build a model. If data quality is insufficient, recommend improving data collection instead. Building on garbage data wastes resources, produces misleading results, can harm people, and damages trust in AI.

Data Quality Audit (Tactical Guide):

Beyond conceptual understanding, practitioners need concrete tools:

• Data drift detection: Compare recent data with the training period, then determine whether observed drift changes decision-relevant performance. Distributional change alone does not establish a need to retrain.
• Facility artifact detection: Systematic differences across reporting sources often indicate coding practices, not clinical variation.
• Imputation vs. exclusion decision: Choose complete-case analysis, imputation, weighting, or sensitivity analysis from the missingness mechanism, estimand, sample size, auxiliary information, and consequences of error (Lee et al., 2021).
• Delay-aware anomaly detection: Standard methods fail on surveillance data. Adjust for reporting delays before flagging anomalies.

See Data Quality Audit: Tactical Guide for Python code implementing these checks.

The Takeaway for Public Health Practitioners: Data quality determines model quality. Investigate outliers before removing them, they might be the most important cases (COVID-19 cytokine storm). Document everything, what you cleaned and why. Feature engineering with domain expertise beats algorithm sophistication. Know when to stop, sometimes the ethical choice is not to build a model. As Box said: “All models are wrong, but some are useful.” If your data is wrong enough, your model will be useless.

Introduction

Here’s the uncomfortable truth: Most AI tutorials use clean, well-formatted datasets. Most public health data is none of those things.

Real public health data is: - Incomplete (missing values everywhere) - Delayed (cases reported days or weeks late) - Biased (who gets tested determines who appears in your data) - Inconsistent (formats change, definitions evolve) - Messy (typos, duplicates, impossible values) - Heterogeneous (multiple sources with different standards)

And yet: This is the data you have to work with. Learning to handle messy data is arguably more important than learning fancy algorithms.

The Iron Law of Machine Learning

Garbage in = Garbage out

The most sophisticated deep learning model cannot overcome fundamentally flawed data. A simple logistic regression on clean, well-understood data will outperform a complex neural network on garbage data.

Data quality and algorithm design are both necessary. This chapter emphasizes data because many development failures originate in measurement, selection, linkage, and temporal alignment rather than model architecture.

Why Public Health Data Is Uniquely Challenging

1. The Surveillance Pyramid Problem

Public health surveillance captures only a fraction of reality:

     [Lab] Lab-Confirmed Cases (Data you have)
     /
    [Hosp] Hospitalized Cases
    /
  [Care] Symptomatic Cases Seeking Care
  /
 [Symp] All Symptomatic Cases
 /
[All] All Infections (Including Asymptomatic)

graph TB
 subgraph " "
 A["Lab-Confirmed Cases<br/>(What surveillance systems capture)"]
 B["Hospitalized Cases<br/>(Severe cases only)"]
 C["Symptomatic Cases Seeking Care<br/>(Access to healthcare varies)"]
 D["All Symptomatic Cases<br/>(Many don't seek care)"]
 E["All Infections<br/>(Including asymptomatic - true burden)"]
 end

 E --> D
 D --> C
 C --> B
 B --> A

 style A fill:#ff6b6b,stroke:#333,stroke-width:2px,color:#fff
 style B fill:#ffa07a,stroke:#333,stroke-width:2px
 style C fill:#ffd166,stroke:#333,stroke-width:2px
 style D fill:#bfe1b0,stroke:#333,stroke-width:2px
 style E fill:#95e1d3,stroke:#333,stroke-width:3px

Figure 6.1: The surveillance pyramid showing how public health data captures only a fraction of true disease burden. Each level represents increasing underascertainment, with lab-confirmed cases (what gets reported) representing only the visible tip of the iceberg. Most infections, especially asymptomatic and mild cases, remain undetected.

The issue: Your dataset represents the tip of the pyramid, but the population of interest is the entire pyramid. This is selection bias by design.

Example: COVID-19 Testing Bias

Early in the pandemic, testing was limited to: - Symptomatic individuals - Healthcare workers - Travelers - Close contacts of confirmed cases

Any model trained on this data would: - Overestimate symptom severity (asymptomatic cases invisible) - Misunderstand demographic risk factors (testing access varied by socioeconomic status) - Produce biased risk predictions (dataset not representative of population)

As documented in Lipsitch et al.’s analysis of surveillance biases, these selection effects can lead to systematic underestimation of disease burden.

When Selection Bias Invalidates AI

If training data comes from one level of the surveillance pyramid but predictions target another, model complexity alone cannot solve the resulting selection problem.

You need either: 1. Data from the correct population (often impossible) 2. Statistical methods to adjust for selection (Hernán’s target trial framework) 3. A different research question that matches your available data

2. Reporting Delays and Temporal Misalignment

Illustrative reporting-delay scenario: You want to predict tomorrow’s case counts based on today’s data. A reporting extract can include observations from several earlier dates in proportions that vary by jurisdiction and reporting system: - Cases from today (20%) - Cases from yesterday, reported today (40%) - Cases from 2 days ago, reported today (30%) - Cases from 3+ days ago, reported today (10%)

The extract labeled “September 15” may therefore mix cases from several occurrence dates, with proportions that must be measured rather than assumed.

Why this matters for ML: - Features and labels are temporally misaligned - Recent trends are systematically underestimated (right-censoring) - Models learn spurious patterns from reporting artifacts

Real example: Early COVID-19 forecasting models struggled because they treated reported case counts as actual case counts, ignoring 5-10 day reporting lags that varied by jurisdiction.

3. Changing Definitions and Data Collection

Public health data collection evolves over time, as extensively documented in Reich et al.’s analysis of forecasting challenges.

Case definition changes: - COVID-19 case definitions changed multiple times in 2020-2021 - Adding antigen tests to confirmed cases - Including probable cases vs. confirmed only

Testing availability changes: - Week 1: Only sick hospitalized patients tested - Week 10: Symptomatic individuals can get tested - Week 20: Asymptomatic screening becomes common

Your dataset appears to show: - Exponential case growth - Changing age distribution - Different symptom profiles over time

But reality: These might be artifacts of changing surveillance, not actual epidemiological changes.

The Fundamental Challenge

Machine learning assumes the relationship between features and outcomes is stationary (constant over time).

Public health surveillance violates this assumption constantly. The data-generating process itself evolves.

This is why pandemic forecasting is so difficult, the ground truth keeps shifting under your feet.

4. The Denominator Problem

You observe:

Region A: 1,000 cases, 50 deaths → 5% CFR
Region B: 500 cases, 100 deaths → 20% CFR

Is Region B truly more dangerous? Or does it have: - Less testing (missing mild cases in denominator) - Older population - Worse healthcare access - Different case definitions

The problem: Your dataset has numerators (cases, deaths) but often lacks good denominators (population at risk, testing rates, exposure levels).

AI models trained on case counts without accounting for testing intensity will confuse “more testing” with “more disease,” as Ioannidis cautioned.

5. Privacy Constraints and Data Aggregation

To protect patient privacy, public health data is often: - Aggregated (county-level instead of individual-level) - Suppressed (cells with <5 cases shown as “<5” or asterisk) - Coarsened (age shown as brackets: “20-29” instead of exact age) - Delayed (real-time data withheld, released weeks later)

Impact on ML: - Loss of granularity reduces predictive power - Non-standard missing data patterns - Cannot link across datasets (no unique identifiers) - Temporal resolution insufficient for some analyses

The HL7 FHIR standard and HIPAA Safe Harbor guidelines constrain what data you can access and how it’s formatted.

Current CDC Data Modernization Milestones

CDC’s 2026 Public Health Data Strategy milestones make the data problem operational. The milestones include receiving syndromic data from at least 86% of U.S. emergency departments, automated bed-capacity feeds from at least 17 state and territorial jurisdictions, and production electronic case reporting processing for at least 70% of ELC-funded health departments connected to AIMS (CDC, April 2026). These are AI prerequisites, not administrative background: models built on delayed, manually reconciled, or inconsistently structured feeds inherit those defects.

The same milestone set places AI readiness inside data modernization by calling for CDC GenAI guidance and informational resources for state, tribal, local, and territorial (STLT) health department partners. For data teams, the practical message is direct: do not treat model selection as the starting point. Data flow automation, shared standards, and governance capacity determine whether AI can be evaluated at all.

The Four Dimensions of Data Quality

When assessing whether your data is suitable for AI, evaluate these four dimensions, as outlined in Weiskopf and Weng’s data quality framework.

1. Completeness: Are Values Missing?

The question: What percentage of values are missing? Is the missingness random or systematic?

Code to assess:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load your data
df = pd.read_csv('../data/examples/surveillance_data.csv')

# Calculate missingness
missing_summary = pd.DataFrame({
 'column': df.columns,
 'missing_count': df.isnull().sum(),
 'missing_percent': (df.isnull().sum() / len(df) * 100).round(2)
}).sort_values('missing_percent', ascending=False)

print("Missing Data Summary:")
print(missing_summary)

# Visualize missingness patterns
import missingno as msno

# Matrix plot: see where missing values cluster
msno.matrix(df, figsize=(12, 6), sparkline=False)
plt.title('Missing Data Pattern')
plt.tight_layout()
plt.savefig('../images/examples/missing_data_matrix.png', dpi=300)

Interpreting results:

Missingness percentage is a useful descriptive measure, not a universal decision threshold. Assess the mechanism, variable role, estimand, available auxiliary information, and uncertainty before choosing a method (Lee et al., 2021).

Types of missingness (Little & Rubin’s taxonomy):

1. Missing Completely At Random (MCAR): Missingness is unrelated to any variables

• Example: Lab results lost due to random system glitches
• Implication: Complete-case analysis may be unbiased under additional assumptions; single-value imputation can still understate uncertainty

2. Missing At Random (MAR): Missingness depends on observed variables

• Example: Income more likely missing for younger respondents
• Implication: Can impute using related variables

3. Missing Not At Random (MNAR): Missingness depends on the missing value itself

• Example: Sickest patients too ill to complete surveys
• Implication: Missing data itself is informative

Testing Missingness Patterns

# Check if missingness is related to other variables
df['symptom_onset_missing'] = df['symptom_onset'].isnull()

# Compare groups with and without missing values
print(df.groupby('symptom_onset_missing')['age'].mean())
print(df.groupby('symptom_onset_missing')['hospitalized'].mean())

# Statistical test
from scipy.stats import chi2_contingency
contingency_table = pd.crosstab(df['symptom_onset_missing'],
         df['hospitalized'])
chi2, p_value, dof, expected = chi2_contingency(contingency_table)

if p_value < 0.05:
 print("Missingness is NOT random - be careful with imputation")

2. Accuracy: Are Values Correct?

Common accuracy issues:

Impossible or Implausible Values

# Detect impossible values
issues = []

# Age checks
if (df['age'] < 0).any() or (df['age'] > 120).any():
 issues.append("Impossible ages detected")

# Date checks
df['symptom_date'] = pd.to_datetime(df['symptom_date'], errors='coerce')
df['test_date'] = pd.to_datetime(df['test_date'], errors='coerce')

# Symptom onset after test date? This can be valid for presymptomatic testing,
# so flag it for context-specific review rather than treating it as impossible.
invalid_dates = df['symptom_date'] > df['test_date']
if invalid_dates.any():
 issues.append(f"{invalid_dates.sum()} cases with symptom onset after test require review")

# Temperature checks
if 'temperature' in df.columns:
 if (df['temperature'] < 35).any() or (df['temperature'] > 42).any():
  issues.append("Implausible temperatures detected")

for issue in issues:
 print(issue)

Data Entry Errors

# Common data entry errors

# 1. Height/weight transpositions
df['bmi'] = df['weight'] / (df['height']/100)**2
suspicious_bmi = (df['bmi'] < 10) | (df['bmi'] > 60)
print(f"Suspicious BMI values: {suspicious_bmi.sum()}")

# 2. Unit inconsistencies
if 'temperature' in df.columns:
 potential_fahrenheit = (df['temperature'] > 50).sum()
 if potential_fahrenheit > 0:
  print(f"Possible unit mixing: {potential_fahrenheit} values >50°")

# 3. Decimal point errors (370 instead of 37.0)
if 'temperature' in df.columns:
 extreme_values = df['temperature'] > 100
 print(f"Possible decimal errors: {extreme_values.sum()}")

# 4. Age heaping (digit preference)
age_last_digit = df['age'] % 10
digit_counts = age_last_digit.value_counts().sort_index()

from scipy.stats import chisquare
expected_freq = [len(df)/10] * 10
chi2, p_value = chisquare(digit_counts, expected_freq)
if p_value < 0.05:
 print("Age heaping detected - rounding to 0s and 5s")

The “Too Clean” Red Flag

If your public health dataset has zero missing values, perfect consistency, and no outliers, be suspicious. Real-world data is messy.

3. Timeliness: Are Values Up-to-Date?

Quantifying reporting delays:

# Calculate reporting lag
df['report_date'] = pd.to_datetime(df['report_date'])
df['symptom_onset'] = pd.to_datetime(df['symptom_onset'])

# Lag from symptom onset to report
df['onset_to_report_days'] = (df['report_date'] - df['symptom_onset']).dt.days

# Visualize
fig, ax = plt.subplots(figsize=(10, 6))
ax.hist(df['onset_to_report_days'].dropna(), bins=50, edgecolor='black')
ax.set_xlabel('Days from Symptom Onset to Report')
ax.set_ylabel('Frequency')
ax.axvline(df['onset_to_report_days'].median(), color='red',
   linestyle='--',
   label=f"Median: {df['onset_to_report_days'].median():.1f} days")
ax.legend()
plt.savefig('../images/examples/reporting_delays.png', dpi=300)

Nowcasting: Adjusting for reporting delays as described in McGough et al.’s work.

Advanced Nowcasting

For rigorous handling of reporting delays: - EpiNow2 R package - CDC’s COVID-19 nowcasting approach

4. Representativeness: Does Data Match Target Population?

Common biases:

# Check for demographic bias

# Compare to census data
population_age_dist = {
 '0-17': 0.22,
 '18-64': 0.62,
 '65+': 0.16
}

sample_age_dist = df['age_group'].value_counts(normalize=True).to_dict()

print("Age Distribution Comparison:")
print("Age Group | Population | Sample | Difference")
print("-" * 50)
for age_group in population_age_dist:
 pop_pct = population_age_dist[age_group] * 100
 sample_pct = sample_age_dist.get(age_group, 0) * 100
 diff = sample_pct - pop_pct
 print(f"{age_group:8} | {pop_pct:6.1f}% | {sample_pct:6.1f}% | {diff:+6.1f}%")

When Representativeness Matters Most

Representativeness is critical when: - Making population-level inferences - Forecasting total burden - Evaluating interventions

Less critical when: - Early outbreak detection (any signal helps) - Comparing relative risks within sample

Common Data Issues and How to Handle Them

Issue 1: Missing Data

When MCAR:

from sklearn.impute import SimpleImputer

# Median imputation for numerical variables
numerical_cols = df.select_dtypes(include=[np.number]).columns
imputer = SimpleImputer(strategy='median')
df[numerical_cols] = imputer.fit_transform(df[numerical_cols])

When MAR:

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# Iterative imputation (like MICE)
imputer = IterativeImputer(max_iter=10, random_state=42)
df_imputed = pd.DataFrame(
 imputer.fit_transform(df[numerical_cols]),
 columns=numerical_cols
)

When MNAR:

# Create missingness indicator
df['income_missing'] = df['income'].isnull().astype(int)

# Impute with flag value
df['income_imputed'] = df['income'].fillna(-999)

Best Practices for Missing Data

1. Never silently drop missing data
2. Compare complete-case vs. imputed analysis
3. Try multiple imputation
4. Report missingness patterns
5. Create missingness indicators when uncertain

Issue 2: Outliers and Extreme Values

Not all outliers are errors!

def detect_outliers_iqr(df, column, multiplier=1.5):
 """Detect outliers using IQR method"""
 Q1 = df[column].quantile(0.25)
 Q3 = df[column].quantile(0.75)
 IQR = Q3 - Q1

 lower_bound = Q1 - multiplier * IQR
 upper_bound = Q3 + multiplier * IQR

 outliers = df[(df[column] < lower_bound) | (df[column] > upper_bound)]
 return outliers, lower_bound, upper_bound

# Investigate before removing
age_outliers, lower, upper = detect_outliers_iqr(df, 'age', multiplier=3)
print(f"Age outliers: {len(age_outliers)}")
print(age_outliers[['age', 'outcome', 'hospital']].head())

Decision tree for outliers:

Is it biologically impossible?
├─ Yes → Error. Correct or remove.
└─ No → Is it a data entry error?
 ├─ Yes → Correct it
 └─ No → Does it represent important subpopulation?
  ├─ Yes → Keep it!
  └─ Uncertain → Sensitivity analysis

The “Interesting Outlier” Trap

Real example: Early COVID-19 models removed patients with “impossibly fast” progression as outliers. Later discovered these were cytokine storm cases, rare but critical.

Lesson: Outliers often teach you the most. Be cautious about automatic removal.

Issue 3: Duplicate Records

# Detect duplicates
exact_dupes = df[df.duplicated(keep=False)]
print(f"Exact duplicates: {len(exact_dupes)}")

patient_dupes = df[df.duplicated(subset=['patient_id'], keep=False)]
print(f"Patients with multiple records: {patient_dupes['patient_id'].nunique()}")

# Handle duplicates
# Option 1: Keep first
df_deduped = df.drop_duplicates(subset=['patient_id'], keep='first')

# Option 2: Keep most recent
df_sorted = df.sort_values('report_date', ascending=False)
df_deduped = df_sorted.drop_duplicates(subset=['patient_id'], keep='first')

Issue 4: Inconsistent Coding

# Standardize categorical variables
sex_mapping = {
 'M': 'Male', 'Male': 'Male', 'MALE': 'Male', 'm': 'Male',
 'F': 'Female', 'Female': 'Female', 'FEMALE': 'Female', 'f': 'Female',
 'Unknown': 'Unknown', 'U': 'Unknown', np.nan: 'Unknown'
}
df['sex_clean'] = df['sex'].map(sex_mapping)

# Standardize dates
df['date_clean'] = pd.to_datetime(df['date'], infer_datetime_format=True, errors='coerce')

Data Quality Audit: Tactical Guide

The conceptual framework above describes what can go wrong. Concrete code and decision frameworks below address these issues in practice.

Detecting Data Drift

Data drift occurs when the statistical properties of your data change over time. In public health, this happens constantly: testing policies change, new reporting systems deploy, case definitions evolve. Models trained on stable data fail when deployed on drifted data.

import numpy as np
from scipy import stats
from sklearn.preprocessing import StandardScaler

def detect_drift(reference_data, current_data, feature_cols,
                 threshold=0.05, method='ks'):
    """
    Detect data drift between reference (training) and current (production) data.

    Parameters:
    - reference_data: DataFrame from training period
    - current_data: DataFrame from recent window
    - feature_cols: List of columns to check
    - threshold: p-value threshold for drift detection
    - method: 'ks' (Kolmogorov-Smirnov) or 'chi2' (categorical)

    Returns: Dictionary of drifted features with test statistics
    """
    drifted_features = {}

    for col in feature_cols:
        ref_vals = reference_data[col].dropna()
        curr_vals = current_data[col].dropna()

        if len(ref_vals) < 30 or len(curr_vals) < 30:
            continue  # Insufficient data

        if reference_data[col].dtype in ['object', 'category']:
            # Chi-square test for categorical
            ref_counts = ref_vals.value_counts()
            curr_counts = curr_vals.value_counts()

            # Align categories
            all_cats = set(ref_counts.index) | set(curr_counts.index)
            ref_aligned = [ref_counts.get(c, 0) for c in all_cats]
            curr_aligned = [curr_counts.get(c, 0) for c in all_cats]

            stat, p_value = stats.chisquare(
                curr_aligned,
                f_exp=[r * sum(curr_aligned)/sum(ref_aligned) for r in ref_aligned]
            )
        else:
            # KS test for numerical
            stat, p_value = stats.ks_2samp(ref_vals, curr_vals)

        if p_value < threshold:
            drifted_features[col] = {
                'statistic': stat,
                'p_value': p_value,
                'ref_mean': ref_vals.mean() if ref_vals.dtype != 'object' else None,
                'curr_mean': curr_vals.mean() if curr_vals.dtype != 'object' else None
            }

    return drifted_features

# Example usage: weekly drift monitoring
training_period = df[df['date'] < '2024-01-01']
recent_week = df[df['date'] >= '2024-06-01']

drift_results = detect_drift(
    training_period,
    recent_week,
    feature_cols=['age', 'test_positivity', 'symptom_count']
)

if drift_results:
    print("WARNING: Data drift detected in:")
    for feature, stats in drift_results.items():
        print(f"  {feature}: p={stats['p_value']:.4f}")
        if stats['ref_mean']:
            print(f"    Training mean: {stats['ref_mean']:.2f}")
            print(f"    Current mean: {stats['curr_mean']:.2f}")

When drift is detected:

Feature-level tests can flag distributional changes for investigation, but counts of significant tests and p-values alone are not severity or retraining rules. Predefine monitoring metrics and action thresholds from expected temporal variation, clinical or program consequences, and prospective performance. Investigate multiplicity, sample size, data-pipeline changes, and decision-relevant performance before changing or suspending a model.

Facility-Specific Data Artifacts

Different reporting facilities often have systematic differences that create spurious patterns. A model might learn “Hospital A predicts worse outcomes” when the real pattern is “Hospital A has different coding practices.”

def detect_facility_artifacts(df, facility_col, outcome_col,
                               feature_cols, min_cases=50):
    """
    Detect systematic differences across facilities that may indicate
    data artifacts rather than true clinical variation.

    Returns facilities with suspicious patterns for investigation.
    """
    artifacts = {}

    # 1. Check for extreme missingness variation
    missing_by_facility = df.groupby(facility_col)[feature_cols].apply(
        lambda x: x.isnull().mean()
    )

    for col in feature_cols:
        facility_missing = missing_by_facility[col]
        overall_missing = df[col].isnull().mean()

        # Flag facilities with 3x higher missingness
        outlier_facilities = facility_missing[
            facility_missing > max(3 * overall_missing, 0.3)
        ]

        if len(outlier_facilities) > 0:
            artifacts[f'{col}_missing'] = outlier_facilities.to_dict()

    # 2. Check for digit preference (age heaping, etc.)
    if 'age' in df.columns:
        for facility in df[facility_col].unique():
            facility_ages = df[df[facility_col] == facility]['age']
            if len(facility_ages) < min_cases:
                continue

            # Check for 0/5 preference
            last_digit = facility_ages % 10
            zero_five_pct = last_digit.isin([0, 5]).mean()

            if zero_five_pct > 0.4:  # Expect 20% if random
                if 'age_heaping' not in artifacts:
                    artifacts['age_heaping'] = {}
                artifacts['age_heaping'][facility] = zero_five_pct

    # 3. Check for implausible outcome rates
    outcome_by_facility = df.groupby(facility_col)[outcome_col].mean()
    overall_rate = df[outcome_col].mean()

    suspicious = outcome_by_facility[
        (outcome_by_facility < overall_rate * 0.25) |
        (outcome_by_facility > overall_rate * 4)
    ]

    if len(suspicious) > 0:
        artifacts['extreme_outcome_rates'] = suspicious.to_dict()

    return artifacts

# Run facility audit
facility_issues = detect_facility_artifacts(
    df,
    facility_col='reporting_hospital',
    outcome_col='hospitalized',
    feature_cols=['symptom_onset', 'comorbidity_count', 'age']
)

for issue_type, facilities in facility_issues.items():
    print(f"\nIssue: {issue_type}")
    for facility, value in facilities.items():
        print(f"  {facility}: {value:.2%}")

Handling Changing Case Definitions Mid-Outbreak

When case definitions change (as happened repeatedly during COVID-19), historical data becomes heterogeneous. Models trained across definition changes learn artifacts, not epidemiology.

Strategy 1: Definition Version Flags

# Document definition changes with explicit version markers
DEFINITION_CHANGES = {
    'v1': {'start': '2020-01-01', 'end': '2020-03-15',
           'criteria': 'lab_confirmed_only'},
    'v2': {'start': '2020-03-16', 'end': '2020-07-31',
           'criteria': 'lab_or_antigen'},
    'v3': {'start': '2020-08-01', 'end': None,
           'criteria': 'lab_antigen_or_clinical'}
}

def assign_definition_version(date):
    """Assign case definition version based on report date."""
    for version, params in DEFINITION_CHANGES.items():
        start = pd.to_datetime(params['start'])
        end = pd.to_datetime(params['end']) if params['end'] else pd.Timestamp.max
        if start <= date <= end:
            return version
    return 'unknown'

df['definition_version'] = df['report_date'].apply(assign_definition_version)

# Option A: Include version as feature (model learns definition-specific patterns)
# Option B: Train separate models per definition version
# Option C: Only use data from single definition version (loses history)

Strategy 2: Harmonization When Possible

def harmonize_to_strictest(df, definition_version_col):
    """
    Harmonize to strictest (most specific) case definition.
    Cases from more inclusive definitions that don't meet strict
    criteria are excluded.
    """
    harmonized = df.copy()

    # Only keep lab-confirmed cases (strictest common criterion)
    harmonized = harmonized[
        (harmonized['lab_confirmed'] == 1) |
        (harmonized[definition_version_col] == 'v1')  # v1 was already strict
    ]

    # Document data loss
    excluded = len(df) - len(harmonized)
    excluded_pct = excluded / len(df) * 100
    print(f"Harmonization excluded {excluded:,} cases ({excluded_pct:.1f}%)")
    print(f"Exclusions by version:")
    for v in df[definition_version_col].unique():
        v_total = len(df[df[definition_version_col] == v])
        v_kept = len(harmonized[harmonized[definition_version_col] == v])
        print(f"  {v}: {v_total - v_kept} excluded of {v_total}")

    return harmonized

Imputation vs. Exclusion: Decision Framework

The choice between imputing missing values and excluding incomplete records is consequential. Neither is universally correct.

Consideration	Questions for the analysis plan
Missingness mechanism	What subject-matter and data-collection evidence supports MCAR, MAR, or MNAR assumptions?
Role of the variable	Is it an outcome, exposure, predictor, confounder, or auxiliary variable?
Estimand and model use	Which records and population must the result represent?
Available information	Which observed variables can support imputation or weighting?
Operational constraints	Can the same inputs and transformation be reproduced at deployment?
Uncertainty	Do complete-case, imputed, weighted, and sensitivity analyses lead to materially different conclusions?

Practical Decision Code:

def imputation_recommendation(df, feature_col, outcome_col=None):
    """
    Recommend imputation strategy based on missingness analysis.
    """
    missing_pct = df[feature_col].isnull().mean() * 100

    # Test for MAR vs MCAR
    df['_is_missing'] = df[feature_col].isnull()

    # Check if missingness correlates with other variables
    mar_indicators = []
    for col in df.select_dtypes(include=[np.number]).columns:
        if col in [feature_col, '_is_missing']:
            continue
        corr = df['_is_missing'].astype(int).corr(df[col])
        if abs(corr) > 0.1:
            mar_indicators.append((col, corr))

    # Association with observed variables can guide analysis, but does not
    # identify MCAR, MAR, or MNAR by itself.
    outcome_association = False
    if outcome_col and outcome_col in df.columns:
        outcome_corr = df['_is_missing'].astype(int).corr(df[outcome_col])
        if abs(outcome_corr) > 0.15:
            outcome_association = True

    df.drop('_is_missing', axis=1, inplace=True)

    # Generate recommendation
    recommendation = {
        'feature': feature_col,
        'missing_pct': missing_pct,
        'mechanism': 'not identified from observed correlations alone',
        'mar_predictors': mar_indicators[:3] if mar_indicators else None
    }

    recommendation['strategy'] = 'analysis_plan_required'
    recommendation['method'] = (
        'Compare justified complete-case, imputation, weighting, and sensitivity '
        'analyses using subject-matter knowledge and the intended estimand.'
    )
    recommendation['outcome_association'] = outcome_association

    return recommendation

# Example usage
for col in ['symptom_onset', 'age', 'comorbidity_count']:
    rec = imputation_recommendation(df, col, outcome_col='hospitalized')
    print(f"\n{col}:")
    print(f"  Missing: {rec['missing_pct']:.1f}%")
        print(f"  Mechanism: {rec['mechanism']}")
    print(f"  Strategy: {rec['strategy']}")

Anomaly Detection with Reporting Delay Awareness

Standard anomaly detection fails on surveillance data because reporting delays create artificial patterns. A spike detected today might be reporting catchup, not a real outbreak.

def delay_aware_anomaly_detection(df, date_col, count_col,
                                   delay_distribution, lookback_days=28):
    """
    Detect anomalies accounting for reporting delays.

    Parameters:
    - df: Time series with date and count columns
    - delay_distribution: Dict mapping days_lag -> proportion reported
      e.g., {0: 0.2, 1: 0.3, 2: 0.25, 3: 0.15, 4: 0.10}
    - lookback_days: Historical window for baseline

    Returns: DataFrame with nowcast estimates and anomaly flags
    """
    df = df.copy().sort_values(date_col)

    # Calculate nowcast adjustment factors
    max_delay = max(delay_distribution.keys())
    cumulative_reported = {}
    cumsum = 0
    for d in range(max_delay + 1):
        cumsum += delay_distribution.get(d, 0)
        cumulative_reported[d] = cumsum

    # Apply nowcast adjustment to recent dates
    today = df[date_col].max()
    df['days_old'] = (today - df[date_col]).dt.days

    df['nowcast_adjustment'] = df['days_old'].apply(
        lambda d: 1 / cumulative_reported.get(min(d, max_delay), 1)
    )

    df['nowcast_count'] = df[count_col] * df['nowcast_adjustment']

    # Calculate baseline from fully-reported historical period
    stable_period = df[df['days_old'] > max_delay + 7]

    if len(stable_period) < lookback_days:
        raise ValueError("Insufficient historical data for baseline")

    baseline_mean = stable_period[count_col].tail(lookback_days).mean()
    baseline_std = stable_period[count_col].tail(lookback_days).std()

    # Flag anomalies (nowcast exceeds baseline by >2 SD)
    df['z_score'] = (df['nowcast_count'] - baseline_mean) / baseline_std
    df['anomaly'] = df['z_score'] > 2.0
    df['anomaly_confidence'] = 'high'

    # Lower confidence for very recent dates (nowcast uncertain)
    df.loc[df['days_old'] < 3, 'anomaly_confidence'] = 'low'
    df.loc[(df['days_old'] >= 3) & (df['days_old'] < max_delay), 'anomaly_confidence'] = 'medium'

    return df[['date_col', count_col, 'nowcast_count', 'z_score',
               'anomaly', 'anomaly_confidence']]

# Example: typical respiratory illness reporting delays
typical_delays = {0: 0.15, 1: 0.25, 2: 0.25, 3: 0.20, 4: 0.10, 5: 0.05}

results = delay_aware_anomaly_detection(
    daily_counts,
    date_col='report_date',
    count_col='case_count',
    delay_distribution=typical_delays
)

# Show recent anomalies with confidence levels
recent_anomalies = results[results['anomaly'] & (results['days_old'] < 14)]
print(recent_anomalies[['report_date', 'case_count', 'nowcast_count',
                        'z_score', 'anomaly_confidence']])

Data Quality Audit Checklist

Run these checks before any modeling project:

• Missingness audit: Calculate per-feature, test MCAR/MAR/MNAR
• Temporal coverage: Any gaps in time series? Definition changes?
• Facility variation: Systematic differences across reporting sources?
• Drift detection: Compare recent data to training period
• Outlier investigation: Document outliers, decision to keep/remove
• Label quality: How reliable is your outcome variable?
• Selection bias: Who’s in your data vs. who’s in your target population?

Document findings in a data quality report before proceeding.

Feature Engineering for Public Health

Domain expertise matters most, as Domingos emphasizes.

Time-Based Features

# Extract temporal features
df['date'] = pd.to_datetime(df['report_date'])

df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day_of_week'] = df['date'].dt.dayofweek
df['week_of_year'] = df['date'].dt.isocalendar().week
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)

# Seasonal indicators
def assign_season(month):
 if month in [12, 1, 2]:
  return 'Winter'
 elif month in [3, 4, 5]:
  return 'Spring'
 elif month in [6, 7, 8]:
  return 'Summer'
 else:
  return 'Fall'

df['season'] = df['month'].apply(assign_season)

# Days since outbreak start
outbreak_start = df['date'].min()
df['days_since_start'] = (df['date'] - outbreak_start).dt.days

Rolling Statistics

# Sort by date first!
df = df.sort_values('date')

# 7-day rolling average
df['cases_7day_avg'] = df['daily_cases'].rolling(window=7, min_periods=1).mean()

# 14-day rolling sum
df['cases_14day_sum'] = df['daily_cases'].rolling(window=14, min_periods=1).sum()

# Growth rate
df['case_growth_rate'] = df['daily_cases'].pct_change(periods=7)

# Lag features
df['cases_lag_7'] = df['daily_cases'].shift(7)
df['cases_lag_14'] = df['daily_cases'].shift(14)

# Ratio features
df['cases_this_week_vs_last'] = df['daily_cases'] / df['cases_lag_7']

Avoiding Data Leakage

NEVER use future information, as warned in Kaufman et al.’s analysis:

# WRONG: Uses future data
df['cases_rolling'] = df['daily_cases'].rolling(window=7, center=True).mean()

# CORRECT: Only past data
df['cases_rolling'] = df['daily_cases'].rolling(window=7, min_periods=1).mean()

For time series splits:

# WRONG for time series
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)

# CORRECT for time series
split_date = df['date'].quantile(0.8)
train_df = df[df['date'] < split_date]
test_df = df[df['date'] >= split_date]

Epidemiological Features

# R0 estimation from growth rate
def estimate_R0(cases, generation_time=5):
 """Estimate R0 from exponential growth"""
 from scipy.optimize import curve_fit

 days = np.arange(len(cases))

 def exponential(x, a, b):
  return a * np.exp(b * x)

 params, _ = curve_fit(exponential, days, cases, p0=[1, 0.1])
 growth_rate = params[1]
 R0 = 1 + growth_rate * generation_time
 return R0

df['R0_estimate'] = df['daily_cases'].rolling(window=14).apply(
 lambda x: estimate_R0(x.values) if len(x) > 7 else np.nan,
 raw=False
)

# Attack rate
population = 1000000
df['attack_rate'] = df['cumulative_cases'] / population

# Case fatality rate
df['CFR'] = df['cumulative_deaths'] / df['cumulative_cases']

# Test positivity rate
df['test_positivity'] = df['positive_tests'] / df['total_tests']

Geographic Features

# Distance to healthcare
from sklearn.metrics.pairwise import haversine_distances

case_coords = df[['latitude', 'longitude']].values
hospital_coords = hospitals[['lat', 'lon']].values

# Convert to radians
case_coords_rad = np.radians(case_coords)
hospital_coords_rad = np.radians(hospital_coords)

# Calculate distances (km)
distances = haversine_distances(case_coords_rad, hospital_coords_rad) * 6371
df['nearest_hospital_km'] = distances.min(axis=1)

# Spatial clustering
from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(radius=5.0, metric='haversine')
nn.fit(case_coords_rad)

distances, indices = nn.radius_neighbors(case_coords_rad)
df['cases_within_5km'] = [len(idx) - 1 for idx in indices]

Case Study: COVID-19 Forecasting Failures

The COVID-19 pandemic provided a natural experiment in real-time forecasting. In the U.S. Forecast Hub evaluation, ensemble forecasts outperformed individual models on average, while error increased with forecast horizon (Cramer et al., 2022).

What went wrong:

1. Reporting delay ignorance

• Models treated “reported March 15” as “occurred March 15”
• Reality: 5-15 day lags, varying by jurisdiction
• Result: Systematic underestimation

2. Changing testing regime

• Week 1: Only hospitalized patients
• Week 8: Asymptomatic screening
• Models confused testing expansion with disease growth

3. Selection bias

• Early models trained on Chinese data (strict lockdowns)
• Applied to US/Europe (different compliance)
• External validity failed

4. Missing confounders

• Behavioral responses not in feature set
• Policy changes (masks, closures) ignored
• Human adaptation breaks stationarity

The numbers:

• Auxiliary indicator analysis: A plain autoregressive model using no external features was already competitive with many of the top COVID-19 forecasting models in use
• IHME model: Repeatedly revised by orders of magnitude
• Ensemble methods outperformed individuals, but still struggled

Lessons learned:

Data Quality > Model Sophistication

A plain autoregressive model using no external features was already competitive with many of the top COVID-19 forecasting models in use, and adding auxiliary data streams produced gains that varied by trend direction rather than transforming accuracy (McDonald et al., 2021).

The pandemic exposed that public health data infrastructure, not algorithms, is the bottleneck.

When Data Problems Make AI Inappropriate

Red flags that should stop you:

1. Extreme selection bias with no adjustment

• Example: Training on hospitalized, deploying on general population
• Why fatal: Model does not generalize

2. Outcome label quality is poor

• Example: Inconsistent disease diagnosis criteria
• Why fatal: Model learns noise, not signal

3. Critical features have unresolved missingness

• Example: Symptom onset missing for most cases
• Why fatal: Cannot build temporal features

4. Data generating process changed mid-dataset

• Example: Pre/post policy change, different testing
• Why fatal: Violates stationarity assumption

5. Sample size too small

• Example: 100 cases, 50 features, trying deep learning
• Why fatal: Severe overfitting

6. External validity concerns

• Example: One hospital’s data, deploying elsewhere
• Why fatal: Context matters, model does not transfer

The Courage to Say “No”

Good data science includes knowing when NOT to build a model.

If data quality is insufficient: 1. Report limitations clearly 2. Recommend improving data collection 3. Suggest alternatives (traditional epi, qualitative research)

Building on garbage data: - Wastes resources - Produces misleading results - Can harm people - Damages trust in AI

As Box said: “All models are wrong, but some are useful.” If your data is wrong enough, your model will be useless.

Key Takeaways

1. Data quality determines model quality. Algorithms cannot overcome flawed data.

2. Understand missingness mechanisms. MCAR, MAR, MNAR require different strategies.

3. Public health data has unique challenges: selection bias, reporting delays, changing surveillance.

4. Investigate, do not automatically fix. Outliers might be the most important cases.

5. Domain and technical expertise are complementary. Domain knowledge guides feature validity; comparative validation tests whether model complexity adds value.

6. Document everything. Future users need to know what you cleaned and why.

7. Know when to stop. Sometimes the ethical choice is not to build a model.

8. Temporal ordering matters. Never use future to predict past.

Practice Exercises

Exercise 1: Missing Data Mechanisms

1. Identify variables with missing data
2. Test if missingness is MCAR, MAR, or MNAR
3. Choose imputation strategies
4. Compare model performance

Exercise 2: Reporting Delay Adjustment

1. Calculate reporting triangle
2. Implement nowcasting
3. Compare nowcast vs. raw counts
4. Evaluate accuracy over time

Exercise 3: Feature Engineering

Create outbreak predictor using: - Daily cases - Temperature - Rainfall - Population density

Engineer 10+ features. Which matter most?

Exercise 4: Data Quality Audit

Audit a “cleaned” dataset: 1. Find 5+ quality issues 2. Determine if fixable or fatal 3. Write quality report

Check Your Understanding

Test your knowledge of the key concepts from this chapter. Click “Show Answer” to reveal the correct response and explanation.

Question 1: Data Quality Foundations

Which dimension of data quality is violated when COVID-19 testing was initially limited to hospitalized patients, but models were deployed to predict community transmission?

1. Completeness
2. Accuracy
3. Timeliness
4. Representativeness

Answer: d) Representativeness

Explanation: This is a representativeness problem. The training data (hospitalized patients) does not represent the target population (community cases). This selection bias means the model learns patterns from a severely ill subpopulation that do not generalize to mild/asymptomatic cases in the community.

Question 2: Missing Data Mechanisms

You notice that income data is missing for 30% of younger respondents but only 5% of older respondents. What type of missingness is this?

1. Missing Completely At Random (MCAR)
2. Missing At Random (MAR)
3. Missing Not At Random (MNAR)
4. Systematic error

Answer: b) Missing At Random (MAR)

Explanation: This is MAR (Missing At Random) because the missingness depends on an observed variable (age), not on the missing value itself (income). You can impute income using age as a predictor since you know the missingness pattern.

Question 3: Temporal Data Leakage

True or False: When creating rolling averages for time series forecasting, using rolling(window=7, center=True) is appropriate because it creates smoother trends.

Answer: False

Explanation: Using center=True causes data leakage by incorporating future values into the calculation. For a value on day 10, it would include data from days 7-13, meaning you’re using days 11-13 (the future) to predict day 10. For forecasting, you must use center=False (the default) to only look backwards in time.

Question 4: Reporting Delays

A surveillance system shows cases declining on the most recent dates. What is the most likely explanation before concluding the outbreak is ending?

1. Effective public health intervention
2. Population-level immunity developing
3. Right-censoring from reporting delays
4. Seasonal weather changes

Answer: c) Right-censoring from reporting delays

Explanation: Right-censoring from reporting delays is the most common explanation for apparent declines in recent data. Cases from recent days have not fully been reported yet, creating an artificial downward trend. Always check reporting lag distributions before interpreting recent trends. This is why nowcasting methods exist.

Question 5: Feature Engineering

Which of these features would be MOST useful for predicting dengue outbreaks and demonstrates good domain expertise?

1. Day of the week
2. Raw daily temperature
3. 14-day cumulative rainfall
4. Month number (1-12)

Answer: c) 14-day cumulative rainfall

Explanation: 14-day cumulative rainfall shows domain expertise because dengue mosquitoes (Aedes) breed in standing water that accumulates over time. Raw daily temperature or simple month number do not capture the mechanism. Day of the week is irrelevant for disease biology. This demonstrates how epidemiological understanding drives effective feature engineering.

Question 6: Outliers

You find 5 patients with “impossibly fast” disease progression (symptom onset to death in <24 hours). What should you do?

1. Remove them as data entry errors
2. Remove them to avoid overfitting on outliers
3. Investigate them carefully, they might be important rare cases
4. Set their values to the median progression time

Answer: c) Investigate them carefully, they might be important rare cases

Explanation: Investigate carefully before removing. These could be: - Real rare but critical cases (e.g., cytokine storm, overwhelming infection) - Important subpopulation with different risk factors - Data entry errors (date fields swapped)

The COVID-19 pandemic taught us that “outliers” like rapidly progressing cases were often the most clinically important to understand. Only remove after investigation confirms they’re truly errors.

Frequently Asked Questions

What determines whether missing public health data can be used?

The percentage missing is only one input. The decision depends on why data are missing, which people and time periods are affected, the estimand, available auxiliary information, sample size, and the consequences of error. Complete-case analysis, imputation, weighting, or sensitivity analysis each rely on assumptions. Those assumptions should be stated and tested rather than selected from a universal missingness cutoff.

What is temporal data leakage?

Temporal leakage occurs when model development uses information that would not have been available at the prediction time, including later diagnoses, revised records, future laboratory results, or random splits that mix earlier and later observations from the same process. Define the prediction time first, reconstruct the available feature set at that moment, and use time-aware validation that mirrors deployment.

Does distributional drift mean a model should be retrained?

Not automatically. A change in inputs may or may not degrade the task’s performance, calibration, or equity. Teams should investigate the cause, evaluate outcomes when labels become available, and follow prespecified action rules. Retraining without understanding the shift can reproduce a data error or erase useful stability. Any updated model requires fresh validation and controlled release.

Discussion Questions

1. Your training data is 80% from one hospital. How does this affect generalizability? What can you do?

2. Model has 95% accuracy on training, 65% on test, with 30% missing labels. How do you interpret this?

3. Colleague wants to impute symptom onset as report date minus median delay. Pros/cons? When appropriate?

4. Forecasting COVID-19 hospitalizations: use raw counts or adjust for test positivity? Justify.

5. When is it acceptable to remove outliers from public health data? Examples of appropriate vs. inappropriate removal.

Further Reading

Data Quality Frameworks

• Weiskopf, N.G., & Weng, C. (2013). “Methods and dimensions of EHR data quality.” JAMIA, 20(1), 144-151.
• Kahn, M.G., et al. (2016). “Harmonized data quality terminology.” JAMIA, 23(6), 1244-1251.

Missing Data

• Little, R.J.A., & Rubin, D.B. (2019). Statistical Analysis with Missing Data. Wiley.
• van Buuren, S. (2018). Flexible Imputation of Missing Data. CRC Press.

Surveillance Challenges

• Lipsitch, M., et al. (2020). “Defining Covid-19 epidemiology.” Science, 368(6490), 565-568.
• Reich, N.G., et al. (2019). “Collaborative flu forecasting assessment.” PNAS, 116(8), 3146-3154.

COVID-19 Lessons

• Cramer, E.Y., et al. (2022). “Evaluation of COVID-19 forecasts.” PNAS, 119(15), e2111453119.
• McGough, S.F., et al. (2020). “Nowcasting by Bayesian smoothing.” PLoS Comp Bio, 16(10), e1008785.

Data Leakage

• Kaufman, S., et al. (2012). “Leakage in data mining.” ACM TKDD, 6(4), 1-21.

---

You now understand why data quality is the foundation of successful AI in public health. Clean data and thoughtful feature engineering matter more than fancy algorithms.

---

Part I Summary: What You Should Now Know

Congratulations! You’ve completed Part I: Foundations. Before moving to applications, ensure you can confidently:

From AI in Healthcare: A Brief History

• Explain how AI has evolved in public health from expert systems to modern machine learning
• Identify when AI adds value vs. when traditional epidemiology is sufficient
• Recognize patterns of AI hype and separate genuine capabilities from marketing
• Understand the unique ethical considerations of AI in population health

From Machine Learning Fundamentals

• Distinguish between supervised, unsupervised, and reinforcement learning paradigms
• Choose appropriate algorithms for different problem types (logistic regression → Random Forests → gradient boosting → deep learning)
• Interpret evaluation metrics (accuracy, sensitivity, specificity, ROC-AUC) and know when each matters
• Identify overfitting and data leakage issues
• Understand why most public health tabular data works best with tree-based methods, not deep learning
• Explain predictions using feature importance and SHAP values

From The Data Problem (This Chapter)

• Assess data quality across four dimensions: completeness, accuracy, timeliness, representativeness
• Identify missing data mechanisms (MCAR, MAR, MNAR) and choose appropriate imputation strategies
• Handle reporting delays and right-censoring in surveillance data
• Engineer meaningful features from public health data using domain expertise
• Recognize when data problems make AI inappropriate (selection bias, insufficient sample size, changing data-generating processes)
• Avoid temporal data leakage when building forecasting models

Core Skills Checklist

Can you: - [ ] Read and modify basic ML code in Python (scikit-learn, pandas)? - [ ] Run a complete ML pipeline from data loading → preprocessing → training → evaluation? - [ ] Interpret confusion matrices and choose metrics appropriate for your use case? - [ ] Detect and handle missing data without blindly dropping rows? - [ ] Create time-based features without leaking future information? - [ ] Explain why a simple model on clean data beats a complex model on dirty data? - [ ] Identify when NOT to build an AI model due to data quality issues?

What’s Next

Part II: Applications applies these foundations to real public health problems: - Disease surveillance and outbreak detection - Epidemic forecasting - Genomic surveillance - Large language models in public health - Clinical decision support

You now have the conceptual foundation and technical skills to understand how AI works in practice. The next chapters show you specific applications where AI creates value, and where it falls short.

If you’re unsure about any foundation concept, review the relevant chapter before proceeding. The applications build directly on these fundamentals.

---

Next chapter: AI in Disease Surveillance and Outbreak Detection