graph TB subgraph " " A["Lab-Confirmed Cases<br/>(What surveillance systems capture)"] B["Hospitalized Cases<br/>(Severe cases only)"] C["Symptomatic Cases Seeking Care<br/>(Access to healthcare varies)"] D["All Symptomatic Cases<br/>(Many don't seek care)"] E["All Infections<br/>(Including asymptomatic - true burden)"] end E --> D D --> C C --> B B --> A style A fill:#ff6b6b,stroke:#333,stroke-width:2px,color:#fff style B fill:#ffa07a,stroke:#333,stroke-width:2px style C fill:#ffd166,stroke:#333,stroke-width:2px style D fill:#bfe1b0,stroke:#333,stroke-width:2px style E fill:#95e1d3,stroke:#333,stroke-width:3px
The Data Problem in Public Health AI
Public health data is incomplete, delayed, biased, and inconsistent by nature. Surveillance captures only the tip of the iceberg. Reporting lags create temporal misalignment, and testing availability determines who appears in a dataset. Learning to assess data quality across four dimensions (completeness, accuracy, timeliness, representativeness) and engineer features with epidemiological insight matters more than algorithm sophistication.
Learning Objectives
By the end of this chapter, readers will be able to:
- Assess data quality dimensions (completeness, accuracy, timeliness, representativeness)
- Distinguish missing data mechanisms (MCAR, MAR, MNAR)
- Recognize how selection bias invalidates predictions (COVID-19 testing)
- Identify when data quality makes AI inappropriate
- Implement practical quality assessments and consistency checks
- Recommend against deployment when data quality does not support it
Prerequisites: History of AI in Healthcare and Public Health, Machine Learning Fundamentals.
Introduction
Most AI tutorials use clean, well-formatted datasets. Most public health data is none of those things.
Real public health data is:
- Incomplete (missing values everywhere)
- Delayed (cases reported days or weeks late)
- Biased (who gets tested determines who appears in your data)
- Inconsistent (formats change, definitions evolve)
- Messy (typos, duplicates, impossible values)
- Heterogeneous (multiple sources with different standards)
This is the data public health teams have to work with. Learning to handle messy data is arguably more important than learning sophisticated algorithms.
The Iron Law of Machine Learning
Garbage in = Garbage out
The most sophisticated deep learning model cannot overcome fundamentally flawed data. A simple logistic regression on clean, well-understood data can outperform a complex neural network on garbage data.
Data quality and algorithm design are both necessary. This chapter emphasizes data because many development failures originate in measurement, selection, linkage, and temporal alignment rather than model architecture.
Why Public Health Data Is Uniquely Challenging
1. The Surveillance Pyramid Problem
Public health surveillance captures only a fraction of reality:
[Lab] Lab-Confirmed Cases (Data you have)
/
[Hosp] Hospitalized Cases
/
[Care] Symptomatic Cases Seeking Care
/
[Symp] All Symptomatic Cases
/
[All] All Infections (Including Asymptomatic)The issue: The dataset represents the tip of the pyramid, but the population of interest is the entire pyramid. This is selection bias by design.
Example: COVID-19 Testing Bias
Early in the pandemic, testing was limited to:
- Symptomatic individuals
- Healthcare workers
- Travelers
- Close contacts of confirmed cases
Any model trained on this data would:
- Overestimate symptom severity (asymptomatic cases invisible)
- Misunderstand demographic risk factors (testing access varied by socioeconomic status)
- Produce biased risk predictions (dataset not representative of population)
As documented in Lipsitch et al.’s analysis of surveillance biases, these selection effects can lead to systematic underestimation of disease burden.
When Selection Bias Invalidates AI
If training data comes from one level of the surveillance pyramid but predictions target another, model complexity alone cannot solve the resulting selection problem.
Addressing it requires one of the following:
- Data from the correct population (often impossible)
- Statistical methods to adjust for selection (Hernán’s target trial framework)
- A different research question that matches your available data
2. Reporting Delays and Temporal Misalignment
Illustrative reporting-delay scenario: A team wants to predict tomorrow’s case counts from today’s data. A reporting extract can include observations from several earlier dates in proportions that vary by jurisdiction and reporting system:
- Cases from today (20%)
- Cases from yesterday, reported today (40%)
- Cases from 2 days ago, reported today (30%)
- Cases from 3+ days ago, reported today (10%)
The extract labeled “September 15” may therefore mix cases from several occurrence dates, with proportions that must be measured rather than assumed.
Why this matters for ML:
- Features and labels are temporally misaligned
- Recent trends are systematically underestimated (right-censoring)
- Models learn spurious patterns from reporting artifacts
Real example: Early COVID-19 forecasting models struggled because they treated reported case counts as actual case counts, ignoring 5-10 day reporting lags that varied by jurisdiction.
3. Changing Definitions and Data Collection
Public health data collection evolves over time, as extensively documented in Reich et al.’s analysis of forecasting challenges.
Case definition changes:
- COVID-19 case definitions changed multiple times in 2020-2021
- Adding antigen tests to confirmed cases
- Including probable cases vs. confirmed only
Testing availability changes:
- Week 1: Only sick hospitalized patients tested
- Week 10: Symptomatic individuals can get tested
- Week 20: Asymptomatic screening becomes common
The dataset appears to show:
- Exponential case growth
- Changing age distribution
- Different symptom profiles over time
In reality: These might be artifacts of changing surveillance, not actual epidemiological changes.
The Fundamental Challenge
Machine learning assumes the relationship between features and outcomes is stationary (constant over time).
Public health surveillance violates this assumption constantly. The data-generating process itself evolves.
This is why pandemic forecasting is so difficult, the ground truth keeps shifting under your feet.
4. The Denominator Problem
Consider two regions:
Region A: 1,000 cases, 50 deaths → 5% CFR
Region B: 500 cases, 100 deaths → 20% CFRIs Region B truly more dangerous? Or does it have:
- Less testing (missing mild cases in denominator)
- Older population
- Worse healthcare access
- Different case definitions
The problem: A typical surveillance dataset has numerators (cases, deaths) but often lacks good denominators (population at risk, testing rates, exposure levels).
AI models trained on case counts without accounting for testing intensity will confuse “more testing” with “more disease,” as Ioannidis cautioned.
5. Privacy Constraints and Data Aggregation
To protect patient privacy, public health data is often:
- Aggregated (county-level instead of individual-level)
- Suppressed (cells with <5 cases shown as “<5” or asterisk)
- Coarsened (age shown as brackets: “20–29” instead of exact age)
- Delayed (real-time data withheld, released weeks later)
Impact on ML:
- Loss of granularity reduces predictive power
- Non-standard missing data patterns
- Cannot link across datasets (no unique identifiers)
- Temporal resolution insufficient for some analyses
The HL7 FHIR standard and HIPAA Safe Harbor guidelines constrain what data you can access and how it’s formatted.
Current CDC Data Modernization Milestones
CDC’s 2026 Public Health Data Strategy milestones make the data problem operational. The milestones include receiving syndromic data from at least 86% of U.S. emergency departments, automated bed-capacity feeds from at least 17 state and territorial jurisdictions, and production electronic case reporting processing for at least 70% of ELC-funded health departments connected to AIMS (CDC, April 2026). These are AI prerequisites, not administrative background: models built on delayed, manually reconciled, or inconsistently structured feeds inherit those defects.
The same milestone set places AI readiness inside data modernization by calling for CDC GenAI guidance and informational resources for state, tribal, local, and territorial (STLT) health department partners. For data teams, the practical message is direct: do not treat model selection as the starting point. Data flow automation, shared standards, and governance capacity determine whether AI can be evaluated at all.
The Four Dimensions of Data Quality
When assessing whether your data is suitable for AI, evaluate these four dimensions. Completeness, accuracy (correctness), and timeliness (currency) are three of the five dimensions identified in Weiskopf and Weng’s review of electronic health record data quality assessment; representativeness is added here because population inference depends on it.
1. Completeness: Are Values Missing?
The question: What percentage of values are missing? Is the missingness random or systematic?
Code to assess:
Hide code
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load your data
df = pd.read_csv('../data/examples/surveillance_data.csv')
# Calculate missingness
missing_summary = pd.DataFrame({
'column': df.columns,
'missing_count': df.isnull().sum(),
'missing_percent': (df.isnull().sum() / len(df) * 100).round(2)
}).sort_values('missing_percent', ascending=False)
print("Missing Data Summary:")
print(missing_summary)
# Visualize missingness patterns
import missingno as msno
# Matrix plot: see where missing values cluster
msno.matrix(df, figsize=(12, 6), sparkline=False)
plt.title('Missing Data Pattern')
plt.tight_layout()
plt.savefig('../images/examples/missing_data_matrix.png', dpi=300)Interpreting results:
Missingness percentage is a useful descriptive measure, not a universal decision threshold. Assess the mechanism, variable role, estimand, available auxiliary information, and uncertainty before choosing a method (Lee et al., 2021).
Types of missingness (Little & Rubin’s taxonomy):
- Missing Completely At Random (MCAR): Missingness is unrelated to any variables
- Example: Lab results lost due to random system glitches
- Implication: Complete-case analysis may be unbiased under additional assumptions; single-value imputation can still understate uncertainty
- Missing At Random (MAR): Missingness depends on observed variables
- Example: Income more likely missing for younger respondents
- Implication: Can impute using related variables
- Missing Not At Random (MNAR): Missingness depends on the missing value itself
- Example: Sickest patients too ill to complete surveys
- Implication: Missing data itself is informative
Testing Missingness Patterns
Hide code
# Check if missingness is related to other variables
df['symptom_onset_missing'] = df['symptom_onset'].isnull()
# Compare groups with and without missing values
print(df.groupby('symptom_onset_missing')['age'].mean())
print(df.groupby('symptom_onset_missing')['hospitalized'].mean())
# Statistical test
from scipy.stats import chi2_contingency
contingency_table = pd.crosstab(df['symptom_onset_missing'],
df['hospitalized'])
chi2, p_value, dof, expected = chi2_contingency(contingency_table)
if p_value < 0.05:
print("Missingness is NOT random - be careful with imputation")2. Accuracy: Are Values Correct?
Common accuracy issues:
Impossible or Implausible Values
Hide code
# Detect impossible values
issues = []
# Age checks
if (df['age'] < 0).any() or (df['age'] > 120).any():
issues.append("Impossible ages detected")
# Date checks
df['symptom_date'] = pd.to_datetime(df['symptom_date'], errors='coerce')
df['test_date'] = pd.to_datetime(df['test_date'], errors='coerce')
# Symptom onset after test date? This can be valid for presymptomatic testing,
# so flag it for context-specific review rather than treating it as impossible.
invalid_dates = df['symptom_date'] > df['test_date']
if invalid_dates.any():
issues.append(f"{invalid_dates.sum()} cases with symptom onset after test require review")
# Temperature checks
if 'temperature' in df.columns:
if (df['temperature'] < 35).any() or (df['temperature'] > 42).any():
issues.append("Implausible temperatures detected")
for issue in issues:
print(issue)Data Entry Errors
Hide code
# Common data entry errors
# 1. Height/weight transpositions
df['bmi'] = df['weight'] / (df['height']/100)**2
suspicious_bmi = (df['bmi'] < 10) | (df['bmi'] > 60)
print(f"Suspicious BMI values: {suspicious_bmi.sum()}")
# 2. Unit inconsistencies
if 'temperature' in df.columns:
potential_fahrenheit = (df['temperature'] > 50).sum()
if potential_fahrenheit > 0:
print(f"Possible unit mixing: {potential_fahrenheit} values >50°")
# 3. Decimal point errors (370 instead of 37.0)
if 'temperature' in df.columns:
extreme_values = df['temperature'] > 100
print(f"Possible decimal errors: {extreme_values.sum()}")
# 4. Age heaping (digit preference)
age_last_digit = df['age'] % 10
digit_counts = age_last_digit.value_counts().sort_index()
from scipy.stats import chisquare
expected_freq = [len(df)/10] * 10
chi2, p_value = chisquare(digit_counts, expected_freq)
if p_value < 0.05:
print("Age heaping detected - rounding to 0s and 5s")
The “Too Clean” Red Flag
If your public health dataset has zero missing values, perfect consistency, and no outliers, be suspicious. Real-world data is messy.
3. Timeliness: Are Values Up-to-Date?
Quantifying reporting delays:
Hide code
# Calculate reporting lag
df['report_date'] = pd.to_datetime(df['report_date'])
df['symptom_onset'] = pd.to_datetime(df['symptom_onset'])
# Lag from symptom onset to report
df['onset_to_report_days'] = (df['report_date'] - df['symptom_onset']).dt.days
# Visualize
fig, ax = plt.subplots(figsize=(10, 6))
ax.hist(df['onset_to_report_days'].dropna(), bins=50, edgecolor='black')
ax.set_xlabel('Days from Symptom Onset to Report')
ax.set_ylabel('Frequency')
ax.axvline(df['onset_to_report_days'].median(), color='red',
linestyle='--',
label=f"Median: {df['onset_to_report_days'].median():.1f} days")
ax.legend()
plt.savefig('../images/examples/reporting_delays.png', dpi=300)Nowcasting: Adjusting for reporting delays as described in McGough et al.’s work.
Advanced Nowcasting
For rigorous handling of reporting delays:
4. Representativeness: Does Data Match Target Population?
Common biases:
Hide code
# Check for demographic bias
# Compare to census data
population_age_dist = {
'0-17': 0.22,
'18-64': 0.62,
'65+': 0.16
}
sample_age_dist = df['age_group'].value_counts(normalize=True).to_dict()
print("Age Distribution Comparison:")
print("Age Group | Population | Sample | Difference")
print("-" * 50)
for age_group in population_age_dist:
pop_pct = population_age_dist[age_group] * 100
sample_pct = sample_age_dist.get(age_group, 0) * 100
diff = sample_pct - pop_pct
print(f"{age_group:8} | {pop_pct:6.1f}% | {sample_pct:6.1f}% | {diff:+6.1f}%")
When Representativeness Matters Most
Representativeness is critical when:
- Making population-level inferences
- Forecasting total burden
- Evaluating interventions
Less critical when:
- Early outbreak detection (any signal helps)
- Comparing relative risks within sample
An NHANES methods study in Frontiers in Digital Health illustrates this population-inference rule. The authors drew hypertension and diabetes training data from 6,024 adults and test data from 1,926 (2007–2014 cycles), and balanced the classes before fitting by random undersampling (RUS) or undersampling based on clustering (SBC). Unweighted AUC on the weight-aware models over- or under-estimated the design-based weighted AUC (wAUC). For the hypertension LASSO model with RUS, ordinary cross-validation gave an AUC and wAUC of 0.838 for separating adults with and without hypertension, against a wAUC of 0.829 under design-based cross-validation (dCV); for the diabetes LASSO model with SBC, the corresponding AUCs for separating adults with and without diabetes were 0.866 and 0.852. Design-based cross-validation and jackknife repeated replication (JKn) also changed how many variables entered the models, and the authors write that unweighted models overestimate or underestimate the true population parameters (Kim, Rogers & Wang, 2026). Sample-weight-adjusted discrimination is not a hypertension or diabetes screening endpoint; random forest is only their fallback when weights cannot be used, and they exclude rare-disease settings. Unweighted AUC ≠ population parameter.
Common Data Issues and How to Handle Them
Issue 1: Missing Data
When MCAR:
Hide code
from sklearn.impute import SimpleImputer
# Median imputation for numerical variables
# Single-value imputation understates variance and uncertainty, even under MCAR
numerical_cols = df.select_dtypes(include=[np.number]).columns
imputer = SimpleImputer(strategy='median')
df[numerical_cols] = imputer.fit_transform(df[numerical_cols])When MAR:
Hide code
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
# Iterative imputation (like MICE)
imputer = IterativeImputer(max_iter=10, random_state=42)
df_imputed = pd.DataFrame(
imputer.fit_transform(df[numerical_cols]),
columns=numerical_cols
)When MNAR:
Hide code
# Create missingness indicator
df['income_missing'] = df['income'].isnull().astype(int)
# Impute with flag value
df['income_imputed'] = df['income'].fillna(-999)
Best Practices for Missing Data
- Never silently drop missing data
- Compare complete-case vs. imputed analysis
- Try multiple imputation
- Report missingness patterns
- Create missingness indicators when uncertain
Issue 2: Outliers and Extreme Values
Not all outliers are errors.
Hide code
def detect_outliers_iqr(df, column, multiplier=1.5):
"""Detect outliers using IQR method"""
Q1 = df[column].quantile(0.25)
Q3 = df[column].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - multiplier * IQR
upper_bound = Q3 + multiplier * IQR
outliers = df[(df[column] < lower_bound) | (df[column] > upper_bound)]
return outliers, lower_bound, upper_bound
# Investigate before removing
age_outliers, lower, upper = detect_outliers_iqr(df, 'age', multiplier=3)
print(f"Age outliers: {len(age_outliers)}")
print(age_outliers[['age', 'outcome', 'hospital']].head())Decision tree for outliers:
Is it biologically impossible?
├─ Yes → Error. Correct or remove.
└─ No → Is it a data entry error?
├─ Yes → Correct it
└─ No → Does it represent important subpopulation?
├─ Yes → Keep it
└─ Uncertain → Sensitivity analysis
The “Interesting Outlier” Trap
Illustrative scenario: A COVID-19 model that removes patients with “impossibly fast” progression as outliers can discard the rare, severe cases a model most needs to learn from.
Lesson: Outliers are often the most informative cases. Be cautious about automatic removal.
Issue 3: Duplicate Records
Hide code
# Detect duplicates
exact_dupes = df[df.duplicated(keep=False)]
print(f"Exact duplicates: {len(exact_dupes)}")
patient_dupes = df[df.duplicated(subset=['patient_id'], keep=False)]
print(f"Patients with multiple records: {patient_dupes['patient_id'].nunique()}")
# Handle duplicates
# Option 1: Keep first
df_deduped = df.drop_duplicates(subset=['patient_id'], keep='first')
# Option 2: Keep most recent
df_sorted = df.sort_values('report_date', ascending=False)
df_deduped = df_sorted.drop_duplicates(subset=['patient_id'], keep='first')Issue 4: Inconsistent Coding
Hide code
# Standardize categorical variables
sex_mapping = {
'M': 'Male', 'Male': 'Male', 'MALE': 'Male', 'm': 'Male',
'F': 'Female', 'Female': 'Female', 'FEMALE': 'Female', 'f': 'Female',
'Unknown': 'Unknown', 'U': 'Unknown', np.nan: 'Unknown'
}
df['sex_clean'] = df['sex'].map(sex_mapping)
# Standardize dates
df['date_clean'] = pd.to_datetime(df['date'], infer_datetime_format=True, errors='coerce')Data Quality Audit: Tactical Guide
The code and decision frameworks below address these issues in practice.
Detecting Data Drift
Data drift occurs when the statistical properties of data change over time. In public health, this happens constantly: testing policies change, new reporting systems deploy, case definitions evolve. Models trained on stable data fail when deployed on drifted data.
Hide code
import numpy as np
from scipy import stats
from sklearn.preprocessing import StandardScaler
def detect_drift(reference_data, current_data, feature_cols,
threshold=0.05, method='ks'):
"""
Detect data drift between reference (training) and current (production) data.
Parameters:
- reference_data: DataFrame from training period
- current_data: DataFrame from recent window
- feature_cols: List of columns to check
- threshold: p-value threshold for drift detection
- method: 'ks' (Kolmogorov-Smirnov) or 'chi2' (categorical)
Returns: Dictionary of drifted features with test statistics
"""
drifted_features = {}
for col in feature_cols:
ref_vals = reference_data[col].dropna()
curr_vals = current_data[col].dropna()
if len(ref_vals) < 30 or len(curr_vals) < 30:
continue # Insufficient data
if reference_data[col].dtype in ['object', 'category']:
# Chi-square test for categorical
ref_counts = ref_vals.value_counts()
curr_counts = curr_vals.value_counts()
# Align categories
all_cats = set(ref_counts.index) | set(curr_counts.index)
ref_aligned = [ref_counts.get(c, 0) for c in all_cats]
curr_aligned = [curr_counts.get(c, 0) for c in all_cats]
stat, p_value = stats.chisquare(
curr_aligned,
f_exp=[r * sum(curr_aligned)/sum(ref_aligned) for r in ref_aligned]
)
else:
# KS test for numerical
stat, p_value = stats.ks_2samp(ref_vals, curr_vals)
if p_value < threshold:
drifted_features[col] = {
'statistic': stat,
'p_value': p_value,
'ref_mean': ref_vals.mean() if ref_vals.dtype != 'object' else None,
'curr_mean': curr_vals.mean() if curr_vals.dtype != 'object' else None
}
return drifted_features
# Example usage: weekly drift monitoring
training_period = df[df['date'] < '2024-01-01']
recent_week = df[df['date'] >= '2024-06-01']
drift_results = detect_drift(
training_period,
recent_week,
feature_cols=['age', 'test_positivity', 'symptom_count']
)
if drift_results:
print("WARNING: Data drift detected in:")
for feature, result in drift_results.items():
print(f" {feature}: p={result['p_value']:.4f}")
if result['ref_mean']:
print(f" Training mean: {result['ref_mean']:.2f}")
print(f" Current mean: {result['curr_mean']:.2f}")When drift is detected:
Feature-level tests can flag distributional changes for investigation, but counts of significant tests and p-values alone are not severity or retraining rules. Predefine monitoring metrics and action thresholds from expected temporal variation, clinical or program consequences, and prospective performance. Investigate multiplicity, sample size, data-pipeline changes, and decision-relevant performance before changing or suspending a model.
Facility-Specific Data Artifacts
Different reporting facilities often have systematic differences that create spurious patterns. A model might learn “Hospital A predicts worse outcomes” when the real pattern is “Hospital A has different coding practices.”
Hide code
def detect_facility_artifacts(df, facility_col, outcome_col,
feature_cols, min_cases=50):
"""
Detect systematic differences across facilities that may indicate
data artifacts rather than true clinical variation.
Returns facilities with suspicious patterns for investigation.
"""
artifacts = {}
# 1. Check for extreme missingness variation
missing_by_facility = df.groupby(facility_col)[feature_cols].apply(
lambda x: x.isnull().mean()
)
for col in feature_cols:
facility_missing = missing_by_facility[col]
overall_missing = df[col].isnull().mean()
# Flag facilities with 3x higher missingness
outlier_facilities = facility_missing[
facility_missing > max(3 * overall_missing, 0.3)
]
if len(outlier_facilities) > 0:
artifacts[f'{col}_missing'] = outlier_facilities.to_dict()
# 2. Check for digit preference (age heaping, etc.)
if 'age' in df.columns:
for facility in df[facility_col].unique():
facility_ages = df[df[facility_col] == facility]['age']
if len(facility_ages) < min_cases:
continue
# Check for 0/5 preference
last_digit = facility_ages % 10
zero_five_pct = last_digit.isin([0, 5]).mean()
if zero_five_pct > 0.4: # Expect 20% if random
if 'age_heaping' not in artifacts:
artifacts['age_heaping'] = {}
artifacts['age_heaping'][facility] = zero_five_pct
# 3. Check for implausible outcome rates
outcome_by_facility = df.groupby(facility_col)[outcome_col].mean()
overall_rate = df[outcome_col].mean()
suspicious = outcome_by_facility[
(outcome_by_facility < overall_rate * 0.25) |
(outcome_by_facility > overall_rate * 4)
]
if len(suspicious) > 0:
artifacts['extreme_outcome_rates'] = suspicious.to_dict()
return artifacts
# Run facility audit
facility_issues = detect_facility_artifacts(
df,
facility_col='reporting_hospital',
outcome_col='hospitalized',
feature_cols=['symptom_onset', 'comorbidity_count', 'age']
)
for issue_type, facilities in facility_issues.items():
print(f"\nIssue: {issue_type}")
for facility, value in facilities.items():
print(f" {facility}: {value:.2%}")Handling Changing Case Definitions Mid-Outbreak
When case definitions change (as happened repeatedly during COVID-19), historical data becomes heterogeneous. Models trained across definition changes learn artifacts, not epidemiology.
Strategy 1: Definition Version Flags
Hide code
# Document definition changes with explicit version markers
DEFINITION_CHANGES = {
'v1': {'start': '2020-01-01', 'end': '2020-03-15',
'criteria': 'lab_confirmed_only'},
'v2': {'start': '2020-03-16', 'end': '2020-07-31',
'criteria': 'lab_or_antigen'},
'v3': {'start': '2020-08-01', 'end': None,
'criteria': 'lab_antigen_or_clinical'}
}
def assign_definition_version(date):
"""Assign case definition version based on report date."""
for version, params in DEFINITION_CHANGES.items():
start = pd.to_datetime(params['start'])
end = pd.to_datetime(params['end']) if params['end'] else pd.Timestamp.max
if start <= date <= end:
return version
return 'unknown'
df['definition_version'] = df['report_date'].apply(assign_definition_version)
# Option A: Include version as feature (model learns definition-specific patterns)
# Option B: Train separate models per definition version
# Option C: Only use data from single definition version (loses history)Strategy 2: Harmonization When Possible
Hide code
def harmonize_to_strictest(df, definition_version_col):
"""
Harmonize to strictest (most specific) case definition.
Cases from more inclusive definitions that don't meet strict
criteria are excluded.
"""
harmonized = df.copy()
# Only keep lab-confirmed cases (strictest common criterion)
harmonized = harmonized[
(harmonized['lab_confirmed'] == 1) |
(harmonized[definition_version_col] == 'v1') # v1 was already strict
]
# Document data loss
excluded = len(df) - len(harmonized)
excluded_pct = excluded / len(df) * 100
print(f"Harmonization excluded {excluded:,} cases ({excluded_pct:.1f}%)")
print(f"Exclusions by version:")
for v in df[definition_version_col].unique():
v_total = len(df[df[definition_version_col] == v])
v_kept = len(harmonized[harmonized[definition_version_col] == v])
print(f" {v}: {v_total - v_kept} excluded of {v_total}")
return harmonizedImputation vs. Exclusion: Decision Framework
The choice between imputing missing values and excluding incomplete records is consequential. Neither is universally correct.
| Consideration | Questions for the analysis plan |
|---|---|
| Missingness mechanism | What subject-matter and data-collection evidence supports MCAR, MAR, or MNAR assumptions? |
| Role of the variable | Is it an outcome, exposure, predictor, confounder, or auxiliary variable? |
| Estimand and model use | Which records and population must the result represent? |
| Available information | Which observed variables can support imputation or weighting? |
| Operational constraints | Can the same inputs and transformation be reproduced at deployment? |
| Uncertainty | Do complete-case, imputed, weighted, and sensitivity analyses lead to materially different conclusions? |
Practical Decision Code:
Hide code
def imputation_recommendation(df, feature_col, outcome_col=None):
"""
Recommend imputation strategy based on missingness analysis.
"""
missing_pct = df[feature_col].isnull().mean() * 100
# Test for MAR vs MCAR
df['_is_missing'] = df[feature_col].isnull()
# Check if missingness correlates with other variables
mar_indicators = []
for col in df.select_dtypes(include=[np.number]).columns:
if col in [feature_col, '_is_missing']:
continue
corr = df['_is_missing'].astype(int).corr(df[col])
if abs(corr) > 0.1:
mar_indicators.append((col, corr))
# Association with observed variables can guide analysis, but does not
# identify MCAR, MAR, or MNAR by itself.
outcome_association = False
if outcome_col and outcome_col in df.columns:
outcome_corr = df['_is_missing'].astype(int).corr(df[outcome_col])
if abs(outcome_corr) > 0.15:
outcome_association = True
df.drop('_is_missing', axis=1, inplace=True)
# Generate recommendation
recommendation = {
'feature': feature_col,
'missing_pct': missing_pct,
'mechanism': 'not identified from observed correlations alone',
'mar_predictors': mar_indicators[:3] if mar_indicators else None
}
recommendation['strategy'] = 'analysis_plan_required'
recommendation['method'] = (
'Compare justified complete-case, imputation, weighting, and sensitivity '
'analyses using subject-matter knowledge and the intended estimand.'
)
recommendation['outcome_association'] = outcome_association
return recommendation
# Example usage
for col in ['symptom_onset', 'age', 'comorbidity_count']:
rec = imputation_recommendation(df, col, outcome_col='hospitalized')
print(f"\n{col}:")
print(f" Missing: {rec['missing_pct']:.1f}%")
print(f" Mechanism: {rec['mechanism']}")
print(f" Strategy: {rec['strategy']}")Anomaly Detection with Reporting Delay Awareness
Standard anomaly detection fails on surveillance data because reporting delays create artificial patterns. A spike detected today might be reporting catchup, not a real outbreak.
Hide code
def delay_aware_anomaly_detection(df, date_col, count_col,
delay_distribution, lookback_days=28):
"""
Detect anomalies accounting for reporting delays.
Parameters:
- df: Time series with date and count columns
- delay_distribution: Dict mapping days_lag -> proportion reported
e.g., {0: 0.2, 1: 0.3, 2: 0.25, 3: 0.15, 4: 0.10}
- lookback_days: Historical window for baseline
Returns: DataFrame with nowcast estimates and anomaly flags
"""
df = df.copy().sort_values(date_col)
# Calculate nowcast adjustment factors
max_delay = max(delay_distribution.keys())
cumulative_reported = {}
cumsum = 0
for d in range(max_delay + 1):
cumsum += delay_distribution.get(d, 0)
cumulative_reported[d] = cumsum
# Apply nowcast adjustment to recent dates
today = df[date_col].max()
df['days_old'] = (today - df[date_col]).dt.days
df['nowcast_adjustment'] = df['days_old'].apply(
lambda d: 1 / cumulative_reported.get(min(d, max_delay), 1)
)
df['nowcast_count'] = df[count_col] * df['nowcast_adjustment']
# Calculate baseline from fully-reported historical period
stable_period = df[df['days_old'] > max_delay + 7]
if len(stable_period) < lookback_days:
raise ValueError("Insufficient historical data for baseline")
baseline_mean = stable_period[count_col].tail(lookback_days).mean()
baseline_std = stable_period[count_col].tail(lookback_days).std()
# Flag anomalies (nowcast exceeds baseline by >2 SD)
df['z_score'] = (df['nowcast_count'] - baseline_mean) / baseline_std
df['anomaly'] = df['z_score'] > 2.0
df['anomaly_confidence'] = 'high'
# Lower confidence for very recent dates (nowcast uncertain)
df.loc[df['days_old'] < 3, 'anomaly_confidence'] = 'low'
df.loc[(df['days_old'] >= 3) & (df['days_old'] < max_delay), 'anomaly_confidence'] = 'medium'
return df[[date_col, count_col, 'days_old', 'nowcast_count', 'z_score',
'anomaly', 'anomaly_confidence']]
# Example: typical respiratory illness reporting delays
typical_delays = {0: 0.15, 1: 0.25, 2: 0.25, 3: 0.20, 4: 0.10, 5: 0.05}
results = delay_aware_anomaly_detection(
daily_counts,
date_col='report_date',
count_col='case_count',
delay_distribution=typical_delays
)
# Show recent anomalies with confidence levels
recent_anomalies = results[results['anomaly'] & (results['days_old'] < 14)]
print(recent_anomalies[['report_date', 'case_count', 'nowcast_count',
'z_score', 'anomaly_confidence']])
Data Quality Audit Checklist
Run these checks before any modeling project:
Document findings in a data quality report before proceeding.
Feature Engineering for Public Health
Domain expertise matters most, as Domingos emphasizes.
Time-Based Features
Hide code
# Extract temporal features
df['date'] = pd.to_datetime(df['report_date'])
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day_of_week'] = df['date'].dt.dayofweek
df['week_of_year'] = df['date'].dt.isocalendar().week
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
# Seasonal indicators
def assign_season(month):
if month in [12, 1, 2]:
return 'Winter'
elif month in [3, 4, 5]:
return 'Spring'
elif month in [6, 7, 8]:
return 'Summer'
else:
return 'Fall'
df['season'] = df['month'].apply(assign_season)
# Days since outbreak start
outbreak_start = df['date'].min()
df['days_since_start'] = (df['date'] - outbreak_start).dt.daysRolling Statistics
Hide code
# Sort by date first!
df = df.sort_values('date')
# 7-day rolling average
df['cases_7day_avg'] = df['daily_cases'].rolling(window=7, min_periods=1).mean()
# 14-day rolling sum
df['cases_14day_sum'] = df['daily_cases'].rolling(window=14, min_periods=1).sum()
# Growth rate
df['case_growth_rate'] = df['daily_cases'].pct_change(periods=7)
# Lag features
df['cases_lag_7'] = df['daily_cases'].shift(7)
df['cases_lag_14'] = df['daily_cases'].shift(14)
# Ratio features
df['cases_this_week_vs_last'] = df['daily_cases'] / df['cases_lag_7']
Avoiding Data Leakage
NEVER use future information, as warned in Kaufman et al.’s analysis:
Hide code
# WRONG: Uses future data
df['cases_rolling'] = df['daily_cases'].rolling(window=7, center=True).mean()
# CORRECT: Only past data
df['cases_rolling'] = df['daily_cases'].rolling(window=7, min_periods=1).mean()For time series splits:
Hide code
# WRONG for time series
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)
# CORRECT for time series
split_date = df['date'].quantile(0.8)
train_df = df[df['date'] < split_date]
test_df = df[df['date'] >= split_date]Epidemiological Features
Hide code
# R0 estimation from growth rate
def estimate_R0(cases, generation_time=5):
"""Estimate R0 from exponential growth"""
from scipy.optimize import curve_fit
days = np.arange(len(cases))
def exponential(x, a, b):
return a * np.exp(b * x)
params, _ = curve_fit(exponential, days, cases, p0=[1, 0.1])
growth_rate = params[1]
R0 = 1 + growth_rate * generation_time
return R0
df['R0_estimate'] = df['daily_cases'].rolling(window=14).apply(
lambda x: estimate_R0(x.values) if len(x) > 7 else np.nan,
raw=False
)
# Attack rate
population = 1000000
df['attack_rate'] = df['cumulative_cases'] / population
# Case fatality rate
df['CFR'] = df['cumulative_deaths'] / df['cumulative_cases']
# Test positivity rate
df['test_positivity'] = df['positive_tests'] / df['total_tests']Geographic Features
Hide code
# Distance to healthcare
from sklearn.metrics.pairwise import haversine_distances
case_coords = df[['latitude', 'longitude']].values
hospital_coords = hospitals[['lat', 'lon']].values
# Convert to radians
case_coords_rad = np.radians(case_coords)
hospital_coords_rad = np.radians(hospital_coords)
# Calculate distances (km)
distances = haversine_distances(case_coords_rad, hospital_coords_rad) * 6371
df['nearest_hospital_km'] = distances.min(axis=1)
# Spatial clustering
from sklearn.neighbors import NearestNeighbors
nn = NearestNeighbors(radius=5.0 / 6371, metric='haversine') # haversine works in radians: 5 km / Earth radius
nn.fit(case_coords_rad)
distances, indices = nn.radius_neighbors(case_coords_rad)
df['cases_within_5km'] = [len(idx) - 1 for idx in indices]Case Study: COVID-19 Forecasting Failures
The COVID-19 pandemic provided a natural experiment in real-time forecasting. In the U.S. Forecast Hub evaluation, ensemble forecasts outperformed individual models on average, while error increased with forecast horizon (Cramer et al., 2022).
What went wrong:
- Reporting delay ignorance
- Models treated “reported March 15” as “occurred March 15”
- Reality: reporting lags of several days that varied by jurisdiction
- Result: Systematic underestimation
- Changing testing regime
- Week 1: Only hospitalized patients
- Week 8: Asymptomatic screening
- Models confused testing expansion with disease growth
- Selection bias
- Early models trained on Chinese data (strict lockdowns)
- Applied to U.S./Europe (different compliance)
- External validity failed
- Missing confounders
- Behavioral responses not in feature set
- Policy changes (masks, closures) ignored
- Human adaptation breaks stationarity
The numbers:
- Auxiliary indicator analysis: A plain autoregressive model using no external features was already competitive with many of the top COVID-19 forecasting models in use
- IHME model: Repeatedly revised by orders of magnitude
- Ensemble methods outperformed individuals, but still struggled
Lessons learned:
Data Quality > Model Sophistication
A plain autoregressive model using no external features was already competitive with many of the top COVID-19 forecasting models in use, and adding auxiliary data streams produced gains that varied by trend direction rather than transforming accuracy (McDonald et al., 2021).
The pandemic exposed public health data infrastructure as the main bottleneck.
When Data Problems Make AI Inappropriate
Red flags that should stop a project:
- Extreme selection bias with no adjustment
- Example: Training on hospitalized, deploying on general population
- Why fatal: Model does not generalize
- Outcome label quality is poor
- Example: Inconsistent disease diagnosis criteria
- Why fatal: Model learns noise, not signal
- Critical features have unresolved missingness
- Example: Symptom onset missing for most cases
- Why fatal: Cannot build temporal features
- Data generating process changed mid-dataset
- Example: Pre/post policy change, different testing
- Why fatal: Violates stationarity assumption
- Sample size too small
- Example: 100 cases, 50 features, trying deep learning
- Why fatal: Severe overfitting
- External validity concerns
- Example: One hospital’s data, deploying elsewhere
- Why fatal: Context matters; the model does not transfer
The Courage to Say “No”
Good data science includes knowing when NOT to build a model.
If data quality is insufficient:
- Report limitations clearly
- Recommend improving data collection
- Suggest alternatives (traditional epi, qualitative research)
Building on garbage data:
- Wastes resources
- Produces misleading results
- Can harm people
- Damages trust in AI
As Box said: “All models are wrong, but some are useful.” If your data is wrong enough, your model will be useless.
Key Takeaways
Data quality determines model quality. Algorithms cannot overcome flawed data.
Understand missingness mechanisms. MCAR, MAR, MNAR require different strategies.
Public health data has unique challenges: selection bias, reporting delays, changing surveillance.
Investigate, do not automatically fix. Outliers might be the most important cases.
Domain and technical expertise are complementary. Domain knowledge guides feature validity; comparative validation tests whether model complexity adds value.
Document everything. Future users need to know what you cleaned and why.
Know when to stop. Sometimes the ethical choice is not to build a model.
Temporal ordering matters. Never use future information to predict the past.
Practice Exercises
Exercise 1: Missing Data Mechanisms
- Identify variables with missing data
- Assess whether MCAR, MAR, or MNAR assumptions are plausible
- Choose imputation strategies
- Compare model performance
Exercise 2: Reporting Delay Adjustment
- Calculate reporting triangle
- Implement nowcasting
- Compare nowcast vs. raw counts
- Evaluate accuracy over time
Exercise 3: Feature Engineering
Create an outbreak predictor using:
- Daily cases
- Temperature
- Rainfall
- Population density
Engineer 10+ features. Which matter most?
Exercise 4: Data Quality Audit
Audit a “cleaned” dataset:
- Find 5+ quality issues
- Determine if fixable or fatal
- Write a quality report
Check Your Understanding
Test your knowledge of the key concepts from this chapter. Click “Show Answer” to reveal the correct response and explanation.
Question 1: Data Quality Foundations
Which dimension of data quality is violated when COVID-19 testing was initially limited to hospitalized patients, but models were deployed to predict community transmission?
- Completeness
- Accuracy
- Timeliness
- Representativeness
Answer: d) Representativeness
Explanation: This is a representativeness problem. The training data (hospitalized patients) does not represent the target population (community cases). This selection bias means the model learns patterns from a severely ill subpopulation that do not generalize to mild/asymptomatic cases in the community.
Question 2: Missing Data Mechanisms
You notice that income data is missing for 30% of younger respondents but only 5% of older respondents. What type of missingness is this?
- Missing Completely At Random (MCAR)
- Missing At Random (MAR)
- Missing Not At Random (MNAR)
- Systematic error
Answer: b) Missing At Random (MAR)
Explanation: This is MAR (Missing At Random) because the missingness depends on an observed variable (age), not on the missing value itself (income). You can impute income using age as a predictor since you know the missingness pattern.
Question 3: Temporal Data Leakage
True or False: When creating rolling averages for time series forecasting, using rolling(window=7, center=True) is appropriate because it creates smoother trends.
Answer: False
Explanation: Using center=True causes data leakage by incorporating future values into the calculation. For a value on day 10, it would include data from days 7–13, meaning you’re using days 11–13 (the future) to predict day 10. For forecasting, you must use center=False (the default) to only look backwards in time.
Question 4: Reporting Delays
A surveillance system shows cases declining on the most recent dates. What is the most likely explanation before concluding the outbreak is ending?
- Effective public health intervention
- Population-level immunity developing
- Right-censoring from reporting delays
- Seasonal weather changes
Answer: c) Right-censoring from reporting delays
Explanation: Right-censoring from reporting delays is the most common explanation for apparent declines in recent data. Cases from recent days have not fully been reported yet, creating an artificial downward trend. Always check reporting lag distributions before interpreting recent trends. This is why nowcasting methods exist.
Question 5: Feature Engineering
Which of these features would be MOST useful for predicting dengue outbreaks and demonstrates good domain expertise?
- Day of the week
- Raw daily temperature
- 14-day cumulative rainfall
- Month number (1–12)
Answer: c) 14-day cumulative rainfall
Explanation: 14-day cumulative rainfall shows domain expertise because dengue mosquitoes (Aedes) breed in standing water that accumulates over time. Raw daily temperature or simple month number do not capture the mechanism. Day of the week is irrelevant for disease biology. This demonstrates how epidemiological understanding drives effective feature engineering.
Question 6: Outliers
You find 5 patients with “impossibly fast” disease progression (symptom onset to death in <24 hours). What should you do?
- Remove them as data entry errors
- Remove them to avoid overfitting on outliers
- Investigate them carefully; they might be important rare cases
- Set their values to the median progression time
Answer: c) Investigate them carefully; they might be important rare cases
Explanation: Investigate carefully before removing. These could be:
- Real rare but critical cases (e.g., cytokine storm, overwhelming infection)
- Important subpopulation with different risk factors
- Data entry errors (date fields swapped)
The COVID-19 pandemic showed that “outliers” like rapidly progressing cases were often the most clinically important to understand. Only remove after investigation confirms they’re truly errors.
Discussion Questions
Your training data is 80% from one hospital. How does this affect generalizability? What can you do?
A model has 95% accuracy on training, 65% on test, with 30% missing labels. How do you interpret this?
A colleague wants to impute symptom onset as report date minus median delay. Pros/cons? When appropriate?
Forecasting COVID-19 hospitalizations: use raw counts or adjust for test positivity? Justify.
When is it acceptable to remove outliers from public health data? Examples of appropriate vs. inappropriate removal.
Further Reading
Data Quality Frameworks
- Weiskopf, N.G., & Weng, C. (2013). “Methods and dimensions of EHR data quality.” JAMIA, 20(1), 144-151.
- Kahn, M.G., et al. (2016). “Harmonized data quality terminology.” JAMIA, 23(6), 1244-1251.
Missing Data
- Little, R.J.A., & Rubin, D.B. (2019). Statistical Analysis with Missing Data. Wiley.
- van Buuren, S. (2018). Flexible Imputation of Missing Data. CRC Press.
Surveillance Challenges
- Lipsitch, M., et al. (2020). “Defining Covid-19 epidemiology.” Science, 368(6490), 565-568.
- Reich, N.G., et al. (2019). “Collaborative flu forecasting assessment.” PNAS, 116(8), 3146-3154.
COVID-19 Lessons
- Cramer, E.Y., et al. (2022). “Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.” PNAS, 119(15), e2113561119.
- McGough, S.F., et al. (2020). “Nowcasting by Bayesian smoothing.” PLoS Comp Bio, 16(10), e1008785.
Data Leakage
- Kaufman, S., et al. (2012). “Leakage in data mining.” ACM TKDD, 6(4), 1-21.
You now understand why data quality is the foundation of successful AI in public health. Clean data and thoughtful feature engineering matter more than fancy algorithms.
Next chapter: AI in Disease Surveillance and Outbreak Detection