Executive Summary
This handbook addresses a gap in public health practice: understanding when and how AI tools work, when they fail, and what infrastructure is needed for responsible deployment. The evidence comes from peer-reviewed research, documented deployments, published evaluations, and implementation case studies.
Key Findings
AI Capabilities in Public Health Are Real, But Limited
Disease surveillance benefits from computational signal detection, with important caveats. Event-based systems such as HealthMap and ProMED and commercial systems such as BlueDot contributed early signals during COVID-19, but the published record does not support treating every alert as an independent AI detection (Bogoch et al., 2020). Models trained on historical data can struggle when pathogen dynamics, behavior, testing, or reporting change. Forecast skill generally weakens as the horizon increases, but the useful horizon depends on the target, data, epidemic phase, and decision.
Google Flu Trends remains the defining cautionary tale. Its original validation showed that search-based estimates could be produced one to two weeks before CDC influenza-like illness reports (Ginsberg et al., 2009). The system later overestimated influenza prevalence, including an estimate more than twice the CDC value during the 2012–2013 season as search behavior and the underlying service changed (Lazer et al., 2014). Google ended public Flu Trends estimates in 2015. The lesson: digital surveillance tools must account for behavioral feedback loops, changes to source platforms, and continuous recalibration.
Genomic surveillance has proven operational value. During SARS-CoV-2, genomic data helped identify and characterize emerging lineages before complete phenotypic evidence was available. Antimicrobial-resistance prediction from whole-genome sequencing can perform well for specific pathogen, drug, and resistance-mechanism combinations, but performance does not transfer automatically across organisms, drugs, or settings.
Clinical AI shows mixed results in deployment. In one external validation of the Epic Sepsis Model, sensitivity was 33%, specificity 83%, and positive predictive value 12% at the evaluated threshold (Wong et al., 2021). Those results belong to that model version, threshold, site, and study period; they should not be transferred to later versions or other settings.
Common Failure Modes Are Predictable
Dataset shift is a common source of deployed-model failure. Models trained on one population, time period, or data source can underperform when conditions change. COVID-19 testing patterns, case definitions, and reporting practices shifted repeatedly, undermining models trained on early pandemic data.
Algorithmic bias reflects training data and measurement inequities. Dermatology models have shown performance gaps across skin tones and evaluation datasets (Daneshjou et al., 2022). Chest radiograph models can also learn site-specific signals that reduce transportability across hospitals (Zech et al., 2018). Language and cultural coverage require the same population-specific evaluation.
Implementation conditions often determine whether technical performance produces public health value. Tools that perform well in validation can fail in practice because alerts arrive at the wrong point in a workflow, users cannot interpret outputs, data pipelines differ from development conditions, or no accountable response pathway exists.
Infrastructure Requirements Are Substantial
Data quality is the binding constraint. Public health data is messy: inconsistent case definitions, variable completeness, reporting delays, duplicate records, linkage errors. AI systems amplify data quality problems rather than solving them. Detection of data drift (when production data diverges from training data) requires statistical methods (Kolmogorov-Smirnov tests, chi-square tests) that most health departments lack capacity to implement. See the Data Quality chapter for tactical detection guides.
Cost is an underappreciated barrier. A defensible total-cost-of-ownership estimate must include more than the license or model. Relevant costs include data acquisition and cleaning, integration, validation, security review, workforce training, monitoring, incident response, model or vendor changes, and exit or replacement. Generic price ranges and return-on-investment timelines should not be transferred across agencies or use cases without a local cost model.
Skilled workforce gaps limit adoption. Health departments need staff who understand both epidemiology and AI capabilities. This hybrid expertise remains rare. Training programs are emerging but scaling slowly.
Computational infrastructure varies widely. State and local health departments operate with vastly different technical capacity. Cloud computing democratizes some capabilities but introduces data governance complexity.
Recommendations by Stakeholder
For Health Department Directors
Require external validation before deployment. Vendor-reported performance metrics are insufficient. Demand validation on your population, with your data quality, under your operational conditions. Budget for pilot studies.
Build evaluation capacity, not just procurement capacity. Your staff needs to assess AI claims critically. Invest in training that covers:
- Performance metrics interpretation
- Bias detection
- Deployment monitoring
- Vendor evaluation
Start with augmentation, not automation. AI tools should support human decision-making before replacing it. Clinicians and epidemiologists need to understand model outputs, recognize errors, and maintain override authority.
Plan for maintenance from day one. Deployed models degrade. Budget for ongoing monitoring, recalibration, and potential replacement. A tool that works today may fail next year.
Budget realistically for total cost of ownership. Include data pipelines, integration, validation, staff training, monitoring, change management, and vendor-exit costs. Establish discontinuation criteria upfront: if the tool fails to meet prespecified performance or safety thresholds within defined timelines, sunset it rather than continuing sunk-cost investment.
For Epidemiologists and Data Scientists
Validate on held-out data that reflects deployment conditions. Time-series split validation (train on earlier data, test on later data) matters more than random cross-validation for surveillance applications. Geographic holdouts reveal transferability.
Document data preprocessing decisions. Choices about missing data imputation, outlier handling, case definition changes, and temporal alignment affect model behavior. Future users need this documentation to interpret and update models.
Measure calibration, not just discrimination. A model that distinguishes high-risk from low-risk (good AUC) may still systematically overestimate or underestimate probabilities (poor calibration). For decision support, calibration often matters more.
Report confidence intervals and uncertainty. Point predictions without uncertainty bounds invite overconfidence. Probabilistic forecasts that communicate uncertainty support better decisions than precise-sounding wrong answers.
For Policymakers
Use risk- and outcome-focused requirements while retaining system-specific controls. Technology-specific labels become obsolete quickly, but governance still needs requirements tied to intended use, evidence, data, human oversight, change control, and measurable outcomes:
- Accuracy requirements
- Bias audits
- Transparency obligations
- Recourse mechanisms for affected individuals
Require algorithmic impact assessments for high-stakes applications. Before deployment in clinical decision support, resource allocation, or outbreak response, mandate structured evaluation of potential harms, affected populations, and mitigation strategies.
Fund public health data infrastructure. AI capabilities depend on data quality. Investment in electronic laboratory reporting, case-based surveillance systems, and health information exchange infrastructure pays dividends across all analytic approaches.
Support workforce development. Create training pipelines for public health informaticists. Integrate AI literacy into MPH curricula. Fund continuing education for current practitioners.
Monitor evolving FDA guidance. FDA issued updated final guidance on Clinical Decision Support (CDS) software in January 2026 (issued January 6, 2026 and re-issued January 29, 2026), clarifying FDA’s interpretation of which CDS software functions are excluded from the definition of a device under section 520(o)(1)(E) (Non-Device CDS criteria) and providing examples (FDA CDS Guidance, Jan 2026; PDF; FDA CDS FAQs). This affects procurement decisions and liability frameworks. Regulatory landscapes shift rapidly; policymakers should track FDA, FTC, and state-level developments.
For AI Developers Working in Public Health
Involve domain experts from the start. Epidemiologists and public health practitioners understand data limitations, workflow constraints, and deployment contexts that pure ML approaches miss. Build interdisciplinary teams, not consulting relationships.
Publish validation on diverse populations. Single-site validations do not demonstrate generalizability. Multi-site studies with explicit subgroup analyses reveal where models work and where they fail.
Document failure modes explicitly. Under what conditions does your model break? What populations are underrepresented in training data? What data quality issues cause problems? Honest documentation builds appropriate trust.
Design for interpretability where decisions affect individuals. Black-box models may achieve higher accuracy, but practitioners need to understand predictions to act appropriately on them. For high-stakes applications, interpretability is not optional.
Evidence Quality Assessment
High-Confidence Findings
These conclusions rest on multiple high-quality studies with consistent results:
- Dataset shift degrades deployed model performance
- Algorithmic bias reflects training data inequities
- External validation can reveal performance loss or calibration failure that development validation misses
- Workflow integration determines real-world impact more than technical performance
Moderate-Confidence Findings
Evidence supports these conclusions but gaps remain:
- Forecast skill generally decreases as the horizon increases, but decision-useful horizons vary by target, disease, data, and epidemic phase
- Diverse ensembles often improve robustness, but each horizon and target requires prospective comparison with individual models and simple baselines
- Active learning can reduce labeling burden while maintaining performance
Emerging Areas (Limited Evidence)
These represent promising directions with insufficient evidence for strong conclusions:
- Large language models for epidemiological literature synthesis (performance can decline when relevant information is buried within long inputs, so position, retrieval, and citation checks require task-specific evaluation)
- Foundation models for clinical decision support
- Federated learning for privacy-preserving multi-site analysis
- Causal inference methods for policy evaluation
- Autonomous AI agents for public health tasks (tool permissions, third-party skills, side effects, and unmonitored decision pathways require explicit security and authority testing)
Priority Investment Areas
An evidence-informed sequencing approach is:
Tier 1: Foundational
- Data quality improvement: Invest in data standardization, completeness monitoring, and linkage infrastructure before advanced analytics
- Workforce development: Build hybrid epidemiology/data science expertise through structured training programs
- Evaluation capacity: Establish methods for ongoing model monitoring and bias detection
Tier 2: Near-Term Applications
- Genomic surveillance integration: AI-accelerated variant detection and antimicrobial resistance prediction have demonstrated value
- Syndromic surveillance enhancement: Automated signal detection with human-in-the-loop verification
- Administrative automation: Reduce burden on routine tasks (data cleaning, report generation) to free staff for analytic work
Tier 3: Longer-Term Development
- Clinical decision support: Requires solving workflow integration and trust challenges first
- Predictive resource allocation: Depends on forecast accuracy improvements and ethical frameworks
- Automated literature surveillance: Promising for tracking emerging evidence but validation needed
What This Handbook Does Not Cover
Clinical AI for individual patient care. This handbook’s primary focus is population health applications. Clinical decision support is covered where it intersects with public health operations (e.g., sepsis alerts affecting hospital capacity, diagnostic AI enabling screening at scale). However, detailed treatment protocols, individual prescribing decisions, and specialty-specific clinical workflows are beyond scope.
Research methodology for developing new AI systems. The focus is on evaluation, deployment, and governance of existing approaches, not novel algorithm development.
Production-grade technical implementation. Code examples demonstrate concepts but production systems require engineering expertise beyond this handbook’s scope.
This executive summary is part of The Public Health AI Handbook. For evidence citations, case studies, and technical details, see the full chapters.