Appendix E: Case Study Library - Overview
A collection of documented public health AI cases and explicitly labeled teaching scenarios, examining successes, failures, and implementation tradeoffs. Reported evidence is separated from illustrative code, outcome tables, and economic models.
- Quick scan: Review the summary table and key themes below
- Specific domain: Jump to relevant sections using the navigation links
- Deep dive: Read the complete case studies
- Code implementation: Cases include illustrative Python-style pseudocode and templates
Case Studies at a Glance
| # | Case Study | Domain | Outcome | Key Metric |
|---|---|---|---|---|
| 1 | BlueDot COVID-19 Detection | Surveillance | Documented alert | Client alert on December 31, 2019 |
| 2 | Google Flu Trends | Surveillance | Failure and discontinuation | More than twofold overestimation (2012–2013) |
| 3 | ProMED + HealthMap | Surveillance | Complementary workflow | Automated aggregation plus expert interpretation |
| 4 | IDx-DR Autonomous Diagnostic | Diagnostics | Authorized device | FDA De Novo classification, 2018 |
| 5 | DeepMind AKI Work | Diagnostics | Distinct systems | Streams was rule-based; VA model was retrospective |
| 6 | Breast Cancer AI | Diagnostics | Mixed | Performance varies by product, population, equipment, threshold, and workflow |
| 7 | Sepsis RL Treatment | Treatment | Controversial | RCT needed |
| 8 | COVID-19 Prediction Models | Treatment | Mostly failed | All 232 at high or unclear risk of bias |
| 9 | Ventilator Allocation | Resources | Ethical and policy analysis | Human review, transparency, and appeals emphasized |
| 10 | Allegheny Child Welfare | Population Health | Controversial | Calibration and threshold-based group differences |
| 11 | NHS-Inspired Disparity Audit | Population Health | Teaching scenario | Illustrative equity-audit workflow |
| 12 | Hospital Bed Allocation | Health Economics | Teaching scenario | Illustrative optimization and ROI model |
| 13 | Crisis Text Line | Mental Health | Documented platform plus teaching scenario | Illustrative evaluation metrics |
| 14 | AlphaFold Drug Discovery | Drug Discovery | Research and development programs | Product-specific evidence required |
| 15 | Project ECHO + AI | Rural Health | Teaching scenario | Hypothetical AI extension |
| 16 | MomConnect | Maternal Health | National platform | SMS messaging with NLP question matching and human escalation |
The full cases provide illustrative pseudocode and source-linked evidence.
Part I: Disease Surveillance and Outbreak Detection
Case Study 1: BlueDot - Early COVID-19 Detection
Context: Automated global disease surveillance using news reports, flight data, and climate patterns.
Key Achievement:
- Alerted clients on December 31, 2019, the same date WHO’s China Country Office identified the Wuhan notice and before WHO’s January 4 public communication (WHO, 2024)
- Predicted initial spread destinations (Bangkok, Hong Kong, Tokyo, Seoul, Singapore)
Limitations:
- Could not predict pandemic severity or spread dynamics
- Early warning alone insufficient without policy action
Technical Highlights:
- Multi-source data integration (news in 65 languages, flight networks, climate data)
- NLP for disease mention extraction
- Geographic risk scoring
- 24/7 automated monitoring
Lesson: AI excels at early detection, not prediction of impact
Case Study 2: Google Flu Trends - Rise and Fall
Context: Search query-based flu surveillance (2008–2015)
Timeline:
- 2008–2011: Accurate predictions (correlation 0.90 with CDC data)
- 2012–2013: Estimates exceeded CDC data more than twofold at the peak
- 2013–2015: Continued overestimation; public estimates ended in 2015
Why It Failed:
- Correlation without causation and an unpublished methodology
- Search behavior changes (media coverage effects)
- No update mechanism (model drift)
- Overfitting to data artifacts
Proposed Remedies:
- Combined with CDC data (ensemble approach)
- Increased transparency
- Regular recalibration
- Acknowledged limitations
Lesson: Correlation-based models need ongoing validation, transparency, and update mechanisms
Case Study 3: ProMED-mail + HealthMap - Human-AI Collaboration
Context: Hybrid human-AI disease surveillance system
Complementary Functions:
- ProMED provides expert-curated outbreak reports and moderator context
- HealthMap aggregates and classifies online disease reports
- A combined workflow requires explicit review criteria and evaluation
Key Innovation: Automated collection can extend breadth while experts provide contextual interpretation
Lesson: Humans + AI > Either alone
Part II: Diagnostic AI
Case Study 4: IDx-DR - First Autonomous AI Diagnostic
Regulatory Milestone:
- First FDA-authorized autonomous diagnostic AI (April 2018)
- Provides a screening result without clinician image interpretation
Performance:
- Sensitivity: 87.2%
- Specificity: 90.7%
- Clinical trial: 900 patients, 10 sites
Real-World Challenges:
- Performance can change with image quality, population, camera, and workflow
- Reimbursement challenges
Regulatory Pathway:
- FDA De Novo classification
- Extensive clinical validation
- Post-market surveillance requirements
Lesson: Autonomous AI possible but requires extensive validation and monitoring
Case Study 5: DeepMind AKI Work - Two Distinct Systems
Evidence boundary:
- Royal Free Streams: A clinically deployed app using the NHS national rule-based AKI algorithm, not machine learning
- VA prediction model: A retrospective recurrent neural network trained on 703,782 US Veterans Affairs patients
- Reported performance: Predicted 55.8% of inpatient AKI and 90.2% of dialysis-requiring AKI up to 48 hours in advance at two false alerts per true alert (Tomašev et al., 2019)
- Clinical impact: Not established by a prospective deployment trial
Why the distinction matters:
- Royal Free’s data-sharing controversy concerns governance of Streams
- The VA model’s reported metrics concern retrospective prediction
- Alert burden, workflow integration, and patient outcomes require separate prospective evidence
Critical Insight: Retrospective prediction performance does not establish clinical utility
Lessons for the Future:
- Design with clinicians, not for them
- Provide actionable recommendations, not just predictions
- Integrate into existing workflows
- Clear value proposition required
Case Study 6: Breast Cancer Detection - Inconsistent Results
Multiple Systems Evaluated:
- Google Health/DeepMind
- Lunit INSIGHT MMG
- iCAD ProFound AI
Performance Variability:
- Performance changes are product-, population-, equipment-, threshold-, and workflow-specific
- Internal validation does not establish transportability
- Multi-site, multi-equipment testing is required before local deployment
Success Story:
- Sweden (Lund): 44% radiologist workload reduction, maintained detection rate
- Key: AI as concurrent reader, not replacement
Lesson: Internal validation insufficient; need multi-site, multi-equipment testing
Part III: Treatment Optimization
Case Study 7: Sepsis Treatment - AI-RL Controversy
The AI Clinician (MIT):
- Learned treatment policy from 100,000 ICU patients
- Recommended less fluid than standard guidelines
- Controversial: Observational data are subject to confounding
The Confounding Problem:
- Sicker patients receive more aggressive treatment → worse outcomes
- AI learns: More treatment → Worse outcomes (confounded)
- Reality: Treatment could not overcome initial severity
Current Status:
- The AI Clinician evidence remains retrospective and observational
- Prospective comparative evidence of clinical benefit is still required
Lesson: Reinforcement learning on observational data is hypothesis-generating, not practice-changing without prospective comparative validation
Case Study 8: COVID-19 Prediction Models - Limited Impact
The Pandemic Rush:
- 232 COVID models published by October 2020
- 98% had high risk of bias
- Only 1 externally validated with low bias
- Most never used clinically
Common Problems:
- Small sample sizes (<500 patients)
- Lack of external validation
- Poor reporting standards
- Overfitting
Models That Worked:
- 4C Mortality Score (UK): 35,000 patients, multiple sites, simple and interpretable
- ISARIC-4C: 75,000 patients, properly validated
Lesson: Urgency does not justify poor methods. Simple, validated models > complex, unvalidated ones
Part IV: Resource Allocation
Case Study 9: Ventilator Allocation - Ethics Meets AI
The Dilemma:
- COVID-19 ventilator shortages required triage decisions
- AI systems proposed for allocation
- Published triage frameworks, such as Pittsburgh’s, used clinical assessment instead
The Trilemma (Cannot maximize all three):
- Utility (save most lives)
- Fairness (equal treatment)
- Autonomy (individual rights)
Reasons to Limit Algorithmic Allocation:
- Insufficient accuracy (COVID-19 survival models had C-statistics of 0.70–0.80)
- Bias concerns (perpetuate historical inequities)
- Legal risks (disability discrimination)
- Trust and legitimacy issues
Features of Published Triage Frameworks:
- Human clinical assessment with ethical oversight
- Triage officers (experienced clinicians)
- Appeals process
- Re-evaluation every 48–120 hours
Lesson: Some decisions should remain human. AI can inform but not decide life-or-death allocation.
Part V: Population Health and Health Equity
Case Study 10: Allegheny Family Screening Tool - Algorithmic Child Welfare
Context: Risk assessment for child welfare referrals (used since 2016)
Performance:
- Predicts child removal risk (AUC 0.76)
- Used by caseworkers to prioritize investigations
Fairness Findings:
- The audit found broadly similar calibration patterns across groups
- Threshold-based placement and error rates varied, with the most pronounced false-positive-rate imbalance reported for the Mixed-race subgroup
- Group-specific errors require direct measurement rather than inference from overall accuracy
The Feedback Loop Problem:
- Historical over-surveillance of Black and low-income families
- More system contact → Higher risk scores
- Higher scores → More investigation
- Cycle perpetuates
Responses:
- Public documentation and transparency
- Community engagement
- Regular fairness audits
- Human override capability maintained
Ongoing Debate:
- Supporters: More consistent than unaided human judgment
- Critics: Automates and scales existing discrimination
- Both perspectives have validity
Lesson: Historical bias in data perpetuates inequality. Transparency and community input are essential.
Case Study 11: Hypothetical NHS-Inspired Disparity Audit
Documented UK COVID-19 disparities motivate this teaching scenario. The AI implementation, care-pathway findings, interventions, and outcome figures below are illustrative, not reported NHS results.
Illustrative premise:
- AI identified disparities in human care delivery, not AI decisions
- Used as diagnostic tool for systemic racism
- Findings led to concrete policy changes
Illustrative disparities found:
- Black patients: 2.5x mortality rate (1.8x after adjusting for comorbidities)
- 8-hour longer admission wait times for Black patients
- Lower ICU admission rates despite similar severity
- Lower guideline-concordant care rates
Illustrative interventions:
- Enhanced translation services (24/7 availability)
- Cultural competency training (mandatory)
- Community health workers
- Care pathway standardization
- Real-time disparity monitoring dashboards
Illustrative results after two years:
- Admission disparities reduced 40%
- ICU access disparities reduced 25%
- Mortality disparities reduced 15%
- Disparities persist despite measurable progress
Lesson: AI can expose systemic problems for intervention. Used correctly, it is a tool for justice, not just a source of bias.
Part VI: Health Economics
Case Study 12: Hypothetical AI-Driven Hospital Bed Allocation
This Johns Hopkins-inspired teaching scenario preserves an optimization and economic-analysis template. The results, equity effects, ROI, and named replication sites are illustrative, not reported Johns Hopkins outcomes.
Illustrative implementation:
Challenge: Balance competing objectives:
- Efficiency (maximize utilization)
- Access (minimize wait times)
- Quality (appropriate care level)
- Equity (fair access across populations)
Illustrative results:
- Bed utilization: 82% → 88% (+6 percentage points)
- ED wait times: 4.2 → 3.0 hours (28% reduction)
- Ambulance diversions: 45% reduction
- Elective surgery delays: 35% reduction
Illustrative economic model:
- 3-Year ROI: 2,054%
- Total costs: $650,000
- Total benefits: $14,004,000
- Net benefit: $13,354,000
- Payback period: 2.3 months
Equity Impact:
- Illustrative reduction in racial disparities: over 80%
- Fairness constraints embedded in optimization
- Wait time disparities: Black patients +1.2 hours → +0.2 hours
Potential evaluation settings:
- Hospitals with comparable capacity constraints
Lesson: Optimization with explicit fairness constraints should be evaluated for both efficiency and equity effects
Part VII: Mental Health AI
Case Study 13: Crisis Text Line and an Illustrative AI-Triage Evaluation
Crisis Text Line has used NLP to support prioritization. The performance, capacity, outcome, and replication figures below are illustrative unless linked to a point source.
Context:
- High volume of text-based crisis conversations
- Suicide is a leading cause of death in the US
- Minutes matter in prevention
Illustrative impact metrics:
- Wait times for highest-risk: 45 min → 3 min (93% reduction)
- Sensitivity: 92% (detecting high-risk)
- Estimated 250 lives saved over 7 years (conservative)
- False negative rate: 8% (a residual risk that requires human safeguards)
Safety Features:
- Multiple screening layers (keywords → ML → human counselor)
- Conservative thresholds (high sensitivity, accept some false positives)
- Human counselor maintains final authority
- Continuous conversation monitoring
- Supervisor alerts for escalation
Illustrative counselor impact:
- 40% efficiency increase
- Better workload management
- Reduced burnout
- Context provided before conversation
Challenges:
- False negatives (8% miss high-risk individuals)
- Privacy concerns (AI analyzing sensitive content)
- Bias risks (addressed through continuous auditing)
- Preventing over-reliance (training emphasizes human judgment)
Potential applicability:
- 988 Suicide & Crisis Lifeline (US)
- Samaritans (UK)
- Lifeline Australia
- Crisis Services Canada
Lesson: High-stakes applications require extreme caution, multiple safety layers, and human authority
Part VIII: Drug Discovery
Case Study 14: AlphaFold and AI-Accelerated Drug Discovery
The AlphaFold Advance:
- Addressed the 50-year protein structure prediction problem
- CASP14 competition: median GDT score of 92.4
- Hours of computation vs months of lab work
- Democratized structural biology
AI drug-discovery evidence boundary:
- Multiple AI-assisted candidates have entered clinical development
- Timeline, cost, and success-rate claims require product-specific evidence
- Trial status should be checked against current registries and sponsor reports
Where AI Helped:
- Virtual screening (faster candidate triage)
- Lead optimization (predict properties)
- Target identification (multi-omics analysis)
- Protein structure prediction (major advance)
Where AI Fell Short of Hype:
- “AI eliminates need for chemists” → Still need expert chemists
- “AI drugs have higher success rates” → Too early to tell
- “AI eliminates animal testing” → Still required by regulators
- “10x faster overall” → Clinical trials remain the rate-limiting step
Examples of AI-assisted candidates:
- Exscientia DSP-1181: 5-HT1A agonist for OCD; entered phase I and was later discontinued
- Insilico INS018_055, now rentosertib: Investigational treatment for idiopathic pulmonary fibrosis
- BenevolentAI BEN-2293: Investigational treatment studied in atopic dermatitis
- Relay Therapeutics RLY-4008: Investigational treatment for FGFR2-altered cancers
Economic evidence boundary:
- Investment, valuation, time-savings, and cost-savings claims require dated product-level or portfolio-level sources
- Trial entry is not equivalent to approval, comparative effectiveness, or positive return on investment
Lesson: Progress is real but more modest than promotional claims. Experimental and clinical validation remain essential.
Part IX: Rural Health
Case Study 15: Hypothetical AI Extension to Project ECHO
Project ECHO’s hub-and-spoke model is evidence-based. The AI implementation, outcome tables, quotations, ROI, and replication claims below are illustrative, not reported Project ECHO results.
Context:
- Limited local specialist access in rural areas
- Longer specialist wait times than in urban areas
- Long travel distances for specialist care
- Higher mortality rates than in urban areas
The ECHO Model:
- Hub-and-spoke (specialists mentor PCPs)
- Case-based learning
- “Moving knowledge, not patients”
- Community of practice
AI Enhancements:
- Clinical decision support for PCPs
- Automated case classification
- Remote monitoring with AI triage
- Predictive analytics for high-risk patients
Illustrative New Mexico pilot results:
Access Improvements:
- PCP confidence: 4.2 → 7.8 out of 10 (+86%)
- Cases managed locally: 45% → 72% (+27 points)
- Specialist referrals: -38% reduction
- Wait times: 6.5 → 2.1 weeks (for cases still needing specialist)
Clinical Outcomes:
- Diabetes control: 32% → 51% at goal (+19 points)
- Hypertension control: 48% → 64% at goal (+16 points)
- Hepatitis C cure rate: 67% → 89% (+22 points)
- Hospitalization rate: -23% reduction
Economic Impact:
- 3-Year ROI: 840%
- Cost per patient/year: $8,500 (traditional) → $6,100 (ECHO+AI)
- Savings: $2,400 per patient per year
- Total savings: $324 million (45,000 patients over 3 years)
Provider Impact:
- Satisfaction: 6.2 → 8.7 out of 10
- Burnout: 58% → 34% reporting burnout
Patient Impact:
- Fewer 3-hour drives to specialists
- Local care with specialist backing
- Satisfaction: 7.1 → 8.9 out of 10
Illustrative scale assumptions:
- A replication plan could span 120 clinics
- It could reach ~200,000 patients
- Payer and Medicaid coverage would follow prospective evaluation
Lesson: Technology + human networks > Either alone. Sustainability, ROI, and equity benefits require measured evaluation.
Key Themes Across the Case Collection
1. Technical Success ≠ Clinical Impact
Evidence: The retrospective VA AKI model and COVID-19 prediction-model reviews show why model metrics must not be treated as clinical outcomes.
Implication: Must measure patient-centered endpoints, not just algorithm performance
2. External Validation is Mandatory
Evidence: Mammography AI (internal AUC 0.95 → external 0.82), COVID models (98% high bias)
Implication: Internal test performance may not transport. Validate on different populations, sites, equipment, and time periods.
3. Fairness Requires Active Design
Evidence: The Allegheny audit documents group-specific performance concerns. The bed-allocation example illustrates how fairness constraints could be evaluated.
Implication: Algorithms can perpetuate bias unless explicitly designed for fairness. Regular auditing is essential.
4. Human-AI Collaboration Optimal
Evidence: ProMED and Crisis Text Line illustrate distinct human-review roles. The Project ECHO extension is a hypothetical model for designing such collaboration.
Implication: AI provides scale and consistency, humans provide judgment and accountability. Hybrid > either alone.
5. Context Matters Profoundly
Evidence: Mammography AI performance varies by equipment and setting; ventilator-allocation frameworks differed across institutions and ethical approaches.
Implication: Same algorithm performs differently in different settings. Must adapt to local context.
6. Economic Value Requires Local Measurement
Evidence: The bed-allocation, Project ECHO, and Crisis Text Line examples show the inputs an economic evaluation would need. Their illustrative calculations are not observed returns.
Implication: Measure implementation costs, operating costs, uncertainty, outcomes, and the relevant comparator before making an ROI claim.
7. Implementation Is Half the Battle
Evidence: Streams highlights governance and implementation requirements. The ECHO scenario illustrates training and change-management questions.
Implication: Algorithm quality alone is insufficient. Implementation must address change management, training, workflow integration.
8. Transparency Builds Trust
Evidence: Allegheny and ProMED provide documented transparency mechanisms. The NHS-inspired scenario illustrates how disparity reporting could support accountability.
Implication: Explainability supports clinician review. Public documentation increases accountability.
9. Continuous Monitoring Required
Evidence: Google Flu Trends illustrates model drift; authorized devices require postmarket surveillance; the bed-allocation scenario illustrates operational monitoring.
Implication: Performance can degrade over time and requires ongoing evaluation and model updates.
10. Some Decisions Should Remain Human
Evidence: Ventilator-allocation frameworks emphasize accountable review; Crisis Text Line illustrates a human escalation pathway.
Implication: Life-or-death decisions require human judgment. AI should inform, not decide.
Evidence Classification
Documented deployments and platforms:
- BlueDot
- ProMED and HealthMap
- IDx-DR
- Google Flu Trends
- Allegheny Family Screening Tool
- Crisis Text Line
- MomConnect
Research-stage or retrospective evidence:
- Veterans Affairs AKI prediction model
- Mammography AI studies
- Sepsis reinforcement-learning research
- COVID-19 prediction models
- AI-assisted drug discovery
Explicitly hypothetical teaching scenarios:
- NHS-inspired disparity audit
- Hospital bed allocation
- AI extension to Project ECHO
Using This Appendix
For Students
- Start here: Read cases relevant to your interests
- Study implementations: Cases include illustrative Python-style pseudocode and templates
- Analyze outcomes: What worked vs what did not, and why
- Extract lessons: Apply to your own projects
For Practitioners
- Before implementation: Review cases in your domain
- Learn from mistakes: Study the failures to avoid repeating them
- Adapt code: Use examples as starting templates
- Evaluate properly: Follow validation frameworks demonstrated
For Researchers
- Identify gaps: What has not been studied yet?
- Deep dives: Follow references for full literature review
- Benchmark your work: Compare against the documented results
- Contribute evidence: Help build the evidence base
For Policymakers
- Understand impact: See real-world effects, not just promises
- Evidence-based policy: Design regulations based on actual outcomes
- Prioritize investments: Distinguish measured economic outcomes from illustrative models
- Equity focus: Learn from documented cases and clearly labeled teaching scenarios
Collection Scope
Coverage
- Geographic: Includes US, UK, South African, and global examples
- Domains: Surveillance, diagnostics, treatment, resource allocation, population health, health economics, mental health, drug discovery, rural health, and maternal health
- Evidence types: Deployments, regulatory records, retrospective studies, reviews, controversies, and teaching scenarios
Technical Depth
- Code examples: Illustrative pseudocode and implementation templates
- Algorithms covered: CNN, RNN, RL, NLP, optimization, ensemble methods
- Frameworks: TensorFlow, PyTorch, scikit-learn, XGBoost, SHAP, Fairlearn
Evidence Base
- References: Primary, regulatory, official, and peer-reviewed sources linked at the point of use
- Evaluation focus: Validation, transportability, workflow, economic assumptions, safety, and fairness
Updates and Contributions
This appendix is continuously updated. For corrections or to suggest additional case studies, contact the author at bryantegomoh.com.
Citation
If you use these case studies in your work, please cite:
@incollection{tegomoh2025casestudies,
title = {Case Study Library: Real-World AI in Public Health},
booktitle = {The Public Health AI Handbook: Evaluating AI Tools for Public Health Practice},
author = {Tegomoh, Bryan},
year = {2025},
doi = {10.5281/zenodo.18263442},
url = {https://publichealthaihandbook.com/appendices/case-study-overview.html}
}See How to Cite This Handbook for additional citation formats.
Next Steps
Continue with the full cases and supporting resources:
→ Read Complete Case Studies: Full technical details, code, and analysis
→ Code Repository Guide: Access companion code and examples
→ Further Reading: Curated resources for continued learning
For complete technical details, methodology, code implementations, and full analysis, see the full case studies.