Appendix D — Case Study Library - Overview
A collection of documented public health AI cases and explicitly labeled teaching scenarios, examining successes, failures, and implementation tradeoffs. Reported evidence is separated from illustrative code, outcome tables, and economic models.
- Quick scan? Review the summary table and key themes below
- Specific domain? Jump to relevant sections using the navigation links
- Deep dive? Read the complete case studies
- Code implementation? Cases include illustrative Python-style pseudocode and templates
Case Studies at a Glance
| # | Case Study | Domain | Outcome | Key Metric | Lines |
|---|---|---|---|---|---|
| 1 | BlueDot COVID-19 Detection | Surveillance | Documented alert | Client alert on December 31, 2019 | ~600 |
| 2 | Google Flu Trends | Surveillance | Failure→Recovery | 135% error (2013) | ~550 |
| 3 | ProMED + HealthMap | Surveillance | Complementary workflow | Automated aggregation plus expert interpretation | ~500 |
| 4 | IDx-DR Autonomous Diagnostic | Diagnostics | Authorized device | FDA De Novo classification, 2018 | ~700 |
| 5 | DeepMind AKI Work | Diagnostics | Distinct systems | Streams was rule-based; VA model was retrospective | ~650 |
| 6 | Breast Cancer AI | Diagnostics | Mixed | Performance varies by product, population, equipment, threshold, and workflow | ~900 |
| 7 | Sepsis RL Treatment | Treatment | Controversial | RCT needed | ~900 |
| 8 | COVID-19 Prediction Models | Treatment | Mostly Failed | 98% high bias | ~700 |
| 9 | Ventilator Allocation | Resources | Ethical and policy analysis | Human review, transparency, and appeals emphasized | ~900 |
| 10 | Allegheny Child Welfare | Population Health | Controversial | Calibration and threshold-based group differences | ~1,100 |
| 11 | NHS-Inspired Disparity Audit | Population Health | Teaching scenario | Illustrative equity-audit workflow | ~900 |
| 12 | Hospital Bed Allocation | Health Economics | Teaching scenario | Illustrative optimization and ROI model | ~1,200 |
| 13 | Crisis Text Line | Mental Health | Documented platform plus teaching scenario | Illustrative evaluation metrics | ~1,100 |
| 14 | AlphaFold Drug Discovery | Drug Discovery | Research and development programs | Product-specific evidence required | ~1,500 |
| 15 | Project ECHO + AI | Rural Health | Teaching scenario | Hypothetical AI extension | ~1,300 |
The full cases provide illustrative pseudocode and source-linked evidence. Fixed counts are omitted because the collection is continuously updated.
Part I: Disease Surveillance and Outbreak Detection
Case Study 1: BlueDot - Early COVID-19 Detection
Context: Automated global disease surveillance using news reports, flight data, and climate patterns.
Key Achievement: - Alerted clients on December 31, 2019, the same date WHO’s China Country Office identified the Wuhan notice and before WHO’s January 4 public communication (WHO, 2024) - Predicted initial spread destinations (Bangkok, Hong Kong, Tokyo, Seoul, Singapore) - Alerted clients immediately
Limitations: - Couldn’t predict pandemic severity or spread dynamics - Early warning alone insufficient without policy action
Technical Highlights: - Multi-source data integration (news in 65 languages, flight networks, climate data) - NLP for disease mention extraction - Geographic risk scoring - 24/7 automated monitoring
Lesson: AI excels at early detection, not prediction of impact
Case Study 2: Google Flu Trends - Rise and Fall
Context: Search query-based flu surveillance (2008-2015)
Timeline: - 2008-2011: Accurate predictions (correlation 0.90 with CDC data) - 2012-2013: Catastrophic failure (135% overprediction) - 2014-2015: Recovery through hybrid human-AI approach
Why It Failed: - Algorithm opacity (correlation without causation) - Search behavior changes (media coverage effects) - No update mechanism (model drift) - Overfitting to data artifacts
Recovery Strategy: - Combined with CDC data (ensemble approach) - Increased transparency - Regular recalibration - Acknowledged limitations
Lesson: Simple correlations fail; need robust, interpretable models with update mechanisms
Case Study 3: ProMED-mail + HealthMap - Human-AI Collaboration
Context: Hybrid human-AI disease surveillance system
Complementary Functions: - ProMED provides expert-curated outbreak reports and moderator context - HealthMap aggregates and classifies online disease reports - A combined workflow requires explicit review criteria and evaluation
Key Innovation: Automated collection can extend breadth while experts provide contextual interpretation
Lesson: Humans + AI > Either alone
Part II: Diagnostic AI
Case Study 4: IDx-DR - First Autonomous AI Diagnostic
Historic Achievement: - First FDA-authorized autonomous diagnostic AI (April 2018) - Can diagnose without clinician interpretation
Performance: - Sensitivity: 87.2% - Specificity: 90.7% - Clinical trial: 900 patients, 10 sites
Real-World Challenges: - Performance can change with image quality, population, camera, and workflow - Dated postmarket deployment scale requires an attributable source - Reimbursement challenges
Regulatory Pathway: - FDA De Novo classification - Extensive clinical validation - Post-market surveillance requirements
Lesson: Autonomous AI possible but requires extensive validation and monitoring
Case Study 5: DeepMind AKI Work - Two Distinct Systems
Evidence boundary: - Royal Free Streams: A clinically deployed app using the NHS national rule-based AKI algorithm, not machine learning - VA prediction model: A retrospective recurrent neural network trained on 703,782 US Veterans Affairs patients - Reported performance: Predicted 55.8% of inpatient AKI and 90.2% of dialysis-requiring AKI up to 48 hours in advance at two false alerts per true alert (Tomašev et al., 2019) - Clinical impact: Not established by a prospective deployment trial
Why the distinction matters: - Royal Free’s data-sharing controversy concerns governance of Streams - The VA model’s reported metrics concern retrospective prediction - Alert burden, workflow integration, and patient outcomes require separate prospective evidence
Critical Insight: Retrospective prediction performance does not establish clinical utility
Lessons for Future: - Design WITH clinicians, not FOR them - Provide actionable recommendations, not just predictions - Integrate into existing workflows - Clear value proposition required
Case Study 6: Breast Cancer Detection - Inconsistent Results
Multiple Systems Evaluated: - Google Health/DeepMind - Lunit INSIGHT MMG - iCAD ProFound AI
Performance Variability: - Performance changes are product-, population-, equipment-, threshold-, and workflow-specific - Internal validation does not establish transportability - Multi-site, multi-equipment testing is required before local deployment
Success Story: - Sweden (Lund): 44% radiologist workload reduction, maintained detection rate - Key: AI as concurrent reader, not replacement
Lesson: Internal validation insufficient; need multi-site, multi-equipment testing
Part III: Treatment Optimization
Case Study 7: Sepsis Treatment - AI-RL Controversy
The AI Clinician (MIT): - Learned treatment policy from 100,000 ICU patients - Recommended less fluid than standard guidelines - Controversial: Observational data biased
The Confounding Problem: - Sicker patients receive more aggressive treatment → worse outcomes - AI learns: More treatment → Worse outcomes (confounded!) - Reality: Treatment couldn’t overcome initial severity
Current Status: - The AI Clinician evidence remains retrospective and observational - Prospective comparative evidence of clinical benefit is still required
Lesson: Reinforcement learning on observational data is hypothesis-generating, not practice-changing without prospective comparative validation
Case Study 8: COVID-19 Prediction Models - Limited Impact
The Pandemic Rush: - 232 COVID models published by October 2020 - 98% had high risk of bias - Only 1 externally validated with low bias - Most never used clinically
Common Problems: - Small sample sizes (<500 patients) - Lack of external validation - Poor reporting standards - Overfitting
Models That Worked: - 4C Mortality Score (UK): 35,000 patients, multiple sites, simple and interpretable - ISARIC-4C: 75,000 patients, properly validated
Lesson: Urgency doesn’t justify poor methods. Simple, validated models > complex, unvalidated ones
Part IV: Resource Allocation
Case Study 9: Ventilator Allocation - Ethics Meets AI
The Dilemma: - COVID-19 ventilator shortages required triage decisions - AI systems proposed for allocation - Most hospitals rejected AI-driven allocation
The Trilemma (Cannot maximize all three): 1. Utility (save most lives) 2. Fairness (equal treatment) 3. Autonomy (individual rights)
Why AI Was Rejected: - Insufficient accuracy (70-80% not enough for life/death) - Bias concerns (perpetuate historical inequities) - Legal risks (disability discrimination) - Trust and legitimacy issues
What Hospitals Did Instead: - Human clinical assessment with ethical oversight - Triage officers (experienced clinicians) - Appeals process - Re-evaluation every 48-120 hours
Lesson: Some decisions should remain human. AI can inform but not decide life-or-death allocation.
Part V: Population Health and Health Equity
Case Study 10: Allegheny Family Screening Tool - Algorithmic Child Welfare
Context: Risk assessment for child welfare referrals (used since 2016)
Performance: - Predicts child removal risk (AUC 0.76) - Used by caseworkers to prioritize investigations
Fairness Findings: - The audit found broadly similar calibration patterns across groups - Threshold-based placement and error rates varied, with the most pronounced false-positive-rate imbalance reported for the Mixed-race subgroup - Group-specific errors require direct measurement rather than inference from overall accuracy
The Feedback Loop Problem: - Historical over-surveillance of Black/poor families - More system contact → Higher risk scores - Higher scores → More investigation - Cycle perpetuates
Responses: - Public documentation and transparency - Community engagement - Regular fairness audits - Human override capability maintained
Ongoing Debate: - Supporters: More consistent than human bias alone - Critics: Automates and scales existing discrimination - Both perspectives have validity
Lesson: Historical bias in data perpetuates inequality. Transparency and community input essential.
Case Study 11: Hypothetical NHS-Inspired Disparity Audit
Documented UK COVID-19 disparities motivate this teaching scenario. The AI implementation, care-pathway findings, interventions, and outcome figures below are illustrative, not reported NHS results.
Illustrative premise: - AI identified disparities in HUMAN care delivery, not AI decisions - Used as diagnostic tool for systemic racism - Findings led to concrete policy changes
Disparities Found: - Black patients: 2.5x mortality rate (1.8x after adjusting for comorbidities) - 8-hour longer admission wait times for Black patients - Lower ICU admission rates despite similar severity - Lower guideline-concordant care rates
Illustrative interventions: - Enhanced translation services (24/7 availability) - Cultural competency training (mandatory) - Community health workers - Care pathway standardization - Real-time disparity monitoring dashboards
Illustrative results after two years: - Admission disparities reduced 40% - ICU access disparities reduced 25% - Mortality disparities reduced 15% - Still work to do, but measurable progress
Lesson: AI can expose systemic problems for intervention. Used correctly, it’s a tool for justice, not just a source of bias.
Part VI: Health Economics
Case Study 12: Hypothetical AI-Driven Hospital Bed Allocation
This Johns Hopkins-inspired teaching scenario preserves an optimization and economic-analysis template. The results, equity effects, ROI, and named replication sites are illustrative, not reported Johns Hopkins outcomes.
Illustrative implementation:
Challenge: Balance competing objectives: - Efficiency (maximize utilization) - Access (minimize wait times) - Quality (appropriate care level) - Equity (fair access across populations)
Illustrative results: - Bed utilization: 82% → 88% (+6 percentage points) - ED wait times: 4.2 → 3.0 hours (28% reduction) - Ambulance diversions: 45% reduction - Elective surgery delays: 35% reduction
Illustrative economic model: - 3-Year ROI: 2,054% - Total costs: $650,000 - Total benefits: $14,004,000 - Net benefit: $13,354,000 - Payback period: 2.3 months
Equity Impact: - REDUCED racial disparities by 80%+ - Fairness constraints embedded in optimization - Wait time disparities: Black patients +1.2 hours → +0.2 hours
Potential evaluation settings: - Mayo Clinic (2020) - Cleveland Clinic (2021) - Mass General Brigham (2022) - Over 50 other hospitals
Lesson: Optimization with explicit fairness constraints delivers both efficiency and equity
Part VII: Mental Health AI
Case Study 13: Crisis Text Line and an Illustrative AI-Triage Evaluation
Crisis Text Line has used NLP to support prioritization. The performance, capacity, outcome, and replication figures below are illustrative unless linked to a point source.
Context: - Over 100,000 crisis texts monthly - 48,000 suicide deaths/year in US - Minutes matter in prevention
Illustrative impact metrics: - Wait times for highest-risk: 45 min → 3 min (93% reduction) - Sensitivity: 92% (detecting high-risk) - Estimated 250 lives saved over 7 years (conservative) - False negative rate: 8% (concerning but unavoidable with current technology)
Safety Features: - Multiple screening layers (keywords → ML → human counselor) - Conservative thresholds (high sensitivity, accept some false positives) - Human counselor maintains final authority - Continuous conversation monitoring - Supervisor alerts for escalation
Illustrative counselor impact: - 40% efficiency increase - Better workload management - Reduced burnout - Context provided before conversation
Challenges: - False negatives (8% miss high-risk individuals) - Privacy concerns (AI analyzing sensitive content) - Bias risks (addressed through continuous auditing) - Preventing over-reliance (training emphasizes human judgment)
Potential applicability: - National Suicide Prevention Lifeline (US) - Samaritans (UK) - Lifeline Australia - Crisis Services Canada
Lesson: High-stakes applications require extreme caution, multiple safety layers, and human authority
Part VIII: Drug Discovery
Case Study 14: AlphaFold and AI-Accelerated Drug Discovery
The AlphaFold Breakthrough: - Solved 50-year protein folding problem - CASP14 competition: 92.4% median accuracy - Hours of computation vs months of lab work - Democratized structural biology
AI drug-discovery evidence boundary: - Multiple AI-assisted candidates have entered clinical development - Timeline, cost, and success-rate claims require product-specific evidence - Trial status should be checked against current registries and sponsor reports
Where AI Helped: - Virtual screening (10-100x faster) - Lead optimization (predict properties) - Target identification (multi-omics analysis) - Protein structure prediction (major advance)
Where AI Fell Short of Hype: - “AI eliminates need for chemists” → Still need expert chemists - “AI drugs have higher success rates” → Too early to tell - “AI eliminates animal testing” → Still required by regulators - “10x faster overall” → More like 2-3x (clinical trials not faster)
Verified examples: - Exscientia DSP-1181: 5-HT1A agonist for OCD; entered phase I and was later discontinued - Insilico INS018_055, now rentosertib: Investigational treatment for idiopathic pulmonary fibrosis - BenevolentAI BEN-2293: Investigational treatment studied in atopic dermatitis - Relay Therapeutics RLY-4008: Investigational treatment for FGFR2-altered cancers
Economic evidence boundary: - Investment, valuation, time-savings, and cost-savings claims require dated product-level or portfolio-level sources - Trial entry is not equivalent to approval, comparative effectiveness, or positive return on investment
Lesson: Real progress, but more modest than hyped. AI is powerful tool, not magic. Experimental validation still essential.
Part IX: Rural Health
Case Study 15: Hypothetical AI Extension to Project ECHO
Project ECHO’s hub-and-spoke model is evidence-based. The AI implementation, outcome tables, quotations, ROI, and replication claims below are illustrative, not reported Project ECHO results.
Context: - 60 million Americans live in rural areas - 2x longer specialist wait times - Many drive over 100 miles for care - Rural mortality rates 20% higher than urban
The ECHO Model: - Hub-and-spoke (specialists mentor PCPs) - Case-based learning - “Moving knowledge, not patients” - Community of practice
AI Enhancements: - Clinical decision support for PCPs - Automated case classification - Remote monitoring with AI triage - Predictive analytics for high-risk patients
Illustrative New Mexico pilot results:
Access Improvements: - PCP confidence: 4.2 → 7.8 out of 10 (+86%) - Cases managed locally: 45% → 72% (+27 points) - Specialist referrals: -38% reduction - Wait times: 6.5 → 2.1 weeks (for cases still needing specialist)
Clinical Outcomes: - Diabetes control: 32% → 51% at goal (+19 points) - Hypertension control: 48% → 64% at goal (+16 points) - Hepatitis C cure rate: 67% → 89% (+22 points) - Hospitalization rate: -23% reduction
Economic Impact: - 3-Year ROI: 840% - Cost per patient/year: $8,500 (traditional) → $6,100 (ECHO+AI) - Savings: $2,400 per patient per year - Total savings: $32.4 million (45,000 patients over 3 years)
Provider Impact: - Satisfaction: 6.2 → 8.7 out of 10 - Burnout: 58% → 34% reporting burnout
Patient Impact: - No more 3-hour drives to specialists - Local care with specialist backing - Satisfaction: 7.1 → 8.9 out of 10
Illustrative scale assumptions: - Now in 120 clinics across 10 states - ~200,000 patients reached - CMS Innovation Award: $50M for national expansion - 15 states cover via Medicaid
Lesson: Technology + human networks > Either alone. Sustainable model with clear ROI and equity benefits.
Key Themes Across the Case Collection
1. Technical Success ≠ Clinical Impact
Evidence: The retrospective VA AKI model and COVID-19 prediction-model reviews show why model metrics must not be treated as clinical outcomes.
Implication: Must measure patient-centered endpoints, not just algorithm performance
2. External Validation is Mandatory
Evidence: Mammography AI (internal AUC 0.95 → external 0.82), COVID models (98% high bias)
Implication: Internal test performance may not transport. Validate on different populations, sites, equipment, and time periods.
3. Fairness Requires Active Design
Evidence: The Allegheny audit documents group-specific performance concerns. The bed-allocation example illustrates how fairness constraints could be evaluated.
Implication: Algorithms perpetuate bias unless explicitly designed for fairness. Regular auditing essential.
4. Human-AI Collaboration Optimal
Evidence: ProMED and Crisis Text Line illustrate distinct human-review roles. The Project ECHO extension is a hypothetical model for designing such collaboration.
Implication: AI provides scale and consistency, humans provide judgment and accountability. Hybrid > either alone.
5. Context Matters Profoundly
Evidence: Mammography AI performance varies by equipment and setting; ventilator-allocation frameworks differed across institutions and ethical approaches.
Implication: Same algorithm performs differently in different settings. Must adapt to local context.
6. Economic Value Can Be Substantial
Evidence: The bed-allocation, Project ECHO, and Crisis Text Line examples show the inputs an economic evaluation would need. Their illustrative calculations are not observed returns.
Implication: Measure implementation costs, operating costs, uncertainty, outcomes, and the relevant comparator before making an ROI claim.
7. Implementation is Half the Battle
Evidence: Streams highlights governance and implementation requirements. The ECHO scenario illustrates training and change-management questions.
Implication: Algorithm quality insufficient. Must address change management, training, workflow integration.
8. Transparency Builds Trust
Evidence: Allegheny and ProMED provide documented transparency mechanisms. The NHS-inspired scenario illustrates how disparity reporting could support accountability.
Implication: Explainable AI preferred by clinicians. Public documentation increases accountability.
9. Continuous Monitoring Required
Evidence: Google Flu Trends illustrates model drift; authorized devices require postmarket surveillance; the bed-allocation scenario illustrates operational monitoring.
Implication: Performance degrades over time. Need ongoing evaluation and model updates.
10. Some Decisions Should Remain Human
Evidence: Ventilator-allocation frameworks emphasize accountable review; Crisis Text Line illustrates a human escalation pathway.
Implication: Life-or-death decisions require human judgment. AI should inform, not decide.
Evidence Classification
Documented deployments and platforms: 1. BlueDot 2. ProMED and HealthMap 3. IDx-DR 4. Google Flu Trends 5. Allegheny Family Screening Tool 6. Crisis Text Line 7. MomConnect
Research-stage or retrospective evidence: 1. Veterans Affairs AKI prediction model 2. Mammography AI studies 3. Sepsis reinforcement-learning research 4. COVID-19 prediction models 5. AI-assisted drug discovery
Explicitly hypothetical teaching scenarios: 1. NHS-inspired disparity audit 2. Hospital bed allocation 3. AI extension to Project ECHO
Using This Appendix
For Students
- Start here: Read cases relevant to your interests
- Study implementations: Cases include illustrative Python-style pseudocode and templates
- Analyze outcomes: What worked vs what didn’t, and why
- Extract lessons: Apply to your own projects
For Practitioners
- Before implementation: Review cases in your domain
- Learn from mistakes: Study the failures to avoid repeating them
- Adapt code: Use examples as starting templates
- Evaluate properly: Follow validation frameworks demonstrated
For Researchers
- Identify gaps: What hasn’t been studied yet?
- Deep dives: Follow references for full literature review
- Benchmark your work: Compare to these real-world results
- Contribute evidence: Help build the evidence base
For Policymakers
- Understand impact: See real-world effects, not just promises
- Evidence-based policy: Design regulations based on actual outcomes
- Prioritize investments: Distinguish measured economic outcomes from illustrative models
- Equity focus: Learn from documented cases and clearly labeled teaching scenarios
Collection Scope
Coverage
- Geographic: Includes US, UK, South African, and global examples
- Domains: Surveillance, diagnostics, treatment, resource allocation, population health, mental health, drug discovery, and rural health
- Evidence types: Deployments, regulatory records, retrospective studies, reviews, controversies, and teaching scenarios
Technical Depth
- Code examples: Illustrative pseudocode and implementation templates
- Algorithms covered: CNN, RNN, RL, NLP, optimization, ensemble methods
- Frameworks: TensorFlow, PyTorch, scikit-learn, XGBoost, SHAP, Fairlearn
Evidence Base
- References: Primary, regulatory, official, and peer-reviewed sources linked at the point of use
- Evaluation focus: Validation, transportability, workflow, economic assumptions, safety, and fairness
Updates and Contributions
This appendix is continuously updated. For corrections or to suggest additional case studies, contact the author at bryantegomoh.com.
Citation
If you use these case studies in your work, please cite:
@incollection{tegomoh2025casestudies,
title = {Case Study Library: Real-World AI in Public Health},
booktitle = {The Public Health AI Handbook: Evaluating AI Tools for Public Health Practice},
author = {Tegomoh, Bryan},
year = {2025},
doi = {10.5281/zenodo.18263442},
url = {https://publichealthaihandbook.com/appendices/case-study-overview.html}
}See How to Cite This Handbook for additional citation formats.
Next Steps
Ready to dive deeper?
→ Read Complete Case Studies - Full technical details, code, and analysis
→ Code Repository Guide - Access companion code and examples
→ Further Reading - Curated resources for continued learning
This overview provides navigation across cases. For complete technical details, methodology, code implementations, and full analysis, see the full case studies.