Public Health AI Deployment Exercises
Applied exercises for MLOps, monitoring, integration, and organizational deployment decisions. The material is maintained separately so each operational question has a stable, focused reference.
- Identify the evidence and controls relevant to this decision area
- Distinguish technical performance from operational and population impact
- Apply the included framework without extending claims beyond the cited evidence
Introduction
This focused reference is part of the broader Public Health AI Deployment Exercises overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.
Check Your Understanding
Test your knowledge of deployment, monitoring, and maintenance for public health AI systems. Each question builds on the key concepts from this chapter.
A hospital deploys a pneumonia prediction model that performed excellently in validation (AUC=0.92). Three months post-deployment, monitoring reveals the model’s precision has dropped from 0.85 to 0.68, while its recall remains stable. Investigation shows the feature distribution hasn’t changed significantly. What is the MOST likely cause and appropriate response?
- Data drift: input features have changed, requiring immediate model retraining with recent data
- Concept drift: the relationship between features and outcomes has changed, requiring model retraining and potential re-architecture
- Label shift: the prevalence of pneumonia has decreased (perhaps due to seasonal patterns), requiring recalibration rather than full retraining
- Model decay: the model is simply aging and needs to be replaced with a newer architecture
Correct Answer: c) Label shift: the prevalence of pneumonia has decreased (perhaps due to seasonal patterns), requiring recalibration rather than full retraining
This question tests understanding of different types of model performance degradation and appropriate diagnostic reasoning. The key clues are: (1) precision dropped but recall remained stable, (2) feature distributions haven’t changed significantly, and (3) the timeframe suggests seasonal variation.
The chapter distinguishes between three types of drift:
Data drift (covariate shift): Input feature distributions change (X changes, P(X) ≠ P’(X)). Example: During COVID-19, chest X-ray volumes and patient demographics shifted dramatically. This would manifest as changes in feature statistics (mean age, vital signs distributions) which monitoring would detect. The question explicitly states features haven’t changed significantly, ruling this out.
Concept drift: The relationship between features and target changes (P(Y|X) ≠ P’(Y|X)). Example: A new pneumonia treatment becomes standard, changing how clinical features relate to outcomes. This typically affects both precision and recall, not just one. The chapter discusses concept drift as requiring substantial response including potential model re-architecture.
Label shift (prior probability shift): The prevalence of the outcome changes (P(Y) ≠ P’(Y)) but relationships remain stable. This is exactly what the scenario describes: if pneumonia cases decrease (perhaps post-flu season), the model still identifies the same patterns (stable recall, it catches pneumonia cases when they occur), but generates more false positives relative to true positives (reduced precision) because it was calibrated for higher prevalence.
The mechanism: A model trained when 10% of patients have pneumonia learns a decision threshold appropriate for that prevalence. If prevalence drops to 5%, the same threshold produces more false positives per true positive, reducing precision while recall stays constant. This is a calibration issue, not a fundamental model failure.
Option (a) is incorrect because data drift would show up in feature monitoring and would typically affect recall as well. Option (b), concept drift, would manifest differently (both precision and recall affected) and the stable feature distributions argue against fundamental relationship changes. Option (d), generic “model decay”, is too vague and doesn’t explain the specific pattern (precision down, recall stable).
The chapter emphasizes appropriate responses: - For label shift: Recalibration using Platt scaling or isotonic regression on recent data. This adjusts decision thresholds without full retraining. Much faster and less resource-intensive than retraining. - For data drift: Retrain with recent data to adapt to new feature distributions - For concept drift: Potentially re-architect the model, collect new training data, and conduct full validation
The broader lesson connects to the chapter’s discussion of monitoring strategies. Effective monitoring tracks: 1. Performance metrics (precision, recall, AUC) - detect problems 2. Feature distributions (mean, std, ranges) - diagnose data drift 3. Prediction distributions (model confidence, predicted probabilities) - detect calibration issues 4. Label distributions (outcome prevalence) - identify label shift
Seasonal patterns are common in public health: respiratory infections peak in winter, vector-borne diseases vary with climate, behavioral patterns change with holidays. The chapter emphasizes that not all performance changes indicate model failure, some reflect genuine population changes requiring recalibration rather than retraining.
For public health practitioners: establish baselines for expected seasonal variation, distinguish between model degradation requiring intervention and normal variation requiring adjustment, and maintain calibration monitoring alongside standard performance metrics. The Wong et al. (2021) sepsis model study cited in the chapter illustrates the cost of inadequate post-deployment monitoring, problems went undetected until external researchers investigated.
This scenario also highlights the value of A/B testing or shadow mode deployment where new calibrations can be validated before fully replacing production models. The chapter’s MLOps framework emphasizes exactly this kind of disciplined, evidence-based maintenance rather than reactive scrambling when metrics degrade.
A public health department is deploying an outbreak prediction system using a blue-green deployment strategy. The “green” environment (new model version 2.0) shows 5% better accuracy than the “blue” environment (current model version 1.5) in testing. However, 2 hours after switching traffic to green, monitoring alerts trigger showing increased prediction latency (400ms vs. previous 150ms) and higher memory usage. What should the operations team do IMMEDIATELY?
- Continue monitoring for 24 hours to ensure the latency issue isn’t a temporary spike before making decisions
- Immediately rollback to the blue environment (v1.5), investigate the performance issue, and only redeploy after resolution
- Scale up the green environment with more resources to handle the latency issue while keeping it in production
- Reduce the prediction frequency to decrease system load while keeping the more accurate model in production
Correct Answer: b) Immediately rollback to the blue environment (v1.5), investigate the performance issue, and only redeploy after resolution
This question tests understanding of production deployment best practices, particularly blue-green deployment strategies and incident response protocols. The scenario presents a common real-world situation: a model that performs better algorithmically but has operational problems in production.
The chapter emphasizes that blue-green deployment’s primary advantage is enabling instant rollback when problems arise. The architecture maintains two complete production environments: “blue” (current stable version) and “green” (new version being deployed). Traffic switches atomically between them, and crucially, blue remains ready to instantly resume service if green fails.
The key principle: When production systems exhibit unexpected behavior post-deployment, rollback first, investigate second. This reflects the chapter’s emphasis on reliability and the healthcare context where system failures can impact patient care.
Why immediate rollback is correct:
Patient safety priority: The chapter emphasizes healthcare’s near-zero tolerance for downtime and performance degradation. Outbreak prediction systems may inform time-critical public health responses. A 2.7x latency increase (150ms → 400ms) could indicate deeper problems that might worsen or cause system failure.
Blue-green enables zero-downtime rollback: This is exactly why the architecture exists. Switching traffic back to blue is essentially instantaneous, no complex migration or risky fixes under pressure.
Unknown root cause: The symptoms (latency + memory) suggest potential issues like memory leaks, inefficient model serving code, or resource contention. These can escalate to complete system failure. Without understanding the cause, keeping green in production is gambling with system stability.
5% accuracy gain doesn’t justify operational risk: The chapter discusses balancing innovation with reliability. Slightly better predictions matter less than reliable predictions. A system that’s down or slow when decisions are needed provides zero value.
Option (a), waiting 24 hours, violates incident response protocols. The chapter’s discussion of monitoring and alerting emphasizes rapid response to anomalies. Latency degradation often worsens as memory issues compound, and 24 hours of degraded performance risks missing critical outbreak signals or causing clinician frustration that undermines trust.
Option (c), scaling up resources, treats symptoms rather than causes. While resource scaling might temporarily mask the problem, it doesn’t address why v2.0 requires 2.7x more resources. This could be a code inefficiency, memory leak, or architectural problem that scaling won’t solve. The chapter emphasizes understanding root causes before applying fixes. Moreover, scaling during an incident adds complexity and risk, you’re modifying a system exhibiting unexpected behavior, potentially making things worse.
Option (d), reducing prediction frequency, degrades system functionality to accommodate a problematic deployment. This is backwards: the system should meet requirements, not have requirements adjusted to accommodate failures. Outbreak prediction systems may need real-time or near-real-time updates; reducing frequency undermines the core value proposition.
The chapter’s deployment checklist and runbook guidance supports this response:
Immediate actions (Incident Response): 1. Rollback to last known good state (blue environment v1.5) 2. Verify restoration of normal operations (latency back to 150ms) 3. Preserve logs and metrics from green (for investigation) 4. Initiate root cause analysis
Investigation phase (Post-Incident): 1. Analyze why v2.0 has higher latency and memory usage 2. Profile the model serving code 3. Compare resource utilization patterns 4. Test fixes in staging environment 5. Conduct load testing before redeployment
Redeployment (After fix validation): 1. Deploy fixed v2.1 to green with resolved issues 2. Canary deployment (5% of traffic first) 3. Monitor latency and memory metrics closely 4. Gradual traffic increase if metrics are stable 5. Full cutover only after validation period
This aligns with the chapter’s discussion of the Epic sepsis model failure, where inadequate post-deployment monitoring and response contributed to widespread deployment of an underperforming system. The lesson: sophisticated deployment strategies only provide value if coupled with disciplined operational practices including rapid rollback when warranted.
The broader principle: Production systems must meet operational requirements (latency, availability, resource efficiency) in addition to algorithmic requirements (accuracy, precision, recall). The chapter’s MLOps framework emphasizes that ML engineering encompasses both. A model that passes validation but fails operationally is not production-ready.
For public health practitioners: establish clear operational SLAs (service level agreements) for your AI systems alongside performance metrics. Define automated rollback triggers (latency > X, error rate > Y) that don’t require human judgment during incidents. Practice rollback procedures during development, don’t discover procedural gaps during real incidents. Maintain runbooks documenting exactly how to rollback, investigate, and redeploy for each production system.
The chapter’s emphasis on observability and monitoring makes incidents like this discoverable. The proper response demonstrates operational maturity: prioritizing system reliability over incremental improvements, using infrastructure designed for safe experimentation, and following disciplined incident response protocols.
A tuberculosis screening AI system deployed 18 months ago originally achieved 89% sensitivity and 84% specificity. Recent monitoring shows sensitivity has dropped to 76% while specificity increased to 91%. Manual review of misclassified cases reveals the model is missing more subtle presentations while correctly rejecting clearer negative cases. Feature distributions show the average image quality score has decreased from 8.2 to 7.1 (scale 1-10). What does this scenario MOST likely indicate?
- The model is performing better overall because specificity increased more than sensitivity decreased
- Data drift: image quality has deteriorated, requiring either improved data collection practices or model retraining on lower-quality images
- Concept drift: TB presentation patterns have changed in the population, requiring retraining with recent cases
- Normal model behavior: the precision-recall tradeoff means sensitivity and specificity naturally vary inversely
Correct Answer: b) Data drift: image quality has deteriorated, requiring either improved data collection practices or model retraining on lower-quality images
This question requires integrating multiple pieces of evidence to diagnose a real-world model performance problem. The scenario provides several clues that point specifically to data drift caused by image quality degradation.
The chapter defines data drift (covariate shift) as changes in input feature distributions: P(X) ≠ P’(X). The critical evidence here is the feature monitoring data: average image quality has decreased from 8.2 to 7.1. This 13% decline represents significant data drift, the model is now operating on inputs systematically different from its training distribution.
The mechanism of failure is illuminated by the manual review: the model misses “more subtle presentations” while correctly rejecting “clearer negative cases.” This pattern is characteristic of quality-dependent model behavior. Machine learning models trained on high-quality images learn to detect subtle features (fine details, early infiltrates, minimal abnormalities) that become harder or impossible to detect in lower-quality images. Lower image quality means: - Reduced spatial resolution limiting fine detail detection - Increased noise masking subtle abnormalities - Compression artifacts obscuring real pathology - Positioning or exposure issues affecting visibility
The model handles clear cases well (high specificity, obvious normal images are still clearly normal even in lower quality) but struggles with subtle positive cases that require fine detail (lower sensitivity, early TB or minimal disease becomes invisible in degraded images).
Option (a) makes a fundamental error in medical screening evaluation. The chapter emphasizes that sensitivity and specificity must be evaluated in context, not traded off mechanically. For TB screening, sensitivity is typically prioritized because missing TB (false negative) has serious consequences: continued disease transmission, delayed treatment, worse patient outcomes. Specificity increases don’t compensate for sensitivity decreases when false negatives are high-stakes. Moreover, the improved specificity likely reflects that obvious negatives are still obvious, not genuine improvement.
Option (c), concept drift, would mean the relationship between images and TB has changed (P(Y|X) ≠ P’(Y|X)). This seems unlikely; TB radiographic presentation hasn’t fundamentally changed in 18 months. The feature monitoring showing image quality decline points to input changes (data drift) rather than relationship changes (concept drift). Concept drift might show different patterns, perhaps a new TB strain with different radiographic appearance, or population changes (HIV co-infection altering presentations), but those wouldn’t correlate with image quality metrics.
Option (d) misunderstands the precision-recall tradeoff and sensitivity-specificity relationship. While these metrics do have mathematical relationships, they shouldn’t spontaneously change in production if the decision threshold is fixed. The chapter discusses that once a model is deployed with a particular threshold, sensitivity and specificity should remain relatively stable unless the underlying data distribution or relationships change. Natural variation exists, but 13-point sensitivity decrease is not “normal variation”, it indicates a real problem.
The chapter discusses exactly this scenario in the context of data quality monitoring. Common causes of image quality degradation in practice include: - Equipment aging or maintenance issues: X-ray machines need calibration and maintenance - Operator variability: Staff turnover, inadequate training, or workflow changes - Patient positioning: Shifts in protocols or patient population (sicker, less mobile patients) - Environmental factors: Power fluctuations, temperature, humidity affecting equipment
The chapter’s monitoring framework recommends tracking feature distributions precisely to catch data drift: - Statistical tests (Kolmogorov-Smirnov, chi-square) comparing current vs. training distributions - Control charts for key features (image quality, patient demographics, vital signs) - Automated alerts when distributions deviate beyond thresholds
Appropriate responses, per the chapter:
Short-term: 1. Investigate root cause of quality decrease: Check imaging equipment, review protocols, interview radiology technicians 2. Address data collection issues: Recalibrate equipment, retrain staff, update protocols 3. Consider lowering quality threshold temporarily: Accept only images meeting minimum standards
Medium-term: 4. Retrain model on mixed-quality data: If quality improvement isn’t feasible, adapt the model to handle degraded inputs 5. Implement quality-aware predictions: Model could output confidence scores that factor quality 6. Establish quality monitoring: Real-time image quality checks with feedback to operators
Long-term: 7. Robust model development: Future models should be trained on diverse quality ranges to handle real-world variability 8. Automated quality control: Systems that reject or flag poor-quality images before prediction
This scenario illustrates a key theme from the chapter: production ML requires monitoring not just model outputs but also inputs. The Wong et al. (2021) COVID-19 model degradation study cited in the chapter showed how external factors (patient population changes, care patterns) cause model performance decay. Effective MLOps catches these issues through comprehensive monitoring rather than waiting for catastrophic failures.
For public health practitioners: implement feature monitoring alongside performance monitoring, investigate sudden metric changes by examining data characteristics, maintain relationships with data generators (radiology departments, lab staff, EHR teams) to quickly identify upstream changes, and design models with robustness to expected real-world variations. The chapter’s emphasis on “production ML is mostly maintenance” applies precisely here, catching and responding to data drift is ongoing operational work, not a one-time deployment task.
A hospital integrates a clinical decision support AI into its EHR system using HL7 FHIR APIs. During integration testing, the AI works perfectly. However, in production, the system frequently returns errors for 15-20% of patients. Investigation reveals these patients have vital signs documented in different FHIR profiles than expected (using FHIR R4 instead of the expected R3 format), or have multiple conflicting vital sign entries from different sources. What does this scenario BEST illustrate about production ML in healthcare?
- FHIR APIs are unreliable and should not be used for clinical AI integration
- The model was inadequately tested and needs to be completely redesigned
- Production healthcare data is messy, heterogeneous, and requires robust data validation, error handling, and graceful degradation
- The EHR vendor’s implementation is non-compliant with FHIR standards and they should be required to fix it
Correct Answer: c) Production healthcare data is messy, heterogeneous, and requires robust data validation, error handling, and graceful degradation
This question addresses a critical theme from the chapter: the gap between idealized testing environments and messy production reality, particularly in healthcare IT integration. The scenario illustrates multiple real-world challenges the chapter emphasizes throughout.
The chapter extensively discusses healthcare integration complexity, noting over 700 different EHR systems in the US, multiple data standards (HL7 v2, v3, FHIR, DICOM), and inconsistent data quality. The specific issues described, multiple FHIR versions, conflicting entries from different sources, are characteristic of real healthcare environments.
Why this represents normal production reality:
FHIR version heterogeneity: Healthcare institutions upgrade systems gradually. Different departments might use different FHIR versions. Third-party systems (lab interfaces, monitoring devices, external records) may use older standards. The chapter notes that real production systems must handle this diversity.
Multiple data sources: Modern EHRs aggregate data from numerous sources: bedside monitors, nursing documentation, physician notes, imported records, manual entry. These sources may conflict (blood pressure entered twice by different providers, vital signs from different time points). The chapter discusses this as characteristic of healthcare data.
Test vs. production gap: Testing typically uses clean, curated data. Production reveals edge cases, legacy data, and integration quirks that testing misses. The chapter cites this as a primary reason why 47% of AI projects fail in deployment, they don’t handle production complexity.
The appropriate response, per the chapter’s MLOps best practices:
Robust data validation: - Implement schema validation for multiple FHIR versions - Parse and normalize data from different profiles - Detect and flag data quality issues - Maintain compatibility layers for multiple standards
Error handling: - Graceful degradation: if vital signs can’t be parsed, can the model work without them or use alternatives? - Clear error messages: tell clinicians why the system can’t make a prediction - Fallback strategies: use last known good values, request manual entry, defer prediction
Logging and monitoring: - Log all parsing failures with specifics (which FHIR profile, which field, which patient) - Track error patterns to identify systematic issues - Alert when error rates exceed thresholds - Aggregate logs to prioritize fixes
Production resilience: - Handle missing data gracefully - Validate inputs before passing to model - Time out gracefully if external systems are slow - Provide partial results when possible
Option (a) incorrectly blames FHIR standards. The chapter discusses FHIR as an improvement over older HL7 versions precisely because it’s more standardized and web-friendly. The problem isn’t FHIR itself but the messy reality of healthcare data regardless of standard used. Option (b) suggests complete redesign, which is overkill. The model works fine when it receives properly formatted data. The issue is the integration layer’s ability to handle data variety. The chapter emphasizes that production ML requires robust engineering around models, not just within models. Option (d) blames the vendor, which might feel satisfying but doesn’t solve the problem. Even if the vendor implements FHIR perfectly, other systems (labs, monitors, external records) won’t. The chapter’s discussion of integration emphasizes that AI systems must adapt to reality rather than demanding reality conform to ideal specifications.
This scenario connects to several themes from the chapter:
“ML is 5% algorithms, 95% infrastructure”: The model itself works. Production failures come from surrounding infrastructure, data parsing, API handling, error management.
Integration complexity as deployment barrier: The chapter lists integration as consuming 30% of deployment effort. This scenario illustrates why, healthcare data ecosystems are inherently complex.
Testing vs. production environments: The chapter emphasizes staged deployment (dev → staging → production) specifically to surface these issues before full deployment. Integration testing on clean test data missed problems that appeared in production.
Observability and monitoring: The error logging that revealed the FHIR version issue exemplifies the chapter’s emphasis on comprehensive logging. Without detailed error messages, debugging would be much harder.
The chapter provides practical guidance for this exact scenario:
Design for heterogeneity: - Support multiple data formats and versions - Build adapters for different EHR systems - Maintain mapping tables for vocabulary differences - Test against real production data dumps (with appropriate de-identification)
Implement progressive validation: 1. Input validation: Is data properly formatted? 2. Semantic validation: Do values make sense? (BP < 300, heart rate < 300) 3. Completeness validation: Are required fields present? 4. Consistency validation: Do related values align? (pediatric patient with adult vital signs?)
Graceful degradation strategies: - Return confidence scores reflecting data quality - Provide predictions with caveats (“limited confidence due to missing data”) - Offer manual override options - Fall back to simpler models if complex models can’t run
The broader lesson: successful production ML requires anticipating and handling real-world messiness. The chapter’s emphasis on observability, monitoring, error handling, and robust engineering reflects exactly these challenges. Public health AI systems must work in the environment they’re deployed to, not the idealized environment we wish existed.
For public health practitioners: allocate significant time and resources to integration and data quality handling, test against real production data (not just clean test sets), build error handling and logging from the start (not as an afterthought), maintain relationships with IT staff who understand institutional data quirks, and expect that initial production deployment will reveal issues testing didn’t catch. The chapter’s staged deployment and canary release strategies exist precisely to discover and address these problems incrementally rather than all at once.
An organization implements an MLOps pipeline with automated retraining: every week, the system automatically trains a new model version using the most recent 12 months of data, validates it on a hold-out set, and if validation metrics exceed thresholds (AUC > 0.85), automatically deploys to production. After 3 months, the data science team discovers the model’s fairness metrics have degraded significantly, the sensitivity gap between white and Black patients has increased from 3% to 12%. The automated system didn’t catch this because fairness metrics weren’t included in the deployment criteria. What does this scenario BEST illustrate?
- Automated retraining is dangerous and should not be used for clinical models
- Validation criteria must include all relevant performance dimensions including fairness, equity, and subgroup performance, not just aggregate metrics
- Fairness metrics are inherently unstable and shouldn’t be used as deployment gates
- The model should be retrained less frequently to allow more thorough manual review
Correct Answer: b) Validation criteria must include all relevant performance dimensions including fairness, equity, and subgroup performance, not just aggregate metrics
This question synthesizes themes from both this chapter (MLOps and automated deployment) and the Ethics, Bias, and Equity chapter. The scenario illustrates a critical failure mode: optimizing for narrow metrics while neglecting broader stakeholder requirements and ethical considerations.
The chapter discusses automated retraining and deployment as essential MLOps capabilities that enable models to stay current with changing data. However, it emphasizes that automation doesn’t mean absence of oversight, it means codifying the right checks into the automated pipeline.
The core problem: The deployment criteria (AUC > 0.85) captured one dimension of model quality (overall discriminative ability) while ignoring another critical dimension (equitable performance across subgroups). Aggregate metrics can mask serious subgroup disparities, a model can have excellent overall AUC while performing much worse for minority populations.
The chapter’s discussion of model validation connects to the Ethics chapter’s emphasis on fairness evaluation. The scenario demonstrates how fairness regressions can occur silently if not explicitly monitored:
Possible mechanisms for fairness degradation: 1. Training data composition changes: If the proportion of Black patients in recent training data decreased, the model may have learned less well for this subgroup 2. Differential label quality: If documentation quality differs across groups, recent data may have better labels for white patients 3. Feature drift differs by subgroup: If certain features become less predictive for one group, performance gaps can emerge 4. Selection bias in outcomes: If treatment access or documentation practices differ across groups, this can affect labels
The chapter’s automated retraining discussion emphasizes validation checklists and gates. A complete validation should include:
Aggregate performance: - AUC-ROC, precision, recall, specificity, sensitivity - Calibration metrics (Brier score, calibration plots) - Performance across confidence thresholds
Subgroup performance: - Metrics stratified by race, ethnicity, gender, age, socioeconomic status - Equalized odds (equal TPR and FPR across groups) - Equal opportunity (equal TPR across groups) - Calibration within subgroups
Fairness criteria: - Maximum allowed performance gaps between groups - Statistical significance tests for disparities - Intersectional analysis (e.g., race × gender combinations)
Data quality: - Sufficient sample sizes for subgroup evaluation - Label quality assessment - Feature completeness by subgroup
Option (a) wrongly concludes that automation itself is the problem. The chapter emphasizes automation as essential for production ML, manual processes don’t scale and introduce human error and delays. The problem isn’t automation but what’s automated. Properly designed automated systems that include fairness checks would have prevented this issue. Option (c) deflects responsibility by blaming fairness metrics. The chapter and its references to the Ethics chapter make clear that fairness metrics can be measured reliably and meaningfully. A 12% sensitivity gap is clinically and ethically significant, dismissing this as “unstable” would mean tolerating substantially worse care for Black patients. Option (d) suggests slowing retraining for manual review. While manual review has value, it’s not guaranteed to catch fairness issues either (this organization had data scientists who presumably could have caught this with manual review but didn’t until later). The solution is better automated checks, not replacing automation with manual processes.
The chapter discusses the FDA’s proposed regulatory framework for adaptive AI systems, which includes requirements for monitoring not just overall performance but also performance across demographic subgroups and fairness metrics. This regulatory direction reflects recognition that automated systems need comprehensive oversight built-in.
Appropriate implementation of automated retraining with fairness safeguards:
1. Comprehensive validation gates:
Deployment gates:
- Overall AUC > 0.85
- Sensitivity gap (max - min across racial groups) < 5%
- Equalized odds satisfied (p < 0.05 for χ² test)
- Calibration ECE < 0.05 within each major subgroup
- Minimum subgroup sample sizes met2. Automated fairness monitoring: - Compute fairness metrics on validation and test sets - Compare to previous model version - Require explicit approval if gaps widen beyond threshold - Generate automated fairness reports
3. Staged deployment with subgroup monitoring: - Deploy to shadow mode initially - Monitor subgroup performance in production - Canary deployment (small fraction of traffic) - Increase traffic only if subgroup metrics remain acceptable
4. Alerting and escalation: - Automatically alert if deployment gates fail - Escalate to human review when fairness metrics borderline - Require data science + ethics review for significant changes
5. Audit trails: - Log all deployment decisions and metrics - Document which checks passed/failed - Enable retrospective analysis - Support regulatory compliance
This scenario also connects to the chapter’s discussion of observability and monitoring. The fairness degradation was discovered eventually, but three months of deployment with widening disparities caused real harm, patients received differentially quality care based on race. Continuous production monitoring of fairness metrics (not just validation checks before deployment) would have detected this sooner.
The chapter’s emphasis on comprehensive MLOps practices means thinking beyond technical performance to all stakeholder requirements: clinicians need reliable and explainable predictions, patients need equitable treatment, regulators need evidence of safety and efficacy, and institutions need to meet ethical obligations. Automated systems must enforce all these requirements, not just the ones easiest to measure.
For public health practitioners: when implementing MLOps pipelines, ensure deployment gates reflect all dimensions of model quality including fairness and equity, stratify all metrics by demographic subgroups and monitor trends over time, involve diverse stakeholders (clinical, ethical, community) in defining deployment criteria, document what checks are performed and why, and audit deployed models regularly even if automated validation passed. The chapter’s production readiness checklist should explicitly include fairness evaluation as a non-negotiable requirement.
The broader lesson: automation amplifies our choices. Automating deployment with only aggregate metrics means rapidly propagating potentially biased models. Automating with comprehensive fairness checks means rapidly ensuring equitable care. The technology is neutral; the responsibility lies in how we design and implement it.
A regional health department deploys a disease forecasting model that requires integration with multiple data sources: hospital EHR systems, pharmacy sales data, and syndromic surveillance feeds. The architecture team proposes three options: (A) Batch processing, collect data overnight, run forecasts, deliver results in morning; (B) Real-time streaming, ingest data continuously, update forecasts every 15 minutes; (C) Hybrid, real-time data ingestion but forecast updates every 6 hours. Which factors should MOST heavily guide the architecture decision?
- Technical sophistication: choose real-time streaming because it represents the most advanced approach
- Cost: choose batch processing because it requires the least infrastructure
- Use case requirements: evaluate how quickly forecasts need updating, whether real-time decisions depend on outputs, and what data latency is acceptable
- Data availability: choose whichever architecture the data sources currently support to minimize integration effort
Correct Answer: c) Use case requirements: evaluate how quickly forecasts need updating, whether real-time decisions depend on outputs, and what data latency is acceptable
This question tests understanding of the chapter’s emphasis on architectural decisions driven by actual requirements rather than technical fashion, cost minimization, or convenience. The scenario presents a realistic public health ML architecture decision requiring thoughtful evaluation of trade-offs.
The chapter discusses deployment architecture choices extensively, emphasizing that different use cases have different requirements. The key is matching system characteristics to actual needs rather than defaulting to either the simplest (batch) or most sophisticated (real-time) option.
Use case requirements analysis for disease forecasting:
1. Decision timing: How quickly do public health officials need to act on forecasts? - If decisions are made in weekly or daily planning meetings, overnight batch processing may be sufficient - If emergency response depends on hour-by-hour updates (novel pathogen, rapidly evolving outbreak), real-time matters - For most endemic disease forecasting (flu, routine surveillance), daily or 6-hour updates likely sufficient
2. Data freshness value: Does newer data significantly improve forecast accuracy? - Epidemic dynamics: some diseases change slowly (TB, HIV) vs. rapidly (novel respiratory pathogen) - Lead time: if forecasts are 2-week ahead predictions, 6-hour vs. 15-minute updates may be negligible - Signal strength: if key signals (hospital admissions) only update daily, sub-daily forecasts don’t gain much
3. Operational complexity: What’s the cost-benefit of complexity? - Real-time streaming requires: Kafka/streaming infrastructure, complex error handling, state management, continuous monitoring - Batch processing: simpler infrastructure, easier debugging, more forgiving failure modes - Hybrid: balanced approach for many scenarios
4. Failure modes: What happens when things break? - Real-time systems: failures cascade quickly, require immediate response, complex recovery - Batch systems: failures are contained (one batch fails, next batch might succeed), easier recovery - Health department operations: do they have 24/7 engineering support for real-time system debugging?
The chapter’s discussion of reliability and operational burden is critical here. Healthcare systems demand high reliability (near-zero downtime tolerance), but achieving this with complex real-time architectures requires significant engineering investment. The quote “ML is 5% algorithms, 95% infrastructure” applies, real-time streaming is mostly infrastructure complexity, not model sophistication.
Option (a), technical sophistication, represents technology-driven rather than requirements-driven decision-making. The chapter warns against this throughout, emphasizing that production systems should use the simplest architecture that meets requirements. Real-time streaming adds significant operational complexity (state management, error handling, monitoring, scaling) that’s only justified if real-time responsiveness actually matters. For many public health use cases, it doesn’t. The chapter references the Epic sepsis model failure partly because sophisticated technology was deployed without evidence it improved outcomes.
Option (b), cost minimization, is too narrow. While cost matters, the cheapest solution that fails to meet requirements wastes money. If real-time decisions genuinely depend on up-to-date forecasts, batch processing could cause harm (delayed response to outbreak acceleration). The chapter discusses cost-effectiveness, not mere cost minimization. However, cost considerations are legitimate: if the use case doesn’t require real-time updates, spending on streaming infrastructure diverts resources from other priorities.
Option (d), data availability, confuses convenience with requirements. The chapter discusses integration challenges extensively but emphasizes systems should be designed around needs, not constraints. If real-time forecasts are critical and data sources don’t currently support it, the solution might be: (1) work with data providers to enable real-time feeds, (2) use hybrid architecture using available real-time sources while batch-processing others, or (3) carefully evaluate whether the use case truly requires real-time given data constraints. This is a trade-off to analyze, not a simple “accept what exists” decision.
The chapter’s decision framework for this scenario:
Analyze requirements: - Interview stakeholders: How do they use forecasts? When do they need updates? - Evaluate decision cadence: Are responses hourly, daily, weekly? - Assess data refresh rates: How often do input data sources update? - Calculate value of timeliness: Does faster meaningfully improve outcomes?
Evaluate architectures:
Batch (overnight processing): - Simpler infrastructure, easier maintenance - Lower operational burden (failures less urgent) - Easier debugging and monitoring - 24-hour latency might miss rapid developments - Appropriate if: Decisions are daily/weekly, disease changes slowly, stakeholders review forecasts during business hours
Real-time streaming (15-minute updates): - Minimum latency for rapid response - Enables real-time dashboards and alerts - Complex infrastructure requiring specialized expertise - Higher operational burden (24/7 monitoring needs) - More failure modes and harder debugging - Appropriate if: Outbreak evolves rapidly, emergency decisions depend on latest data, stakeholders actively monitor in real-time
Hybrid (6-hour updates): - Balance of timeliness and simplicity - Catches intra-day trends without full streaming complexity - Enables meaningful response within business day - Simpler than full streaming, more timely than overnight batch - Appropriate if: Moderate update frequency needed, stakeholders check multiple times daily, infrastructure capacity limited
The chapter’s emphasis on staged deployment and iteration is relevant: start with simpler architecture (batch or hybrid), validate it meets needs, and only increase complexity if evidence shows value. The Gartner statistic (53% of AI projects fail to reach production) often reflects over-engineering, building complex systems that collapse under their own weight.
For public health practitioners: resist both technology fashion and false economy, define clear requirements before choosing architecture (work backward from decisions that depend on system outputs), consider operational capacity honestly (can you maintain complex systems?), start simple and add complexity only when necessary, and measure whether architectural sophistication actually improves outcomes. The chapter’s MLOps principles emphasize reliability over sophistication, a simple batch system that runs reliably provides more value than a real-time system that frequently fails.
This decision embodies the chapter’s philosophy: production ML succeeds by matching technical implementation to real-world requirements, organizational capabilities, and stakeholder needs, not by deploying the most impressive technology regardless of context.