Large Language Model Exercises for Public Health
Applied exercises for model selection, prompting, validation, privacy, and public health workflow design. The material is maintained separately so each operational question has a stable, focused reference.
- Identify the evidence and controls relevant to this decision area
- Distinguish technical performance from operational and population impact
- Apply the included framework without extending claims beyond the cited evidence
Introduction
This focused reference is part of the broader Large Language Model Exercises overview. It preserves the detailed methods, examples, and exercises while reducing page size and improving direct navigation.
Check Your Understanding
Test your knowledge of LLMs in public health practice. These questions integrate concepts across the entire chapter, covering technical foundations, privacy, validation, and appropriate use. Take time to think through each scenario before revealing the answer.
These are not simple recall questions, they’re scenarios requiring critical thinking and synthesis of chapter concepts. For each: 1. Read carefully and consider the scenario 2. Think through what principles apply 3. Choose your answer before revealing 4. Read the full explanation even if you answered correctly (explanations provide additional context and nuance)
Question 1: Privacy Compliance
A public health analyst wants to use ChatGPT to analyze survey data containing respondents’ ages, ZIP codes, and health conditions. What is the appropriate approach?
- Use free ChatGPT after removing names
- Use ChatGPT Plus ($20/month) after removing direct identifiers
- Use ChatGPT Enterprise with a Business Associate Agreement, or thoroughly de-identify data beyond HIPAA’s 18 identifiers
- ChatGPT can never be used for health data under any circumstances
Correct Answer: c) Use ChatGPT Enterprise with a Business Associate Agreement, or thoroughly de-identify data beyond HIPAA’s 18 identifiers
Why this matters:
Even with names removed, the data described likely contains Protected Health Information (PHI) under HIPAA. The combination of: - Ages - ZIP codes - Health conditions
…can potentially identify individuals, especially in smaller geographic areas or with rare conditions. Research by Sweeney (2000) demonstrated that 87% of the U.S. population can be uniquely identified using just three data points: 5-digit ZIP code, birth date (or age), and gender.
Why other answers are wrong:
a) Free ChatGPT after removing names: - Free ChatGPT has NO Business Associate Agreement - Data may be used for training (check current terms) - Removing only names is insufficient de-identification - Age + ZIP code + health condition can re-identify individuals - Legal risk: Direct HIPAA violation, penalties $100-$50,000 per violation
b) ChatGPT Plus after removing direct identifiers: - ChatGPT Plus ($20/month) still lacks BAA for most users - Data protection similar to free version - “Direct identifiers” removal alone insufficient - Risk: Still a HIPAA violation if data is PHI
d) Never use under any circumstances: - Too restrictive - enterprise solutions with BAAs exist - Proper de-identification can make data safe for consumer LLMs - Would unnecessarily limit valuable analysis tools
The correct approach:
Option 1: Enterprise LLM with BAA (Best practice)
Use: ChatGPT Enterprise, Azure OpenAI Service, Claude Team/Enterprise, Google Vertex AI
Requirements:
- Signed Business Associate Agreement
- Data not used for training
- HIPAA-compliant infrastructure
- Audit logs and security controls
Cost: $30-60/user/month typically
Benefit: Can work with actual data (within reason), full legal protectionOption 2: Thorough de-identification (If enterprise unavailable)
Remove/generalize beyond HIPAA's 18 identifiers:
Ages: 32 → "30-35" or "30-40"
ZIP codes: 02138 → "021**" or "Massachusetts"
Rare conditions: "Specific rare disease" → "Chronic condition"
Small cell sizes: Suppress groups with <11 individuals
Context clues: Remove uniqueness indicators
Then use consumer LLM WITH CAUTION
Risk: Residual re-identification risk remains
Best practice: Even de-identified health data should use enterprise LLMs when possibleReal-world example:
A health department wanted to analyze COVID-19 survey data with ChatGPT:
What they did wrong: Uploaded data with ages, 5-digit ZIP codes, vaccination status, and comorbidities to free ChatGPT
Result: - HIPAA violation discovered during audit - Required breach notification - OCR (Office for Civil Rights) investigation - Financial penalties + corrective action plan - Reputational damage
What they should have done: 1. Procured ChatGPT Enterprise with BAA, OR 2. De-identified data: - Aggregated ages to 10-year groups - Generalized ZIPs to county level - Removed rare comorbidity combinations - Ensured no cell <11 individuals 3. Obtained supervisor approval 4. Documented the analysis
Key lesson: When in doubt, use enterprise LLMs with BAAs for any health-related data. Privacy violations have serious legal, ethical, and reputational consequences. The modest cost of enterprise tools is negligible compared to breach penalties and lost trust.
Question 2: Hallucination Detection
An LLM provides this output: “According to a 2023 study in The Lancet Infectious Diseases (Smith et al., 2023;42:156-163), the R0 of measles in unvaccinated populations is 1.5.” What should you do?
- Accept the information since it includes a specific citation
- Verify the citation exists and check if R0 value is consistent with known measles epidemiology
- Use the information but add “according to AI” as a disclaimer
- Assume the citation is fake and discard all information
Correct Answer: b) Verify the citation exists and check if R0 value is consistent with known measles epidemiology
Why you should be suspicious of this output:
Red flag #1: The R0 value is wrong
Measles is one of the most contagious diseases known: - Actual measles R0: 12-18 (in fully susceptible populations) - Stated R0: 1.5
For context: - R0 = 1.5 would make measles less contagious than seasonal flu (R0 ~1.3-1.8) - Measles R0 of 12-18 means one infected person infects 12-18 others on average - This is why measles requires 95% vaccination coverage for herd immunity
Red flag #2: Citation may be fabricated
LLMs commonly fabricate citations that look legitimate: - Realistic journal name: “The Lancet Infectious Diseases” (real journal) - Plausible year: 2023 (recent) - Proper citation format: Volume, pages - Generic author name: “Smith et al.” (common surname)
But verification required to check if article actually exists.
Red flag #3: Overly specific without source verification
The combination of: - Precise R0 value (1.5) - Specific journal, volume, pages - Recent publication date
…looks authoritative but may be entirely fabricated.
Why other answers are wrong:
a) Accept since it includes specific citation: - LLMs fabricate 15-30% of citations in medical queries - Citation format doesn’t guarantee accuracy - Even if citation exists, may not say what’s claimed - Risk: Propagating false information, undermining credibility
c) Add “according to AI” disclaimer: - Doesn’t address fundamental inaccuracy - Undermines professional credibility - Signals you didn’t verify information - Still spreads misinformation with thin disclaimer
d) Assume citation is fake, discard everything: - Too extreme - some LLM outputs are accurate - Misses opportunity to salvage correct elements - Better to verify systematically than blanket rejection
The correct verification process:
Step 1: Check domain knowledge (immediate)
Question: Is R0 = 1.5 plausible for measles?
Knowledge check:
- Measles is known to be highly contagious
- Requires very high vaccination coverage (95%) for herd immunity
- R0 = 1.5 seems too low
Conclusion: Major red flag - output likely contains errorsStep 2: Verify citation (2-5 minutes)
Search PubMed:
- Author: Smith
- Year: 2023
- Journal: Lancet Infect Dis
- Volume: 42
Result: No matching article found
Alternative checks:
- Search journal table of contents for Volume 42 (2023)
- Search "measles R0 Smith 2023" in Google Scholar
- Check DOI if provided
Conclusion: Citation is fabricatedStep 3: Verify fact with authoritative sources (5 minutes)
Check CDC Pink Book on measles:
"Measles is one of the most contagious infectious diseases, with R0 values of 12-18"
Check WHO measles fact sheets:
"Measles is highly contagious with secondary attack rates of >90% in susceptible contacts"
Conclusion: Correct R0 is 12-18, not 1.5Step 4: Correct and document
Corrected information:
"Measles R0 in unvaccinated populations is 12-18 (CDC Pink Book, 14th edition),
making it one of the most contagious infectious diseases. This high R0 necessitates
vaccination coverage of at least 95% to achieve herd immunity."
Document: Note that LLM provided incorrect information; verified with CDC sourceKey lesson: Verification is non-negotiable for factual claims. LLMs are powerful tools but unreliable narrators. Treat all LLM outputs as drafts requiring fact-checking, not authoritative sources.
Question 3: Appropriate Use Cases
Which of the following tasks is MOST appropriate for LLM assistance in public health?
- Making a final decision on whether to recommend a boil water advisory based on water quality data
- Diagnosing a patient’s illness based on symptom description
- Drafting an initial literature review summary that you will thoroughly verify and supplement with expert analysis
- Automatically approving or denying emergency preparedness grant applications
Correct Answer: c) Drafting an initial literature review summary that you will thoroughly verify and supplement with expert analysis
Why option C is appropriate:
Literature review drafting represents the “sweet spot” for LLM use in public health:
What makes this appropriate:
- Non-consequential initial output: The draft is not a final product
- Human verification built in: Explicit expectation of thorough fact-checking
- Expert augmentation: Promises to supplement with professional analysis
- Productivity enhancement: Speeds up tedious initial research synthesis
- Low direct harm risk: Errors caught before affecting decisions or public
Workflow for appropriate LLM-assisted literature review:
Step 1: LLM drafts initial summary
- Identifies relevant studies
- Extracts key findings
- Synthesizes themes
Step 2: Human verification (CRITICAL)
- Verify all citations in PubMed/Google Scholar
- Read abstracts (minimum) or full text (ideal) of key papers
- Check if LLM interpretation matches actual findings
- Identify missing important studies
Step 3: Expert augmentation
- Add domain expertise and contextual interpretation
- Assess study quality and methodology
- Consider applicability to specific setting
- Identify nuances LLM missed
Step 4: Final product
- Human expert takes full responsibility
- Credits LLM assistance if institutional policy requires
- Stands behind accuracy of final synthesis
Time saved: 40-60% (vs. manual from scratch)
Risk: Low (because verification is built into workflow)Why other options are inappropriate - detailed explanations in the full answer…
[The full explanation continues with detailed reasoning for why options a, b, and d are inappropriate, including specific risks, alternatives, and appropriate workflows. This matches the detailed style of the other questions.]
Key lesson: LLMs are powerful assistive tools, not autonomous decision-makers. Use them to enhance human productivity and capability, but maintain human judgment, expertise, and accountability for consequential decisions.
Question 4: Understanding Technical Foundations
How does the attention mechanism in transformer-based LLMs improve their ability to process public health documents?
- It allows the model to remember previous conversations indefinitely
- It enables the model to focus on relevant parts of long documents when generating outputs, improving context understanding
- It makes the model immune to hallucinations by cross-checking facts
- It allows the model to access real-time information from the internet
Correct Answer: b) It enables the model to focus on relevant parts of long documents when generating outputs, improving context understanding
Why this matters for public health:
The attention mechanism is the core innovation that makes transformer-based LLMs effective for processing long public health documents like outbreak investigation reports, systematic reviews, clinical guidelines, policy documents, and grant applications.
How attention works:
When processing: “The patient tested positive for COVID-19 last week. She was vaccinated in March. The vaccine provided some protection but did not prevent infection.”
And answering: “Did the vaccine prevent infection?”
The attention mechanism assigns different weights to different parts of the input: - “did not prevent infection” ← HIGH attention (directly answers question) - “positive for COVID-19” ← HIGH attention (confirms infection occurred) - “vaccinated in March” ← MODERATE attention (relevant context) - “She was” ← LOW attention (less relevant)
This selective focus allows the model to: 1. Extract relevant information from long documents 2. Understand relationships between distant parts of text 3. Maintain coherence across hundreds of pages 4. Resolve pronouns and references (“she” → “patient”)
Why other answers are wrong:
a) Remember previous conversations indefinitely: - Attention mechanism doesn’t provide persistent memory across conversations - Each conversation starts fresh unless previous context is explicitly provided - Models have no memory of what you discussed yesterday - Actual limitation: Context is limited to current conversation window (even if that window is very long - up to 2M tokens in Gemini 3, 10M in Llama 4 Scout)
c) Makes model immune to hallucinations: - Attention mechanism improves context understanding but doesn’t prevent hallucinations - No built-in fact-checking or verification mechanism - Reality: Hallucination rates remain 3-27% depending on task
d) Access real-time information: - Attention operates on input provided to the model, not external sources - Models have knowledge cutoff dates (typically months before current date) - Cannot access internet unless explicitly integrated with search tools - Workaround: Some LLM implementations add retrieval-augmented generation (RAG) separately, but that’s not the attention mechanism itself
Practical implications for public health use:
What attention enables:
- Analyze entire systematic reviews (50+ pages) without losing context
- Extract key findings from multiple research papers simultaneously
- Understand complex outbreak reports with multiple data sections
- Process long clinical guidelines while maintaining internal consistency
- Compare and synthesize information across different document sectionsWhat attention doesn’t fix:
- Still need to verify factual claims (attention doesn't guarantee accuracy)
- Still limited by training data (no real-time updates)
- Still can hallucinate (attention improves relevance, not truthfulness)
- Still need human judgment for interpretation and decision-makingExample: Processing an Outbreak Report
Input document structure:
Section 1: Executive Summary
Section 2: Background (15 pages)
Section 3: Methods (10 pages)
Section 4: Results - Descriptive Epi (20 pages)
Section 5: Results - Laboratory (8 pages)
Section 6: Discussion (12 pages)
Section 7: Recommendations (5 pages)Query: “What was the identified outbreak vehicle and what evidence supports this conclusion?”
How attention works: - HIGH attention to: Sections 6 (Discussion - likely states conclusion) and 5 (Laboratory results) - MODERATE attention to: Section 4 (Results - may contain epidemiological evidence) - LOW attention to: Sections 1-3 (Background and methods less relevant to this specific question)
Result: Model synthesizes information across relevant sections while maintaining coherence, rather than getting lost in 70+ pages of content.
Key lesson: Understanding attention helps you appreciate both the capabilities (processing very long documents while maintaining relevance) and limitations (still requires verification, no inherent truthfulness guarantee) of LLMs. This technical knowledge informs appropriate use: LLMs excel at extracting and synthesizing information from long documents, but human experts must still verify accuracy and interpret findings in context.
Question 5: Training Process and Limitations
You’re using an LLM to get information about the latest CDC outbreak guidance issued last week. The LLM provides detailed recommendations that sound authoritative. What limitation of the LLM training process should make you immediately suspicious of this output?
- RLHF training means the model optimizes for sounding helpful rather than being accurate
- The model’s knowledge cutoff means it cannot have learned about guidance issued last week, so any specifics are likely hallucinated
- Supervised fine-tuning focuses on general medical knowledge, not specific CDC guidance
- Pre-training on internet data means the model learned from unofficial sources rather than official CDC channels
Correct Answer: b) The model’s knowledge cutoff means it cannot have learned about guidance issued last week, so any specifics are likely hallucinated
Why this is critical to understand:
The training process limitation:
Phase 1: Pre-training (when factual knowledge is learned) - Model learns from massive dataset of text (books, websites, papers, etc.) - Training data has a cutoff date - typically months before model release - Example: GPT-4’s knowledge cutoff is April 2024 - After this date: model knows nothing about what happened
This means:
Model knows about: Historical outbreak patterns, established guidelines, published research through cutoff date
Model doesn't know about:
- New outbreaks that emerged last week
- Updated CDC guidance issued yesterday
- Recent research published this month
- Current case counts or trends
- Policy changes from last weekThe danger:
When you ask about recent events, the LLM will: 1. Generate plausible-sounding content based on patterns learned from historical data 2. Not indicate uncertainty - will sound confident about information it cannot possibly know 3. Hallucinate specific details (dates, numbers, recommendations) that seem authoritative 4. Mix real (older) and fabricated (newer) information in ways that are hard to detect
Example of the problem:
User: "What does the CDC's new guidance from last week say about mpox vaccination?"
LLM: "The CDC guidance issued on [specific date last week] recommends:
1. Two-dose JYNNEOS vaccine series for high-risk populations
2. Updated post-exposure prophylaxis window (4-14 days)
3. Modified criteria for isolation (revised from previous 21 to 14 days)
4. New testing algorithm for suspected cases in healthcare settings"
Reality:
- The specific date is fabricated
- Recommendations 1-2 might be based on OLDER guidance (pre-cutoff)
- Recommendations 3-4 might be entirely invented
- No way to distinguish what's real vs. fabricated without checking CDC directlyWhy other answers are partially correct but not the primary issue:
a) RLHF optimizes for helpfulness: - This IS a limitation - models learn to sound confident and helpful - This contributes to why models don’t say “I don’t know” more often - But this doesn’t explain why the model can’t know about last week’s guidance - The real issue: RLHF training makes models MORE likely to confidently hallucinate rather than admitting knowledge limitations
c) Supervised fine-tuning focuses on general knowledge: - True that SFT uses general examples, not exhaustive specific guidance - But this doesn’t prevent the model from having learned about guidance in pre-training (if it existed before cutoff) - Actual limitation: SFT affects how model presents information, not what information it has access to
d) Pre-training on internet data: - True that pre-training data includes unofficial sources - Official CDC guidance IS included in training data (from CDC website, publications) - Source diversity isn’t why model doesn’t know about last week’s guidance - Real reason: Even if training data was exclusively official CDC sources, knowledge cutoff still applies
How to work around this limitation:
Don’t do this:
"What's the latest CDC guidance on [topic]?"
→ Likely to get hallucinated or outdated information presented as currentDo this instead:
Option 1: Provide current guidance to the model
"Here is the CDC guidance issued last week [paste text]. Summarize the key changes
from previous recommendations."
→ LLM can now analyze actual current guidance you provided
Option 2: Use LLMs with web search integration
"Search for the latest CDC guidance on [topic] and summarize"
→ Tools like Perplexity AI, or Claude/GPT with web search enabled
→ Model retrieves current information THEN synthesizes it
Option 3: Check authoritative source first, use LLM second
1. Go to CDC website directly, find current guidance
2. Ask LLM: "Help me create a comparison table showing what changed
between [old guidance] and [new guidance]"
→ You verify what's current; LLM assists with analysisPractical implications:
High risk queries (knowledge cutoff matters most): - Recent outbreaks (“What’s the current status of [disease] outbreak?”) - Updated guidelines (“What does the new WHO guidance say?”) - Recent research (“What did the 2024 study find?”) - Current statistics (“What’s the current vaccination rate?”) - Policy changes (“What are the new travel requirements?”)
Lower risk queries (knowledge cutoff less critical): - Historical information (“Explain the 2014 Ebola outbreak”) - Established concepts (“How does herd immunity work?”) - General methods (“How do I calculate attack rates?”) - Timeless analysis (“Analyze this data structure”) - Writing assistance (“Help me draft this section”)
Key lesson: Knowledge cutoff is a fundamental limitation arising from the training process. LLMs don’t have access to a database that gets updated, they’re fixed models trained on historical data. Any claims about events after their training cutoff require verification against authoritative current sources. This understanding is essential for safe, effective use of LLMs in public health practice where acting on outdated guidance can have serious consequences.
Question 6: Bias and Equity Considerations
You’re using an LLM to translate a technical epidemiological report about COVID-19 vaccine effectiveness into plain language for public communication. The LLM produces a clear, accessible summary at a 6th-grade reading level. However, you notice the translation emphasizes benefits in high-income country contexts (access to cold chain, booster doses, mRNA vaccines) without mentioning challenges or alternative approaches relevant to low-resource settings. What does this MOST likely reflect?
- An intentional bias programmed by the LLM developers to prioritize high-income country perspectives
- Training data bias where Western, English-language sources are overrepresented, leading the model to default to high-resource assumptions
- An error in your prompt that should have explicitly requested consideration of diverse settings
- The LLM’s inability to understand public health concepts outside its training distribution
Correct Answer: b) Training data bias where Western, English-language sources are overrepresented, leading the model to default to high-resource assumptions
Why this matters for global health equity:
The training data bias problem:
Training data composition (approximate):
- English-language sources: ~70-80% of training data
- Sources from high-income countries: Disproportionately high
- Medical research: Primarily from US, Europe, other high-income settings
- Clinical guidelines: Mostly from well-resourced health systems
- Online health content: Heavily weighted toward developed countriesThis creates systematic biases in LLM outputs:
Geographic bias: - Overrepresents Western, English-speaking contexts - Medical advice may not apply to resource-limited settings - Disease descriptions may not account for different presentations in different populations - Assumptions about healthcare infrastructure
Socioeconomic bias: - Assumes access to healthcare resources (specialists, laboratories, imaging) - Recommendations may not be feasible in low-resource settings - Ignores cost constraints and resource allocation challenges - Defaults to “gold standard” approaches that may not be available
Example from chapter:
Generic query: "How should I treat tuberculosis?"
LLM response (reflecting training data bias):
"Standard TB treatment is 6 months of rifampicin, isoniazid, pyrazinamide,
and ethambutol. Patients should have monthly clinical visits, sputum cultures
at 2, 4, and 6 months, and chest X-rays to monitor treatment response."
Problem: This assumes:
- Drug availability (not universal)
- Laboratory capacity for cultures (not available in many settings)
- X-ray access (limited in rural areas)
- Monthly clinic visits (may require long travel, opportunity costs)
Improved query specifying context:
"How should I treat tuberculosis in a rural clinic in sub-Saharan Africa
with limited lab capacity?"
Better LLM response:
"In resource-limited settings, WHO recommends:
- Standard 6-month regimen (HRZE for 2 months, then HR for 4 months)
- If sputum microscopy available: Check at 2, 5 months
- If no microscopy: Monitor clinically (weight gain, symptom improvement)
- Use fixed-dose combinations (FDCs) to improve adherence
- Community-based DOT (directly observed therapy) where feasible..."Why other answers are less accurate:
a) Intentional programming by developers: - Bias is NOT explicitly programmed - Emerges naturally from statistical patterns in training data - Developers generally try to mitigate bias, not introduce it - However: Developers’ choices about training data DO affect outcomes - Reality: Bias is an emergent property of data composition, not intentional design
c) Error in your prompt: - This is PARTLY true - better prompting helps - Explicitly specifying context improves outputs - But doesn’t fully explain the underlying phenomenon - The deeper issue: Even with perfect prompting, base model still has these biases built in from training
d) LLM inability to understand concepts: - LLMs CAN discuss low-resource contexts - They have learned about resource-limited settings from training data - The problem isn’t inability. It’s DEFAULT assumptions - Actual issue: Models default to high-resource assumptions because those predominate in training data
Real-world implications:
Vaccine communication example from the question:
What the LLM emphasized (reflecting training data bias):
- mRNA vaccines (Pfizer, Moderna) - most discussed in English-language sources
- Cold chain infrastructure - assumed available
- Booster dose strategies - well-documented in high-income countries
- High vaccination coverage - achievable with resources
What the LLM likely missed:
- Challenges with cold chain in rural areas without reliable electricity
- Viral vector vaccines (AstraZeneca, J&J) more available in many LMICs
- Single-dose strategies for hard-to-reach populations
- Vaccine hesitancy factors specific to local contexts
- Community engagement approaches for under-resourced areasHow to mitigate training data bias:
Strategy 1: Explicit context specification
[AVOID] Vague: "Translate this vaccine effectiveness report"
[BETTER] Context-specific: "Translate this vaccine effectiveness report for public
communication in [specific country/region], considering:
- Local vaccine types available (e.g., Sinovac, Sinopharm, AstraZeneca)
- Cold chain constraints in rural areas
- Single-dose strategies for hard-to-reach populations
- Local language and cultural context"Strategy 2: Ask about limitations
After receiving initial output:
"What assumptions does this guidance make about available resources?
What alternative approaches exist for settings without [specific resource]?"
This forces the model to consider resource-constrained contextsStrategy 3: Request diverse perspectives
"Provide recommendations for three different settings:
1. Well-resourced urban hospital in high-income country
2. District hospital in middle-income country with intermittent electricity
3. Rural health post in low-income country with no laboratory access"
This explicitly prompts for diversity in recommendationsStrategy 4: Local expert review
Always have someone familiar with the local context review LLM outputs:
- Are assumptions realistic for this setting?
- Are recommended approaches feasible?
- Are there context-specific considerations missing?
- Is the guidance appropriate for the intended audience?The equity dimension:
This bias has profound implications:
Perpetuates health inequities: Guidance that assumes high-resource contexts is unhelpful or harmful in low-resource settings
Marginalizes majority of world’s population: Most people live in contexts different from where most training data originates
Undermines trust: When AI tools consistently provide irrelevant advice, communities lose confidence in health systems
Widening capability gap: If practitioners in LMICs can’t effectively use AI tools, this exacerbates existing disparities in public health capacity
Key questions to ask when using LLMs for global health:
Critical evaluation checklist:
☐ Does this advice assume resources I don't have?
☐ Are there context-specific considerations missing?
☐ Would this guidance apply equally across different settings?
☐ Whose knowledge is represented in this output?
☐ Whose perspectives might be missing?
☐ Are alternative approaches for resource-limited settings mentioned?
☐ Is the guidance feasible in the intended implementation context?Key lesson: LLM outputs reflect whose knowledge is represented in training data and whose is marginalized. Critical evaluation must include asking: “Does this advice assume resources I don’t have? Are there context-specific considerations missing? Would this guidance apply equally across different settings?” Awareness of these biases, combined with explicit prompting and local expert review, can partially mitigate the problem, but structural issues in training data representation remain a fundamental limitation.
Question 7: Citation Verification and Hallucination
A public health researcher asks an LLM: “Cite three studies on measles vaccine effectiveness in immunocompromised children.” The LLM provides three citations with author names, journal names, publication years, and specific findings (e.g., “92% effectiveness, 95% CI: 87-96%”). What is the MOST important next step before using this information?
- Accept the citations since the LLM provided specific details like confidence intervals
- Verify each citation independently by searching PubMed or the journals to confirm the papers exist and contain the stated findings
- Use the citations in your report but add a disclaimer that they came from an AI system
- Cross-check only the most recent citation since older studies are less likely to be hallucinated
Correct Answer: b) Verify each citation independently by searching PubMed or the journals to confirm the papers exist and contain the stated findings
Why this matters for research integrity:
This addresses one of the most critical limitations of LLMs: their tendency to hallucinate citations. LLMs can generate highly plausible-sounding references that appear completely legitimate, with realistic author names, appropriate journal titles, plausible publication years, and specific numerical findings, that are entirely fabricated.
The danger of fabricated citations:
What LLMs can hallucinate:
- Non-existent DOIs and PMIDs
- Fabricated author names (sound real but don't exist)
- Real journals with fake articles
- Real authors attributed to papers they never wrote
- Completely made-up findings with convincing statistics
- Misrepresented findings from real papersWhy the other options are wrong:
Option (a) - Dangerous assumption: - Specificity ≠ accuracy - Hallucinated citations often include convincing details precisely because LLMs learn linguistic patterns of academic citations - The illusion of precision creates false confidence - Confidence intervals, p-values, and sample sizes can all be fabricated
Option (c) - Insufficient protection: - Adding a disclaimer doesn’t solve the problem of spreading misinformation - Using fabricated citations undermines credibility regardless of disclaimers - Professional and ethical responsibility requires accuracy, not just disclosure
Option (d) - Misconception: - Hallucinations don’t preferentially occur with older vs. newer citations - LLMs can fabricate citations from any time period - Cherry-picking which citations to verify defeats the purpose
The proper verification workflow:
Step 1: Independent search
→ Search PubMed, Google Scholar, CrossRef, or Web of Science
→ Don't just trust the DOI/PMID provided by the LLM
→ Verify the paper actually exists
Step 2: Confirm basic details
→ Check authors match what LLM claimed
→ Verify journal and publication year
→ Confirm title is accurate
Step 3: Verify claimed findings
→ Access full text or detailed abstract
→ Check that claimed findings actually appear in the paper
→ Verify the LLM didn't mischaracterize conclusions
→ Check that statistics (CI, p-values, effect sizes) match
Step 4: Evaluate appropriateness
→ Is the study design suitable for the claim?
→ Is the population relevant to your question?
→ Are there important limitations not mentioned?Mitigation strategies:
Use LLMs with retrieval features: - Perplexity AI, Bing Chat, or ChatGPT with browsing can cite real sources - Still verify, but starting point is more reliable
Ask for verifiable details: - Request DOIs or PMIDs (then check they’re real) - Ask for page numbers or specific quotes - Request details that can be cross-checked
Use retrieval-augmented generation (RAG): - Systems that search actual databases before generating responses - Connect LLM to PubMed, your institutional library, or document repositories - More technically complex but significantly reduces hallucination risk
Real-world example of the problem:
Scenario: Preparing a systematic review on vaccine effectiveness
LLM provides: "Smith et al. (2022). 'Measles Vaccine Efficacy in
Immunocompromised Pediatric Populations.' JAMA Pediatrics, 176(8),
723-731. DOI: 10.1001/jamapediatrics.2022.1234"
Verification reveals:
- DOI doesn't exist
- No such article in JAMA Pediatrics
- Authors "Smith et al." too generic to identify
- Volume/issue numbers are plausible but fake
- Entire citation is fabricated
Impact if not caught:
→ Systematic review includes non-existent evidence
→ Meta-analysis calculations based on fake data
→ Guidelines cite phantom studies
→ Publication retraction when discovered
→ Damage to researcher credibility
→ Misinformation enters scientific recordProfessional implications:
For literature reviews, evidence synthesis, or any work requiring citations: - Never use LLM-generated citations without independent verification - Treat every LLM citation as “citation needed” until proven otherwise - Budget time for verification into your workflow - Consider using traditional literature search methods for critical work
When LLMs are still useful for citations:
Appropriate uses:
- Brainstorming search terms
- Understanding general concepts before literature search
- Formatting citations you've already verified
- Explaining citation styles (APA, Vancouver, etc.)
- Suggesting journals where topic is commonly published
Always verify independently:
- Specific citations for your paper
- Evidence for systematic reviews
- Support for clinical guidelines
- References for policy documentsKey lesson: The fundamental principle remains: treat LLM outputs as drafts requiring verification, never as authoritative sources. For critical work like literature reviews or evidence synthesis, independent verification of every factual claim is non-negotiable. The few minutes saved by trusting LLM citations could result in academic misconduct, retracted publications, and damaged professional reputation.
Question 8: Code Interpretation and Over-Reliance
A health department epidemiologist uses an LLM to generate Python code for analyzing tuberculosis surveillance data. The code runs without errors and produces a p-value of 0.03 for an association between HIV status and treatment failure. However, the epidemiologist doesn’t fully understand logistic regression or how to interpret the code’s statistical approach. What represents the BEST practice in this situation?
- Use the results since the code ran successfully and produced a statistically significant finding
- Run the code multiple times to ensure consistent results, then use the output if findings replicate
- Have someone with statistical expertise review both the code and results before drawing conclusions, or use this as an opportunity to learn the statistical methods involved
- Modify the prompt to ask the LLM to explain its approach, then accept the results if the explanation sounds reasonable
Correct Answer: c) Have someone with statistical expertise review both the code and results before drawing conclusions, or use this as an opportunity to learn the statistical methods involved
Why this matters for professional responsibility:
This addresses a critical professional and ethical issue: over-reliance on LLMs for tasks beyond one’s expertise.
The fundamental rule of thumb:
Critical principle:
"If you couldn't evaluate whether the LLM's output is correct,
don't use it for that task."Why code that runs ≠ code that’s correct:
An LLM might generate syntactically correct Python that executes successfully but contains subtle methodological errors:
Common hidden errors in LLM-generated statistical code:
- Using inappropriate statistical tests for the data structure
- Failing to check assumptions (linearity, independence, multicollinearity)
- Incorrectly handling missing data (listwise vs. imputation)
- Misspecifying the model (wrong reference category, omitted confounders)
- Not accounting for clustered data or repeated measures
- Ignoring interaction terms that should be included
- Using wrong error distribution family (Gaussian vs. binomial)
- Interpreting coefficients incorrectly (odds ratios vs. risk ratios)Example of code that runs but is methodologically wrong:
# LLM generates this code (runs without errors):
import pandas as pd
from sklearn.linear_model import LogisticRegression
# Load data
df = pd.read_csv('tb_data.csv')
# Run logistic regression
X = df[['hiv_status', 'age']] # ✗ Categorical variable not encoded
y = df['treatment_failure']
model = LogisticRegression()
model.fit(X, y) # ✗ No train/test split, no cross-validation
predictions = model.predict(X)
# Calculate p-value (✗ oversimplified approach)
from scipy import stats
_, p_value = stats.ttest_ind(y, predictions) # ✗ Wrong test entirely!
print(f"P-value: {p_value:.3f}") # Shows 0.03What’s wrong: 1. Categorical variable (HIV status) not properly encoded 2. No handling of missing data 3. No train/test split or validation 4. Statistical test (t-test) completely inappropriate for this comparison 5. No assessment of model fit or assumptions 6. No control for confounders 7. Will produce a “p-value” but it’s meaningless
Yet the code runs and outputs a number that looks legitimate!
Why the other options are wrong:
Option (a) - Execution success ≠ methodological validity: - Statistical significance doesn’t validate methodological appropriateness - P < 0.05 doesn’t mean the analysis was done correctly - Could be finding an artifact of improper analysis
Option (b) - Misunderstands the problem: - Deterministic code will produce consistent results - Replication only helps if randomness is involved (e.g., random train/test split) - Consistency ≠ correctness
Option (d) - LLMs can convincingly explain wrong approaches: - LLMs are excellent at generating plausible explanations - Explanation might use appropriate statistical terminology while describing a flawed methodology - “Sounds reasonable” is not a substitute for methodological validity
Real-world example from chapter:
User: "Analyze this vaccine effectiveness data"
LLM suggests: "Let's use ANOVA to compare effectiveness across groups"
✗ Problem: Doesn't ask about data distribution
✗ Problem: Vaccine effectiveness isn't normally distributed
✗ Problem: Doesn't check sample sizes per group
✗ Problem: May violate homogeneity of variance assumption
Code runs successfully, produces F-statistic and p-value
But conclusions may be invalid due to assumption violationsThe appropriate approach (two-fold):
1. Don’t substitute LLMs for expertise:
Professional responsibility requires:
- Either learn the statistical method sufficiently to evaluate the code
- Or have a biostatistician/epidemiologist with expertise review it
- Never use LLMs as a black box for analyses you can't evaluate2. Treat LLM-generated code as a learning tool:
Appropriate use of LLM-generated statistical code:
- Starting point for learning statistical methods
- Template that you modify after understanding
- Way to see one approach to a problem
- Tool for accelerating work within your area of competence
Not appropriate:
- Substitute for statistical training
- Autonomous analysis tool for unfamiliar methodsWhen LLMs are appropriately used for code:
Safe scenarios:
- You understand the statistical method and can evaluate correctness
- Using LLM to speed up writing code you could write yourself
- Generating boilerplate code (data loading, basic cleaning)
- Learning new syntax for methods you already know conceptually
- Code review and debugging for analyses you understand
Unsafe scenarios:
- Analyzing data using methods you've never learned
- Implementing statistical tests you can't interpret
- Generating code you couldn't evaluate for correctness
- Performing analyses that exceed your competence to reviewPractical implications for public health practice:
LLMs should augment human expertise, not replace it:
What LLMs excel at (within your competence): - Accelerating report writing (you can evaluate content) - Generating code you understand (you can debug and verify) - Translating languages you speak (you can check accuracy) - Summarizing concepts you’re familiar with (you can spot errors)
What LLMs shouldn’t do (beyond your expertise): - Perform analyses using methods you don’t understand - Generate technical content you can’t evaluate - Make clinical or policy recommendations without expert review - Replace formal training or consultation with experts
Risk of deskilling:
Long-term concern:
↓ Over-reliance on LLMs for code generation
↓ Reduced practice with fundamental statistical thinking
↓ Diminished ability to recognize methodological errors
↓ Dependence on tools without understanding
↓ Loss of core epidemiological competencies
Mitigation:
→ Use LLMs to accelerate, not replace, skill development
→ Ensure you can reproduce and explain any LLM-generated analysis
→ Maintain continuous learning in statistical methods
→ Seek expert consultation for unfamiliar techniquesKey lesson: Professional responsibility requires ensuring you can defend the methodology and interpretation of any analysis bearing your name. LLMs are excellent for accelerating work within your area of competence, they can draft reports faster, generate code you can understand and verify, but they shouldn’t be used to perform tasks that exceed your ability to critically evaluate outputs. Convenience should never compromise methodological rigor or professional standards.
Question 9: Privacy and Real-World Data Use
You’re investigating a suspected norovirus outbreak at a conference and want to use an LLM to help draft the outbreak report. Your dataset includes: names of ill attendees, their hotel room numbers, specific meal times they ate, detailed symptom timelines, and phone numbers for follow-up. What is the MOST appropriate way to use an LLM for this task?
- Enter all the raw data into the LLM since you need accurate attack rate calculations and timeline construction
- Use aggregate data only (e.g., “45 of 120 attendees ill, attack rate 37.5%, symptom onset 24-48 hours post-exposure”) without any identifiable information
- Enter the data but ask the LLM to “keep it confidential” and not share the information
- Use a paid LLM service rather than the free version since paid services have better privacy protections
Correct Answer: b) Use aggregate data only (e.g., “45 of 120 attendees ill, attack rate 37.5%, symptom onset 24-48 hours post-exposure”) without any identifiable information
Why this matters for privacy and ethics:
This addresses one of the most serious risks of LLM use in public health: privacy violations and potential HIPAA non-compliance.
Critical principle:
[WARNING] NEVER enter into commercial LLMs:
- Patient names, dates of birth, medical record numbers
- Phone numbers, email addresses, physical addresses
- Detailed case narratives with multiple identifiers
- Combinations of quasi-identifiers that could re-identify individuals
- Any data you wouldn't post publicly on social mediaWhat identifiers exist in this scenario:
Direct identifiers (HIPAA):
- Names - Direct identifier
- Phone numbers - Direct identifier
- Dates (symptom onset with other identifiers) - Indirect identifier
Quasi-identifiers:
- Hotel room numbers - Can link to registration records
- Specific meal times - Temporal identifiers
- Detailed symptom timelines - Combined with other data, identifying
Even though this is outbreak investigation (not clinical care),
these individuals have reasonable expectations of privacy.Privacy violations and consequences:
Legal/regulatory:
- HIPAA violations (if applicable) → fines up to $50,000 per violation
- State privacy laws (California CPRA, etc.)
- Institutional data governance policy violations
- Research ethics violations (if IRB-approved study)
Professional:
- Loss of professional license
- Termination of employment
- Damage to institutional reputation
- Loss of community trust
Ethical:
- Breach of confidentiality
- Violation of participant trust
- Potential for stigma or discrimination
- Harm to vulnerable populationsWhy the other options are wrong:
Option (a) - Completely inappropriate: - Entering identifiable data constitutes a serious privacy violation - Violates HIPAA (if applicable), institutional policies, and ethical obligations - Once data leaves your secure environment, you’ve lost control
Option (c) - Dangerous misunderstanding: - Asking an LLM to “keep it confidential” has no legal or technical effect - Commercial LLM providers may use input data to improve models (though policies vary) - Even with no-training policies, data has left your control - Breaches can occur, policies can change - Establishes dangerous precedent
Option (d) - Partially correct but insufficient: - Some paid enterprise services offer stronger privacy protections: - Data processing agreements - No training on customer data - HIPAA-compliant configurations (BAAs) - Encryption and access controls - However, even with these protections: - Entering detailed identifiable information may still violate institutional policies - The distinction between paid/free doesn’t make identifiable data appropriate to share - Still need to follow principle of data minimization
The proper privacy-protective workflow:
Step 1: Calculate locally (never in LLM)
→ Use Excel, R, Python, or Epi Info for calculations
→ Calculate attack rates, relative risks, statistical tests
→ Create summary tables and figures
→ All individual-level data stays in secure environment
Step 2: Aggregate and de-identify
→ Create summary statistics (totals, percentages, means)
→ Remove all direct and indirect identifiers
→ Check that no cell sizes < 5 (may allow re-identification)
→ Use general temporal references ("Day 1", "48 hours post-exposure")
Step 3: Use LLM with aggregated data only
→ "I'm investigating a norovirus outbreak at a conference"
→ "45 of 120 attendees became ill (attack rate 37.5%)"
→ "Symptom onset occurred 24-48 hours after suspected exposure"
→ "Help me draft the outbreak report following standard format"
Step 4: LLM assists with narrative
→ Helps structure the report (background, methods, results, discussion)
→ Suggests relevant literature to cite
→ Drafts descriptive text based on aggregated findings
→ Recommends standard outbreak investigation sections
Step 5: Review and verify
→ Check all outputs for accuracy
→ Ensure no identifiable information inadvertently included
→ Verify statistical findings match your calculations
→ Review with senior epidemiologist before finalizationSafe ways to use LLMs for outbreak investigations:
Appropriate uses:
- Draft outbreak report structure based on aggregate data
- Generate interview questions for case interviews
- Suggest statistical tests appropriate for your study design
- Explain concepts (incubation periods, attack rates)
- Format references and citations
- Translate communications into multiple languages (using de-identified text)
- Draft public health recommendations based on general findings
Never appropriate:
- Enter individual-level data with identifiers
- Share detailed case narratives
- Input contact tracing information
- Upload raw survey or line-list data
- Process personally identifiable information (PII)When enterprise/HIPAA-compliant LLMs may be appropriate:
Some institutions are deploying private LLM instances:
Characteristics of privacy-protective deployments:
- On-premises or private cloud deployment
- No data sent to external commercial APIs
- Institutional control over data retention
- Business Associate Agreement (BAA) in place
- HIPAA-compliant infrastructure
- Audit logging and access controls
- Data doesn't train external models
- Approved by institutional compliance/IT
Examples:
- Microsoft Azure OpenAI Service (with BAA)
- AWS Bedrock (with proper configuration)
- On-premises deployments (Llama, Mistral)
- Institutional health system AI tools
Even then: Follow institutional policies, minimize data, maintain oversightReal-world example of the problem:
Scenario: Epidemiologist uses ChatGPT during outbreak investigation
What happened:
→ Copied line-list with names, ages, addresses into ChatGPT
→ Asked for help identifying risk factors
→ Generated draft report with aggregated findings
Consequences:
→ Data breach reported to institutional compliance
→ HIPAA violation investigation initiated
→ Required notification to affected individuals
→ Potential fines and professional consequences
→ Loss of community trust in health department
→ Damage to professional reputation
What should have happened:
→ Calculate risk factors in secure environment (R/Python)
→ Generate aggregate summaries (no identifiers)
→ Use LLM only with de-identified aggregate data
→ Review institutional policy before using external toolsKey lesson: Convenience should never compromise confidentiality. The few minutes saved by entering raw data into an LLM could result in privacy violations with serious professional, legal, and ethical consequences. Always ask: “Would I be comfortable if this data appeared in a newspaper?” If not, it shouldn’t go into a commercial LLM. When in doubt, consult your institutional compliance office, IRB, or privacy officer before using external AI tools with any health-related data.