Precision, Recall, and Class Imbalance in Health ML
Many healthcare prediction tasks are rare-event problems. A model can look strong on ROC-AUC and still create
too many false alarms to be useful. This page explains how precision, recall, and prevalence work together
so students can judge screening models more realistically.
Beginner
Learn the difference between catching true cases and sending too many unnecessary alerts.
Intermediate
Compare precision, recall, specificity, and why PR curves often matter more than ROC curves for rare outcomes.
Advanced
Think about prevalence shift, resampling, operating-point choice, and how workload changes across hospitals.
2Questions to ask: how many true cases did we catch, and how many alerts were correct?
1Key warning: lower prevalence usually lowers precision, even with the same model behavior
PRPrecision-recall views are often more informative than ROC views in rare-disease settings
Why this page matters
The portal already covers ROC curves, evaluation metrics, calibration, and threshold choice. The next gap is
the evaluation problem students meet constantly in health research: the outcome is rare, false positives add
workload, and the usual headline metric can hide that.
Simple rule: recall asks how many real cases you caught, while precision asks how trustworthy a positive alert is.
Do not mix up these three ideas
Beginner
Recall
Of all patients who truly have the outcome, how many did the model flag?
High recall means fewer missed cases.
Useful in screening and early detection.
It says nothing about how many flagged patients are false alarms.
Intermediate
Precision
Of all patients the model flagged, how many actually have the outcome?
High precision means alerts are more believable.
Very sensitive models can have poor precision in rare outcomes.
Strongly affected by prevalence.
Advanced
Class imbalance
The positive class is much rarer than the negative class.
Common in disease detection, adverse events, and mortality models.
Can make accuracy and ROC summaries feel better than the clinical workflow really is.
Requires metric choices that respect workload and rarity.
ROC versus PR: both are useful, but not for the same question
When ROC is helpful
Comparing ranking ability across thresholds.
General view of sensitivity versus false positive rate.
Good first check of discrimination.
When PR is more revealing
Rare outcomes where positive predictions are scarce.
Tasks where false alarms create real downstream burden.
Cases where the user asks, "If we alert on someone, how often are we right?"
Healthcare takeaway
Use ROC to study ranking.
Use PR to study alert quality in imbalanced settings.
Choose the threshold based on intended use, not a single favorite metric.
Reality check: a model can keep the same sensitivity and specificity, yet precision can collapse when deployed to a lower-prevalence population.
Imbalance lab: same model behavior, different prevalence
Pick a clinical scenario, then switch between operating points. The sensitivity and specificity change with
threshold choice, but prevalence changes precision too. That is why model evaluation must be tied to the target population.
Toy cohort of 1,000 patients
Rare outcome, balanced threshold
0%Precision
0%Recall
0%Specificity
0Patients flagged
True positives
0
False positives
0
False negatives
0
True negatives
0
Interpretation tip: when the disease is rare, even a good screen may generate many more false positives than true positives.
Practical workflow for imbalanced health outcomes
1
Start from the decision
Clarify whether the model supports screening, triage, confirmatory testing, or specialist referral.
2
Report prevalence explicitly
Precision depends on how common the outcome is in the evaluated population.
3
Show recall and precision together
A model that catches many cases but overwhelms the clinic may still be a poor operating choice.
4
Use validation data for threshold choice
Pick the operating point before final test reporting.
5
Consider workload
Translate false positives into follow-up visits, lab tests, and clinician time.
6
Recheck after transport
If the model moves to a new hospital or time period, precision can change even if the ranking stays good.
Common mistakes
Reporting accuracy alone
With 1% prevalence, a model that predicts everyone negative can still look 99% accurate.
Relying on ROC-AUC only
ROC can stay impressive while precision stays too low for practical use.
Ignoring prevalence shift
A model validated in a referral center may look much less precise in primary care.
Assuming resampling fixes everything
Oversampling or class weights can help training, but they do not remove the need for honest evaluation on realistic data.
Deployment warning: if you rebalance classes during training, remember that the deployed population is still imbalanced. Evaluation must return to that real population.
Quick self-check
These short cases check whether the metric logic is clear before students move on.
Case 1
A rare-disease model catches most true cases but only a small fraction of alerted patients really have the disease. Which metric is low?
Correct answer: Precision. The alerts are not very trustworthy even if recall is high.
Case 2
You deploy the same model in a lower-prevalence population. Which metric is most likely to drop first?
Correct answer: Precision. With fewer true cases in the population, each false positive hurts more.
Case 3
Which summary is often more helpful than ROC for rare-event alert systems?
Correct answer: Precision-recall view. It focuses attention on alert quality and missed cases.
How this connects to other portal pages
After ROC and metrics
This page explains why ROC alone is not enough when the positive class is rare.
After calibration and thresholds
Threshold choice changes recall and precision, while prevalence determines how those alerts feel in practice.