A model is not finished when training ends. In health ML, the hard question is what happens after deployment: do predictions still match patients, does alert workload stay manageable, and when should a team recalibrate, retrain, or retire the model?
The portal already covers safe splitting, class imbalance, calibration, and external validation. The next missing step is what students need for real-world health AI: once a model goes live, teams must monitor whether predictions remain trustworthy, clinically useful, and safe for the surrounding workflow.
The incoming patients, prevalence, or feature values are no longer similar to the development data.
The ranking may still look acceptable, but predicted probabilities become too high or too low.
The meaning of the outcome or the use of the prediction changes because practice changed.
Select a live deployment scenario. Watch how discrimination, calibration, alert burden, and recommended action move together. This is a simplified monitoring dashboard, but it teaches the right habit: do not follow one metric in isolation.
The launch setting still resembles the development data. The goal is to establish a baseline before interpreting later changes.
Use this when shifts are small and performance remains clinically acceptable.
Use this when ranking still works, but probabilities are systematically too high or too low.
Use this when feature relationships or measurements changed enough that calibration alone is not enough.
Use this when the workflow changed so much that the model no longer matches the intended decision.
Define who the model is for, when it runs, who sees the output, and what action follows.
Measure early deployment performance and workload before deciding what counts as abnormal.
Track discrimination, calibration, prevalence, missingness, alert rate, and downstream workload together.
Ask whether the change came from patients, measurements, labels, or workflow before changing the model.
Escalate from monitoring to recalibration, update, or retirement based on evidence and governance.
Version the model, thresholds, and dates of updates so future evaluations remain interpretable.
A model may keep decent ranking while predicted risks or alert workload become unreliable.
Threshold changes alter workload and safety. They should be governed, not improvised during daily operations.
If documentation or adjudication changes, observed performance may shift even without a real biological change.
Every model should have criteria for update, pause, or retirement.
Use these short cases to confirm the main deployment logic before moving on.
A deployed model keeps similar AUC, but predicted risks are now too high for most patients. What is the most likely issue?
A hospital adopts a new lab analyzer and several model inputs now shift abruptly. What should you suspect first?
Which response is strongest when the clinical workflow itself changed and the model no longer matches the decision point?
Use external validation to decide whether a model travels before it goes live.
Use the calibration guide to understand which deployment changes require recalibration or a threshold review.
Read the class imbalance guide when prevalence shift changes precision and alert burden.
This page prepares students for deeper topics such as model updating, governance, impact studies, and reporting standards.