A model that works in one hospital, cohort, scanner, or calendar period may not work the same way somewhere else. This guide explains how to test generalization beyond the development data and how to reason about whether a health ML model is transportable to a new setting.
The portal already covers train/validation/test splits, leakage, metrics, calibration, thresholds, and class imbalance. The missing next step is what health researchers need before claiming clinical usefulness: testing the model on data that differs from the development environment.
Train/test splits or cross-validation within one dataset estimate performance after model development.
Train on earlier patients and test on later patients from the same system.
Test in a different institution, registry, country, device, or care pathway.
When a model moves, the question is not simply "does AUC stay high?" The key is whether the model, outcome definition, measurements, and decision workflow still match the intended use.
Age, comorbidity, disease severity, referral patterns, or prevalence differ across settings.
Lab assays, scanners, coding practices, missingness patterns, or wearable devices are not equivalent.
The outcome definition, follow-up window, adjudication process, or diagnostic intensity changes.
The model is used at a different decision point, by different staff, or with a different downstream action.
Select a target setting. The toy metrics show a common pattern: apparent performance is strongest near the development hospital, while calibration and precision can change sharply when prevalence, measurement, or workflow shifts.
Internal performance is useful, but the model has not yet proven that it travels beyond the development environment.
State the patient group, prediction time, outcome, decision maker, and action supported by the model.
Do not tune features, thresholds, or preprocessing after seeing external outcomes.
Report age, sex, disease severity, outcome prevalence, missingness, and measurement differences.
Show discrimination, calibration, precision/recall, decision thresholds, and uncertainty intervals.
If risks are systematically too high or low, consider recalibration before clinical use.
Say exactly which settings were tested and avoid claiming transportability to untested contexts.
A random split from one dataset is internal validation, even when the test rows are untouched.
If the external data guides feature selection or threshold choice, it becomes part of development.
AUC can remain acceptable while calibration, precision, or workload becomes unsafe.
A model transported from a referral center to primary care may produce far fewer useful alerts.
Use these short cases to test whether the transportability logic is clear.
A model was trained and tested with cross-validation in one hospital only. What validation type is this?
A risk model has similar AUC in a new clinic, but predicted risks are consistently too high. What should you check?
Which claim is safest after testing in two urban academic hospitals?
Start with safe train/validation/test design, then ask whether the model travels.
Use the calibration guide to interpret transported risk estimates and action thresholds.
Read the precision-recall guide when target prevalence changes across sites.
Use interpretability tools to investigate whether feature effects changed plausibly.