Quantix Bio Labs / Paper 11
Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed.
Background. When a clinical model fails at a new hospital, the standard diagnosis is distribution shift, and the standard measurement is a single external AUC drop. Neither identifies which parts of the model failed or why. We decomposed cross hospital transport at the feature family level on real ICU data from two hospital systems, and tested the assumption implicit in most transportability analysis: that the families whose distributions shift most transport worst.
Methods. Hourly vitals from the PhysioNet/CinC 2019 sepsis cohorts: 4,978 hospital A patients (3,484 train, 1,494 held out) and 3,147 hospital B patients, 119,386 training observation hours, predicting sepsis onset within six hours. Models were trained on hospital A restricted to feature families: absolute vitals (plus age); four hour trends; measurement frequency (the fraction of vitals actually observed in the trailing four hours, a practice pattern signal); and combinations. Each family's transported share of signal was measured as (AUC_B − 0.5)/(AUC_A − 0.5), with logistic regression and gradient boosting in parallel. Separately, a site discrimination probe measured how identifiable the hospital is from each family (AUC of classifying A versus B observations), quantifying covariate shift per family.
Results. The two measurements dissociated. Absolute vitals were massively site identifiable (site discrimination AUC 0.99), yet their outcome signal transported robustly (logistic: internal 0.600, external 0.591, 95 percent patient bootstrap CI 0.550 to 0.641; transported share 0.91). Measurement frequency showed the mirror image: nearly indistinguishable in distribution across hospitals (site AUC 0.57), yet its outcome relationship failed to transport at all: internal 0.534, external 0.492 with CI 0.455 to 0.524 spanning chance, a point estimate below 0.5. Trends sat between (external 0.517, CI 0.491 to 0.545). Row level discriminations were modest throughout (0.53 to 0.61), so all conclusions are about relative transport of signal, not absolute performance.
Conclusions. How different a feature looks across hospitals did not predict whether its predictive meaning survived the move; what predicted failure was whether the feature encodes local clinical practice rather than physiology (the practice pattern family retained no externally detectable signal despite being the least distribution shifted family). Transportability auditing should therefore classify features by their causal provenance (physiological versus operational) rather than by distributional distance, and practice pattern features, however predictive in derivation, should be treated as site local until proven otherwise. This gives a concrete, testable design rule for portable clinical models and a measured counterexample to covariate shift centred transfer analysis.
Keywords. transportability, external validation, dataset shift, measurement frequency, sepsis, multi site
External validation failures are the central credibility problem of clinical prediction, and the customary post mortem stops at "distribution shift" [1, 2]. The term conflates two different failures with different remedies. Covariate shift, features distributed differently at the new site while their relationship to outcome is preserved, is comparatively benign: a model may even benefit if the new site's case mix separates better. Concept shift, the feature to outcome relationship itself changing, is the dangerous one, and no amount of input distribution matching detects it.
A specific hypothesis about where concept shift lives in EHR models comes from the observation that many of their strongest signals are not physiology but process: how often clinicians measure, order, and document tracks their concern, and concern tracks outcomes [3, 4]. These practice pattern signals are real information, our own sepsis alerting study profited from measurement frequency [5], but their semantics are set by local staffing, protocols and habits, which do not travel. The hypothesis: physiological features transport, practice pattern features do not, and distributional distance predicts neither.
The two hospital structure of the PhysioNet 2019 sepsis data [6] permits a direct test, which is this study's design: restrict models to feature families of known provenance, measure each family's transported share of signal, and separately measure each family's covariate shift with a site discrimination probe, so the two axes can be compared.
Hospital A (Beth Israel Deaconess): 4,978 patients, split at patient level into 3,484 training and 1,494 internal test. Hospital B (Emory): 3,147 patients, evaluation only. Observation rows are patient hours (119,386 in training), labelled positive within six hours before sepsis onset (onset defined as in our companion study, correcting the challenge label's six hour lead). Six vitals forward filled, with observation masks retained.
Families: absolute vitals (six current values plus age); trends (four hour slopes of the six vitals); measurement frequency (fraction of the six vitals actually observed in the trailing four hours); absolute plus trends; all. Models per family: L2 logistic regression on standardized features, and gradient boosted trees, trained on hospital A training rows only. Transported share of signal: (AUC_B − 0.5)/(AUC_A internal − 0.5), the fraction of above chance discrimination surviving the site change; values above 1 mean the signal discriminates better externally (case mix differences make absolute AUC incomparable across sites, which is why the ratio, not the difference, is the estimand, and why both are reported).
External AUC confidence intervals use the patient cluster bootstrap (resampling hospital B patients with replacement, 300 resamples), respecting the within patient correlation of observation hours. Reporting standard. This manuscript follows the TRIPOD+AI reporting guidance for clinical prediction model studies (title through limitations); the released code maps each reported number to the line that produces it.
For each family, a gradient boosted classifier was trained to identify which hospital an observation came from (balanced sample, 70/30 split). Its AUC is a direct measure of covariate shift as seen through that family: 0.5 means the family's joint distribution is indistinguishable across sites; approaching 1 means the family fingerprints the site.
Table 1. Cohort characteristics.
| Characteristic | Hospital A (derivation) | Hospital B (external) |
|---|---|---|
| Patients | 4,978 | 3,147 |
| Age, mean ± SD | 62.7 ± 16.2 | 60.4 ± 16.5 |
| Record length, hours, mean ± SD | 39 ± 23 | 38 ± 22 |
| Sepsis (challenge label) | 8.9% | 5.9% |
| Heart rate, mean | 85 | 84 |
| Respiratory rate, mean | 19 | 19 |
| Vitals observed per hour | 80% | 76% |
The prevalence difference (9 percent versus 6 percent) and the lower observation density at hospital B are visible in Table 1 and are part of the transport story rather than nuisance facts.
Table 2. Transport of outcome signal versus covariate shift, by feature family (logistic regression primary; external CIs by patient cluster bootstrap, 300 resamples).
| Family | Site discrimination AUC | Internal AUC | External AUC (95% CI) | Transported share |
|---|---|---|---|---|
| Absolute vitals | 0.993 | 0.600 | 0.591 (0.550 to 0.641) | 0.91 |
| Trends | 0.892 | 0.543 | 0.517 (0.491 to 0.545) | 0.41 |
| Measurement frequency | 0.567 | 0.534 | 0.492 (0.455 to 0.524) | none (point estimate below chance) |
| Absolute plus trends | 0.994 | 0.602 | 0.591 (0.544 to 0.635) | 0.89 |
| All | 0.996 | 0.609 | 0.590 (0.541 to 0.635) | 0.83 |
Table 2b. Gradient boosting secondary analysis (point estimates).
| Family | Internal AUC | External AUC |
|---|---|---|
| Absolute vitals | 0.576 | 0.597 |
| Trends | 0.550 | 0.565 |
| Measurement frequency | 0.525 | 0.528 |
| Absolute plus trends | 0.589 | 0.613 |
| All | 0.602 | 0.577 |
Under gradient boosting the frequency family is at chance on both sides, so its transported share is undefined rather than contradictory; the interpretable primary analysis carries the comparison, and both model classes agree on the ordering of the physiological families.
The ordering by covariate shift and the ordering by transport failure are opposite. Absolute vitals fingerprint their hospital almost perfectly, through unit conventions, rounding, device and charting differences, yet retain 91 percent of their above chance signal externally, with a confidence interval comfortably above chance. Measurement frequency is the one family a site classifier can barely distinguish, and it is the one family with no detectable external signal: its interval spans chance with a point estimate below it, despite reliable internal signal. At the earlier interim cohort size the logistic point estimate was significantly below chance, an inversion; at full power the honest statement is collapse to chance with inversion not excluded, and either reading breaks the covariate shift account equally.
Measurement frequency encodes staffing ratios, monitoring protocols and escalation habits. A hospital that intensifies observation early in deterioration writes one sign into the data; a hospital whose baseline monitoring is denser, or whose intensification comes later, writes another. The feature's predictive content is a fact about the institution, not the patient, so there is no reason its meaning should be preserved across institutions, and empirically it was not: the family retained no external signal. This is concept shift in pure culture, occurring in the family where covariate shift is smallest.
Row level AUCs here are modest (0.53 to 0.61) because six forward filled vitals at hourly resolution carry limited six hour horizon information, consistent with the challenge literature's reliance on laboratory values for higher discriminations [6]. The study's claims are therefore about the relative transport of each family's above chance signal, for which the modest scale is sufficient and the dissociation is large; they are not claims of a deployable sepsis model, for which our companion alerting study is the relevant reference [5].
Distributional distance, the quantity transportability analyses usually monitor, predicted transport failure with the wrong sign in these data: the most shifted families transported best and the least shifted family retained no detectable external signal. The predictive variable was provenance: physiology transported, practice patterns did not. This is a measured counterexample to covariate shift centred reasoning about external validity, in the direction that theory anticipates once covariate and concept shift are distinguished [1, 7].
The result converts into an auditable rule for portable clinical models: classify every candidate feature by provenance; treat operational features (measurement patterns, ordering behaviour, documentation cadence) as site local by default, either excluding them from transported models or re estimating their coefficients at each site from local data; and evaluate site probes not as failure alarms but as what they are, detectors of fingerprinting that say nothing about semantic stability. In our companion sepsis study, measurement frequency improved alerting within site [5]; both results are true, and together they say: use such features where they were learned, distrust them where they were not.
The informativeness of healthcare process signals is established [3, 4], as is the taxonomy of dataset shift [1, 7] and the general finding that EHR models travel poorly [2]. The contribution is the explicit two axis measurement, transport of signal versus site identifiability, family by family on a two hospital cohort, and the demonstration that they dissociate completely. We are not aware of a prior measured example where the least distribution shifted feature family is the one that loses all signal.
Two hospital systems; the dissociation should be re measured wherever multi site data exists, and the released code makes that a data substitution. Vitals only, hourly, forward filled; richer inputs would raise absolute AUCs and could change family attribution at the margin. The transported share ratio is unstable when internal AUC is near 0.5, which is why raw AUCs with clustered confidence intervals are the primary report and both model classes are shown. The measurement frequency family is one dimensional here; richer operational features (lab ordering, note cadence) deserve the same test on data that contains them. Case mix differences between hospitals affect absolute AUCs; they do not selectively erase one family's signal while preserving the others'.
What a portable model must fear is not that the new hospital's data looks different, but that some of its features never meant what it thought they meant anywhere but home. Provenance, not distance, is the audit axis, and it is measurable with the two probes this paper releases.
The PhysioNet/CinC 2019 dataset is open. Complete source regenerates every number with one command.
The author declares no competing interests. The work received no funding.
Figure 1. Internal versus external AUC by feature family (transport.svg).