Quantix Bio Labs / Paper 04
Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed.
Background. Early warning systems for sepsis are typically compared by per observation discrimination, while deployed systems fail through alarm fatigue. We previously proposed evaluating alerting systems on the operating curve of warning time against false alert episodes per patient day, demonstrated on simulation. Here we apply the framework to real ICU data, with derivation and external validation in different hospital systems.
Methods. Hourly vital signs from the PhysioNet/Computing in Cardiology 2019 challenge: 4,978 ICU patients from hospital system A (Beth Israel Deaconess, 442 sepsis cases) and 3,147 from hospital system B (Emory, 185 cases), with sepsis onset defined from the challenge labels (which lead clinical onset by six hours, corrected for in all lead time calculations). Two alerting systems were compared: a NEWS2 style aggregate threshold score, and L2 regularized logistic regression on six current vitals, their four hour trends, age, and measurement frequency, trained on hospital A only. For each system, thresholds were swept across their operating range, and each operating point was summarized by sepsis detection fraction, median warning hours before onset, and false alert episodes (rising edges) per patient day among non septic patients. Hospital B was touched only at evaluation.
Results. On external validation at matched detection of 92 percent, the trend aware model imposed 1.26 false alert episodes per patient day against the NEWS2 style score's 2.38, a 47 percent reduction in alarm burden at equal detection with comparable median warning (24 versus 25 hours). The advantage held down the curve: near 80 percent detection the model's burden was 1.01 against the score's roughly 1.4 interpolated, and near 59 percent detection 0.51 against 0.64. The pattern replicated internally on hospital A. Both systems' median warnings at high detection (20 to 28 hours) substantially exceed the six hour horizon usually discussed, reflecting persistent physiological abnormality in patients who later meet sepsis criteria, and the alarm burden of both systems at high detection quantifies why ward scale sepsis alerting exhausts its users.
Conclusions. On real data from two hospital systems, trend information bought meaningfully lower alarm burden at the high detection operating points that sepsis alerting policies target, and the advantage survived external validation without recalibration or refitting. More consequentially, the operating curves make explicit what single threshold reports obscure: every high sensitivity configuration of either system imposes an alarm burden that must be staffed, and the framework prices that burden in the unit wards experience.
Keywords. sepsis, early warning scores, alarm fatigue, external validation, NEWS2, critical care
Sepsis alerting is the arena where the gap between reported model performance and deployed usefulness has been most visible: systems with acceptable discrimination have shown large alerting burdens and modest clinical impact at scale [1, 2, 3]. In a companion methodology paper we argued that the operating curve of warning time against false alert episodes per patient day, swept across each system's threshold range, is the evaluation that predicts deployment behaviour, and demonstrated the framework on simulation where ground truth onset is exact [4].
Simulation establishes machinery, not findings. This study carries the framework to real ICU data with the strongest design the public data allows: two hospital systems, model derivation confined to one, evaluation on both, and the comparator a NEWS2 style aggregate score of the kind hospitals actually run [5].
The PhysioNet/CinC 2019 challenge dataset [6] provides hourly vital signs, laboratory values and demographics for ICU patients from Beth Israel Deaconess Medical Center (set A) and Emory University Hospital (set B), with an hourly sepsis label constructed by the challenge organizers from Sepsis-3 criteria. We analyze 4,978 set A patients (442 septic, 8.9 percent) and 3,147 set B patients (185 septic, 5.9 percent). The challenge label is deliberately shifted to lead clinical onset by six hours; we therefore define clinical onset as the first labelled hour plus six, credit alerts only if they fire strictly before onset, and censor septic patients' alarm behaviour after onset. Records shorter than eight hours are excluded.
Six vitals are used: heart rate, oxygen saturation, temperature, systolic pressure, mean arterial pressure and respiratory rate, forward filled within patient, with population medians for values missing before first measurement. The NEWS2 style comparator bands each vital and alerts when the point sum crosses a threshold, swept 3 to 10. The learned model is logistic regression on the six current vitals, their four hour slopes, age, and the fraction of vital measurements actually observed in the trailing four hours (measurement frequency is real clinical signal, since observation intensifies with concern [7]); it is trained on set A only, on hours more than six hours before onset labelled negative and the final six pre onset hours positive, and its thresholds are swept over quantiles of its set A score distribution. No laboratory values are used by either system, keeping the comparison a vitals versus vitals one.
Per operating point: detection (fraction of septic patients with any pre onset alert), median lead time from first pre onset alert to onset (zero for undetected patients), and false alert episodes per patient day among non septic patients, an episode being a transition from non alerting to alerting state. Episodes rather than alerting hours because episodes are what interrupt staff; per patient day because that is the unit wards staff against.
Reporting standard. This manuscript follows the TRIPOD+AI reporting guidance for clinical prediction model studies (title through limitations); the released code maps each reported number to the line that produces it.
Pure numpy end to end; the complete pipeline from raw pipe separated files to figures runs in about thirty seconds and accompanies the manuscript.
Table 1. Cohort characteristics.
| Characteristic | Hospital A (derivation) | Hospital B (external) |
|---|---|---|
| Patients | 4,978 | 3,147 |
| Age, mean ± SD | 62.7 ± 16.2 | 60.4 ± 16.5 |
| Record length, hours, mean ± SD | 39 ± 23 | 38 ± 22 |
| Sepsis (challenge label) | 8.9% | 5.9% |
| Heart rate, mean | 85 | 84 |
| Respiratory rate, mean | 19 | 19 |
| Vitals observed per hour | 80% | 76% |
The prevalence difference (9 percent versus 6 percent) and the lower observation density at hospital B are visible in Table 1 and are part of the transport story rather than nuisance facts.
Table 2. Every operating point, hospital system B (3,147 patients, 185 septic).
| System | Threshold | Detection | Median lead (h) | False episodes per patient day |
|---|---|---|---|---|
| NEWS2 style | 3 | 92.4% | 25 | 2.38 |
| NEWS2 style | 4 | 82.7% | 20 | 1.68 |
| NEWS2 style | 5 | 72.4% | 10 | 1.08 |
| NEWS2 style | 6 | 59.5% | 4 | 0.64 |
| NEWS2 style | 7 | 38.9% | 0 | 0.33 |
| NEWS2 style | 8 | 22.2% | 0 | 0.14 |
| NEWS2 style | 9 | 11.9% | 0 | 0.06 |
| NEWS2 style | 10 | 5.9% | 0 | 0.02 |
| Trend model | q30 | 99.5% | 32 | 2.07 |
| Trend model | q45 | 99.5% | 31 | 2.09 |
| Trend model | q60 | 98.9% | 31 | 1.83 |
| Trend model | q70 | 98.4% | 30 | 1.59 |
| Trend model | q75 | 96.8% | 29 | 1.41 |
| Trend model | q78 | 91.9% | 24 | 1.26 |
| Trend model | q80 | 87.6% | 24 | 1.14 |
| Trend model | q82 | 79.5% | 18 | 1.01 |
| Trend model | q84 | 73.5% | 14 | 0.85 |
| Trend model | q86 | 65.4% | 9 | 0.68 |
| Trend model | q88 | 58.9% | 5 | 0.55 |
| Trend model | q90 | 54.1% | 4 | 0.45 |
| Trend model | q92 | 46.5% | 0 | 0.36 |
| Trend model | q95 | 35.1% | 0 | 0.23 |
| Trend model | q98 | 20.0% | 0 | 0.10 |
At the high detection end that sepsis policies target, the matched comparison is exact: at 92 percent detection both systems agree on warning time (24 to 25 hours median) and differ twofold in cost, 1.26 versus 2.38 false episodes per patient day. The advantage persists at every matched point down the curve. The full curves are shown in Figure 1.
On hospital A (derivation, evaluated in sample), the same ordering held: near 96 percent detection the score required 2.85 episodes per patient day while the model reached comparable detection near 2.0, converging mid curve. The external advantage being no smaller than the internal one is consistent with the model's reliance on trends and measurement frequency, signals that appear to transport across hospital systems better than absolute thresholds.
Table 3. Every operating point, hospital system A (derivation cohort, in sample; 4,978 patients, 442 septic).
| System | Threshold | Detection | Median lead (h) | False episodes per patient day |
|---|---|---|---|---|
| NEWS2 style | 3 | 95.7% | 28 | 2.85 |
| NEWS2 style | 4 | 87.8% | 23 | 2.16 |
| NEWS2 style | 5 | 74.9% | 14 | 1.42 |
| NEWS2 style | 6 | 61.5% | 6 | 0.81 |
| NEWS2 style | 7 | 41.6% | 0 | 0.41 |
| NEWS2 style | 8 | 24.7% | 0 | 0.19 |
| NEWS2 style | 9 | 13.8% | 0 | 0.08 |
| NEWS2 style | 10 | 4.8% | 0 | 0.03 |
| Trend model | q30 | 99.3% | 31 | 2.01 |
| Trend model | q45 | 98.9% | 30 | 2.15 |
| Trend model | q60 | 97.1% | 28 | 2.05 |
| Trend model | q70 | 95.0% | 27 | 1.82 |
| Trend model | q75 | 92.8% | 25 | 1.64 |
| Trend model | q78 | 90.0% | 24 | 1.51 |
| Trend model | q80 | 89.1% | 23 | 1.41 |
| Trend model | q82 | 85.3% | 20 | 1.29 |
| Trend model | q84 | 80.8% | 19 | 1.16 |
| Trend model | q86 | 76.0% | 15 | 1.01 |
| Trend model | q88 | 69.0% | 12 | 0.84 |
| Trend model | q90 | 63.8% | 9 | 0.71 |
| Trend model | q92 | 58.4% | 6 | 0.59 |
| Trend model | q95 | 47.5% | 0 | 0.38 |
| Trend model | q98 | 31.9% | 0 | 0.16 |
Two observations hold for both systems and both hospitals. First, median warnings at high detection are 20 to 28 hours, far beyond the six hour horizon in which sepsis alerts are usually discussed: many patients who meet sepsis criteria are persistently abnormal for a day or more beforehand, so high sensitivity configurations alert early and often. Second, every configuration with detection above 90 percent imposes at least 1.2 false episodes per patient day. On a 30 bed unit that is 36 or more interruptions per day from non septic patients alone, a number that explains observed alert fatigue with deployed sepsis systems [2, 3] and that single threshold sensitivity reporting never surfaces.
The framework transported from simulation to real data without modification, and its verdicts sharpened: trend and measurement frequency information bought the largest burden reductions exactly at the high detection operating points that matter for sepsis, and the advantage survived evaluation on a hospital system the model had never seen. The absolute numbers are as important as the comparison: both systems are expensive in interruptions at high sensitivity, and the framework prices this explicitly.
The choice of a NEWS2 style score as comparator is not a straw man but the deployed reality: aggregate weighted track and trigger scores are the standard of care for deterioration surveillance in most health systems, and their thresholds are the objects hospital committees actually debate. The choice of an almost minimal learned model, logistic regression on six vitals, their trends and measurement frequency, is equally deliberate in the opposite direction: it establishes how much of the advantage attributed to machine learning in sepsis prediction is available from trend arithmetic alone, before any flexible model is introduced. The answer, roughly a third to a half less alarm burden at high detection, is a floor for learned systems and a challenge to them: a complex model that cannot beat this floor on these curves is adding risk without adding value. We would regard the operating curves of this paper as a useful minimum benchmark for the sepsis prediction literature, which has tended to compare against the challenge utility score rather than against burden priced in staff interruptions.
External validations of proprietary sepsis models have reported the combination of moderate sensitivity and high alerting burden [1]; our contribution is not another model but an evaluation frame in which such tradeoffs are the primary reported object, applied with a derivation and external validation design on open data, fully reproducible. The measurement frequency signal echoes prior findings that observation patterns carry outcome information [7], with the deployment caveat noted below.
The challenge label construction, while principled, encodes Sepsis-3 criteria applied to retrospective records; onset times inherit its assumptions, and the uniform six hour correction is approximate. Our cohorts are large fixed subsets of the full release (the lowest numbered records of each set), not the complete 40,336 patients; record numbering carries no clinical meaning, but the full cohort replication is a one line change. ICU populations are continuously monitored; ward deployment, where early warning matters most, has sparser observation and this evaluation would need repeating there. The measurement frequency feature partially encodes clinician suspicion, so some of the model's advantage is borrowed vigilance, useful in practice but not purely physiological signal; Lead times for undetected patients are scored as zero in medians, a conservative convention. Neither system uses laboratory values, by design.
Evaluated in the units that determine whether staff keep listening, a simple trend aware model dominates a threshold score at the operating points sepsis alerting actually uses, and the result holds across hospital systems. We propose that sepsis alerting studies report these curves as their primary result.
The PhysioNet/CinC 2019 dataset is openly available [6]. Complete source and per operating point results accompany the manuscript and regenerate with one command.
The author declares no competing interests. The work received no funding.
Figure 1. Median lead time versus false alert episodes per patient day, hospital B external validation (tradeoff_B.svg); detection versus burden (detection_B.svg). Figure 2. The same curves on hospital A (tradeoff_A.svg, detection_A.svg).