Quantix Bio Labs / Paper 09
Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed.
Background. Our companion study showed that a diabetes risk model frozen in 2002 kept its discrimination but lost its calibration over twenty years of NHANES, tracking rising prevalence. Aggregate drift, however, is an average over a heterogeneous population, and the fairness question is whether the probabilities decay equally for everyone. We measured calibration drift within race and ethnicity, sex, and age subgroups on every cycle, and replicated the aggregate result under NHANES sampling weights.
Methods. The same frozen logistic model (age, sex, body mass index, waist circumference, systolic blood pressure; fit on 1999 to 2002, 8,201 adults) was evaluated on each two year cycle through 2017 to 2018, with expected calibration error (ECE) and AUC computed within subgroups: sex, age bands (20 to 44, 45 to 64, 65 and over), and the three largest race and ethnicity groups (non Hispanic white, non Hispanic Black, Mexican American), requiring at least 300 members and 20 events per cell. Survey weighted AUC and ECE, implemented from first principles with examination sampling weights, replicated the aggregate trajectory. All analysis is pure numpy on public data and regenerates with one command.
Results. The aggregate drift decomposed into starkly unequal parts. By 2017 to 2018, the frozen model's ECE was 0.016 for non Hispanic white participants against 0.098 for Mexican American, 0.069 for non Hispanic Black, 0.093 for ages 45 to 64 and 0.100 for 65 and over, while remaining 0.011 for ages 20 to 44: a sixfold disparity between the best and worst monitored groups that the aggregate value of 0.057 does not reveal. The disparity was present from the first evaluation cycle (Mexican American 0.065 versus white 0.034 in 2003 to 2004) and widened with time. Under sampling weights, aggregate drift replicated in direction but with smaller magnitude (weighted ECE 0.012 to 0.021, a 1.8 fold rise, versus 0.015 to 0.057 unweighted), because weighting shifts mass toward younger participants for whom the model remained calibrated; weighted AUC was stable near 0.80 throughout.
Conclusions. A model can pass aggregate calibration monitoring while being materially miscalibrated for exactly the demographic groups bearing the highest diabetes burden. Subgroup calibration belongs alongside subgroup discrimination in fairness auditing, and drift surveillance that ignores it will systematically underestimate harm to minority and older patients. The weighted replication separately shows that sample level and population level drift can differ severalfold, so monitoring conclusions depend on the population the monitor is weighted to represent.
Keywords. algorithmic fairness, calibration, subgroup analysis, drift, NHANES, health equity
Fairness analyses of clinical prediction models have concentrated on discrimination: whether AUC, sensitivity or error rates differ across demographic groups [1, 2]. Calibration disparity, whether the stated probabilities are equally trustworthy across groups, has received less attention despite being the property that treatment thresholds act on, and despite theoretical results showing calibration and error rate parities cannot generally coexist [3]. A model miscalibrated for one group silently misprices risk for that group at every clinical decision that consumes its output.
Our companion study established that a frozen diabetes risk model's calibration decayed fourfold over twenty years of NHANES while its discrimination stayed flat, tracking the population's rising prevalence [4]. That result poses the fairness question in a form long horizon national data can actually answer: as the population drifted, did the model's probabilities decay for everyone, or did the aggregate hide unequal decay? Diabetes prevalence trends in the United States differ substantially by race, ethnicity and age [5, 6], so the hypothesis that drift is unequal is epidemiologically motivated, not merely possible.
This study answers it by decomposing the drift within sex, age and the three largest race and ethnicity groups on every cycle, and adds the survey weighted replication of the aggregate result, which doubles as a demonstration that the population a monitor represents changes the drift it reports.
Data, model, outcome definition and the frozen evaluation protocol are identical to the companion study [4]: the model is fit once on 1999 to 2002 and never updated; each cycle from 2003 to 2004 onward is scored by the frozen model. Two additions.
Subgroup evaluation. Within each cycle, ECE (ten equal width bins) and AUC are computed inside subgroups defined by sex, age bands (20 to 44, 45 to 64, 65 and over) and race and ethnicity (non Hispanic white, non Hispanic Black, Mexican American, from the RIDRETH1 classification; smaller categories lack stable cell sizes). Cells require at least 300 participants and 20 events. Because age and sex are model inputs while race and ethnicity is not, the subgroup analysis probes both within feature miscalibration and miscalibration along an unmodelled axis.
Weighted replication. Aggregate AUC and ECE per cycle are recomputed under the examination sampling weights, with the weighted Mann Whitney statistic and weighted binned calibration error implemented from first principles (both released). The unweighted analysis describes the examined sample; the weighted one estimates the civilian noninstitutionalized population, which is the population a deployed national score would face.
Table 1. ECE of the frozen model within subgroups, selected cycles.
| Subgroup | 2003-04 | 2007-08 | 2011-12 | 2015-16 | 2017-18 |
|---|---|---|---|---|---|
| Overall | 0.015 | 0.032 | 0.044 | 0.046 | 0.057 |
| Age 20-44 | 0.015 | 0.013 | 0.006 | 0.006 | 0.011 |
| Age 45-64 | 0.036 | 0.072 | 0.084 | 0.088 | 0.093 |
| Age 65 plus | 0.037 | 0.047 | 0.070 | 0.075 | 0.100 |
| Female | 0.018 | 0.040 | 0.045 | 0.036 | 0.047 |
| Male | 0.014 | 0.029 | 0.044 | 0.059 | 0.067 |
| Non Hispanic white | 0.034 | 0.019 | 0.028 | 0.023 | 0.016 |
| Non Hispanic Black | 0.042 | 0.105 | 0.083 | 0.071 | 0.069 |
| Mexican American | 0.065 | 0.061 | 0.068 | 0.104 | 0.098 |
Three facts stand out. The model never drifted for younger adults: ages 20 to 44 end the period better calibrated than they began. The entire aggregate drift is carried by adults over 45, for whom prevalence rose most. And along the unmodelled race and ethnicity axis, the disparity is large, persistent and worsening: by the final cycle the model's probabilities are six times more miscalibrated for Mexican American participants (0.098) and four times for non Hispanic Black participants (0.069) than for white participants (0.016), whose calibration actually improved as the aggregate deteriorated.
The aggregate ECE of 0.057 in 2017 to 2018 sits far from every subgroup: it understates the situation of every group over 45 and both minority groups, and overstates it for the young and for white participants. More pointedly, an auditor applying the common practice of monitoring aggregate calibration with a 0.05 alarm threshold would have raised a first alarm around 2015, a decade after the Mexican American subgroup crossed the same threshold (2003 to 2004, at 0.065).
Within subgroup AUC ranged roughly 0.72 to 0.81 across cells and cycles without a strong trend (full grid in released results), consistent with the companion finding that drift attacks calibration, not ranking, and reinforcing that a fairness audit restricted to discrimination metrics would find little to report here while the probability disparity grew.
Under examination sampling weights, aggregate ECE rose from 0.012 to 0.021 (1.8 fold) against 0.015 to 0.057 unweighted, and weighted AUC held near 0.80 throughout. The direction replicates; the magnitude is population dependent, because NHANES deliberately oversamples groups, including older adults and minority populations, in whom the model drifts most, and weighting dilutes exactly those groups toward their population shares. Both numbers are correct answers to different questions, and the gap between them is itself a finding: drift surveillance inherits the demographics of its monitoring sample, and a monitor built on a convenience sample that over represents drifting subgroups will alarm early, while one weighted to a majority population will alarm late for minorities, which is section 3.2 restated at the level of study design.
Population drift did not degrade this model's probabilities uniformly: it left them intact for the young and for white participants and progressively corrupted them for older, Black and Mexican American participants, with a sixfold disparity by the final cycle, along an axis the model does not even take as input. Aggregate calibration, the quantity a maintenance program would monitor, averaged the harmed and unharmed groups into a number that triggered alarm a decade after the worst affected group crossed the same line.
The mechanism is the companion study's, applied unevenly: prevalence rose fastest in the groups where the model drifts most, and a frozen intercept cannot follow a prevalence trajectory that differs by group. The remedy is correspondingly cheap: subgroup intercept updates, or at minimum subgroup calibration surveillance with the same few hundred labelled cases per group that aggregate monitoring already requires. The decision rule evaluated in the companion study extends group wise without modification; what is required is the will to compute it group wise.
Calibration disparity has been raised as a fairness criterion in principle [2, 3] and observed in single timepoint audits; the contribution here is longitudinal: twenty years of drift measured within fixed subgroups under a fixed model on a measurement stable national survey, showing disparity that originates early and compounds. The weighted versus unweighted contrast connects the fairness literature to the survey methodology one: whose population the monitor represents is a fairness decision, not a technicality.
Subgroup cells, particularly Mexican American in later cycles, are in the hundreds, so single cell ECE values carry sampling noise; the conclusions rest on trajectories, not single cells. RIDRETH1 categories are coarse instruments for lived heterogeneity. The unmodelled axis argument identifies association, not mechanism: the race and ethnicity disparity plausibly routes through unmeasured socioeconomic and access covariates. The weighted ECE uses the examination weights without design based variance estimation; formal survey inference is the natural extension. All companion study limitations apply.
Calibration drift is a fairness problem before it is a maintenance problem: it arrives earliest and largest for the groups already bearing the highest disease burden, and aggregate monitoring is structurally blind to it. Subgroup calibration should be a standing row in any model monitoring report.
All NHANES files are public. Complete source, including the weighted metric implementations and the full per cell results grid, regenerates with one command.
The author declares no competing interests. The work received no funding.
Figure 1. ECE trajectories within subgroups, 2003 to 2018 (subgroup_ece.svg). Figure 2. Weighted versus unweighted aggregate drift (weighted_check.svg).