Quantix Bio Labs / Paper 01
Karanvir Singh
Independent researcher, Quantix Bio, Cambridge, MA, USA
Correspondence: karanvir.singh@quantixmind.com
Manuscript draft, August 2026. Not yet peer reviewed.
Background. Post hoc recalibration methods such as Platt scaling, beta calibration and isotonic regression are widely recommended when a clinical risk model's predicted probabilities disagree with observed event rates. The sample size needed for recalibration to help rather than hurt has received far less attention than the sample size needed to fit the model itself, and recalibration is often applied by default to holdout sets of whatever size happens to be available.
Methods. Using eight public biomedical classification datasets (coronary disease in two cohorts, breast mass diagnosis by two feature sets, diabetes onset, liver disease, blood donation return, and five year cancer survival; n = 303 to 768, prevalence 0.24 to 0.71), we trained L2 regularized logistic regression models and applied four strategies to a held out calibration set: no recalibration, Platt scaling, beta calibration and isotonic regression. Calibration set size was varied over 25, 50, 100 and 200 observations. For each dataset and size we ran 50 repeated random partitions into training, calibration and test sets, measuring expected calibration error (ECE) and Brier score on the test set, and recording the minority class event count of each calibration set. All methods were implemented from first principles in numpy and all code and data sources are public.
Results. With 25 calibration observations, recalibration increased test ECE across the eight datasets by a random effects pooled +0.024 (95% CI 0.012 to 0.035) for Platt scaling, +0.025 (0.013 to 0.036) for beta calibration and +0.034 (0.018 to 0.050) for isotonic regression, harming calibration in 65, 68 and 72 percent of repeats respectively. On the diabetes dataset, isotonic regression with 25 calibration observations increased mean ECE from 0.071 to 0.131 (paired change +0.060, standard error 0.007), harming 96 percent of repeats, and worsened the Brier score from 0.163 to 0.191. Harm shrank monotonically with calibration set size for all three methods: at 100 observations the pooled mean ECE change was within +0.002 of zero with the fraction harmed near one half, and at 200 observations all three methods produced small pooled improvements that excluded zero for Platt (−0.002, CI −0.003 to −0.000) and isotonic regression (−0.004, CI −0.008 to −0.001). Indexed by events, calibration sets at n = 25 contained on average only 5 to 10 minority class events, and mean harm crossed zero between roughly 25 and 45 minority events. Three further analyses establish the mechanism. A simulation with a perfectly calibrated model showed that recalibration's excess ECE decays with calibration set size with Platt scaling decaying faster than isotonic regression, as estimation theory predicts, and with a known miscalibration of 0.05 the simulated break even fell at 100 (Platt) and 200 (isotonic) observations, independently reproducing the empirical range. Replacing the base learner with deliberately miscalibrated Gaussian naive Bayes (mean ECE 0.16) reversed the effect entirely: isotonic recalibration then helped even at 25 observations (pooled change −0.043, 40 percent harmed) and strongly at 200 (−0.099, 9 percent harmed), confirming that the danger zone is a property of the bias to variance ratio, not of recalibration as such.
Conclusions. Post hoc recalibration is an estimator whose variance can exceed the bias it removes. For a near calibrated base model, harm dominated below roughly 100 calibration observations or 25 minority events in every dataset studied, a threshold reproduced independently by theory driven simulation, while for a badly miscalibrated base model recalibration helped at every size tested. The decision to recalibrate should therefore be driven by the measured miscalibration of the base model relative to the estimation noise its correction will introduce, for which we give a practical rule. Recalibration should be treated as an intervention with a measurable failure mode and a sample size requirement, not as a default step, and reporting guidelines for prediction models should ask authors to justify calibration set size.
Keywords. calibration, Platt scaling, beta calibration, isotonic regression, clinical prediction models, sample size, expected calibration error
Clinical risk models are increasingly required to be calibrated: a predicted probability of 0.8 should correspond to an observed event rate near 80 percent in comparable patients. Calibration has been called the Achilles heel of predictive analytics [1], and miscalibrated models can be worse than useless at the bedside because treatment thresholds act on the stated probabilities [1, 2].
The standard remedies are post hoc recalibration maps fit on held out data: Platt scaling, a two parameter logistic transformation of the model's scores [3]; beta calibration, its three parameter generalization [8]; and isotonic regression, a nonparametric monotone map fit by pool adjacent violators [4, 5, 6]. All are simple, widely implemented, and routinely applied.
The question this study addresses is narrower than whether these methods work, and more practical: how much held out data do they need before they help more often than they hurt? The sample size required to develop a prediction model has received systematic treatment [7], and it is known qualitatively that isotonic regression can overfit small calibration sets [6, 8]. But recalibration is often applied to whatever holdout set is available, and in clinical settings, where events are scarce and cohorts small, that set may contain only a few dozen observations and a handful of events. We are not aware of a systematic empirical answer across multiple clinical datasets to the question of where the break even point lies.
Our motivation was an observed failure. In a study of coronary disease risk on the Cleveland cohort (n = 303), isotonic recalibration fit on roughly 55 held out observations per fold doubled the expected calibration error of a logistic model that was already close to calibrated. This paper asks whether that failure was an anomaly or the expected outcome at that sample size.
Eight public biomedical binary classification datasets were used (Table 1). All are fully deidentified research datasets in the public domain, so no ethics approval is required; each is cited to its originating study where one exists.
Table 1. Datasets.
| Dataset | n | Outcome | Prevalence | Source |
|---|---|---|---|---|
| Coronary disease (Cleveland) | 303 | angiographic coronary disease | 0.46 | Detrano et al [9], UCI [10] |
| Breast mass, morphometry (WDBC) | 569 | malignancy | 0.37 | Street et al [11], UCI [10] |
| Diabetes (Pima) | 768 | diabetes onset | 0.35 | Smith et al [12], UCI [10] |
| Blood donation (Transfusion) | 748 | donation in target month | 0.24 | Yeh et al [13], UCI [10] |
| Cancer survival (Haberman) | 306 | death within five years | 0.27 | UCI [10] |
| Breast mass, cytology (Wisconsin original) | 699 | malignancy | 0.34 | Wolberg and Mangasarian [15], UCI [10] |
| Liver disease (ILPD) | 583 | liver disease diagnosis | 0.71 | UCI [10] |
| Coronary disease (South Africa) | 462 | coronary heart disease | 0.35 | Rossouw et al [16] |
The base model was logistic regression with an L2 penalty (lambda = 1.0 on standardized features), fit by Newton's method. Logistic regression was chosen deliberately: it is the most common clinical modelling class and it is approximately calibrated by construction when not badly misspecified [6], so it isolates the effect of the recalibration step rather than compensating for a poorly calibrated base learner.
Four strategies were compared. None: the base model's probabilities used directly. Platt scaling: a two parameter logistic map fit on the base model logits of the calibration set [3]. Beta calibration: the three parameter map of Kull, Silva Filho and Flach, fit as logistic regression on ln(p) and −ln(1−p), with a negative slope handled by dropping the offending term and refitting, following the reference implementation [8]. Isotonic regression: a monotone step function fit on the calibration set by the pool adjacent violators algorithm [4, 5], applied to test probabilities with linear interpolation between knots.
For each dataset and each calibration set size n_cal in {25, 50, 100, 200}: the data were randomly partitioned into 30 percent training, 30 percent test, and a remaining pool from which n_cal observations were drawn. The base model was fit on the training partition, each recalibration map was fit on the calibration partition only, and all metrics were computed on the test partition only. This was repeated 50 times with different random partitions; the minority class event count of each calibration set was recorded. Repeats in which either the training or the calibration partition contained fewer than two or three observations of either class were redrawn. Cells where the pool could not supply n_cal observations (n_cal = 200 for the three smallest datasets) were omitted.
Expected calibration error (ECE) with ten equal width bins was the primary outcome, and Brier score the secondary outcome, both on the held out test partition. For each dataset, size and method we report the mean over the 50 repeats, its standard error, the paired mean change versus no recalibration, and the fraction of repeats in which recalibration increased ECE ("fraction harmed"). ECE with a fixed binning is an imperfect estimator of true miscalibration [14]; it is used here as a comparative measure between methods evaluated on identical test sets, where its biases are shared.
Per dataset paired effects (mean ECE change and its standard error over the 50 repeats) were pooled per method and size by DerSimonian and Laird random effects meta analysis [17], reporting the pooled effect, its 95 percent confidence interval, and the I squared heterogeneity statistic.
To test whether the observed harm is specific to near calibrated base models, the full grid was repeated with Gaussian naive Bayes as the base learner, whose conditional independence assumption produces characteristically overconfident probabilities [6].
If the base model is perfectly calibrated, any recalibration map adds pure estimation noise: for a two parameter map the parameter error is of order n^(−1/2), while for isotonic regression the minimax risk for monotone regression implies an L1 error scale of order n^(−1/3), slower because the function class is larger. Recalibration should therefore help only when intrinsic miscalibration epsilon exceeds this estimation noise, and the break even should come earlier for Platt scaling than for isotonic regression at any epsilon. We validated these predictions in a controlled simulation with Gaussian class conditional scores where the true calibration map is known exactly: first measuring the decay of excess ECE under a perfectly calibrated model (300 repeats per size, test sets of 20,000), then measuring the break even under a fixed known miscalibration (logit scale distortion of 1.5, intrinsic ECE 0.049, chosen to match the typical miscalibration scale of the real datasets).
All methods, including the logistic solver, beta calibration, the pool adjacent violators algorithm and the metrics, are implemented directly in numpy with no modelling libraries, so every step is inspectable. The full experiment runs from raw public data with one command and completes in seconds on a laptop. Code, figures and per cell results are available with the manuscript.
Pooled across the eight datasets at n_cal = 25, Platt scaling increased test ECE by +0.025 and harmed calibration in 65 percent of repeats; beta calibration by +0.026 with 68 percent harmed; isotonic regression by +0.035 with 72 percent harmed (Table 2). Every one of the eight datasets showed mean harm from all three methods at n_cal = 25.
Table 2. Mean change in test ECE versus no recalibration, pooled across datasets (k = number of datasets contributing).
| n_cal | Platt ΔECE | Platt harmed | Beta ΔECE | Beta harmed | Isotonic ΔECE | Isotonic harmed | k |
|---|---|---|---|---|---|---|---|
| 25 | +0.025 | 65% | +0.026 | 68% | +0.035 | 72% | 8 |
| 50 | +0.014 | 67% | +0.013 | 63% | +0.019 | 67% | 8 |
| 100 | +0.001 | 52% | +0.001 | 51% | +0.001 | 49% | 8 |
| 200 | −0.001 | 46% | −0.002 | 46% | −0.003 | 42% | 5 |
The worst single cell was the diabetes dataset at n_cal = 25 under isotonic regression: mean ECE rose from 0.071 (uncalibrated) to 0.131, with harm in 96 percent of the 50 repeats (paired ΔECE +0.060, standard error 0.007), and the Brier score worsened from 0.163 to 0.191. The harm is visible directly in the pooled reliability diagram for this cell (Figure 3): the isotonic curve departs from the diagonal that the uncalibrated model tracks. The harm was not confined to the calibration metric: the probability estimates were made genuinely worse.
Mean harm shrank monotonically with calibration set size for all three methods on every dataset (Table 3, per dataset figures). At n_cal = 100 the pooled effect was within +0.002 of zero for all methods, with the fraction harmed near one half, meaning recalibration at that size was close to a coin flip. At n_cal = 200 all three methods produced small average benefits, and isotonic regression began to show its expected advantage where genuine miscalibration existed to correct: on the breast cytology dataset it reduced ECE from 0.032 to 0.024 and was harmful in only 24 percent of repeats, and on the breast morphometry dataset from 0.032 to 0.025 (30 percent harmed). The same early crossover appeared on the two coronary datasets at n_cal = 100 (Cleveland 0.109 to 0.095, harmed 30 percent; South Africa 0.104 to 0.094, harmed 40 percent).
Table 3. Mean test ECE by dataset, method and calibration set size (50 repeats per cell; percent of repeats harmed in parentheses for isotonic).
| Dataset | n_cal | None | Platt | Beta | Isotonic |
|---|---|---|---|---|---|
| Cleveland | 25 | 0.118 | 0.120 | 0.129 | 0.126 (58%) |
| Cleveland | 50 | 0.109 | 0.114 | 0.113 | 0.114 (60%) |
| Cleveland | 100 | 0.109 | 0.106 | 0.107 | 0.095 (30%) |
| WDBC | 25 | 0.033 | 0.044 | 0.044 | 0.040 (60%) |
| WDBC | 50 | 0.033 | 0.042 | 0.040 | 0.040 (66%) |
| WDBC | 100 | 0.031 | 0.033 | 0.031 | 0.029 (58%) |
| WDBC | 200 | 0.032 | 0.031 | 0.029 | 0.025 (30%) |
| Pima | 25 | 0.071 | 0.121 | 0.117 | 0.131 (96%) |
| Pima | 50 | 0.073 | 0.096 | 0.093 | 0.105 (80%) |
| Pima | 100 | 0.076 | 0.080 | 0.082 | 0.089 (68%) |
| Pima | 200 | 0.072 | 0.073 | 0.076 | 0.076 (56%) |
| Transfusion | 25 | 0.064 | 0.102 | 0.107 | 0.125 (80%) |
| Transfusion | 50 | 0.064 | 0.091 | 0.096 | 0.103 (90%) |
| Transfusion | 100 | 0.067 | 0.074 | 0.075 | 0.080 (60%) |
| Transfusion | 200 | 0.071 | 0.066 | 0.066 | 0.069 (58%) |
| Haberman | 25 | 0.100 | 0.138 | 0.141 | 0.155 (74%) |
| Haberman | 50 | 0.101 | 0.107 | 0.104 | 0.117 (56%) |
| Haberman | 100 | 0.093 | 0.093 | 0.093 | 0.091 (40%) |
| Wisconsin orig | 25 | 0.035 | 0.039 | 0.038 | 0.036 (52%) |
| Wisconsin orig | 50 | 0.034 | 0.040 | 0.039 | 0.038 (50%) |
| Wisconsin orig | 100 | 0.033 | 0.035 | 0.031 | 0.029 (36%) |
| Wisconsin orig | 200 | 0.032 | 0.030 | 0.026 | 0.024 (24%) |
| ILPD | 25 | 0.077 | 0.107 | 0.104 | 0.127 (76%) |
| ILPD | 50 | 0.073 | 0.092 | 0.089 | 0.100 (74%) |
| ILPD | 100 | 0.074 | 0.076 | 0.080 | 0.084 (62%) |
| ILPD | 200 | 0.077 | 0.077 | 0.077 | 0.073 (44%) |
| SA heart | 25 | 0.101 | 0.126 | 0.125 | 0.140 (80%) |
| SA heart | 50 | 0.098 | 0.112 | 0.115 | 0.124 (60%) |
| SA heart | 100 | 0.104 | 0.094 | 0.097 | 0.094 (40%) |
Calibration sets of 25 observations contained on average only 5 to 10 minority class events across these prevalences. Re expressing the grid by the mean minority event count of each cell (Figure 2), pooled harm crossed zero between roughly 25 and 45 minority events for all three methods. This framing explains why the same n_cal behaved differently across datasets: the lowest prevalence dataset (blood donation, 0.24) showed the largest harm at small n_cal, its 25 observation calibration sets holding barely five events.
At every size of 50 and below, the harm ordered by flexibility: Platt scaling least harmful, beta calibration close behind, isotonic regression worst, consistent with two, three and effectively n parameters respectively overfitting in proportion [6, 8]. But less is not none: at n_cal = 25 even Platt scaling harmed two thirds of repeats. The parametric methods reduce the variance of the failure without removing it.
The pooled isotonic effect at n_cal = 25 was +0.034 (95% CI 0.018 to 0.050), at 100 was null (+0.001, CI −0.005 to +0.006), and at 200 was a significant benefit (−0.004, CI −0.008 to −0.001). Platt scaling followed the same trajectory with smaller magnitudes (+0.024 at 25; −0.002 at 200, both excluding zero). Heterogeneity was high at small sizes (I squared 0.91 to 0.96 at n = 25), as expected when datasets differ in prevalence and hence event count at fixed n; heterogeneity fell to near zero for Platt at 100 and 200.
With Gaussian naive Bayes as the base learner (mean base ECE 0.16 across datasets, five times the logistic base), isotonic recalibration helped from the smallest size tested: pooled change −0.043 at n = 25 with 40 percent of repeats harmed, strengthening monotonically to −0.099 at n = 200 with 9 percent harmed (Figure 4). The harm documented in sections 3.1 to 3.4 is therefore not a property of recalibration in general but of applying it where there is little bias to remove: the variance of the map must be paid either way, and only a large bias makes the purchase worthwhile.
Under a perfectly calibrated model, the simulated excess ECE decayed smoothly with calibration set size, with Platt scaling decaying faster than isotonic regression at every size (log log slopes −0.65 and −0.48; the asymptotic rates of −0.50 and −0.33 are approached from below in this pre asymptotic range, and the predicted ordering holds throughout). Under a fixed intrinsic miscalibration of 0.049, matching the typical scale of the real datasets, simulated recalibration first beat the uncalibrated model at 100 observations for Platt scaling and 200 for isotonic regression: the same break even range the eight real datasets produced. The empirical rule of thumb is thus not an accident of these cohorts but the visible edge of a bias variance tradeoff whose two sides can be independently measured.
Table 4. Random effects pooled change in test ECE versus no recalibration, all methods and sizes (DerSimonian and Laird; k = 8 datasets at sizes 25 to 100, k = 5 at 200).
| Method | n_cal | Pooled ΔECE | 95% CI | I² |
|---|---|---|---|---|
| platt | 25 | +0.0236 | 0.0123 to 0.0349 | 0.91 |
| platt | 50 | +0.0135 | 0.0078 to 0.0192 | 0.75 |
| platt | 100 | +0.0016 | -0.0007 to 0.0039 | 0.24 |
| platt | 200 | -0.0016 | -0.0031 to -0.0000 | 0.00 |
| beta | 25 | +0.0248 | 0.0134 to 0.0362 | 0.91 |
| beta | 50 | +0.0125 | 0.0059 to 0.0191 | 0.79 |
| beta | 100 | +0.0011 | -0.0019 to 0.0042 | 0.51 |
| beta | 200 | -0.0030 | -0.0061 to 0.0000 | 0.62 |
| isotonic | 25 | +0.0339 | 0.0176 to 0.0502 | 0.96 |
| isotonic | 50 | +0.0188 | 0.0089 to 0.0287 | 0.90 |
| isotonic | 100 | +0.0005 | -0.0051 to 0.0061 | 0.80 |
| isotonic | 200 | -0.0044 | -0.0083 to -0.0005 | 0.61 |
Two features deserve note. Heterogeneity is very high at n = 25 (I squared 0.91 to 0.96) and collapses as size grows, which is itself informative: at small sizes the harm depends strongly on each dataset's prevalence, and hence its event count at fixed n, exactly as the event indexed analysis of section 3.3 predicts. And the confidence intervals at n = 200 exclude zero on the benefit side for Platt scaling and isotonic regression despite the small pooled magnitudes, because the paired design removes between split variance.
Table 5. Gaussian naive Bayes base learner: isotonic recalibration effect per dataset.
| Dataset | Base ECE (NB) | ΔECE isotonic, n=25 | harmed | ΔECE isotonic, n=200 | harmed |
|---|---|---|---|---|---|
| cleveland | 0.164 | -0.0218 | 26% | n/a | n/a |
| wdbc | 0.068 | -0.0027 | 48% | -0.0179 | 18% |
| pima | 0.126 | +0.0144 | 56% | -0.0518 | 2% |
| transfusion | 0.126 | +0.0067 | 66% | -0.0653 | 2% |
| haberman | 0.140 | +0.0088 | 58% | n/a | n/a |
| wisc_original | 0.042 | -0.0018 | 48% | -0.0040 | 24% |
| ilpd | 0.429 | -0.2920 | 0% | -0.3575 | 0% |
| saheart | 0.191 | -0.0569 | 16% | n/a | n/a |
The contrast is uniform: with base miscalibration of 0.08 to 0.28 (five to ten times the logistic base), isotonic recalibration produced mean benefit on every dataset at every size, already at 25 observations on six of eight datasets, and overwhelmingly by 200 (4 to 24 percent of repeats harmed). Read jointly with Table 3, the pair of grids brackets the practitioner's situation: the question is never "does recalibration work" but "is my base model's bias larger than the correction's variance at my sample size."
Applied to a well specified base model, post hoc recalibration on small holdout sets systematically worsened calibration on all eight datasets studied, with random effects pooling excluding zero in both directions of the effect: significant harm at 25 observations, significant benefit at 200. The mechanism is the ordinary bias variance tradeoff, and the paper measures both sides of it: the variance side directly, as the decay of excess ECE when there is no bias to remove, and the bias side through a miscalibrated contrast learner for which recalibration paid at every size. A simulation with the two sides set to the scales observed in the real data reproduced the empirical break even of 100 to 200 observations. The practical rule that follows: estimate the base model's miscalibration on the available holdout data, and recalibrate only if it clearly exceeds the estimation noise at that sample size, roughly 0.07 at 25 observations and 0.02 at 200 for isotonic regression on these data.
That isotonic regression needs more data than sigmoid scaling is known from the machine learning literature [6], beta calibration was proposed in part to tame that flexibility [8], and clinical methodologists have warned in general terms against recalibration on small samples while providing sample size guidance for model development [1, 2, 7]. Our contribution is a systematic, replicated measurement of the phenomenon across eight real clinical datasets with a shared protocol, expressed in the units that matter practically: the number of held out observations, and events, at which the intervention stops hurting.
The results compress into a decision procedure a practitioner can execute in five lines. Measure the base model's ECE on the available holdout. Look up, or simulate once for the model class, the expected excess ECE of the intended recalibration map at that holdout size (for isotonic regression on these data: roughly 0.09 at 25 observations, 0.05 at 100, 0.036 at 200). Recalibrate only if measured miscalibration clearly exceeds that noise floor, with a factor of 1.5 as a reasonable margin. Our companion study applies exactly this rule prospectively across twenty years of national survey data, where it outperforms both always and never recalibrating.
Three broader implications follow. First, recalibration should be justified, not reflexive: if the base model is a regularized logistic regression, the prior expectation should be that it is close to calibrated, and a measured miscalibration should motivate the recalibration step. Second, calibration set size, and its event count, should be reported alongside the choice of method; sets below roughly 100 observations or 25 minority events should be presumed harmful, with isotonic regression in particular avoided there. Third, when holdout data is scarce, the choice is not between recalibrating and not recalibrating on principle, but between a known small bias and a demonstrated large variance.
The base learner was a single model class, chosen because it is common and approximately calibrated; badly miscalibrated base learners such as naive Bayes or boosted trees would shift the break even point downward, since there is more genuine error to correct, and our results should not be read as advice against recalibrating such models. ECE with fixed binning is a biased estimator of miscalibration, though the bias is shared across the compared methods and the Brier score corroborated the direction of effects. The datasets are modest sized public research cohorts, not contemporary clinical registries, and event mechanisms differ across them. The 30/30/40 partition design holds training size proportional to dataset size, so base model quality varies across datasets. The event indexed thresholds are derived from cell means rather than a per repeat regression. A supplementary experiment subsampling five of the datasets to prevalences of 0.05 and 0.10 shows the framework extends but the dominant term changes: at rare prevalence the base model itself is often miscalibrated by the imbalance, and where it was (two datasets), recalibration helped from as few as five minority events, while where the base model stayed calibrated (one dataset), isotonic harm persisted at every size tested. The bias variance account predicts exactly this, but a dedicated rare event study with event level regression is warranted before quoting thresholds below prevalence 0.1. The simulation validates the theory in a Gaussian score model, and its measured decay exponents are steeper than the asymptotic rates, as expected pre asymptotically; the theory is used here to predict orderings and break evens, not exact constants.
Recalibration is an estimator with its own variance, and on small clinical holdout sets that variance dominates. A model that is already close to calibrated is more often damaged than improved by Platt scaling, beta calibration or isotonic regression below roughly 100 calibration observations or 25 minority events. The step should be earned by measured miscalibration and backed by enough data, and its sample size should be reported as routinely as the model's.
All eight datasets are public and cited above. Complete source code, including the numpy implementations of every method, the experiment protocol, per cell results and figure generation, is available from the author and will be archived with the preprint.
The author declares no competing interests. The work received no funding.
Figure 1. Mean change in test ECE versus calibration set size, pooled across eight datasets, three recalibration methods (aggregate_delta.svg).
Figure 2. Mean change in test ECE indexed by mean minority class events in the calibration set (delta_by_events.svg).
Figure 3. Reliability diagram for the worst cell, diabetes dataset with 25 calibration observations, pooled over 50 repeats (reliability_worst_cell.svg).
Figure 4. Pooled isotonic effect under a near calibrated versus a deliberately miscalibrated base learner (base_model_contrast.svg).
Figure 5. Simulation: decay of excess ECE under a perfectly calibrated model, and the break even under known miscalibration (exp09 scaling.svg, breakeven.svg).
Supplementary. Per dataset ECE versus calibration set size, four strategies (ece_*.svg).