6  Calibration, and a metric that lies

Chapter 4 established that AUROC cannot see the numbers, only their order. This chapter asks the question it cannot answer: when the model says 0.8, does the event happen 80% of the time?

6.1 Two ways to measure it

The calibration slope regresses the outcome on the logit of the predicted probability. Perfect calibration gives slope 1, intercept 0. Slope below 1 means predictions are too extreme; above 1, too conservative.

calibration
function (label, score, eps = 1e-06) 
{
    s <- pmin(pmax(score, eps), 1 - eps)
    fit <- glm(label ~ qlogis(s), family = binomial())
    c(intercept = unname(coef(fit)[1]), slope = unname(coef(fit)[2]))
}
<bytecode: 0x113db4988>

Expected calibration error bins the predictions and averages the absolute gap between predicted and observed rates within each bin. It is the metric most commonly reported in the machine-learning literature.

ece
function (label, score, n_bins = 10L) 
{
    bins <- cut(score, breaks = seq(0, 1, length.out = n_bins + 
        1L), include.lowest = TRUE)
    parts <- tapply(seq_along(label), bins, function(ix) {
        if (length(ix) == 0L) 
            return(0)
        (length(ix)/length(label)) * abs(mean(label[ix]) - mean(score[ix]))
    })
    sum(unlist(parts), na.rm = TRUE)
}
<bytecode: 0x10da89b28>

Both are reasonable. On this data they disagree completely.

6.2 The disagreement

Group Calibration slope ECE Brier AUROC
overall 1.595 0.351 0.246 0.867
I-II 1.849 0.355 0.241 0.897
III-IV 1.501 0.356 0.251 0.855
V-VI 0.502 0.293 0.265 0.655

The slope says V-VI is badly miscalibrated: 0.50 against 1.85 in I-II. These are opposite failures: predictions too extreme in one group, too conservative in the other. Pooled, they average to 1.60, a number describing no one in the dataset.

ECE says all three groups are the same, to within 0.06.

6.3 Why ECE misses it

ECE takes an absolute value inside each bin. Systematic over-prediction in one part of the score range and under-prediction in another therefore contribute in the same direction, and the tilt in the whole relationship never shows up. It measures how far off predictions are, not in which direction, systematically.

The slope, being a regression coefficient, sees the tilt directly.

Observed event rate against predicted probability, by stratum. The diagonal is perfect calibration.

The V-VI line is visibly flatter than the diagonal: as the model’s confidence rises, the observed rate barely follows. That is a slope of 0.5, drawn.

6.4 Why this matters more than discrimination

If the model feeds a ranked worklist, miscalibration is survivable, because order is preserved. If it feeds a threshold, it is not. A referral rule of “investigate if p > 0.7” assumes 0.7 means the same thing in every patient. Here it does not, and the group where it means least is the group already least likely to have been examined by a specialist.

This is the standard signature of a transported model, and it is why calibration is the first thing to break and the last thing to be reported (1). TRIPOD+AI now asks for calibration to be reported explicitly, with a plot, for exactly this reason (2).

NoteReport both, and the plot

The slope and intercept catch systematic tilt; ECE catches local wobble; the plot catches things neither summarises. Anyone reporting a single calibration number is choosing which failures they are willing to miss.

1.
Ben Van Calster, David J. McLernon, Maarten van Smeden, Laure Wynants, Ewout W. Steyerberg. Calibration: The Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. doi:10.1186/s12916-019-1466-7
2.
Gary S. Collins, Karel G. M. Moons, Paula Dhiman, Richard D. Riley, Andrew L. Beam, Ben Van Calster, others. TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378