2  Three questions, not one

A model that outputs a probability is asked to do three different jobs, and it can do any one of them well while failing the others. “Is it a good model?” is really three questions wearing one coat (1).

2.1 1. Discrimination: can it rank?

Given an event and a non-event, does the model score the event higher? This is ranking, and nothing else. It asks whether the model puts patients in the right order, and it is completely indifferent to what the numbers on that order actually are. A model whose every probability is divided by ten discriminates exactly as well as before.

AUROC measures this, and only this. Chapter 4 builds it. Its blind spot is everything to do with the numbers themselves, which is the whole of the next question.

2.2 2. Calibration: do the numbers mean what they say?

When the model says 0.8, does the event happen about 80% of the time? A model can rank perfectly and still be badly calibrated (confident when it should be cautious, or the reverse), and you cannot tell from the AUROC, because ranking survives any monotone rescaling of the score.

Calibration is the question that matters the moment a probability is read as a probability: fed to a threshold, shown to a clinician, multiplied into an expected cost. It is also, reliably, the first thing to break when a model trained in one population meets another, and the last thing anyone reports (2). Chapter 6 builds two ways to measure it, and shows them disagreeing.

2.3 3. Utility: is it worth using here?

Even a well-ranked, well-calibrated model has to earn its place against the decision it informs. Acting on a prediction has costs (a false alarm is not free, and neither is a missed case), and those costs are not symmetric and not the same in every setting. Decision-analytic tools such as net benefit weigh them explicitly (3).

This book concentrates on the first two questions, where most evaluation mistakes are made, and treats utility through its most common proxy: the operating threshold. Chapter 7 shows that a single fixed threshold (“refer if p > 0.7”) is itself a utility judgement, and one that quietly costs different groups very different amounts.

2.4 Why the separation matters

Keeping the three apart is what lets you locate a failure.

A model can be… …while being… Symptom
well-ranked miscalibrated AUROC fine, probabilities untrustworthy
calibrated on average miscalibrated per subgroup pooled numbers reassuring, group numbers not
ranked and calibrated wrong for this decision good metrics, poor net benefit at the used threshold

Every chapter that follows is an instance of this table. The tools differ; the lesson does not: a single number answers a single question, and there are three.

With the questions named, we need somewhere to answer them where we already know the answer. That is the simulator.

1.
Ewout W. Steyerberg, Andrew J. Vickers, Nancy R. Cook, Thomas Gerds, Mithat Gonen, Nancy Obuchowski, Michael J. Pencina, Michael W. Kattan. Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38. doi:10.1097/EDE.0b013e3181c30fb2
2.
Ben Van Calster, David J. McLernon, Maarten van Smeden, Laure Wynants, Ewout W. Steyerberg. Calibration: The Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. doi:10.1186/s12916-019-1466-7
3.
Andrew J. Vickers, Elena B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565–74. doi:10.1177/0272989X06295361