3  The data, and its known answer

To trust a measuring instrument, you calibrate it against something whose length you already know. Real evaluation data can never be that reference, because with real data the truth (the model’s actual discrimination in each subgroup) is precisely what you are trying to estimate and never get to see. So we simulate, and we simulate in a way that hands us the answer up front.

3.1 A generator with a closed-form truth

The design is deliberately plain. Within each stratum, non-events get a latent score drawn from a standard normal, and events get one shifted up by a separation d:

simulate_predictions
function (groups = GROUPS, prevalence = PREVALENCE, seed = SEED) 
{
    set.seed(seed)
    out <- vector("list", length(groups))
    for (i in seq_along(groups)) {
        g <- names(groups)[i]
        n <- groups[[i]]$n
        d <- groups[[i]]$d
        y <- rbinom(n, 1, prevalence)
        latent <- rnorm(n, mean = y * d, sd = 1)
        out[[i]] <- data.frame(id = sprintf("%s_%05d", g, seq_len(n)), 
            group = g, label = y, latent = latent, score = plogis(latent), 
            stringsAsFactors = FALSE)
    }
    df <- do.call(rbind, out)
    df[sample(nrow(df)), ]
}
<bytecode: 0x1301a7e70>

The move that makes the whole book possible is this: for two normals of equal variance separated by d, the probability that a draw from the event distribution exceeds a draw from the non-event distribution is the AUROC, and it has a closed form.

\[ \text{AUROC} \;=\; P(X_{\text{event}} > X_{\text{non-event}}) \;=\; \Phi\!\left(\frac{d}{\sqrt{2}}\right) \]

true_auroc
function (d) 
{
    pnorm(d/sqrt(2))
}
<bytecode: 0x1329a3a88>

So we do not have to wonder whether an estimate is good. We can subtract it from the right answer.

3.2 The three strata

The separations and sizes are chosen to caricature a common situation: a model that is sharp in the group that dominates the data and dull in the group that does not. The small group is also the one where the signal is genuinely weaker.

Group n Separation d True AUROC
I-II 1800 1.75 0.892
III-IV 900 1.45 0.847
V-VI 160 0.80 0.714

Two things are held fixed on purpose. The prevalence is the same 0.22 in every stratum, so that no performance gap can later be blamed on a difference in base rates. The gap will be about separation and sample size, nothing else. And the seed is fixed, so every number in this book is reproducible to the digit.

Latent score distributions by outcome, within each stratum. As the separation d shrinks, the two distributions overlap more, and the model’s job gets harder.

In I-II the two distributions barely touch; in V-VI they sit almost on top of each other. That overlap is the true AUROC of 0.714, drawn.

The score column the rest of the book uses is just this latent value squeezed through plogis() onto the 0–1 probability scale: a stand-in for a model that is imperfectly calibrated, which becomes the subject of Chapter 6. Everything downstream reads only label and score, exactly as it would for a real model’s output.

With a world whose answer we know, we can now write an estimator and prove it recovers that answer.