4  Discrimination from scratch

The AUROC is usually drawn as the area under a curve of sensitivity against one-minus-specificity, swept over every threshold. That picture is fine, but it hides how simple the quantity is. The area under the ROC curve is exactly the probability that a randomly chosen event scores higher than a randomly chosen non-event. Stated that way, it is a rank statistic, and it can be computed in one line.

4.1 The whole estimator

auroc
function (label, score) 
{
    if (length(unique(label)) < 2L) 
        return(NA_real_)
    n1 <- sum(label == 1L)
    n0 <- sum(label == 0L)
    r <- rank(score)
    (sum(r[label == 1L]) - n1 * (n1 + 1)/2)/(n1 * n0)
}

Rank all the scores; sum the ranks belonging to the events; subtract the ranks those events would occupy if they were all crammed at the bottom (\(n_1(n_1+1)/2\)); divide by the number of event/non-event pairs \(n_1 n_0\). What comes out is the fraction of pairs the model orders correctly: the Mann-Whitney U statistic, rescaled. Ties contribute exactly one half, and using average ranks builds that half-credit in with no special case.

4.2 Is it right? Two checks

A one-line estimator is easy to get subtly wrong, so we never take it on faith. First, against a completely different code path (wilcox.test, which computes the same U statistic through R’s own rank machinery):

w <- wilcox.test(score ~ factor(label), data = df, exact = FALSE)
u_ref <- unname(w$statistic) / (sum(df$label == 0) * sum(df$label == 1))
c(ours = auroc(df$label, df$score), wilcox = 1 - u_ref)
     ours    wilcox 
0.8674829 0.8674829 

Identical to machine precision. That pins the implementation. But a numerically perfect estimator of the wrong quantity is still wrong. The second check is against the truth the simulator planted, the analytic \(\Phi(d/\sqrt{2})\) from the previous chapter:

Group True AUROC Estimate Error
I-II I-II 0.892 0.897 0.005
III-IV III-IV 0.847 0.855 0.007
V-VI V-VI 0.714 0.655 -0.059

Close everywhere, and closest where the sample is largest. That is exactly the behaviour a consistent estimator should show. The estimate in V-VI is 0.655 against a truth of 0.714: an error of -0.059 on 160 patients. Whether that gap is alarming or just sampling noise is not a question the point estimate can answer, and it is the subject of the next chapter.

4.3 What discrimination cannot see

Because it is built from ranks, the AUROC is invariant to any strictly increasing transformation of the score. Running it on the raw latent value and on the squeezed probability gives the same answer to the last digit:

c(on_probability = auroc(df$label, df$score),
  on_latent      = auroc(df$label, df$latent))
on_probability      on_latent 
     0.8674829      0.8674829 

This is a feature and a warning at once. A feature, because it means a badly scaled or miscalibrated model can still discriminate perfectly, and ranking is the one property recalibration cannot destroy. A warning, because it means the AUROC is structurally blind to whether the probabilities mean anything at all. A model can post an AUROC of 0.87 and still be uninterpretable and unsafe at any given threshold.

AUROC sees the order and nothing else. To find out whether the numbers themselves can be trusted (and whether the 0.655 in V-VI is real), we need to put a confidence interval on it, and then a second question entirely.