5  Uncertainty is the finding

Chapter 4 produced an estimate of 0.655 in the smallest stratum against a truth of 0.714. The question this chapter answers is whether that discrepancy was detectable and, more usefully, whether the study could have detected anything at all.

5.1 The stratified bootstrap

Resample with replacement, recompute, repeat, take percentiles. The only subtlety worth caring about is that resampling is done within outcome class:

boot_ci
function (label, score, fn, n_boot = 2000L, alpha = 0.05, seed = 1L, 
    ...) 
{
    point <- fn(label, score, ...)
    ix1 <- which(label == 1L)
    ix0 <- which(label == 0L)
    if (length(ix1) < 2L || length(ix0) < 2L) {
        return(c(estimate = point, lower = NA_real_, upper = NA_real_))
    }
    set.seed(seed)
    stats <- numeric(n_boot)
    for (b in seq_len(n_boot)) {
        ix <- c(sample(ix1, replace = TRUE), sample(ix0, replace = TRUE))
        stats[b] <- fn(label[ix], score[ix], ...)
    }
    stats <- stats[is.finite(stats)]
    q <- quantile(stats, c(alpha/2, 1 - alpha/2), names = FALSE)
    c(estimate = point, lower = q[1], upper = q[2])
}
<bytecode: 0x10ca22e10>

Resampling rows without regard to outcome regularly produces, in a stratum of 39 events, replicates containing no events at all. The metric is then undefined, those replicates get quietly dropped, and the interval becomes a statement about which replicates happened to survive. Stratifying fixes it.

5.2 What the intervals look like

Group n Events True Estimate Lower Upper Width
overall 2860 597 NA 0.867 0.852 0.883 0.031
I-II 1800 377 0.892 0.897 0.878 0.913 0.035
III-IV 900 181 0.847 0.855 0.824 0.883 0.058
V-VI 160 39 0.714 0.655 0.554 0.750 0.196

Subgroup AUROC with 95% bootstrap intervals. Crosses mark the planted truth.

The interval in V-VI is 0.196 wide, six times the pooled interval. It spans from “barely better than chance” to “clinically respectable”. Reported as “AUROC 0.66 in Fitzpatrick V–VI”, which is how it would appear in a table, that estimate conveys a precision it does not have.

5.3 The gap you cannot see is the moderate one

The natural conclusion is that the smallest group is the problem. The more useful conclusion is about the middle row. How many events would each comparison need?

base <- tt$true_auroc[tt$group == "I-II"]
data.frame(
  group        = c("III-IV", "V-VI"),
  true_gap     = round(base - tt$true_auroc[match(c("III-IV","V-VI"), tt$group)], 3),
  events_need  = sapply(c("III-IV","V-VI"), function(g)
                   n_events_for_gap(base, tt$true_auroc[tt$group == g])),
  events_have  = rep$events[match(c("III-IV","V-VI"), rep$group)]
)
        group true_gap events_need events_have
III-IV III-IV    0.045         445         181
V-VI     V-VI    0.178          40          39

The dramatic gap in V-VI is, just barely, detectable: 39 events against roughly 40 required. The moderate 0.045 gap in III-IV needs about 445 events per group and the data contain 181.

So the real gap in III-IV will be found “not statistically significant”, and reported as reassurance. This is the failure mode that matters, because moderate gaps are the common ones, and the study is systematically blind to exactly the range of effects most likely to exist.

Warning“No significant difference between subgroups” is not a finding

It is the expected result of an underpowered comparison, and subgroup comparisons are nearly always underpowered. The honest report states the smallest gap the analysis could have detected, then says whether it detected it. Absence of evidence, as usual, is doing no work here.

5.4 The bootstrap is not free either

The intervals above are percentile intervals, which are known to undercover in small samples (1). So measure it. Simulate 200 fresh datasets and count how often the nominal 95% interval actually contains the truth:

covered <- 0; n_rep <- 200
truth <- true_auroc(GROUPS[["V-VI"]]$d)
for (r in seq_len(n_rep)) {
  set.seed(10000 + r)
  y <- rbinom(160, 1, PREVALENCE)
  s <- plogis(rnorm(160, mean = y * GROUPS[["V-VI"]]$d, sd = 1))
  ci <- boot_ci(y, s, auroc, n_boot = 400L, seed = r)
  if (!is.na(ci[["lower"]]) && truth >= ci[["lower"]] && truth <= ci[["upper"]])
    covered <- covered + 1
}
covered / n_rep
[1] 0.895

89.5%, not 95%. The intervals in this chapter are, if anything, too narrow, which strengthens the argument. BCa intervals or the analytic Hanley–McNeil variance (2) would be the next step.

1.
Bradley Efron. Better bootstrap confidence intervals. Journal of the American Statistical Association. 1987;82(397):171–85. doi:10.1080/01621459.1987.10478410
2.
James A. Hanley, Barbara J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. doi:10.1148/radiology.143.1.7063747