Discrimination and calibration are properties of the scores. A decision is not: it needs a threshold, a line past which the model says act. The moment you draw that line, a new question opens: is one line right for everyone? Whenever the subgroups differ in separability, the honest answer is no.
One threshold, measured two ways
Start with the naive policy: the same cutoff, 0.5, for every group. The equalised-odds view reports the true-positive and false-positive rates each group actually gets under that shared rule (1).
function (df, group_col = "group", threshold = 0.5)
{
gs <- sort(unique(df[[group_col]]))
tpr <- sapply(gs, function(g) {
s <- df[df[[group_col]] == g, ]
sensitivity(s$label, s$score, threshold)
})
fpr <- sapply(gs, function(g) {
s <- df[df[[group_col]] == g, ]
1 - specificity(s$label, s$score, threshold)
})
list(tpr = tpr, fpr = fpr, tpr_gap = max(tpr) - min(tpr),
fpr_gap = max(fpr) - min(fpr), worst = names(which.min(tpr)))
}
<bytecode: 0x122246240>
| I-II |
I-II |
0.960 |
0.506 |
| III-IV |
III-IV |
0.923 |
0.497 |
| V-VI |
V-VI |
0.744 |
0.521 |
Read the two columns separately, because they tell opposite stories. The false-positive rates are close: the non-event scores are drawn from the same distribution in every stratum, so a fixed cutoff catches roughly the same fraction of them everywhere. It is the true-positive rate that collapses: a gap of 0.217 between the best and worst group. A shared threshold does not share the harm evenly; it hands the worst-separated group a much lower chance of being caught. That is the disparate impact hiding inside a group-blind rule.
Fixing the clinical standard instead
So flip it. Fix the thing a clinician actually cares about (“detect 90% of events”) and let each group have whatever threshold reaches it. Now the sensitivity is equal by construction, and the cost surfaces somewhere else.
function (df, group_col = "group", target = 0.9)
{
gs <- sort(unique(df[[group_col]]))
do.call(rbind, lapply(gs, function(g) {
sub <- df[df[[group_col]] == g, ]
thr <- unname(quantile(sub$score[sub$label == 1L], 1 -
target))
data.frame(group = g, threshold = thr, sensitivity = sensitivity(sub$label,
sub$score, thr), specificity = specificity(sub$label,
sub$score, thr), row.names = NULL)
}))
}
<bytecode: 0x121f7ed88>
| I-II |
0.617 |
0.899 |
0.699 |
| III-IV |
0.564 |
0.901 |
0.574 |
| V-VI |
0.256 |
0.897 |
0.132 |
Every group now catches 90% of events. But look at what that costs in specificity. In I-II it is 0.699, so you wave through most true negatives. In V-VI it is 0.132: to reach the same sensitivity on a flatter ROC curve, the threshold slides down into the non-event mass and specificity falls off a cliff.
Put in the terms a service feels: at the same clinical standard, the best-served group refers about 30% of its negatives for follow-up, and the worst-served group refers about 87% of its negatives. Same model, same rule, wildly different burden. That is a service-design problem, not a rounding error.
The trade-off is forced, not incidental
You cannot escape by being cleverer with the cutoff. Equalised odds asks for equal TPR and equal FPR at once; when two groups sit on ROC curves of different quality, you can satisfy that only where the curves happen to cross, which is nowhere near a clinically useful operating point. A shared threshold fails on the TPR leg; equalising sensitivity fails on the FPR leg. Every choice absorbs the disparity somewhere.
Which failure to accept (more missed cases in the worst group, or more false alarms there) is a value judgement about who bears the cost of the model’s weaker grip on that group. It is not a judgement a metric can make for you, and it is certainly not one a single shared threshold makes honestly; it just makes it silently.
p > 0.5 looks neutral and is not. It encodes equal false-positive rates and unequal sensitivities, and it hides that choice behind a round number. Report the operating point you actually use, the sensitivity and specificity it produces in each subgroup, and who pays for the gap.
1.
Moritz Hardt, Eric Price, Nathan Srebro. Equality of opportunity in supervised learning. In: Advances in neural information processing systems (NeurIPS). 2016. p. 3315–23.