Pablo Zavala · AI Safety Evaluation · Research Engineering

Judgment by Algorithm: When Aggregate Fairness Hides Subgroup Harm

A graduate exercise on synthetic pretrial data shows disparate error rates across race that aggregate fairness summaries miss, a failure mode a safety review should catch and an auditable decision record would surface.

November 11, 2024 · 6 min

Two error rates tell the real story of an automated judgment. My graduate team ran a synthetic pretrial exercise across three constructed county datasets, and the exercise reported the two rates pulling in opposite directions by race: for Black defendants a false-positive rate more than five times the White rate, and for White defendants a false-negative rate more than four times the Black rate. A single fairness average would have hidden both.

Uneven error distribution, rather than a bad headline number, marks the auditability concern that should worry anyone who trusts an automated decision. A system can meet a global target and still route its mistakes toward the people least able to contest them.

A Teaching Exercise Rather Than a Field Audit

Honesty about the evidence comes first. My team, working under a responsible-AI course, examined three synthetic datasets rather than records from a real jurisdiction. The datasets carried county names as labels while filling them with constructed values: the Claiborne set skewed predominantly Black, the Copiah set predominantly White, and the Warren set split roughly evenly, and the three matched closely on population size, gender, and education. Synthetic data supports a teaching claim about mechanism rather than a field audit of any deployed tool. Every number below reports an output of that synthetic exercise built to expose a pattern, so I read the results as an illustration of how error distributes rather than as a measurement of any real place or vendor. The coursework materials, including the three synthetic datasets and the group reflective essay, sit behind the responsible-AI course and remain available on request.

Scores Varied by Race While Gender Stayed Flat

The disparity started in the scores themselves. Across all three datasets, Black defendants received higher risk scores than White and other defendants, while male and female defendants received closely matched scores. Gender behaves like a control in this exercise: the same pipeline that treated men and women alike produced a consistent racial gap. A gap that survives across datasets with different demographic mixes stays consistent with a scoring-process mechanism rather than any one population.

Labeled Decisions Moved the Threshold

The exercise's synthetic decision labels widened the gap instead of closing it. Among the synthetic records denied bail, Black defendants formed a large majority; among those released, White defendants did. At an identical synthetic risk score of 5 or 6, the labeled decisions denied Black defendants while releasing White defendants, so the coursework threshold variable the exercise reported sat at 4 for Black defendants and 6 for White defendants. A Black defendant whom the model scored less risky than a White defendant still met a stricter labeled cutoff. The synthetic labels encode a decision bias the score alone would understate; the exercise supplied a number that the labeled decisions then re-weighted.

Error Rates Expose the Subgroup Failure Mode

The threshold gap converted directly into divergent error rates, and the error rates carry the safety lesson. Defining a false positive as detention without re-offense and a false negative as release followed by re-offense, the synthetic exercise reported a Black false-positive rate over five times the White rate and a White false-negative rate over four times the Black rate. Each synthetic group carried a different kind of mistake: wrongful detention concentrated on Black defendants, and missed risk concentrated on White defendants. A model tuned to a single aggregate objective can post a respectable overall score while distributing its two failure modes this unevenly, which is why subgroup error rates, rather than a global average, belong at the center of any safety review.

One Metric Fails to Certify Fairness

Two fairness definitions collided, and the two resist holding together. Demographic parity asks for equal outcome rates across groups; equal opportunity asks for equal error rates, chiefly equal false-negative rates, among people who behave alike. The exercise failed both, and correcting for one worked against the other. Forcing demographic parity would have required releasing higher-scored Black defendants while detaining lower-scored White defendants, breaking equal opportunity; forcing equal opportunity would have required re-weighting the scores themselves, which exclude race explicitly yet let proxies such as neighborhood, income, and family history quietly encode it.

The broader literature formalizes the bind. Kleinberg, Mullainathan, and Raghavan and Chouldechova show that when base rates differ, calibration and equal error rates trade off against each other, and Hardt, Price, and Srebro define the equal-opportunity target the trade-off strains against. ProPublica's COMPAS analysis put the same conflict in front of the public. Calibration in aggregate, the reassurance that a score means the same thing on average, fails by itself to certify that a subgroup escapes concentrated harm.

Biased Labels Corrupt the Ground Truth

Real pretrial ground truth deserves its own suspicion. In real pretrial systems, re-offense labels rest on arrest data, and arrest data reflects who gets policed as much as who breaks the law. Because policing falls unequally, a Black defendant faces a higher chance of re-arrest regardless of underlying conduct, so arrest-based labels embed enforcement bias before any model reads them. A model trained on such labels learns to reproduce the enforcement pattern and then launders it as prediction. Threshold adjustment alone fails to repair a corrupted target; the fix has to reach the measurement, rather than just the cutoff. The synthetic exercise here sidesteps that bias, since its labels are constructed, so the point stands as external context a real deployment would confront.

A Decision Record Would Expose the Gap

The exercise points toward the design question I work on elsewhere. In this scenario, the stricter cutoff for Black defendants stayed implicit, buried in outcomes rather than stated as a rule a person could challenge. An auditable decision record would surface it: the score, the threshold applied, the recomputed subgroup error rate, and any objection raised would travel with the case instead of dissolving into a disposition. My Tribunal prototype pursues exactly that discipline, recording the premise, the rule, and the dissent so a reviewer can contest a judgment after the fact. A ledger leaves a biased decision unfair; a ledger makes the bias legible, which is the precondition for challenging it.

Boundaries

  • The dataset is synthetic and the analysis is coursework; the results illustrate a mechanism and stop short of certifying any deployed system.
  • The findings describe how error and threshold interact by race in a constructed scenario, separate from any claim about a named jurisdiction, vendor, or tool.
  • The fairness impossibility follows from differing base rates and biased measurement, rather than from a verdict that any single remedy is correct.

Sources