Criminal Justice Risk Assessments with Unreliable Class Assignments
Abstract
Quantitative risk assessments have been used for decades to help inform criminal justice decisions. A set of predictors and a categorical response variable are used to train a statistical classifier. The trained classifier can be employed to compute risk scores for a new, unlabeled case for which a forecast is needed, one risk score for each response variable class. In the spirit of Bayes classifiers, the class with the largest risk score conventionally becomes the forecasted outcome label for that case. The largest and next largest risk scores can be very similar or very different. However, when they are much the same, the outcome class is essentially being assigned by a coin flip; the classification is unreliable. Forecasting error becomes more likely, and concerns about fairness can arise. This paper shows how Mondrian conformal prediction sets can increase classification reliability when the classifier is indecisive and provide valid estimates of forecast uncertainty. Risk forecasts for a large dataset of offenders on probation are used to illustrate the issues.
Related
- Noisy-Label Calibration with Multiple Unreliable Labels
- Exact Labelled Layer-Order Classification for Period-2 Mountain--Valley Assignments on 2 x 2k Maps
- Who Blocks Whom? Probabilistic Pass-Blocking Assignments for Evaluating Blockers and Pass Rushers in American Football
- Fixed Hall-Call Assignments and Floor-Ordered Pair Costs in Elevator Dispatch
- Closed Lindelöf Σ Kernels of Neighborhood Assignments in Charming Spaces
BibTeX
@misc{berk2026criminal,
title = {{Criminal Justice Risk Assessments with Unreliable Class Assignments}},
author = {Richard Berk},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.22718},
eprint = {2609.22718},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.22718},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Criminal Justice Risk Assessments with Unreliable Class Assignments
AU - Richard Berk
PY - 2026
DA - 2026/09/19
PB - arXiv
DO - 10.48550/arxiv.2609.22718
UR - https://arxiv.org/abs/2609.22718
AB - Quantitative risk assessments have been used for decades to help inform criminal justice decisions. A set of predictors and a categorical response variable are used to train a statistical classifier. The trained classifier can be employed to compute risk scores for a new, unlabeled case for which a forecast is needed, one risk score for each response variable class. In the spirit of Bayes classifiers, the class with the largest risk score conventionally becomes the forecasted outcome label for that case. The largest and next largest risk scores can be very similar or very different. However, when they are much the same, the outcome class is essentially being assigned by a coin flip; the classification is unreliable. Forecasting error becomes more likely, and concerns about fairness can arise. This paper shows how Mondrian conformal prediction sets can increase classification reliability when the classifier is indecisive and provide valid estimates of forecast uncertainty. Risk forecasts for a large dataset of offenders on probation are used to illustrate the issues.
ER -