Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models
- 1State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing 100191, China
Abstract
We study conformal prediction (CP) for multiple-choice question answering (MCQA) with large language models (LLMs) accessed only through application programming interfaces (APIs), where internal logits and token probabilities are unavailable. Candidate probabilities are often estimated from repeated answer samples, but finite sampling yields coarse estimates and tied nonconformity scores. We examine an extreme consequence of this discreteness, which we call efficiency collapse. It occurs when the correct option receives zero count on enough calibration examples, forcing the threshold to its maximum and producing full prediction sets. We derive the exact finite-sample condition and show that the problem stems from the empirical probability proxy rather than the split conformal rule. To address this issue, we propose Semantic Prior Smoothing (SemPrior), which uses semantic evidence from candidate rationales to allocate a nonuniform Dirichlet prior and combines it with observed answer counts. SemPrior leaves the standard nonconformity score and split conformal calibration unchanged. Experiments on four datasets show that SemPrior generally produces smaller prediction sets while keeping empirical coverage close to nominal levels. Results are also stable across the tested prior settings and calibration-set sizes.
Related
- Multi-Variable Conformal Prediction: Optimizing Prediction Sets using Multiple Calibration Variables
- SCL: Semantic Coherence Learning for Video Question Answering
- Asymptotics and finite sample bounds for prediction and smoothing in Wright-Fisher hidden Markov models
- Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
- Batch Universal Prediction via Conditional Additive Smoothing
BibTeX
@article{chen2026semantic,
title = {{Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models}},
author = {Yikai Chen and Bo Lang and Yi Wei},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14203761},
url = {https://www.mdpi.com/2227-7390/14/20/3761},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models
AU - Yikai Chen
AU - Bo Lang
AU - Yi Wei
PY - 2026
DA - 2026/10/16
JO - Mathematics
DO - 10.3390/math14203761
UR - https://www.mdpi.com/2227-7390/14/20/3761
AB - We study conformal prediction (CP) for multiple-choice question answering (MCQA) with large language models (LLMs) accessed only through application programming interfaces (APIs), where internal logits and token probabilities are unavailable. Candidate probabilities are often estimated from repeated answer samples, but finite sampling yields coarse estimates and tied nonconformity scores. We examine an extreme consequence of this discreteness, which we call efficiency collapse. It occurs when the correct option receives zero count on enough calibration examples, forcing the threshold to its maximum and producing full prediction sets. We derive the exact finite-sample condition and show that the problem stems from the empirical probability proxy rather than the split conformal rule. To address this issue, we propose Semantic Prior Smoothing (SemPrior), which uses semantic evidence from candidate rationales to allocate a nonuniform Dirichlet prior and combines it with observed answer counts. SemPrior leaves the standard nonconformity score and split conformal calibration unchanged. Experiments on four datasets show that SemPrior generally produces smaller prediction sets while keeping empirical coverage close to nominal levels. Results are also stable across the tested prior settings and calibration-set sizes.
SN - 2227-7390
ER -