Bayesian Consensus Calibration of Continuously Evolving IRT Item Banks
Abstract
AI-based item generation and NLP-based prediction of item parameters are producing item banks that are substantially larger, sparser, and more frequently updated than conventional banks. Hierarchical Bayesian item response theory (IRT) is a natural calibration framework for such banks, but the common practice of refitting the entire accumulated response history at each update is costly and can exceed available memory. We describe \emph{consensus calibration}, a divide-and-conquer procedure that calibrates each time period independently and reconstructs the pooled posterior in two layers. First, the posterior draws of each period are mapped to a common metric by a robust characteristic-curve linking (Haebara) that is solved separately for each draw, which propagates the uncertainty of the linking transformation into the linked posteriors. Second, the linked item posteriors are combined as a product of Gaussian densities from which the population prior contributed by each period is removed and a single prior---obtained by consensus across the per-period population posteriors---is reinstated. The correction targets the posterior dispersion, not only its location. As evidence for consensus calibration, we compare it to a pooled single-run analysis on a large operational assessment in terms of item-parameter recovery, an uncertainty-by-exposure diagnostic, and the ability distributions.
Related
- Large Language Models Meet Bayesian Optimization: Towards Self-evolving Wireless Digital Twins
- An Information Freshness-Based Method for Quantum Bayesian Networks in the Consensus-Reaching Process Under Relative Deprivation
- Flexible Transformations for Bayesian Score Calibration
- Rényi-Regularized Flow-Based Variational Inference for Bayesian Calibration
- Posterior Geometry and Identifiability in Multi-Response Bayesian Calibration
BibTeX
@misc{jewsbury2026bayesian,
title = {{Bayesian Consensus Calibration of Continuously Evolving IRT Item Banks}},
author = {Paul A. Jewsbury and Steven W. Nydick and Manqian Liao and Siyuan and Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.13590},
eprint = {2609.13590},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.13590},
}
RIS
TY - UNPB
TI - Bayesian Consensus Calibration of Continuously Evolving IRT Item Banks
AU - Paul A. Jewsbury
AU - Steven W. Nydick
AU - Manqian Liao
AU - Siyuan
AU - Chen
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2609.13590
UR - https://arxiv.org/abs/2609.13590
AB - AI-based item generation and NLP-based prediction of item parameters are producing item banks that are substantially larger, sparser, and more frequently updated than conventional banks. Hierarchical Bayesian item response theory (IRT) is a natural calibration framework for such banks, but the common practice of refitting the entire accumulated response history at each update is costly and can exceed available memory. We describe \emph{consensus calibration}, a divide-and-conquer procedure that calibrates each time period independently and reconstructs the pooled posterior in two layers. First, the posterior draws of each period are mapped to a common metric by a robust characteristic-curve linking (Haebara) that is solved separately for each draw, which propagates the uncertainty of the linking transformation into the linked posteriors. Second, the linked item posteriors are combined as a product of Gaussian densities from which the population prior contributed by each period is removed and a single prior---obtained by consensus across the per-period population posteriors---is reinstated. The correction targets the posterior dispersion, not only its location. As evidence for consensus calibration, we compare it to a pooled single-run analysis on a large operational assessment in terms of item-parameter recovery, an uncertainty-by-exposure diagnostic, and the ability distributions.
ER -