Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
Abstract
Large language models (LLMs) are proposed as synthetic respondents for pilot testing, but their usefulness depends on whether they supply the evidence assessment development requires. We calibrated rating scale models on 14 digital-use skill items from 6,245 adults and used the human item parameters to evaluate responses generated for 1,300 demographically matched personas. LLM responses had high internal consistency (α≈ .94) but used the lowest category in 0.2-0.3% of responses versus 13.7-21.5% for humans, and no persona chose it on every item. These gaps changed the response-category judgment on both subscales and the PC targeting judgment; on the human-calibrated scales, 12 of 14 items had infit below 0.70, indicating responses more predictable than the Rasch model expects. Preregistered changes to the prompt, category order, and persona information did not restore the human lower range. High internal consistency is insufficient evidence that LLM responses can replace human pilot data.
Related
- Application of spectroscopic and multispectral imaging technologies on the assessment of ready-to-eat pineapple quality: A performance evaluation study of machine learning models generated from two commercial data analytics tools
- Benchmark Prediction with Sparse and Unpaired Cached Responses
- Financial Resilience Under a Major Gold-Price Regime Shift: A Benchmark-Based Assessment of Machine Learning, Deep Learning, and Econometric Models
- A Methodology for Integrating Life Cycle Assessment into a Multidisciplinary Design Analysis and Optimization Framework for Sustainable Launcher Development
- Covariance-Calibrated Prediction-Powered Inference
BibTeX
@misc{song2026large,
title = {{Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark}},
author = {Eunjeong Song and Sehee Hong},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.25059},
eprint = {2609.25059},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.25059},
}
RIS
TY - UNPB
TI - Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
AU - Eunjeong Song
AU - Sehee Hong
PY - 2026
DA - 2026/09/10
PB - arXiv
DO - 10.48550/arxiv.2609.25059
UR - https://arxiv.org/abs/2609.25059
AB - Large language models (LLMs) are proposed as synthetic respondents for pilot testing, but their usefulness depends on whether they supply the evidence assessment development requires. We calibrated rating scale models on 14 digital-use skill items from 6,245 adults and used the human item parameters to evaluate responses generated for 1,300 demographically matched personas. LLM responses had high internal consistency (α≈ .94) but used the lowest category in 0.2-0.3% of responses versus 13.7-21.5% for humans, and no persona chose it on every item. These gaps changed the response-category judgment on both subscales and the PC targeting judgment; on the human-calibrated scales, 12 of 14 items had infit below 0.70, indicating responses more predictable than the Rasch model expects. Preregistered changes to the prompt, category order, and persona information did not restore the human lower range. High internal consistency is insufficient evidence that LLM responses can replace human pilot data.
ER -