Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
Abstract
Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.
Related
- Bayesian Change-Point Detection and Fault Scoring in District Heating Systems
- Assessing the impact of climate change and rising temperatures on life insurance portfolios
- Penalties for change-points and outliers in linear models: localisation cost versus dimension cost
- Climate Change Finance and Carbon Offsets: Addressing Global Climate Challenges through Innovative Funding and Emissions Reduction Mechanisms
- Robust Quickest Change Detection with Cost-Efficient Experiment Design and Unknown Post-Change Model
BibTeX
@misc{jo2026change,
title = {{Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring}},
author = {Wonjeong Jo and Gyuwan Kim and Ming Shan Hee},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.22187},
eprint = {2610.22187},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.22187},
}
RIS
TY - UNPB
TI - Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
AU - Wonjeong Jo
AU - Gyuwan Kim
AU - Ming Shan Hee
PY - 2026
DA - 2026/10/15
PB - arXiv
DO - 10.48550/arxiv.2610.22187
UR - https://arxiv.org/abs/2610.22187
AB - Large language models (LLMs) are increasingly used in computational social science to assign scalar scores to social and psychological constructs. However, studies use different rating scales, and whether their scores remain comparable across scale designs is understudied. To address this gap, we define and systematically investigate scale sensitivity, the variation in LLM-generated scores caused by rating scale design. We distinguish numerical sensitivity, changes in normalized scores, from semantic sensitivity, changes in score interpretation. To assess both, our experimental design crosses 2 formats (continuous vs. ordinal), 4 ranges (3, 5, 11, and 21 points), and 2 signs (signed vs. unsigned) across 3 tasks (political leaning, sentiment, and toxicity), 2 prompting strategies (zero-shot and rubric-based), and 5 LLMs. We find that changing one design element shifts normalized scores by about 0.10 on average and changes the category of roughly 20% of texts. Sensitivity is greatest for toxicity, the only unipolar construct examined. Also, rubric-based prompting does not reduce sensitivity compared with zero-shot prompting. We provide practical guidance for designing more reliable LLM-based text scoring and highlight the need for robustness checks across alternative scales.
ER -