AI-based scoring systematically underestimates conceptual understanding of linguistically weak students' explanations in physics
Abstract
Students' explanations of scientific phenomena provide important evidence of their conceptual understanding. However, because conceptual understanding can only be inferred through language rather than observed directly, distinguishing conceptual understanding from linguistic quality represents a fundamental challenge for assessment. This challenge can also be expected to arise when artificial intelligence (AI) is used to score students' text-based explanations. We investigated whether AI-based scoring approaches assess students' conceptual understanding independently of the linguistic quality of their explanations about a phenomenon taken from a physics context. Conceptual understanding scores assigned by nine machine learning-based scoring approaches and two large language model-based scoring approaches were compared with expert-assigned scores for 116 explanations produced by secondary-school students in Germany. Multinomial logistic regression analyses examined whether expert-rated linguistic quality was associated with under- or overestimation of conceptual understanding. Across all eleven AI-based scoring approaches, lower linguistic quality was associated with a greater likelihood of underestimation, although the strength and statistical significance of this association varied between approaches. AI-based scoring approaches may thus systematically underestimate the conceptual understanding expressed in linguistically weaker explanations. This pattern resembles language bias previously documented in STEM teachers' assessment practices. The findings highlight the importance of examining not only agreement with expert ratings but also whether AI-based scoring approaches inadvertently rely on construct-irrelevant information. Addressing language bias is therefore essential for the validity and fairness of AI-supported assessment in physics education and STEM education more broadly.
Related
- Enhancing Students’ Conceptual Understanding of Elasticity Through IoT Based Real Time Multiple Representations: A Quasi Experimental Study
- Development of A Google Sites-Based Granular Mechanics E-Module to Facilitate Students' Conceptual Understanding
- Development of A STEM-Integrated Guided Inquiry E-Module on Global Warming to Enhance Students’ Conceptual Understanding
- Comparative Effectiveness of Learning Cycle 5E-Based and Regular Worksheets on Conceptual Understanding of Current Electricity
- Targeted search shows that random-device testing underestimates worst-case error in a simulated wave-based neural operator
BibTeX
@misc{feser2026aibased,
title = {{AI-based scoring systematically underestimates conceptual understanding of linguistically weak students' explanations in physics}},
author = {Markus S. Feser and Paul L. Tschisgale},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2607.28210},
eprint = {2607.28210},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2607.28210},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - AI-based scoring systematically underestimates conceptual understanding of linguistically weak students' explanations in physics
AU - Markus S. Feser
AU - Paul L. Tschisgale
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2607.28210
UR - https://arxiv.org/abs/2607.28210
AB - Students' explanations of scientific phenomena provide important evidence of their conceptual understanding. However, because conceptual understanding can only be inferred through language rather than observed directly, distinguishing conceptual understanding from linguistic quality represents a fundamental challenge for assessment. This challenge can also be expected to arise when artificial intelligence (AI) is used to score students' text-based explanations. We investigated whether AI-based scoring approaches assess students' conceptual understanding independently of the linguistic quality of their explanations about a phenomenon taken from a physics context. Conceptual understanding scores assigned by nine machine learning-based scoring approaches and two large language model-based scoring approaches were compared with expert-assigned scores for 116 explanations produced by secondary-school students in Germany. Multinomial logistic regression analyses examined whether expert-rated linguistic quality was associated with under- or overestimation of conceptual understanding. Across all eleven AI-based scoring approaches, lower linguistic quality was associated with a greater likelihood of underestimation, although the strength and statistical significance of this association varied between approaches. AI-based scoring approaches may thus systematically underestimate the conceptual understanding expressed in linguistically weaker explanations. This pattern resembles language bias previously documented in STEM teachers' assessment practices. The findings highlight the importance of examining not only agreement with expert ratings but also whether AI-based scoring approaches inadvertently rely on construct-irrelevant information. Addressing language bias is therefore essential for the validity and fairness of AI-supported assessment in physics education and STEM education more broadly.
ER -