Testing the Validity of Embedding-Based Similarity and Clustering for Handwritten Physics Solutions
Abstract
Text embeddings are increasingly used in physics education research to organize, compare, and cluster large collections of written text. Their appeal is clear: once student responses have been mapped into a vector space, similarity comparisons and clustering become computationally inexpensive. However, in assessment contexts, the relevant question is not merely whether clusters can be produced, but whether the geometry of the embedding space preserves grading-relevant distinctions. We tested this premise using 992 handwritten student-problem solutions from a high-stakes engineering thermodynamics exam, transcribed into five textual representations and embedded using nine embedding mechanisms. We compared embedding similarity and embedding-based hierarchical clusters against human-assigned scores. Across models, representations, and clustering choices, embedding similarity showed a consistent but modest relationship to score similarity, and the resulting clusters were score-enriched but not score-equivalent. Experiments with a synthetic data set suggest that this may be due to embeddings behaving like novices when categorizing physics-problem solutions, that is, their similarity geometry is strongly influenced by surface features rather than conceptual, semantic structure. These findings suggest that state-of-the-art embeddings can support exploratory organization and human-in-the-loop review of physics solutions, but they do not provide an unsupervised basis for grading without external validation against the assessment construct of interest.
Versions
Related
- Probabilistic, Structure-Intrinsic Clustering with an Hierarchical Embedding (PSICHE) in Time and Space Applied In Particle Physics Jet Reconstruction
- FPGA-Based Implementation of Artificial Neural Network for Accelerated Handwritten Digit Recognition
- Dynamical large deviations, self-similarity, and criticality
- Extending the Similarity Spectrum to the SWIR for quality control of above water reflectance measurements
- Generalized Similarity Theory for Plasmas
BibTeX
@misc{tauschhuber2026testing,
title = {{Testing the Validity of Embedding-Based Similarity and Clustering for Handwritten Physics Solutions}},
author = {Maike Tauschhuber and Gerd Kortemeyer},
howpublished = {arXiv},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.28872},
eprint = {2606.28872},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.28872},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Testing the Validity of Embedding-Based Similarity and Clustering for Handwritten Physics Solutions
AU - Maike Tauschhuber
AU - Gerd Kortemeyer
PY - 2026
DA - 2026/06/27
PB - arXiv
DO - 10.48550/arxiv.2606.28872
UR - https://arxiv.org/abs/2606.28872
AB - Text embeddings are increasingly used in physics education research to organize, compare, and cluster large collections of written text. Their appeal is clear: once student responses have been mapped into a vector space, similarity comparisons and clustering become computationally inexpensive. However, in assessment contexts, the relevant question is not merely whether clusters can be produced, but whether the geometry of the embedding space preserves grading-relevant distinctions. We tested this premise using 992 handwritten student-problem solutions from a high-stakes engineering thermodynamics exam, transcribed into five textual representations and embedded using nine embedding mechanisms. We compared embedding similarity and embedding-based hierarchical clusters against human-assigned scores. Across models, representations, and clustering choices, embedding similarity showed a consistent but modest relationship to score similarity, and the resulting clusters were score-enriched but not score-equivalent. Experiments with a synthetic data set suggest that this may be due to embeddings behaving like novices when categorizing physics-problem solutions, that is, their similarity geometry is strongly influenced by surface features rather than conceptual, semantic structure. These findings suggest that state-of-the-art embeddings can support exploratory organization and human-in-the-loop review of physics solutions, but they do not provide an unsupervised basis for grading without external validation against the assessment construct of interest.
ER -