Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
Abstract
The Potts model extends the Ising model to multinomial data. We introduce a Rater Ising-Potts model that uses agreement indicators between pairs of ratings and category labels, with weights derived from LLM embeddings. The model does not presuppose ordered category thresholds or equidistant scoring; instead, it focuses on pairwise agreement among ratings and assigns category-specific positive weights, making it suited for multi-category scoring reliability. We evaluate the model on three constructed-response datasets spanning a corpus of K=14,466 short answers on a three-level rubric and two AERA essay prompts of roughly 1,200-1,400 responses on four-point rubrics. We compare three strategies for sharpening the similarity signal: top-K pruning, min-max normalization with a power transformation, and ColBERT late-interaction similarities. Top-K pruning, which replaces the dense similarity graph with a sparse local network of strongest semantic neighbors, consistently yields the highest accuracy and Cohen's kappa, and the selected neighborhoods are always a small fraction of the corpus. Power tuning consistently ranks second, while ColBERT is competitive on longer essay prompts and adds little on short answers. Across all settings, most misclassifications occur between adjacent score levels, confirming that the model preserves the ordinal structure of scoring rubrics without imposing rigid assumptions. These findings suggest that LLM-derived similarities, combined with a parsimonious Potts formulation and a sparse local graph, offer a robust and interpretable framework for reliability auditing in educational assessment. We discuss extensions to multiple raters and hierarchical rating designs.
Related
- Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing
- llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R
- Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatments
- The Akaike Information Criterion and Artificial Intelligence
- A Note on Threshold Principle, Artificial Neural Network Connections and Econometric Legacy
BibTeX
@misc{davier2026bridging,
title = {{Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights}},
author = {Matthias von Davier},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.08797},
eprint = {2609.08797},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.08797},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
AU - Matthias von Davier
PY - 2026
DA - 2026/09/14
PB - arXiv
DO - 10.48550/arxiv.2609.08797
UR - https://arxiv.org/abs/2609.08797
AB - The Potts model extends the Ising model to multinomial data. We introduce a Rater Ising-Potts model that uses agreement indicators between pairs of ratings and category labels, with weights derived from LLM embeddings. The model does not presuppose ordered category thresholds or equidistant scoring; instead, it focuses on pairwise agreement among ratings and assigns category-specific positive weights, making it suited for multi-category scoring reliability. We evaluate the model on three constructed-response datasets spanning a corpus of K=14,466 short answers on a three-level rubric and two AERA essay prompts of roughly 1,200-1,400 responses on four-point rubrics. We compare three strategies for sharpening the similarity signal: top-K pruning, min-max normalization with a power transformation, and ColBERT late-interaction similarities. Top-K pruning, which replaces the dense similarity graph with a sparse local network of strongest semantic neighbors, consistently yields the highest accuracy and Cohen's kappa, and the selected neighborhoods are always a small fraction of the corpus. Power tuning consistently ranks second, while ColBERT is competitive on longer essay prompts and adds little on short answers. Across all settings, most misclassifications occur between adjacent score levels, confirming that the model preserves the ordinal structure of scoring rubrics without imposing rigid assumptions. These findings suggest that LLM-derived similarities, combined with a parsimonious Potts formulation and a sparse local graph, offer a robust and interpretable framework for reliability auditing in educational assessment. We discuss extensions to multiple raters and hierarchical rating designs.
ER -