Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing
Abstract
Scoring consistency for constructed-response items in large-scale assessments is typically estimated through double-scoring, which uses small samples and assumes independence among responses. We present an integrated framework combining network psychometrics with the Linguistic-Integrated Reliability Audit (LiRA) via a modified Ising model. The model defines a joint distribution over binary correctness labels with pairwise interactions set to the cosine similarity of sentence embeddings and a global bias parameter for item difficulty. LiRA's weighted majority voting over semantic neighborhoods is shown to approximate the conditional logistic distributions of this Ising model. The parametric framework supports benchmark score generation, uncertainty quantification, and missing label imputation; parameters are estimated by maximum pseudo-likelihood. The approach uses the full dataset without requiring extensive double-scoring, accounts for semantic dependencies, and provides diagnostics for rater inconsistencies. The integration of LiRA's scalable methodology with a probabilistic graphical model offers a comprehensive tool for reliability assessment in international assessments such as PIRLS, PISA, and TIMSS.
Related
- Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
- llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R
- Spectral Dynamics of DeepWalk Embeddings for Dynamic Network Change-Point Detection
- The latent structure of diversification from relatedness and network embeddings
- Spectral Characterization and Network Systems Dynamics of Zero-Divisor Topologies over 𝔽p[x]/⟨ x4⟩: Consensus, Reliability, and Transport
BibTeX
@misc{davier2026integrating,
title = {{Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing}},
author = {Matthias von Davier},
howpublished = {arXiv},
year = {2026},
month = aug,
doi = {10.48550/arxiv.2608.26790},
eprint = {2608.26790},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.26790},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing
AU - Matthias von Davier
PY - 2026
DA - 2026/08/28
PB - arXiv
DO - 10.48550/arxiv.2608.26790
UR - https://arxiv.org/abs/2608.26790
AB - Scoring consistency for constructed-response items in large-scale assessments is typically estimated through double-scoring, which uses small samples and assumes independence among responses. We present an integrated framework combining network psychometrics with the Linguistic-Integrated Reliability Audit (LiRA) via a modified Ising model. The model defines a joint distribution over binary correctness labels with pairwise interactions set to the cosine similarity of sentence embeddings and a global bias parameter for item difficulty. LiRA's weighted majority voting over semantic neighborhoods is shown to approximate the conditional logistic distributions of this Ising model. The parametric framework supports benchmark score generation, uncertainty quantification, and missing label imputation; parameters are estimated by maximum pseudo-likelihood. The approach uses the full dataset without requiring extensive double-scoring, accounts for semantic dependencies, and provides diagnostics for rater inconsistencies. The integration of LiRA's scalable methodology with a probabilistic graphical model offers a comprehensive tool for reliability assessment in international assessments such as PIRLS, PISA, and TIMSS.
ER -