llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R
Abstract
Text classification has become a central measurement problem in psychology and the social sciences, and the arrival of large language models (LLMs) multiplied the number of available strategies, from contextual encoders and embedding services to prompted generative models and, more recently, general-purpose classification models such as Jev, which return probabilities for typed questions about a text. The present paper introduces the R package llmClassificR, which brings these strategies together in a single, consistent interface, and shows how they can be combined with network psychometrics, complexity science, and modern uncertainty quantification. The package implements (a) unsupervised classification of texts with exploratory graph analysis applied to the embeddings of local encoder models, decoder LLMs, and embedding APIs, including a prime-weight encoding that combines the networks estimated from every transformer layer into a single multiplex representation; (b) supervised classification heads on frozen embeddings, evaluated by cross-validation; (c) prompted classification with generative LLMs, with shuffled batching, label validation, strict JSON Schema outputs, ensembles, and self-consistency; (d) retrieval-augmented, automated, and human-guided topic discovery; (e) Jev-based classification with Choice, Noul, and Score questions; (f) do-it-yourself Jev-like classifiers built from label descriptions and local or API embeddings; and (g) calibration, temperature scaling, Platt scaling, selective classification, and conformal prediction sets. The tutorial uses synthetic texts generated by LLMs, descriptions of emotional situations labeled by the people who wrote them (EmpatheticDialogues), and Reddit comments labeled by several raters (GoEmotions), which makes it possible to show how methods that look nearly perfect on synthetic data behave on real text, how the reliability of the labels limits the accuracy that any classifier can demonstrate, and how model uncertainty relates to human disagreement.
Related
- Bagged Martingale Posteriors: Calibrated Uncertainty Quantification for Predictive Resampling
- No-Harm Physics-Informed Inverse Learning with Residual-Calibrated Uncertainty
- Think Short, Defer Smart, Act, and Repeat: Calibrated Reasoning and Uncertainty-Aware Deferral for Edge LLM Agents
- Calibrated for Which Policy? Conformal Uncertainty for Fixed World Models Under Intervention
- Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
BibTeX
@misc{golino2026llmclassificr,
title = {{llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R}},
author = {Hudson Golino},
howpublished = {PsyArXiv},
year = {2026},
month = nov,
doi = {10.31234/osf.io/259gt_v1},
url = {https://osf.io/preprints/psyarxiv/259gt_v1/},
note = {CC0},
}
RIS
TY - UNPB
TI - llmClassificR: Text Classification with Large Language Models, Network Psychometrics, and Calibrated Uncertainty in R
AU - Hudson Golino
PY - 2026
DA - 2026/11/02
PB - PsyArXiv
DO - 10.31234/osf.io/259gt_v1
UR - https://osf.io/preprints/psyarxiv/259gt_v1/
AB - Text classification has become a central measurement problem in psychology and the social sciences, and the arrival of large language models (LLMs) multiplied the number of available strategies, from contextual encoders and embedding services to prompted generative models and, more recently, general-purpose classification models such as Jev, which return probabilities for typed questions about a text. The present paper introduces the R package llmClassificR, which brings these strategies together in a single, consistent interface, and shows how they can be combined with network psychometrics, complexity science, and modern uncertainty quantification. The package implements (a) unsupervised classification of texts with exploratory graph analysis applied to the embeddings of local encoder models, decoder LLMs, and embedding APIs, including a prime-weight encoding that combines the networks estimated from every transformer layer into a single multiplex representation; (b) supervised classification heads on frozen embeddings, evaluated by cross-validation; (c) prompted classification with generative LLMs, with shuffled batching, label validation, strict JSON Schema outputs, ensembles, and self-consistency; (d) retrieval-augmented, automated, and human-guided topic discovery; (e) Jev-based classification with Choice, Noul, and Score questions; (f) do-it-yourself Jev-like classifiers built from label descriptions and local or API embeddings; and (g) calibration, temperature scaling, Platt scaling, selective classification, and conformal prediction sets. The tutorial uses synthetic texts generated by LLMs, descriptions of emotional situations labeled by the people who wrote them (EmpatheticDialogues), and Reddit comments labeled by several raters (GoEmotions), which makes it possible to show how methods that look nearly perfect on synthetic data behave on real text, how the reliability of the labels limits the accuracy that any classifier can demonstrate, and how model uncertainty relates to human disagreement.
ER -