Embedding qualitative data in LLM semantic space: A tutorial on conceptualization, measurement, and validation
Abstract
Large language model embeddings give any text coordinates in a semantic space, where the cosine between two coordinates indexes their similarity. Standard qualitative practice cannot quantify such closeness: open-ended answers become nominal variables, which represent only difference under dummy codes. Psychometrics obtains such closeness from responses, as the correlations among a scale's items. Whether the cosine can substitute for the correlation is therefore open. We tested it against established findings in open data, each with an agreed structure: the Big Five, PANAS in two languages, Schwartz's values, Pineo-Porter occupational prestige, and the ISEAR and Kjell narratives. Linguistic structure alone did not reproduce the psychometric structure. The item texts recovered part of it, and none of it where a relation carries a sign. They fall far short of a random split-sample ceiling. Read from the other side, that is a finding about questionnaires: a scale's wording does not carry what its respondents give it. Embeddings supply a semantic layer between raw language and conventional coding, where the material that neither psychometrics nor hand coding reaches belongs. A category label—an occupation, a diagnosis, a brand—becomes coordinates rather than a dummy variable. An open-ended answer becomes coordinates without a coder. Cut into segments, a long answer becomes a path, along which departures, returns, and drifts can be drawn and tested. What any of it means, however, stays with the analyst. We supply the conceptual background, an R package, and a tutorial: no training, no Python.
Related
- Cross-cultural adaptation, validation, and measurement invariance of the First Aid Willingness Questionnaire (FAWQ) among Turkish adolescents
- Development and psychometric validation of the Sexual Health Literacy Measurement Tool for Undergraduate Students (SHL-US) in Thailand
- Exact dg categories II: The embedding theorem
- ISOMETRIC EMBEDDING OF NONCOMMUTATIVE L_p -SPACES INTO NONCOMMUTATIVE SYMMETRIC SPACES
- Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation
BibTeX
@misc{muta2026embedding,
title = {{Embedding qualitative data in LLM semantic space: A tutorial on conceptualization, measurement, and validation}},
author = {Toshizumi Muta},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/6hxsw_v2},
url = {https://osf.io/preprints/psyarxiv/6hxsw_v2/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Embedding qualitative data in LLM semantic space: A tutorial on conceptualization, measurement, and validation
AU - Toshizumi Muta
PY - 2026
DA - 2026/09/21
PB - PsyArXiv
DO - 10.31234/osf.io/6hxsw_v2
UR - https://osf.io/preprints/psyarxiv/6hxsw_v2/
AB - Large language model embeddings give any text coordinates in a semantic space, where the cosine between two coordinates indexes their similarity. Standard qualitative practice cannot quantify such closeness: open-ended answers become nominal variables, which represent only difference under dummy codes. Psychometrics obtains such closeness from responses, as the correlations among a scale's items. Whether the cosine can substitute for the correlation is therefore open. We tested it against established findings in open data, each with an agreed structure: the Big Five, PANAS in two languages, Schwartz's values, Pineo-Porter occupational prestige, and the ISEAR and Kjell narratives. Linguistic structure alone did not reproduce the psychometric structure. The item texts recovered part of it, and none of it where a relation carries a sign. They fall far short of a random split-sample ceiling. Read from the other side, that is a finding about questionnaires: a scale's wording does not carry what its respondents give it. Embeddings supply a semantic layer between raw language and conventional coding, where the material that neither psychometrics nor hand coding reaches belongs. A category label—an occupation, a diagnosis, a brand—becomes coordinates rather than a dummy variable. An open-ended answer becomes coordinates without a coder. Cut into segments, a long answer becomes a path, along which departures, returns, and drifts can be drawn and tested. What any of it means, however, stays with the analyst. We supply the conceptual background, an R package, and a tutorial: no training, no Python.
ER -