Skip to content
DraconiticResearch
Search Sources API
201,683 studies31 sourcesUpdated 20 min ago
Peer-reviewedCC BY 4.0Royal Society Open Science

Applying psychometrics to large language model simulated populations: recreating the HEXACO personality inventory experiment with generative agents

  • Sarah Mercer1
  • Daniel Martin1
  • Phil Swatton1
  1. 1The Alan Turing Institute , London, UK
Read full textPDFDOICite

Abstract

Generative agents powered by large language models (LLMs) demonstrate human-like characteristics through sophisticated natural language interactions. Their ability to assume roles and personalities based on predefined character biographies has positioned them as cost-effective substitutes for human participants in social science research. This paper explores the validity of such persona-based agents1 in representing human populations; we recreate the HEXACO personality inventory experiment by surveying 310 GPT-4 powered agents, conducting factor analysis on their responses, and comparing these results to the original findings presented by Ashton et al. (Ashton et al. 2004 J. Pers. Soc. Psychol. 87, 707–721 (doi:10.1037/0022-3514.87.5.707). Our results found (i) a coherent and reliable personality structure was recoverable from the agents’ responses demonstrating partial alignment to the HEXACO framework, (ii) the derived personality dimensions were consistent and reliable within GPT-4, when coupled with a sufficiently curated population, and (iii) cross-model analysis revealed variability in personality profiling, suggesting model-specific biases and limitations. We discuss the practical considerations and challenges encountered during the experiment. This study contributes to the ongoing discourse on the potential benefits and limitations of using generative agents in social science research and provides useful guidance on designing consistent and representative agent personas to maximize coverage and representation of human personality traits, while highlighting potential limitations of applying human psychometric frameworks to generative agents.

Related

  1. Flavorists Are Recreating Smoke to Save Europe’s Snacks
    Matt Blois
    ACS Central SciencePeer-reviewedCC BY 4.0Full textPhysiologyCrossref
  2. Fear without a dose-response: real-time crowdsourced psychometrics show a moderate earthquake’s psychological footprint is decoupled from shaking intensity in an anticipation-primed megacity
    Yasemin Didem Aktas, Remy Bossu, Simon Issartel +1
    EarthArXivPreprintCC BY 4.0Full textPsychiatry & mental healthEarthArXiv
  3. Adolescent-Reported National Norms, Psychometrics, and Clinical Considerations for the Nationwide Quality of Life Scale (NQLS)
    E. A. Nienaber, Y. A. Liu, E. A. Youngstrom +1
    medRxivPreprintCC BY-NCFull textPsychiatry & mental healthmedRxiv
  4. Canine-Assisted Interventions and Symptoms Associated with Borderline Personality Disorder: A Scoping Review
    Emma Pestell, Richard Gorman
    PsyArXivPreprintAll rights reservedFull textPsychiatry & mental healthOSF
  5. Piloting New Medications for Borderline Personality Disorder: The Use of Artificial Intelligence in Psychopharmacological Research
    Carlo Lazzari, Marco Rabottini
    Psychopharmacology BulletinPeer-reviewedLicence not statedFull textPsychiatry & mental healthPMC

Cite

BibTeXRIS
BibTeX
@article{mercer2026applying,
  title = {{Applying psychometrics to large language model simulated populations: recreating the HEXACO personality inventory experiment with generative agents}},
  author = {Sarah Mercer and Daniel Martin and Phil Swatton},
  journal = {Royal Society Open Science},
  year = {2026},
  month = oct,
  doi = {10.1098/rsos.251802},
  url = {https://royalsocietypublishing.org/rsos/article/13/10/251802/483818/Applying-psychometrics-to-large-language-model},
  note = {CC BY 4.0},
}
RIS
TY  - JOUR
TI  - Applying psychometrics to large language model simulated populations: recreating the HEXACO personality inventory experiment with generative agents
AU  - Sarah Mercer
AU  - Daniel Martin
AU  - Phil Swatton
PY  - 2026
DA  - 2026/10/21
JO  - Royal Society Open Science
DO  - 10.1098/rsos.251802
UR  - https://royalsocietypublishing.org/rsos/article/13/10/251802/483818/Applying-psychometrics-to-large-language-model
AB  - Generative agents powered by large language models (LLMs) demonstrate human-like characteristics through sophisticated natural language interactions. Their ability to assume roles and personalities based on predefined character biographies has positioned them as cost-effective substitutes for human participants in social science research. This paper explores the validity of such persona-based agents1 in representing human populations; we recreate the HEXACO personality inventory experiment by surveying 310 GPT-4 powered agents, conducting factor analysis on their responses, and comparing these results to the original findings presented by Ashton et al. (Ashton et al. 2004 J. Pers. Soc. Psychol. 87, 707–721 (doi:10.1037/0022-3514.87.5.707). Our results found (i) a coherent and reliable personality structure was recoverable from the agents’ responses demonstrating partial alignment to the HEXACO framework, (ii) the derived personality dimensions were consistent and reliable within GPT-4, when coupled with a sufficiently curated population, and (iii) cross-model analysis revealed variability in personality profiling, suggesting model-specific biases and limitations. We discuss the practical considerations and challenges encountered during the experiment. This study contributes to the ongoing discourse on the potential benefits and limitations of using generative agents in social science research and provides useful guidance on designing consistent and representative agent personas to maximize coverage and representation of human personality traits, while highlighting potential limitations of applying human psychometric frameworks to generative agents.
SN  - 2054-5703
ER  -