Personality Inference From Semistructured Interviews Using Large Language Models: Comparative Evaluation Study
Abstract
BackgroundPersonality traits are central to understanding human behavior and life outcomes, yet their assessment remains challenging due to biases and limited expressiveness in traditional self-report inventories. The emergence of large language models (LLMs) creates new opportunities to infer personality from naturalistic text, but it remains unclear whether these systems can reliably predict validated personality traits or whether their predictions primarily reflect superficial linguistic patterns. ObjectiveThis study evaluates the ability of LLM-based methods to predict continuous Big Five personality scores (1‐5 Likert scale) from semistructured interview transcripts and compares the performance of different modeling paradigms. MethodsWe analyzed semistructured interview transcripts from 518 participants, each paired with validated self-reported Big Five scores measured using the 10-item Big Five Inventory (BFI-10). We compared four paradigms: (1) zero-shot and chain-of-thought prompting with GPT-5 mini; (2) parameter-efficient fine-tuning (low-rank adaptation) applied to open-source RoBERTa-base and LLaMA-2-7B models; (3) supervised fine-tuning of GPT-4.1 mini across two independent training runs; and (4) embedding-based regression using 4 pretrained encoders (all-mpnet-base-v2, E5-large-v2, text-embedding-3-small, and text-embedding-3-large) with a multioutput ridge regression. ResultsSupervised fine-tuning yielded the strongest overall performance among the evaluated approaches. In the best-performing run, correlations reached rr ConclusionsLLM-based approaches exhibit trait-dependent performance in predicting personality from naturalistic dialogue, with fine-tuning substantially improving correspondence with validated psychometric scores. These findings suggest that personality inference from naturalistic dialogue is feasible but depends strongly on modeling strategy, trait domain, and the availability of supervised fine-tuning with validated personality measures.
Related
- Sentiment Analysis of Clinical Interviews Identifies Language Markers of Identity Diffusion in Adolescents with Personality Disorder
- Normative and Maladaptive Personality Traits in Borderline Personality Disorder: A Comparative and Hierarchical Analysis of HEXACO and PID-5
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Quality, Readability, Understandability, and Actionability of AI Chatbots’ Responses to Dental Patient Questions: A Comparative Evaluation
- CPT/HCPCS Code Recommendation from Clinical Notes: A Comparative Evaluation of AI Methods
BibTeX
@article{zhu2026personality,
title = {{Personality Inference From Semistructured Interviews Using Large Language Models: Comparative Evaluation Study}},
author = {Jianfeng Zhu and Julina Maharjan and Xinyu Li and Karin G Coifman and Ruoming Jin},
journal = {Journal of Medical Internet Research},
year = {2026},
month = oct,
doi = {10.2196/87129},
url = {https://doi.org/10.2196/87129},
note = {CC BY},
}
RIS
TY - JOUR
TI - Personality Inference From Semistructured Interviews Using Large Language Models: Comparative Evaluation Study
AU - Jianfeng Zhu
AU - Julina Maharjan
AU - Xinyu Li
AU - Karin G Coifman
AU - Ruoming Jin
PY - 2026
DA - 2026/10/23
JO - Journal of Medical Internet Research
DO - 10.2196/87129
UR - https://doi.org/10.2196/87129
AB - BackgroundPersonality traits are central to understanding human behavior and life outcomes, yet their assessment remains challenging due to biases and limited expressiveness in traditional self-report inventories. The emergence of large language models (LLMs) creates new opportunities to infer personality from naturalistic text, but it remains unclear whether these systems can reliably predict validated personality traits or whether their predictions primarily reflect superficial linguistic patterns. ObjectiveThis study evaluates the ability of LLM-based methods to predict continuous Big Five personality scores (1‐5 Likert scale) from semistructured interview transcripts and compares the performance of different modeling paradigms. MethodsWe analyzed semistructured interview transcripts from 518 participants, each paired with validated self-reported Big Five scores measured using the 10-item Big Five Inventory (BFI-10). We compared four paradigms: (1) zero-shot and chain-of-thought prompting with GPT-5 mini; (2) parameter-efficient fine-tuning (low-rank adaptation) applied to open-source RoBERTa-base and LLaMA-2-7B models; (3) supervised fine-tuning of GPT-4.1 mini across two independent training runs; and (4) embedding-based regression using 4 pretrained encoders (all-mpnet-base-v2, E5-large-v2, text-embedding-3-small, and text-embedding-3-large) with a multioutput ridge regression. ResultsSupervised fine-tuning yielded the strongest overall performance among the evaluated approaches. In the best-performing run, correlations reached rr ConclusionsLLM-based approaches exhibit trait-dependent performance in predicting personality from naturalistic dialogue, with fine-tuning substantially improving correspondence with validated psychometric scores. These findings suggest that personality inference from naturalistic dialogue is feasible but depends strongly on modeling strategy, trait domain, and the availability of supervised fine-tuning with validated personality measures.
SN - 1438-8871
ER -