AI-generated language remains systematically distinguishable from human communication.
5 affiliations
- 1Department of Informatics, University of Zurich, Zurich CH-8006, Switzerland.
- 2Institute for Logic, Language and Computation, University of Amsterdam, Amsterdam 1098-xg, The Netherlands.
- 3Department of Sociology, Computer Science, Political Science, and Public Policy, Duke University, Durham, NC 27708.
- 4Department of Sociology, New York University, New York, NY 10003.
- 5Department of Sociology, University of Oxford, Oxford OX1 1JD, United Kingdom.
Abstract
Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior and communication, yet validation methods for these applications remain limited. Existing evaluations often rely on human judgments of plausibility, a coarse, difficult-to-scale, and often unreliable benchmark for distinguishing AI-generated from human text. As a result, researchers lack robust tools for assessing the realism of LLM-generated language and for calibrating models to real-world data. We address this gap in two ways. First, we introduce the Conversational Human Authenticity Test (CHAT), a validation framework that combines aggregate metrics of detectability and semantic similarity with interpretable linguistic features, including stylistic markers and topical patterns, to assess how closely LLMs approximate human language in a given dataset. Second, we benchmark nine open-weight LLMs across five calibration strategies, including fine-tuning, stylistic prompting, and context retrieval, on user interactions from X, Bluesky, and Reddit. We find that even after calibration, LLM outputs remain clearly distinguishable from human text, especially in affective tone and emotional expression. Instruction-tuned models generally underperform base models, and larger models do not obviously improve human-likeness. We also identify a trade-off between realism and semantic fidelity: methods that make outputs seem more human often move them further from the content of human responses. Together, these results provide a scalable framework for validation and calibration in LLM-based social simulation while underscoring current limits in reproducing human communication.
Related
- RNAactDrug 2.0: an updated resource for systematically exploring molecular features associated with drug sensitivity across multi-omics datasets
- The intrinsically disordered C-terminal domain of the measles virus nucleoprotein interacts with the C-terminal domain of the phosphoprotein via two distinct sites and remains predominantly unfolded.
- Experimental benchmarking remains crucial to the interpretation of sequence annotation data
- Electrohydraulic fields generated by active transport at tissue interfaces
- 3D MERFISH reveals cell-type diversity generated by mRNA partitioning in spiralian embryogenesis
BibTeX
@article{pagan2026aigenerated,
title = {{AI-generated language remains systematically distinguishable from human communication.}},
author = {Nicolò Pagan and Petter Törnberg and Christopher A Bail and Christopher Barrie},
journal = {Proceedings of the National Academy of Sciences of the United States of America},
year = {2026},
month = oct,
doi = {10.1073/pnas.2531748123},
url = {https://europepmc.org/article/MED/42863061},
note = {CC BY},
}
RIS
TY - JOUR
TI - AI-generated language remains systematically distinguishable from human communication.
AU - Nicolò Pagan
AU - Petter Törnberg
AU - Christopher A Bail
AU - Christopher Barrie
PY - 2026
DA - 2026/10/12
JO - Proceedings of the National Academy of Sciences of the United States of America
DO - 10.1073/pnas.2531748123
UR - https://europepmc.org/article/MED/42863061
AB - Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior and communication, yet validation methods for these applications remain limited. Existing evaluations often rely on human judgments of plausibility, a coarse, difficult-to-scale, and often unreliable benchmark for distinguishing AI-generated from human text. As a result, researchers lack robust tools for assessing the realism of LLM-generated language and for calibrating models to real-world data. We address this gap in two ways. First, we introduce the Conversational Human Authenticity Test (CHAT), a validation framework that combines aggregate metrics of detectability and semantic similarity with interpretable linguistic features, including stylistic markers and topical patterns, to assess how closely LLMs approximate human language in a given dataset. Second, we benchmark nine open-weight LLMs across five calibration strategies, including fine-tuning, stylistic prompting, and context retrieval, on user interactions from X, Bluesky, and Reddit. We find that even after calibration, LLM outputs remain clearly distinguishable from human text, especially in affective tone and emotional expression. Instruction-tuned models generally underperform base models, and larger models do not obviously improve human-likeness. We also identify a trade-off between realism and semantic fidelity: methods that make outputs seem more human often move them further from the content of human responses. Together, these results provide a scalable framework for validation and calibration in LLM-based social simulation while underscoring current limits in reproducing human communication.
SN - 0027-8424
SN - 1091-6490
ER -