The collective turing test: large language models can generate realistic multi-user discussions
4 affiliations
- 1Department of Informatics, University of Zurich
- 2Department of Politics and Public Administration, University of Konstanz
- 3Complexity Science Hub
- 4Department of Innovative Technologies, University of Applied Sciences and Arts of Southern Switzerland
Abstract
Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.
Related
- Leveraging large language models to generate multiple-choice questions for epilepsy education
- A Spectrum of Care: The Failings of Binary Code Status Discussions
- Combined Use of Topic-Analysis and LLMs on Online Peer-to-Peer Medicaid Discussions to Detect the Impact of HR1 on Individuals Medicaid Experience Across The US
- EIN-RECAP: Evidence-Grounded Multi-Agent Summarization and Retrieval of Infectious-Disease Expert Discussions
- PCOS discussions on a Turkish online forum: a qualitative netnographic study of concerns, peer advice, and information gaps
BibTeX
@article{bouleimen2026collective,
title = {{The collective turing test: large language models can generate realistic multi-user discussions}},
author = {Azza Bouleimen and Giordano De Marzo and Taehee Kim and Nicolò Pagan and Hannah Metzler and Silvia Giordano and Anikó Hannák and David Garcia},
journal = {Scientific Reports},
year = {2026},
month = jul,
doi = {10.1038/s41598-026-62286-9},
url = {https://doi.org/10.1038/s41598-026-62286-9},
note = {CC BY},
}
RIS
TY - JOUR
TI - The collective turing test: large language models can generate realistic multi-user discussions
AU - Azza Bouleimen
AU - Giordano De Marzo
AU - Taehee Kim
AU - Nicolò Pagan
AU - Hannah Metzler
AU - Silvia Giordano
AU - Anikó Hannák
AU - David Garcia
PY - 2026
DA - 2026/07/01
JO - Scientific Reports
DO - 10.1038/s41598-026-62286-9
UR - https://doi.org/10.1038/s41598-026-62286-9
AB - Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.
SN - 2045-2322
ER -