A multi-agent large language model framework for structured clinical interviewing and psychiatric screening: a proof-of-concept
10 affiliations
- 1AI for Mental Health Program, St. Michael’s Hospital, Unity Health Toronto, Toronto, ON Canada
- 2Temerty Centre for Artificial Intelligence Research and Education in Medicine, University of Toronto, Toronto, ON Canada
- 3Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, ON Canada
- 4Adult Neurodevelopment and Geriatric Psychiatry Division, Centre for Addiction and Mental Health, Toronto, ON Canada
- 5Department of Psychology, Toronto Metropolitan University, Toronto, ON Canada
- 6Krembil Centre for Neuroinformatics & BMO Chair in Artificial Intelligence and Mental Health, Centre for Addiction and Mental Health, Toronto, ON Canada
- 7Department of Computer Science, University of Toronto, Toronto, ON Canada
- 8Department of Electrical, Computer, and Biomedical Engineering, Toronto Metropolitan University, Toronto, ON Canada
- 9Department of Mathematics and Statistics, York University, Toronto, ON Canada
- 10Department of Psychiatry, University of Toronto, Toronto, ON Canada
Abstract
Structured psychiatric interviews improve diagnostic reliability but are resource-intensive to administer. We developed a multi-agent large language model (LLM) framework to support protocolized structured interviewing and module-level psychiatric screening. The system orchestrates four agents, Questioner, Evaluator, Navigator, and Diagnoser, to administer interview modules, interpret free-text responses, enforce bounded clarification loops, identify safety-related red flags and trigger escalation, and apply encoded decision rules to produce screening outcomes. Prior to any model processing, responses undergo automated redaction of personally identifiable and protected health information. We conducted a simulation-based technical evaluation using 1350 interviews across 15 neuropsychiatric screening modules (e.g., anxiety, depression, and bipolar disorders) with systematic variation in symptom direction, disclosure depth, and emotional tone. To test robustness to language variability, we additionally evaluated reconstructed conversation bundles derived from the deidentified MentalChat16K dataset. In the primary evaluation, module-level decisions showed 87.8% concordance with rule-based expected outcomes (95% CI: 85.9–89.6), with sensitivity 88.9% (95% CI: 86.5–91.2) and specificity 86.7% (95% CI: 84.0–89.2) relative to predefined item-level endorsements. Post hoc comparator analyses showed higher F2 for the multi-agent framework (88.5%) than single-pass LLM (49.5%) or deterministic rule-based (61.9%) transcript-scoring baselines. On a 2000-sentence red flag benchmark stratified by explicit/implicit and active/passive ideation, the classifier achieved 93.9% accuracy, 94.2% sensitivity, and 93.6% specificity. In pilot feedback from mental health clinicians, descriptive ratings suggested favorable perceptions of clarity, workflow fit, and perceived safety for clinician-supervised use. These results support proof-of-concept technical feasibility for a protocol-adherent, auditable agentic LLM framework for structured psychiatric screening.
Related
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- Effectiveness of a Sequential Simulation and Online Learning Intervention on Psychiatric Mental Health Nurses’ Motivational Interviewing Knowledge, Confidence, and Skill Performance for Medication Adherence: A Cluster-randomized Controlled Trial
- A Risk-Formulation Framework and Candidate Screening Items for Dangerous Use of Large Language Models in Clinical Psychiatry
- Citation Fabrication and Accuracy in AI-Generated Psychiatric Research: A Comparative Evaluation of Four Contemporary Large Language Models
- Predicting One-Year Psychiatric Rehospitalization Using Natural Language Processing and Machine Learning Methods Based on Clinical Texts
BibTeX
@article{kamaleddin2026multiagent,
title = {{A multi-agent large language model framework for structured clinical interviewing and psychiatric screening: a proof-of-concept}},
author = {Mohammad Amin Kamaleddin and Mina Mirjalili and Reza Barzegar and Nghia Trung Le and Areez Visram and Fatemeh Gholami Nezhad and Alice Rueda and Bazen Gashaw Teferra and Candice Monson and Tristan Glatard and Sri Krishnan and Divya Sharma and Benoit Mulsant and Venkat Bhat},
journal = {Translational Psychiatry},
year = {2026},
month = jul,
doi = {10.1038/s41398-026-04335-5},
url = {https://doi.org/10.1038/s41398-026-04335-5},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - A multi-agent large language model framework for structured clinical interviewing and psychiatric screening: a proof-of-concept
AU - Mohammad Amin Kamaleddin
AU - Mina Mirjalili
AU - Reza Barzegar
AU - Nghia Trung Le
AU - Areez Visram
AU - Fatemeh Gholami Nezhad
AU - Alice Rueda
AU - Bazen Gashaw Teferra
AU - Candice Monson
AU - Tristan Glatard
AU - Sri Krishnan
AU - Divya Sharma
AU - Benoit Mulsant
AU - Venkat Bhat
PY - 2026
DA - 2026/07/31
JO - Translational Psychiatry
DO - 10.1038/s41398-026-04335-5
UR - https://doi.org/10.1038/s41398-026-04335-5
AB - Structured psychiatric interviews improve diagnostic reliability but are resource-intensive to administer. We developed a multi-agent large language model (LLM) framework to support protocolized structured interviewing and module-level psychiatric screening. The system orchestrates four agents, Questioner, Evaluator, Navigator, and Diagnoser, to administer interview modules, interpret free-text responses, enforce bounded clarification loops, identify safety-related red flags and trigger escalation, and apply encoded decision rules to produce screening outcomes. Prior to any model processing, responses undergo automated redaction of personally identifiable and protected health information. We conducted a simulation-based technical evaluation using 1350 interviews across 15 neuropsychiatric screening modules (e.g., anxiety, depression, and bipolar disorders) with systematic variation in symptom direction, disclosure depth, and emotional tone. To test robustness to language variability, we additionally evaluated reconstructed conversation bundles derived from the deidentified MentalChat16K dataset. In the primary evaluation, module-level decisions showed 87.8% concordance with rule-based expected outcomes (95% CI: 85.9–89.6), with sensitivity 88.9% (95% CI: 86.5–91.2) and specificity 86.7% (95% CI: 84.0–89.2) relative to predefined item-level endorsements. Post hoc comparator analyses showed higher F2 for the multi-agent framework (88.5%) than single-pass LLM (49.5%) or deterministic rule-based (61.9%) transcript-scoring baselines. On a 2000-sentence red flag benchmark stratified by explicit/implicit and active/passive ideation, the classifier achieved 93.9% accuracy, 94.2% sensitivity, and 93.6% specificity. In pilot feedback from mental health clinicians, descriptive ratings suggested favorable perceptions of clarity, workflow fit, and perceived safety for clinician-supervised use. These results support proof-of-concept technical feasibility for a protocol-adherent, auditable agentic LLM framework for structured psychiatric screening.
SN - 2158-3188
ER -