A Specialized Model versus a Frontier Model in Emotional Support: Blind Evaluation by Psychology Professionals
Abstract
General-purpose conversational assistants are also used for emotional support and mental health concerns, even though they were not designed as clinical interventions. In this work we evaluate whether a model specialized in emotional support is preferred, by psychology professionals and under blind conditions, over a frontier model. We compare Pía, the conversational system of the TerapIA application (https://terapiaconpia.com), with the most recent version of OpenAI's ChatGPT — a general-purpose, large-scale frontier model, in the versions GPT-5.5 Instant and GPT-5.6 Sol depending on the period, both with the health and mental-health improvements publicly announced by OpenAI. Both systems receive the same system message, an extensive prompt designed by clinical psychologists for this domain, and the same conversational context, and both are run without explicit reasoning (reasoning disabled). The analysis is carried out on irreversibly anonymized data. Over n=771 blind comparisons corresponding to 101 real conversations of 100 distinct users, evaluated between 19 June and 7 August 2026, Pía was preferred in 52.7% of cases against 39.7% for the reference system (7.7% ties), which amounts to a tie-adjusted preference rate of 56.5% (95% CI bootstrap by conversation: [51.0%, 61.9%]; p=0.017). The observed advantage does not appear to be explained by a preference for longer responses: Pía produces responses less than half as long as the reference system. Nor by greater scale or general capability, since the reference system surpasses Pía in both. An exploratory analysis by position of the turn within the conversation shows preference indistinguishable from parity in the first third (49.8%) and clearly above it in the middle third (60.6%) and the final third (58.4%), with a significant positive trend (p=0.022); his pattern is compatible with differences between systems becoming more visible when more conversational context is available, without the design being able to identify the responsible mechanism. A second exploratory analysis by risk level A second exploratory analysis by risk level of the conversation shows a descriptive upward pattern that does not reach statistical significance.
Related
- Evaluation of the psychology of earthquake victims referred to plastic surgery service after crush injury: a cross-sectional study
- Professionals' perspectives on barriers to improving psychosocial care for adolescents with emotional and behavioural problems - a concept mapping case study.
- Emotional meanings of clinical practice with head and neck cancer patients: a qualitative study with Brazilian professionals.
- Managing Cardiac Amyloidosis in a Rural Cardiology Practice: Bridging Gaps in Specialized Evaluation and Disease-Modifying Treatment
- Automation of literature screening for meta-analyses in the field of psychology: an evaluation of two text-mining approaches.
BibTeX
@misc{serrano2026specialized,
title = {{A Specialized Model versus a Frontier Model in Emotional Support: Blind Evaluation by Psychology Professionals}},
author = {Alejandro Vaca Serrano and Alberto Vaca Escribano and José María González de Echávarri},
howpublished = {PsyArXiv},
year = {2026},
month = sep,
doi = {10.31234/osf.io/mb3p7_v1},
url = {https://osf.io/preprints/psyarxiv/mb3p7_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - A Specialized Model versus a Frontier Model in Emotional Support: Blind Evaluation by Psychology Professionals
AU - Alejandro Vaca Serrano
AU - Alberto Vaca Escribano
AU - José María González de Echávarri
PY - 2026
DA - 2026/09/03
PB - PsyArXiv
DO - 10.31234/osf.io/mb3p7_v1
UR - https://osf.io/preprints/psyarxiv/mb3p7_v1/
AB - General-purpose conversational assistants are also used for emotional support and mental health concerns, even though they were not designed as clinical interventions. In this work we evaluate whether a model specialized in emotional support is preferred, by psychology professionals and under blind conditions, over a frontier model. We compare Pía, the conversational system of the TerapIA application (https://terapiaconpia.com), with the most recent version of OpenAI's ChatGPT — a general-purpose, large-scale frontier model, in the versions GPT-5.5 Instant and GPT-5.6 Sol depending on the period, both with the health and mental-health improvements publicly announced by OpenAI. Both systems receive the same system message, an extensive prompt designed by clinical psychologists for this domain, and the same conversational context, and both are run without explicit reasoning (reasoning disabled). The analysis is carried out on irreversibly anonymized data. Over n=771 blind comparisons corresponding to 101 real conversations of 100 distinct users, evaluated between 19 June and 7 August 2026, Pía was preferred in 52.7% of cases against 39.7% for the reference system (7.7% ties), which amounts to a tie-adjusted preference rate of 56.5% (95% CI bootstrap by conversation: [51.0%, 61.9%]; p=0.017). The observed advantage does not appear to be explained by a preference for longer responses: Pía produces responses less than half as long as the reference system. Nor by greater scale or general capability, since the reference system surpasses Pía in both. An exploratory analysis by position of the turn within the conversation shows preference indistinguishable from parity in the first third (49.8%) and clearly above it in the middle third (60.6%) and the final third (58.4%), with a significant positive trend (p=0.022); his pattern is compatible with differences between systems becoming more visible when more conversational context is available, without the design being able to identify the responsible mechanism. A second exploratory analysis by risk level A second exploratory analysis by risk level of the conversation shows a descriptive upward pattern that does not reach statistical significance.
ER -