Echo-Bench: A Chinese Replication and Cross-Lingual Extension of Psychosis-Bench
Abstract
Psychosis-Bench recently showed that production chat models confirm and elaborate users' delusion-like beliefs under multi-turn pressure, but its evidence is English-only, Western-flagship-only, and rests on an uncalibrated LLM judge. We present Echo-Bench, a Chinese replication and cross-lingual extension: the full 16-case, 12-turn protocol mirrored into Chinese under a fidelity-first translation protocol with an independent re-translation audit, run against two widely deployed Chinese chat models (DeepSeek V4.1-Flash, qwen-plus) in both languages — 192 sessions, 4,032 turn-level ratings. The original qualitative finding replicates in both languages (RQ1). All six pre-committed cross-language contrasts are significant after Benjamini–Hochberg control: the Chinese arm is directionally worse on every dimension, with model-differentiated failure channels (RQ2). Because our pipeline required a bilingual rater, we placed the measurement instrument itself under test (RQ3): a five-layer rater trust chain — entrance examination against human gold, extreme-score arbitration, red-line audit of the arbiter, blind human adjudication, and a spot check of the non-extreme stratum — shows that automated raters fail concentratedly in the extreme band, with two named failure modes (disclaimer laundering, eloquence inflation), and that a strong general-purpose arbiter is disqualified by pre-specified red lines. A negative control bounds rater language sensitivity below what the observed gaps require. Safety behavior certified in English does not certify safety behavior in Chinese; the instruments used to check either must themselves be checked.
Related
- Circulating miR-1260a as a potential biomarker for First-Episode of Psychosis: a replication study in a treatment-Naïve Romanian cohort
- Massive Warfarin-Induced Lingual Hematoma: A Case Report
- Lingual Varix: Case Series
- Identification and operative vs. conservative vs. no treatment trial of breastfeeding-impeding lingual frenulum in newborn infants (LINNE): protocol for a randomized controlled trial (RCT) embedded in a prospective observational cohort study in Northern Finland
- A mini-review of medical question answering for Bangla: the methodological landscape and cross-lingual perspectives.
BibTeX
@misc{zhao2026echobench,
title = {{Echo-Bench: A Chinese Replication and Cross-Lingual Extension of Psychosis-Bench}},
author = {Yucheng Zhao},
howpublished = {PsyArXiv},
year = {2026},
month = oct,
doi = {10.31234/osf.io/thvpz_v1},
url = {https://osf.io/preprints/psyarxiv/thvpz_v1/},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Echo-Bench: A Chinese Replication and Cross-Lingual Extension of Psychosis-Bench
AU - Yucheng Zhao
PY - 2026
DA - 2026/10/21
PB - PsyArXiv
DO - 10.31234/osf.io/thvpz_v1
UR - https://osf.io/preprints/psyarxiv/thvpz_v1/
AB - Psychosis-Bench recently showed that production chat models confirm and elaborate users' delusion-like beliefs under multi-turn pressure, but its evidence is English-only, Western-flagship-only, and rests on an uncalibrated LLM judge. We present Echo-Bench, a Chinese replication and cross-lingual extension: the full 16-case, 12-turn protocol mirrored into Chinese under a fidelity-first translation protocol with an independent re-translation audit, run against two widely deployed Chinese chat models (DeepSeek V4.1-Flash, qwen-plus) in both languages — 192 sessions, 4,032 turn-level ratings. The original qualitative finding replicates in both languages (RQ1). All six pre-committed cross-language contrasts are significant after Benjamini–Hochberg control: the Chinese arm is directionally worse on every dimension, with model-differentiated failure channels (RQ2). Because our pipeline required a bilingual rater, we placed the measurement instrument itself under test (RQ3): a five-layer rater trust chain — entrance examination against human gold, extreme-score arbitration, red-line audit of the arbiter, blind human adjudication, and a spot check of the non-extreme stratum — shows that automated raters fail concentratedly in the extreme band, with two named failure modes (disclaimer laundering, eloquence inflation), and that a strong general-purpose arbiter is disqualified by pre-specified red lines. A negative control bounds rater language sensitivity below what the observed gaps require. Safety behavior certified in English does not certify safety behavior in Chinese; the instruments used to check either must themselves be checked.
ER -