SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies
9 affiliations
- 1School of Educational Sciences, Huangshan University
- 2Faculty of Health Sciences, Universiti Teknologi MARA
- 3School of Economics and Management, Huangshan University
- 4College of Education, Zhejiang Normal University
- 5School of Computer Science and Technology, Zhejiang Normal University
- 6Security Department, Zhejiang Normal University
- 7School of Psychology, Zhejiang Normal University
- 8College of Mathematical Medicine, Zhejiang Normal University
- 9AIFORIA (Hangzhou) Intelligent Technology Co., Ltd
Abstract
Introduction: Large language models (LLMs) can render structured questionnaire variables as natural-language profiles, but their value in mental-health screening cannot be judged from a single fixed-threshold score. We evaluated LLM prompt conditions in a cross-questionnaire benchmark for predicting screen-positive status on the Self-Rating Depression Scale (SDS) among undergraduates. The intended application was an offline methodological benchmark using previously collected questionnaire data, not a shorter alternative to direct SDS administration.
Methods: De-identified records were assigned to Development (n=1,951), Test (n=2,186), and Holdout (n=2,258) grade cohorts. We evaluated logistic regression, XGBoost, and 12 model-prompt conditions at the fixed 0.5 threshold and at thresholds selected only in Development-validation. We also examined paired bootstrap differences, ranking stability, scale correlations, grouped ablations, cohort shift, calibration intercepts and slopes, alternative expected calibration error specifications, representative repeated calls at temperatures 0 and 0.1, sex-stratified performance, and per-1,000 screening scenarios.
Results: At the fixed 0.5 threshold on Test, GPT E1, Gemini E2, and XGBoost achieved macro-F1 values of 0.732, 0.730, and 0.674. At Development-derived thresholds of 0.38, 0.50, and 0.174, the corresponding macro-F1 values were 0.702, 0.730, and 0.660. At the Development-selected specificity target, XGBoost achieved a Test macro-F1 of 0.738. Pairwise intervals showed no clear fixed-0.5 macro-F1 difference among the three highest-ranked LLM conditions. XGBoost yielded the lowest Brier scores. The combined-sample Spearman correlation between SDS and SAS was 0.60. Together with the ablation results, this correlation indicated predictive information distributed across overlapping questionnaire groups rather than dependence on a single measured group. In repeated calls across the complete Holdout cohort, inconsistent-case rates were 23.6% for GPT E1 and 1.5% for Gemini E2. Temperature 0 did not clearly eliminate GPT E1 instability in the representative 400-case subset.
Discussion: Model conclusions depended on both the threshold and the metric. Certain LLM prompt conditions were competitive at specific operating points, but the evidence did not establish a stable or general advantage over structured-data models. Nor did it establish external validity, clinical diagnostic utility, or reduced questionnaire burden. Threshold choice, calibration, output validity, and repeated-call stability therefore require separate evaluation before prospective use.
Related
- Comparative validity and reproducibility of questionnaires in assessing types, frequency, and quantity of sweetened beverage consumption among Mahidol University students
- Intelligent-assistant support versus usual self-scheduling for extracurricular physical activity among university students: a comparative evaluation.
- Proactive Prediction of Alarms in Hemodialysis Machines: A Comparative Evaluation of AI Models for Clinical Prognostic Support
- Inter-rater reliability during live pediatric formative Objective Structured Clinical Examination (OSCE) in a French University
- Stage-structured, distributional prediction of IVF outcomes with conditional updating
BibTeX
@article{yang2026screenpositive,
title = {{SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies}},
author = {Xiao Yang and Ying Cao and Haibin Wang and Huan Li and Xin Jiang and Huiying Xu and Tao Yao and Xiaolei Zhang and Yekai Chen and Jiaming Li and Bin Zhou and Lingwei Xu and Zhendong Chen and Xinzhong Zhu},
journal = {Frontiers in Psychiatry},
year = {2026},
month = oct,
doi = {10.3389/fpsyt.2026.1942164},
url = {https://www.frontiersin.org/articles/10.3389/fpsyt.2026.1942164/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - SDS screen-positive prediction from structured questionnaires in university students: a comparative evaluation of large language model prompting strategies
AU - Xiao Yang
AU - Ying Cao
AU - Haibin Wang
AU - Huan Li
AU - Xin Jiang
AU - Huiying Xu
AU - Tao Yao
AU - Xiaolei Zhang
AU - Yekai Chen
AU - Jiaming Li
AU - Bin Zhou
AU - Lingwei Xu
AU - Zhendong Chen
AU - Xinzhong Zhu
PY - 2026
DA - 2026/10/19
JO - Frontiers in Psychiatry
DO - 10.3389/fpsyt.2026.1942164
UR - https://www.frontiersin.org/articles/10.3389/fpsyt.2026.1942164/full
AB - Introduction: Large language models (LLMs) can render structured questionnaire variables as natural-language profiles, but their value in mental-health screening cannot be judged from a single fixed-threshold score. We evaluated LLM prompt conditions in a cross-questionnaire benchmark for predicting screen-positive status on the Self-Rating Depression Scale (SDS) among undergraduates. The intended application was an offline methodological benchmark using previously collected questionnaire data, not a shorter alternative to direct SDS administration. Methods: De-identified records were assigned to Development (n=1,951), Test (n=2,186), and Holdout (n=2,258) grade cohorts. We evaluated logistic regression, XGBoost, and 12 model-prompt conditions at the fixed 0.5 threshold and at thresholds selected only in Development-validation. We also examined paired bootstrap differences, ranking stability, scale correlations, grouped ablations, cohort shift, calibration intercepts and slopes, alternative expected calibration error specifications, representative repeated calls at temperatures 0 and 0.1, sex-stratified performance, and per-1,000 screening scenarios. Results: At the fixed 0.5 threshold on Test, GPT E1, Gemini E2, and XGBoost achieved macro-F1 values of 0.732, 0.730, and 0.674. At Development-derived thresholds of 0.38, 0.50, and 0.174, the corresponding macro-F1 values were 0.702, 0.730, and 0.660. At the Development-selected specificity target, XGBoost achieved a Test macro-F1 of 0.738. Pairwise intervals showed no clear fixed-0.5 macro-F1 difference among the three highest-ranked LLM conditions. XGBoost yielded the lowest Brier scores. The combined-sample Spearman correlation between SDS and SAS was 0.60. Together with the ablation results, this correlation indicated predictive information distributed across overlapping questionnaire groups rather than dependence on a single measured group. In repeated calls across the complete Holdout cohort, inconsistent-case rates were 23.6% for GPT E1 and 1.5% for Gemini E2. Temperature 0 did not clearly eliminate GPT E1 instability in the representative 400-case subset. Discussion: Model conclusions depended on both the threshold and the metric. Certain LLM prompt conditions were competitive at specific operating points, but the evidence did not establish a stable or general advantage over structured-data models. Nor did it establish external validity, clinical diagnostic utility, or reduced questionnaire burden. Threshold choice, calibration, output validity, and repeated-call stability therefore require separate evaluation before prospective use.
SN - 1664-0640
ER -