Development and Internal Validation of an Interpretable Machine Learning Model for Identifying Past-Year Nonsuicidal Self-Injury Among Adolescents With Depression: Retrospective Study.
7 affiliations
- 1School of Nursing, Anqing Medical College, Anqing, Anhui, China.
- 2School of Mental Health, Wenzhou Medical University, Wenzhou, Zhejiang, China.
- 3School of Technology and Data, Yantai Nanshan University, Yantai, Shandong, China.
- 4Wenzhou Seventh People's Hospital, Wenzhou, Zhejiang, China.
- 5School of Nursing, Wenzhou Medical University, Wenzhou, Zhejiang, China.
- 6Zhejiang Provincial Clinical Research Center for Mental Disorders, Kangning Hospital of Wenzhou Medical University, Wenzhou, Zhejiang, China.
- 7Key Research Center of Philosophy and Social Sciences of Zhejiang Province (Institute of Medical Humanities), Wenzhou Medical University, Wenzhou, Zhejiang, China.
Abstract
Background
Nonsuicidal self-injury (NSSI) is common among adolescents with depressive disorders and is associated with substantial clinical burden. Machine learning may assist in identifying multidimensional patterns associated with NSSI, although its incremental value beyond direct clinical assessment remains uncertain.Objective
This study compares 7 supervised machine learning model variants for classifying past-year NSSI status among adolescents with depressive disorders and conducts detailed analyses of interpretability, sensitivity, subgroup performance, and clinical utility using a random forest model as the focal exploratory model.Methods
We retrospectively identified 437 adolescents aged 10-19 years who received inpatient or outpatient psychiatric care at 4 hospitals in Zhejiang Province, China, between January 2021 and December 2023. A total of 410 eligible participants were included, of whom 274 reported NSSI during the preceding year. The primary analysis included 67 assessment-time sociodemographic, physiological, biochemical, and clinical-behavioral features. The sample was divided into a training set (287/410, 70%) and an untouched held-out test set (123/410, 30%) using stratified random sampling. Hyperparameters were optimized using 5-fold cross-validation within the training set. Performance was evaluated using discrimination, classification metrics, calibration, and decision-curve analysis. Random forest model behavior was examined using Gini importance, permutation importance, and Shapley Additive Explanations (SHAP).Results
The radial basis function support vector machine achieved the highest mean cross-validated area under the receiver operating characteristic curve (AUROC) in the training set (0.720), whereas the random forest showed the highest observed AUROC and area under the precision-recall curve (AUPRC) and the lowest Brier score in the held-out test set. The random forest achieved an AUROC of 0.681 (95% CI 0.583-0.778), an AUPRC of 0.808 (95% CI 0.741-0.874), an accuracy of 0.667 (95% CI 0.585-0.748), a sensitivity of 0.720 (95% CI 0.622-0.817), a specificity of 0.561 (95% CI 0.415-0.707), and a Brier score of 0.212 (95% CI 0.188-0.236). Its calibration slope was 1.013. Pairwise AUROC differences between the random forest and alternative models were not statistically significant after Holm correction (Holm-adjusted P values ranged from .17 to .81). The full random forest model did not outperform a suicidal-ideation-only model (AUROC 0.701), whereas excluding suicidal ideation and previous suicide attempt reduced the AUROC to 0.540. A biochemical-only model showed near-chance discrimination (AUROC 0.513). Decision-curve analysis indicated that the random forest provided greater net benefit than the assess-all and assess-none strategies across threshold probabilities of approximately 0.51-0.80. Suicidal ideation showed the most consistent importance across Gini, SHAP, and permutation analyses.Conclusions
The random forest had the highest observed AUROC and AUPRC and the lowest Brier score in the held-out test set among the evaluated algorithms, although the pairwise AUROC differences were not statistically significant after correction for multiple comparisons. Suicidality-related information contributed substantially to model discrimination, whereas routinely collected biochemical variables did not demonstrate clear incremental value. Because this was a retrospective cross-sectional classification study, the findings do not establish prospective risk prediction. External and prospective temporal validation is required before clinical implementation.
Related
- Eye-movement-based machine learning for identifying Parkinson’s disease dementia in a retrospective cohort: an interpretable model-development study
- Lifetime Prevalence of Nonsuicidal Self-Injury in Adolescents: Reliability of a Dichotomous Survey Item.
- Development and validation of an interpretable machine learning model for predicting early acute kidney injury and altered consciousness in cardiac surgery patients admitted to the intensive care unit: a retrospective study.
- Interpretable machine learning for identifying poor physical fitness in adults across three high-altitude regions of Tibet: a cross-sectional study
- Radiomics-based interpretable machine learning model for identifying high-risk patients with small bowel obstruction: a multicenter study.
BibTeX
@article{liu2026development,
title = {{Development and Internal Validation of an Interpretable Machine Learning Model for Identifying Past-Year Nonsuicidal Self-Injury Among Adolescents With Depression: Retrospective Study.}},
author = {Gaoyang Liu and Chenxi Yang and Wendan Chen and Ningning Ding and Wenwen Tian and Xinwu Ye and Mengdan Luo and Xinnan Mao and Guohua Zhang and Ke Zheng},
journal = {JMIR pediatrics and parenting},
year = {2026},
month = oct,
doi = {10.2196/95340},
url = {https://europepmc.org/article/MED/42849897},
note = {CC BY},
}
RIS
TY - JOUR
TI - Development and Internal Validation of an Interpretable Machine Learning Model for Identifying Past-Year Nonsuicidal Self-Injury Among Adolescents With Depression: Retrospective Study.
AU - Gaoyang Liu
AU - Chenxi Yang
AU - Wendan Chen
AU - Ningning Ding
AU - Wenwen Tian
AU - Xinwu Ye
AU - Mengdan Luo
AU - Xinnan Mao
AU - Guohua Zhang
AU - Ke Zheng
PY - 2026
DA - 2026/10/08
JO - JMIR pediatrics and parenting
DO - 10.2196/95340
UR - https://europepmc.org/article/MED/42849897
AB - Background Nonsuicidal self-injury (NSSI) is common among adolescents with depressive disorders and is associated with substantial clinical burden. Machine learning may assist in identifying multidimensional patterns associated with NSSI, although its incremental value beyond direct clinical assessment remains uncertain.Objective This study compares 7 supervised machine learning model variants for classifying past-year NSSI status among adolescents with depressive disorders and conducts detailed analyses of interpretability, sensitivity, subgroup performance, and clinical utility using a random forest model as the focal exploratory model.Methods We retrospectively identified 437 adolescents aged 10-19 years who received inpatient or outpatient psychiatric care at 4 hospitals in Zhejiang Province, China, between January 2021 and December 2023. A total of 410 eligible participants were included, of whom 274 reported NSSI during the preceding year. The primary analysis included 67 assessment-time sociodemographic, physiological, biochemical, and clinical-behavioral features. The sample was divided into a training set (287/410, 70%) and an untouched held-out test set (123/410, 30%) using stratified random sampling. Hyperparameters were optimized using 5-fold cross-validation within the training set. Performance was evaluated using discrimination, classification metrics, calibration, and decision-curve analysis. Random forest model behavior was examined using Gini importance, permutation importance, and Shapley Additive Explanations (SHAP).Results The radial basis function support vector machine achieved the highest mean cross-validated area under the receiver operating characteristic curve (AUROC) in the training set (0.720), whereas the random forest showed the highest observed AUROC and area under the precision-recall curve (AUPRC) and the lowest Brier score in the held-out test set. The random forest achieved an AUROC of 0.681 (95% CI 0.583-0.778), an AUPRC of 0.808 (95% CI 0.741-0.874), an accuracy of 0.667 (95% CI 0.585-0.748), a sensitivity of 0.720 (95% CI 0.622-0.817), a specificity of 0.561 (95% CI 0.415-0.707), and a Brier score of 0.212 (95% CI 0.188-0.236). Its calibration slope was 1.013. Pairwise AUROC differences between the random forest and alternative models were not statistically significant after Holm correction (Holm-adjusted P values ranged from .17 to .81). The full random forest model did not outperform a suicidal-ideation-only model (AUROC 0.701), whereas excluding suicidal ideation and previous suicide attempt reduced the AUROC to 0.540. A biochemical-only model showed near-chance discrimination (AUROC 0.513). Decision-curve analysis indicated that the random forest provided greater net benefit than the assess-all and assess-none strategies across threshold probabilities of approximately 0.51-0.80. Suicidal ideation showed the most consistent importance across Gini, SHAP, and permutation analyses.Conclusions The random forest had the highest observed AUROC and AUPRC and the lowest Brier score in the held-out test set among the evaluated algorithms, although the pairwise AUROC differences were not statistically significant after correction for multiple comparisons. Suicidality-related information contributed substantially to model discrimination, whereas routinely collected biochemical variables did not demonstrate clear incremental value. Because this was a retrospective cross-sectional classification study, the findings do not establish prospective risk prediction. External and prospective temporal validation is required before clinical implementation.
SN - 2561-6722
ER -