Temporal and cross-site validation of an AI system for self-harm detection
6 affiliations
- 1Department of Data Science and AI, Monash University, Melbourne, Australia
- 2School of Computing and Information Systems, University of Melbourne, Melbourne, Australia
- 3Centre for Digital Transformation of Health, University of Melbourne, Melbourne, Australia
- 4Centre for Youth Mental Health, The University of Melbourne, Melbourne, Australia
- 5Orygen, Melbourne, Australia
- 6School of Computing Technologies, RMIT University, Melbourne, Australia
Abstract
Adequate self-harm surveillance is a key part of suicide prevention. Our previous research demonstrated that an artificial intelligence (AI)-based system could effectively detect self-harm in emergency department triage notes. However, the system was developed using data from a single hospital, raising concerns about its generalisability. Here, we aim to validate the system prospectively and externally to better understand its portability across hospitals. We leveraged emergency department data from two Australian hospitals, with free-text triage notes manually annotated for self-harm. The AI system was developed using data from a major metropolitan hospital in Melbourne, spanning 2012–2017. The system combined extensive text normalisation with a self-harm classification model utilising 1931 selected features. We prospectively validated the model on 329,655 triage notes from the same hospital over the following four years. For external validation, we used 316,877 triage notes from 2012 to 2021 from a regional hospital 150 km outside Melbourne. On the test set, the model achieved an area under the precision-recall curve (PR AUC) of 0.84 with a 95% confidence interval of [0.82, 0.86]. This performance remained stable at the development site with a PR AUC of 0.84, 95% CI [0.83, 0.85]. When applied in the regional context, the model’s ability to distinguish self-harm cases declined, resulting in an overall PR AUC of 0.78, 95% CI [0.77, 0.79]. While the text normalisation component was equally effective across the datasets, our analysis revealed that in regional settings, self-harm presentations are more likely to involve medication ingestion. At the metropolitan hospital, the AI system for self-harm detection maintained its epidemiological utility. At the regional hospital, the text normalisation process was effective, but performance was unstable primarily due to linguistic domain shift and differences in self-harm presentations. Author summary: We tackle a practical challenge in clinical AI: ensuring text-based systems work well across different healthcare settings. Local documentation practices, semantics, local jargon, and patient demographics greatly affect how well an AI model works in new environments. We built an AI system for self-harm detection in emergency departments using real-world data from a large metropolitan hospital in Australia. We assessed the temporal stability and regional variability of the tool, comparing its performance at the development site and a regional hospital. The results showed the AI system is suitable for public health use at the metropolitan hospital. However, performance was unstable at the regional hospital due to linguistic variation and differences in methods patients use to self-harm.
Related
- Mutation-resolved ABL1 inhibitor resistance prediction under mutation-disjoint and temporal cross-study validation
- Development and validation of a laboratory-developed test on Panther Fusion(R) system for detection of macrolide resistance in Bordetella pertussis
- Machine Learning with Social Determinants of Health for Cross-Sectional Diabetes Classification: Development and Temporal Validation.
- Development and multicenter validation of an AI-driven detection system for DLBCL cells in bone marrow smears
- Performance Assessment of an Artificial Intelligence System Used for Case Detection in a Large Community-based Diabetic Retinopathy Screening Program: A Retrospective Diagnostic Accuracy Validation Study.
BibTeX
@article{rozova2026temporal,
title = {{Temporal and cross-site validation of an AI system for self-harm detection}},
author = {Vlada Rozova and Liuliu Chen and Katrina Witt and Mike Conway and Jo Robinson and Karin Verspoor},
journal = {PLOS Digital Health},
year = {2026},
month = sep,
doi = {10.1371/journal.pdig.0001667},
url = {https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001667},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Temporal and cross-site validation of an AI system for self-harm detection
AU - Vlada Rozova
AU - Liuliu Chen
AU - Katrina Witt
AU - Mike Conway
AU - Jo Robinson
AU - Karin Verspoor
PY - 2026
DA - 2026/09/11
JO - PLOS Digital Health
DO - 10.1371/journal.pdig.0001667
UR - https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001667
AB - Adequate self-harm surveillance is a key part of suicide prevention. Our previous research demonstrated that an artificial intelligence (AI)-based system could effectively detect self-harm in emergency department triage notes. However, the system was developed using data from a single hospital, raising concerns about its generalisability. Here, we aim to validate the system prospectively and externally to better understand its portability across hospitals. We leveraged emergency department data from two Australian hospitals, with free-text triage notes manually annotated for self-harm. The AI system was developed using data from a major metropolitan hospital in Melbourne, spanning 2012–2017. The system combined extensive text normalisation with a self-harm classification model utilising 1931 selected features. We prospectively validated the model on 329,655 triage notes from the same hospital over the following four years. For external validation, we used 316,877 triage notes from 2012 to 2021 from a regional hospital 150 km outside Melbourne. On the test set, the model achieved an area under the precision-recall curve (PR AUC) of 0.84 with a 95% confidence interval of [0.82, 0.86]. This performance remained stable at the development site with a PR AUC of 0.84, 95% CI [0.83, 0.85]. When applied in the regional context, the model’s ability to distinguish self-harm cases declined, resulting in an overall PR AUC of 0.78, 95% CI [0.77, 0.79]. While the text normalisation component was equally effective across the datasets, our analysis revealed that in regional settings, self-harm presentations are more likely to involve medication ingestion. At the metropolitan hospital, the AI system for self-harm detection maintained its epidemiological utility. At the regional hospital, the text normalisation process was effective, but performance was unstable primarily due to linguistic domain shift and differences in self-harm presentations. Author summary: We tackle a practical challenge in clinical AI: ensuring text-based systems work well across different healthcare settings. Local documentation practices, semantics, local jargon, and patient demographics greatly affect how well an AI model works in new environments. We built an AI system for self-harm detection in emergency departments using real-world data from a large metropolitan hospital in Australia. We assessed the temporal stability and regional variability of the tool, comparing its performance at the development site and a regional hospital. The results showed the AI system is suitable for public health use at the metropolitan hospital. However, performance was unstable at the regional hospital due to linguistic variation and differences in methods patients use to self-harm.
SN - 2767-3170
ER -