Automatic extraction of multiple contributing factors in medication-related incidents based on the P-mSHELL model: development and evaluation of a BERT-based multilabel classifier
4 affiliations
- 1Keio University Faculty of Pharmacy, Minato-ku, Tokyo, Japan
- 2Institute of Library, Information and Media Science, University of Tsukuba, Tsukuba, Ibaraki, Japan
- 3Nara Institute of Science and Technology, Ikoma, Nara, Japan
- 4Kameda General Hospital, Kamogawa, Chiba, Japan
Abstract
Objectives: Manual analysis of contributing factors in free-text incident narratives is labour intensive. We aimed to develop and internally validate a multilabel natural language processing (NLP) classifier that extracts multiple contributing factors from medication-related incident reports using the P-mSHELL human factors framework.
Methods: We analysed 5058 sentences from 1192 medication-related incident reports (2020–2022) from the Japan Council for Quality Health Care Medical Accident Information Collection Project. Eight contributing factor categories were defined based on a modified P-mSHELL model: Medical Professional (MP), Patient Family (PF), Patient (P), Management (M), Software (S), Hardware (H), Environment (E) and Liveware-Liveware (L-L). Inter-annotator agreement was assessed using Cohen’s kappa (κ) on a random subset. A multilabel classifier was developed by fine-tuning a pretrained Japanese Bidirectional Encoder Representations from Transformers (BERT) model and evaluated using report-level fivefold cross-validation, F1 scores and exact match accuracy.
Results: Substantial inter-annotator agreement (κ=0.676–1.000) was observed across all categories. Multiple labels were assigned to 67.0% of sentences, with Medical Professional (62.9%) and Software (59.9%) most frequent. The classifier achieved F1 scores from 0.490 (Hardware) to 0.902 (Medical Professional), with sentence-level exact match accuracy of 0.512 (95% CI 0.496 to 0.527).
Discussion: The classifier performed well for major factor categories, although its accuracy does not yet support fully automated analysis; pending external validation, it is best positioned as a decision support tool to assist—rather than replace—expert review of incident reports.
Conclusion: This study demonstrates the feasibility of NLP-based multilabel extraction of contributing factors within the P-mSHELL framework, supporting scalable and consistent medication-safety incident analysis.
Related
- Development and psychometric validation of a barrier assessment tool for adherence to prescribed medication regimens in post-PCI patients
- Development of a nomogram for identifying prevalent medication discrepancy in maintenance hemodialysis patients: a multicenter cross-sectional study
- A precision health approach to medication management in neurodevelopmental conditions: a model development and validation study using four international cohorts
- Process evaluation of the INTENSE study aimed at improving adherence to oral diabetes and/or hypertension medication in people with type 2 diabetes
- GLLaucoMed: A Secure Large Language Model-Powered Agentic Workflow for Automated Medication Extraction from Free-Text Glaucoma Clinical Notes.
BibTeX
@article{ebara2026automatic,
title = {{Automatic extraction of multiple contributing factors in medication-related incidents based on the P-mSHELL model: development and evaluation of a BERT-based multilabel classifier}},
author = {Sayaka Ebara and Hayato Kizaki and Yuri Takamatsu and Satoshi Watabe and Shungo Imai and Shuntaro Yada and Eiji Aramaki and Osamu Yasumuro and Ryohkan Funakoshi and Satoko Hori},
journal = {BMJ Health \& Care Informatics},
year = {2026},
month = nov,
doi = {10.1136/bmjhci-2026-102260},
url = {https://doi.org/10.1136/bmjhci-2026-102260},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Automatic extraction of multiple contributing factors in medication-related incidents based on the P-mSHELL model: development and evaluation of a BERT-based multilabel classifier
AU - Sayaka Ebara
AU - Hayato Kizaki
AU - Yuri Takamatsu
AU - Satoshi Watabe
AU - Shungo Imai
AU - Shuntaro Yada
AU - Eiji Aramaki
AU - Osamu Yasumuro
AU - Ryohkan Funakoshi
AU - Satoko Hori
PY - 2026
DA - 2026/11/11
JO - BMJ Health & Care Informatics
DO - 10.1136/bmjhci-2026-102260
UR - https://doi.org/10.1136/bmjhci-2026-102260
AB - Objectives: Manual analysis of contributing factors in free-text incident narratives is labour intensive. We aimed to develop and internally validate a multilabel natural language processing (NLP) classifier that extracts multiple contributing factors from medication-related incident reports using the P-mSHELL human factors framework. Methods: We analysed 5058 sentences from 1192 medication-related incident reports (2020–2022) from the Japan Council for Quality Health Care Medical Accident Information Collection Project. Eight contributing factor categories were defined based on a modified P-mSHELL model: Medical Professional (MP), Patient Family (PF), Patient (P), Management (M), Software (S), Hardware (H), Environment (E) and Liveware-Liveware (L-L). Inter-annotator agreement was assessed using Cohen’s kappa (κ) on a random subset. A multilabel classifier was developed by fine-tuning a pretrained Japanese Bidirectional Encoder Representations from Transformers (BERT) model and evaluated using report-level fivefold cross-validation, F1 scores and exact match accuracy. Results: Substantial inter-annotator agreement (κ=0.676–1.000) was observed across all categories. Multiple labels were assigned to 67.0% of sentences, with Medical Professional (62.9%) and Software (59.9%) most frequent. The classifier achieved F1 scores from 0.490 (Hardware) to 0.902 (Medical Professional), with sentence-level exact match accuracy of 0.512 (95% CI 0.496 to 0.527). Discussion: The classifier performed well for major factor categories, although its accuracy does not yet support fully automated analysis; pending external validation, it is best positioned as a decision support tool to assist—rather than replace—expert review of incident reports. Conclusion: This study demonstrates the feasibility of NLP-based multilabel extraction of contributing factors within the P-mSHELL framework, supporting scalable and consistent medication-safety incident analysis.
SN - 2632-1009
ER -