Comparing Machine Learning Methods for Detecting Caregiver Burden in Transcribed Conversations with Clinicians
- 1University of Pennsylvania
Abstract
Dementia caregivers' elevated burden is associated with earlier patient institutionalization and greater caregiver morbidity, yet no standardized, objective method exists for detecting burden during routine clinical encounters. We compared four machine learning and AI approaches for detecting caregiver burden in de-identified caregiver-clinician transcripts from the COPE in PACE study using Zarit burden scores. A naive baseline predicting burden in every transcript achieved precision/F1 = 0.40/0.57. LLM prompting achieved precision/F1 = 0.43/0.60. Weak supervision achieved precision/F1 = 0.48/0.58. The fully supervised model, trained on sentence-level labels with only 1.1% burden-positive sentences, achieved precision/F1=0.19/0.25; this result is not directly comparable because of different labels and units of analysis. LLM prompting modestly outperformed weak supervision and naive baseline without fine-tuning, while class imbalance limited fully supervised sentence-level classification. Larger annotated corpora and refined labeling strategies are needed for clinical deployment.
Related
- Exploring How Staff Engage in Conversations Around Sexuality With Adults With Intellectual Disabilities: A Qualitative Systematic Review.
- From Echo Chambers to Echo Machines? Endorsement, Evidence, and the Certainty of Belief in Conversations With AI
- Relationships between Counselor Statements and Student Connection to Mental Health Care: A Cross-sectional Analysis of Digital Screening Platform Conversations
- Beyond Initial Crisis Recognition: Suicide-Risk Responses of ChatGPT, Claude, and Gemini Across 14-Turn Ukrainian-Language Conversations
- Time, talk, and teamwork: Perceptions of personalised dementia care planning conversations in primary care
BibTeX
@misc{klein2026comparing,
title = {{Comparing Machine Learning Methods for Detecting Caregiver Burden in Transcribed Conversations with Clinicians}},
author = {A. Klein and W. He and H. Chen and B. Tong and E. Summerhayes and L. Shen and N. Hodgson},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.30.26366925},
url = {https://www.medrxiv.org/content/10.64898/2026.10.30.26366925v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Comparing Machine Learning Methods for Detecting Caregiver Burden in Transcribed Conversations with Clinicians
AU - A. Klein
AU - W. He
AU - H. Chen
AU - B. Tong
AU - E. Summerhayes
AU - L. Shen
AU - N. Hodgson
PY - 2026
DA - 2026/10/30
PB - medRxiv
DO - 10.64898/2026.10.30.26366925
UR - https://www.medrxiv.org/content/10.64898/2026.10.30.26366925v1
AB - Dementia caregivers' elevated burden is associated with earlier patient institutionalization and greater caregiver morbidity, yet no standardized, objective method exists for detecting burden during routine clinical encounters. We compared four machine learning and AI approaches for detecting caregiver burden in de-identified caregiver-clinician transcripts from the COPE in PACE study using Zarit burden scores. A naive baseline predicting burden in every transcript achieved precision/F1 = 0.40/0.57. LLM prompting achieved precision/F1 = 0.43/0.60. Weak supervision achieved precision/F1 = 0.48/0.58. The fully supervised model, trained on sentence-level labels with only 1.1% burden-positive sentences, achieved precision/F1=0.19/0.25; this result is not directly comparable because of different labels and units of analysis. LLM prompting modestly outperformed weak supervision and naive baseline without fine-tuning, while class imbalance limited fully supervised sentence-level classification. Larger annotated corpora and refined labeling strategies are needed for clinical deployment.
ER -