An explainable multimodal framework for exploring depression-associated speech representations in Alzheimer's disease
- 1University of Huddersfield
Abstract
Depression in Alzheimers disease (AD) is clinically challenging to characterise because affective and cognitive manifestations frequently overlap with the underlying neurodegenerative condition. This study investigates depression-related speech patterns in AD using an explainable multimodal framework that integrates linguistic and acoustic features from elicited picture-description speech. To address the absence of directly observed depression labels in the target AD cohort, we employ a source-informed cross-dataset transfer strategy. Unsupervised clustering identified two distinct speech sub-phenotypes within the AD cohort. By comparing these unsupervised representations to a depression-labelled source corpus, one cluster demonstrated mathematical proximity (measured via cosine similarity in a shared latent space) to the source depression centroid. We explicitly acknowledge that this proximity reflects geometric similarity in a shared latent space, which may encapsulate overlapping domain characteristics alongside affective variance. To validate the structural integrity and separability of these discovered clusters, a proxy supervised classification task was employed. The retained multimodal artificial neural network (ANN) integrated information from both modalities and achieved 92.73% cluster separability accuracy. This demonstrates that the representation-derived two-cluster partition was highly learnable on the independent internal hold-out test set; it does not establish clinical validity, longitudinal stability, or diagnostic validity of the identified sub-phenotypes. For interpretability, Shapley Additive exPlanations (SHAP) analysis was utilised to extract feature attributions and organise influential acoustic and linguistic features into a multimodal taxonomy. Furthermore, a Multimodal SHAP Attribution Magnitude (MSAM) score was derived as an exploratory model-based attribution score rather than a clinically validated severity measure. Ultimately, this study provides an interpretable computational baseline for characterising depression-related speech variance in AD, establishing a structural foundation for future prospective validation using independently labelled clinical cohorts.
Author SummaryDepression can be difficult to recognise in people with Alzheimers disease because changes in mood, motivation, language, and behaviour may overlap with symptoms of cognitive impairment. Speech analysis offers a possible way of studying these changes, but datasets containing both Alzheimers disease and independently verified depression labels are limited. We therefore explored whether speech representations from an Alzheimers disease cohort could be compared with a separate depression-labelled speech dataset. We used acoustic and linguistic features to identify groups of participants with similar speech characteristics and then examined whether these groups could be reliably reproduced by supervised machine-learning models. The best multimodal model reproduced the representation-derived grouping with 92.73% accuracy on an independent participant-level test set. Importantly, this result does not demonstrate that the participants had depression, because depression was not independently assessed in the Alzheimers disease cohort. Instead, the study provides an exploratory computational framework for studying cross-corpus speech similarities and identifies features that contributed to the learned speech representations. Prospective studies with clinically assessed depression status are needed before such methods can be considered for clinical use. Such studies should also examine whether the observed speech patterns remain informative after accounting for cognitive impairment, age, medication, speaking task, and recording conditions. This work therefore represents a hypothesis-generating computational step rather than a diagnostic tool.
Related
- SYNERGY-VAE: An explainable generative deep learning framework for discovering depression subgroups from multimodal population health data
- Hybrid Explainable Deep Ensemble Framework with self-attention mechanism for early prediction of depression severity using PHQ-9 clinical and behavioral data
- Explainable AI framework for Alzheimer’s disease prediction using multimodal data
- Mechanical ventilation–associated AKI: toward a multimodal and kidney-protective ventilatory framework
- Multimodal fusion of SpO₂ time series and clinical data for explainable classification of obstructive sleep apnea severity
BibTeX
@misc{hassan2026explainable,
title = {{An explainable multimodal framework for exploring depression-associated speech representations in Alzheimer's disease}},
author = {H. Hassan and D. Patel and H. Aagela and A. Johnson},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.29.26364278},
url = {https://www.medrxiv.org/content/10.64898/2026.09.29.26364278v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - An explainable multimodal framework for exploring depression-associated speech representations in Alzheimer's disease
AU - H. Hassan
AU - D. Patel
AU - H. Aagela
AU - A. Johnson
PY - 2026
DA - 2026/09/29
PB - medRxiv
DO - 10.64898/2026.09.29.26364278
UR - https://www.medrxiv.org/content/10.64898/2026.09.29.26364278v1
AB - Depression in Alzheimers disease (AD) is clinically challenging to characterise because affective and cognitive manifestations frequently overlap with the underlying neurodegenerative condition. This study investigates depression-related speech patterns in AD using an explainable multimodal framework that integrates linguistic and acoustic features from elicited picture-description speech. To address the absence of directly observed depression labels in the target AD cohort, we employ a source-informed cross-dataset transfer strategy. Unsupervised clustering identified two distinct speech sub-phenotypes within the AD cohort. By comparing these unsupervised representations to a depression-labelled source corpus, one cluster demonstrated mathematical proximity (measured via cosine similarity in a shared latent space) to the source depression centroid. We explicitly acknowledge that this proximity reflects geometric similarity in a shared latent space, which may encapsulate overlapping domain characteristics alongside affective variance. To validate the structural integrity and separability of these discovered clusters, a proxy supervised classification task was employed. The retained multimodal artificial neural network (ANN) integrated information from both modalities and achieved 92.73% cluster separability accuracy. This demonstrates that the representation-derived two-cluster partition was highly learnable on the independent internal hold-out test set; it does not establish clinical validity, longitudinal stability, or diagnostic validity of the identified sub-phenotypes. For interpretability, Shapley Additive exPlanations (SHAP) analysis was utilised to extract feature attributions and organise influential acoustic and linguistic features into a multimodal taxonomy. Furthermore, a Multimodal SHAP Attribution Magnitude (MSAM) score was derived as an exploratory model-based attribution score rather than a clinically validated severity measure. Ultimately, this study provides an interpretable computational baseline for characterising depression-related speech variance in AD, establishing a structural foundation for future prospective validation using independently labelled clinical cohorts. Author SummaryDepression can be difficult to recognise in people with Alzheimers disease because changes in mood, motivation, language, and behaviour may overlap with symptoms of cognitive impairment. Speech analysis offers a possible way of studying these changes, but datasets containing both Alzheimers disease and independently verified depression labels are limited. We therefore explored whether speech representations from an Alzheimers disease cohort could be compared with a separate depression-labelled speech dataset. We used acoustic and linguistic features to identify groups of participants with similar speech characteristics and then examined whether these groups could be reliably reproduced by supervised machine-learning models. The best multimodal model reproduced the representation-derived grouping with 92.73% accuracy on an independent participant-level test set. Importantly, this result does not demonstrate that the participants had depression, because depression was not independently assessed in the Alzheimers disease cohort. Instead, the study provides an exploratory computational framework for studying cross-corpus speech similarities and identifies features that contributed to the learned speech representations. Prospective studies with clinically assessed depression status are needed before such methods can be considered for clinical use. Such studies should also examine whether the observed speech patterns remain informative after accounting for cognitive impairment, age, medication, speaking task, and recording conditions. This work therefore represents a hypothesis-generating computational step rather than a diagnostic tool.
ER -