Estimating presenteeism from repeated smartphone-based multimodal behavioral responses
9 affiliations
- 1Graduate School of Science and Technology, University of Tsukuba, Tsukuba, Ibaraki, Japan
- 2Institute of Medicine, University of Tsukuba, Tsukuba, Ibaraki, Japan
- 3Minds in Co., Ltd., Chuo-ku, Tokyo, Japan
- 4Data Science Laboratories, NEC Corporation, Meguro-ku, Tokyo, Japan
- 5Center for Cybernics Research, University of Tsukuba, Tsukuba, Ibaraki, Japan
- 6Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Ibaraki, Japan
- 7Retail & Consumer Service Division, Sojitz Corporation, Chiyoda-ku, Tokyo, Japan
- 8Human Capital Dept. 2, Sojitz Corporation, Chiyoda-ku, Tokyo, Japan
- 9Public Relations Dept., Sojitz Corporation, Chiyoda-ku, Tokyo, Japan
Abstract
Presenteeism, attending work despite physical or mental health problems, is a major source of productivity loss worldwide, yet its early detection in everyday settings remains challenging. Conventional self-report instruments are time-consuming and require psychiatrist interpretation, which limits their scalability. We developed a multimodal machine-learning model that estimates work-function impairment as an indicator of presenteeism from brief smartphone-based dialogues. 38 employees recorded short self-report videos (10–30 s) using front-facing smartphone cameras, which were independently rated on an ordinal three-level work-function scale (Healthy / Moderate / Unwell) by three psychiatrists; the consensus label served as supervision. We tested whether integrating multimodal behavioural signals (acoustic, facial, and linguistic) would provide additional cues beyond linguistic content alone, particularly when verbal cues are sparse. Acoustic, facial, and linguistic features were extracted from each clip and integrated using a three-stream Attention-based Multiple Instance Learning (Attention-MIL) framework with learned late fusion, an ordinal-regression head, and class-prior logit adjustment. Generalisation was assessed under a participant-disjoint protocol (25-fold repeated GroupKFold; n = 1,768 out-of-fold clips from 29 participants). On the Full configuration, the proposed multimodal framework achieved macro-F1 = 0.772 (95% CI [0.697, 0.817]), accuracy = 0.840, macro-AUROC = 0.940, and low expected calibration error (ECE ≈0.024). At the aggregate level, the four text-containing configurations (Full / Audio+Text / Face+Text / Text-only) were mutually indistinguishable on macro-F1 (Holm–Bonferroni padj > 0.40), whereas text-free configurations performed substantially worse (Δ<−0.33, padj < 0.001), confirming the necessity of the linguistic channel. In an exploratory subgroup analysis, Audio+Text outperformed Text-only in the clinically ambiguous subgroup—short-speech responses from non-Healthy participants (n = 77; Δ macro-F1 =+0.030, exploratory, requires replication). These findings provide localised but clinically meaningful support for the multimodal hypothesis under low-information conditions and motivate prospective replication.
Related
- Passive Smartphone-Based Digital Phenotyping for the Identification of PTSD Symptoms: Initial Validation of a Digital Behavioral Marker
- Behavioral Smartphone Orientation Is Associated with Upper Limb Dysfunction Among University Students: A Cross-Sectional Study
- Repeated prefrontal transcranial direct current stimulation improves behavioral regulation and modulates resting-state EEG microstate dynamics in young adults with problematic smartphone use: a randomized sham-controlled longitudinal study
- Artificial Intelligence in Ocular Surface Oncology: A Narrative Review of Multimodal Imaging, Smartphone-Based Screening, and Clinical Decision Support
- Job Coaching and Smartphone-Based Cognitive Behavioral Techniques for Returning to Work and Coping With Unemployment (DRIVEN): Randomized Controlled Trial.
BibTeX
@article{noguchi2026estimating,
title = {{Estimating presenteeism from repeated smartphone-based multimodal behavioral responses}},
author = {Taiga Noguchi and Shotaro Doki and Masakazu Hirokawa and Soma Nishimura and Katsuya Hotta and Yuya Iwata and Naoko Kouda and Shota Matsumoto and Kenji Suzuki},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0354342},
url = {https://doi.org/10.1371/journal.pone.0354342},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Estimating presenteeism from repeated smartphone-based multimodal behavioral responses
AU - Taiga Noguchi
AU - Shotaro Doki
AU - Masakazu Hirokawa
AU - Soma Nishimura
AU - Katsuya Hotta
AU - Yuya Iwata
AU - Naoko Kouda
AU - Shota Matsumoto
AU - Kenji Suzuki
PY - 2026
DA - 2026/09/16
JO - PLOS One
DO - 10.1371/journal.pone.0354342
UR - https://doi.org/10.1371/journal.pone.0354342
AB - Presenteeism, attending work despite physical or mental health problems, is a major source of productivity loss worldwide, yet its early detection in everyday settings remains challenging. Conventional self-report instruments are time-consuming and require psychiatrist interpretation, which limits their scalability. We developed a multimodal machine-learning model that estimates work-function impairment as an indicator of presenteeism from brief smartphone-based dialogues. 38 employees recorded short self-report videos (10–30 s) using front-facing smartphone cameras, which were independently rated on an ordinal three-level work-function scale (Healthy / Moderate / Unwell) by three psychiatrists; the consensus label served as supervision. We tested whether integrating multimodal behavioural signals (acoustic, facial, and linguistic) would provide additional cues beyond linguistic content alone, particularly when verbal cues are sparse. Acoustic, facial, and linguistic features were extracted from each clip and integrated using a three-stream Attention-based Multiple Instance Learning (Attention-MIL) framework with learned late fusion, an ordinal-regression head, and class-prior logit adjustment. Generalisation was assessed under a participant-disjoint protocol (25-fold repeated GroupKFold; n = 1,768 out-of-fold clips from 29 participants). On the Full configuration, the proposed multimodal framework achieved macro-F1 = 0.772 (95% CI [0.697, 0.817]), accuracy = 0.840, macro-AUROC = 0.940, and low expected calibration error (ECE ≈0.024). At the aggregate level, the four text-containing configurations (Full / Audio+Text / Face+Text / Text-only) were mutually indistinguishable on macro-F1 (Holm–Bonferroni padj > 0.40), whereas text-free configurations performed substantially worse (Δ<−0.33, padj < 0.001), confirming the necessity of the linguistic channel. In an exploratory subgroup analysis, Audio+Text outperformed Text-only in the clinically ambiguous subgroup—short-speech responses from non-Healthy participants (n = 77; Δ macro-F1 =+0.030, exploratory, requires replication). These findings provide localised but clinically meaningful support for the multimodal hypothesis under low-information conditions and motivate prospective replication.
SN - 1932-6203
ER -