{
  "abstract": "Presenteeism, attending work despite physical or mental health problems, is a major source of productivity loss worldwide, yet its early detection in everyday settings remains challenging. Conventional self-report instruments are time-consuming and require psychiatrist interpretation, which limits their scalability. We developed a multimodal machine-learning model that estimates work-function impairment as an indicator of presenteeism from brief smartphone-based dialogues. 38 employees recorded short self-report videos (10–30 s) using front-facing smartphone cameras, which were independently rated on an ordinal three-level work-function scale (Healthy / Moderate / Unwell) by three psychiatrists; the consensus label served as supervision. We tested whether integrating multimodal behavioural signals (acoustic, facial, and linguistic) would provide additional cues beyond linguistic content alone, particularly when verbal cues are sparse. Acoustic, facial, and linguistic features were extracted from each clip and integrated using a three-stream Attention-based Multiple Instance Learning (Attention-MIL) framework with learned late fusion, an ordinal-regression head, and class-prior logit adjustment. Generalisation was assessed under a participant-disjoint protocol (25-fold repeated GroupKFold; n = 1,768 out-of-fold clips from 29 participants). On the Full configuration, the proposed multimodal framework achieved macro-F1 = 0.772 (95% CI [0.697, 0.817]), accuracy = 0.840, macro-AUROC = 0.940, and low expected calibration error (ECE ≈0.024). At the aggregate level, the four text-containing configurations (Full / Audio+Text / Face+Text / Text-only) were mutually indistinguishable on macro-F1 (Holm–Bonferroni padj > 0.40), whereas text-free configurations performed substantially worse (Δ<−0.33, padj < 0.001), confirming the necessity of the linguistic channel. In an exploratory subgroup analysis, Audio+Text outperformed Text-only in the clinically ambiguous subgroup—short-speech responses from non-Healthy participants (n = 77; Δ macro-F1 =+0.030, exploratory, requires replication). These findings provide localised but clinically meaningful support for the multimodal hypothesis under low-information conditions and motivate prospective replication.",
  "abstract_shown": true,
  "affiliations": [
    "Graduate School of Science and Technology, University of Tsukuba, Tsukuba, Ibaraki, Japan",
    "Institute of Medicine, University of Tsukuba, Tsukuba, Ibaraki, Japan",
    "Minds in Co., Ltd., Chuo-ku, Tokyo, Japan",
    "Data Science Laboratories, NEC Corporation, Meguro-ku, Tokyo, Japan",
    "Center for Cybernics Research, University of Tsukuba, Tsukuba, Ibaraki, Japan",
    "Graduate School of Comprehensive Human Sciences, University of Tsukuba, Tsukuba, Ibaraki, Japan",
    "Retail & Consumer Service Division, Sojitz Corporation, Chiyoda-ku, Tokyo, Japan",
    "Human Capital Dept. 2, Sojitz Corporation, Chiyoda-ku, Tokyo, Japan",
    "Public Relations Dept., Sojitz Corporation, Chiyoda-ku, Tokyo, Japan"
  ],
  "article_type": "Research Article",
  "arxiv": null,
  "author_total": 9,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Taiga Noguchi"
    },
    {
      "affs": [
        1
      ],
      "name": "Shotaro Doki"
    },
    {
      "affs": [
        2
      ],
      "name": "Masakazu Hirokawa"
    },
    {
      "affs": [
        3
      ],
      "name": "Soma Nishimura"
    },
    {
      "affs": [
        4
      ],
      "name": "Katsuya Hotta"
    },
    {
      "affs": [
        5
      ],
      "name": "Yuya Iwata"
    },
    {
      "affs": [
        6
      ],
      "name": "Naoko Kouda"
    },
    {
      "affs": [
        7
      ],
      "name": "Shota Matsumoto"
    },
    {
      "affs": [
        8
      ],
      "name": "Kenji Suzuki"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/20vpgkg98r8rb.bib",
  "cited_by": 0,
  "doi": "10.1371/journal.pone.0354342",
  "fulltext_url": null,
  "id": "20vpgkg98r8rb",
  "issns": [
    "1932-6203"
  ],
  "landing_url": "https://doi.org/10.1371/journal.pone.0354342",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "openalex",
      "3205|Emotion and Mood Recognition"
    ],
    [
      "plos",
      "/Biology and life sciences/Anatomy/Head/Face"
    ],
    [
      "plos",
      "/Biology and life sciences/Psychology/Behavior"
    ],
    [
      "plos",
      "/Biology and life sciences/Psychology/Psychometrics"
    ],
    [
      "plos",
      "/Computer and information sciences/Software engineering/Preprocessing"
    ],
    [
      "plos",
      "/Engineering and technology/Signal processing/Speech signal processing"
    ],
    [
      "plos",
      "/Engineering and technology/Software engineering/Preprocessing"
    ],
    [
      "plos",
      "/Medicine and health sciences/Anatomy/Head/Face"
    ],
    [
      "plos",
      "/Physical sciences/Physics/Acoustics"
    ],
    [
      "plos",
      "/Physical sciences/Physics/Acoustics/Acoustic signals"
    ],
    [
      "plos",
      "/Social sciences/Linguistics/Speech"
    ],
    [
      "plos",
      "/Social sciences/Psychology/Behavior"
    ],
    [
      "plos",
      "/Social sciences/Psychology/Psychometrics"
    ]
  ],
  "openalex": "W7213426614",
  "pdf_url": "https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0354342&type=printable",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-16",
  "publisher": "Public Library of Science",
  "ris": "https://research.draconitic.com/paper/20vpgkg98r8rb.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:49Z",
      "native_id": "10.1371/journal.pone.0354342",
      "source": "plos",
      "url": "https://doi.org/10.1371/journal.pone.0354342"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "plos:anatomy",
      "subfield": "physiology"
    }
  ],
  "title": "Estimating presenteeism from repeated smartphone-based multimodal behavioral responses",
  "title_text": "Estimating presenteeism from repeated smartphone-based multimodal behavioral responses",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T12:17:49Z",
  "url": "https://research.draconitic.com/paper/20vpgkg98r8rb",
  "venue": "PLOS One",
  "versions": []
}