Cross-Modal Organization of Facial and Vocal Behavior Revealed by Coordination Metrics Derived from Temporal Alignment
- 1Showa Medical University
Abstract
Facial and vocal behaviors jointly convey affective and communicative information, yet their cross-modal organization remains difficult to characterize quantitatively. We propose a framework for characterizing facial-vocal organization by aligning facial and vocal dynamics using canonical time warping. Multivariate facial action unit trajectories and vocal prosodic trajectories were extracted from recordings of spontaneous speech and aligned in a shared latent space. From the alignment process, we derived three cross-modal coordination metrics: "scaling factor", reflecting the relative scaling between modalities; "time shift ratio", reflecting the frequency of temporal shifts; and "time shift magnitude", reflecting their size. We evaluated these metrics in verbally fluent adults with autism spectrum disorder (ASD; N = 35) and controls (N = 44). The proposed metrics did not differ significantly between diagnostic groups but showed associations with individual differences in communication and anxiety. Scaling factor was linked to trait anxiety in ASD, whereas time shift ratio was associated with communication characteristics across participants. The proposed metrics also showed little association with conventional vocal pitch variability, suggesting that they capture aspects of expressive organization not reflected by vocal variability alone. Our findings support alignment-derived metrics as a complementary approach for characterizing individual variability in facial-vocal expressive behavior.
Related
- Functional organization and natural scene responses across mouse visual cortical areas revealed with encoding manifolds
- Overlapping perceptual benefits of temporal attention in aperiodic and periodic streams are mediated by delta phase alignment, but not alpha suppression, in sensory circuits
- Shared Fine-Scale Functional Organization in the Human Subcortex Revealed by Connectivity Hyperalignment
- Coordinated spatial organization of basal ganglia structures revealed by dopamine subtype projectome mapping
- Auditory tracking magnitude, temporal alignment, and anatomical distribution during wakefulness and propofol-based anesthesia in human sEEG
BibTeX
@misc{shikauchi2026crossmodal,
title = {{Cross-Modal Organization of Facial and Vocal Behavior Revealed by Coordination Metrics Derived from Temporal Alignment}},
author = {Y. Shikauchi and T. Naoe and L. M. Puhlmann and R. Kalisch and F. Homae and M. Nakamura},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.26.754599},
url = {https://www.biorxiv.org/content/10.64898/2026.09.26.754599v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Cross-Modal Organization of Facial and Vocal Behavior Revealed by Coordination Metrics Derived from Temporal Alignment
AU - Y. Shikauchi
AU - T. Naoe
AU - L. M. Puhlmann
AU - R. Kalisch
AU - F. Homae
AU - M. Nakamura
PY - 2026
DA - 2026/09/26
PB - bioRxiv
DO - 10.64898/2026.09.26.754599
UR - https://www.biorxiv.org/content/10.64898/2026.09.26.754599v1
AB - Facial and vocal behaviors jointly convey affective and communicative information, yet their cross-modal organization remains difficult to characterize quantitatively. We propose a framework for characterizing facial-vocal organization by aligning facial and vocal dynamics using canonical time warping. Multivariate facial action unit trajectories and vocal prosodic trajectories were extracted from recordings of spontaneous speech and aligned in a shared latent space. From the alignment process, we derived three cross-modal coordination metrics: "scaling factor", reflecting the relative scaling between modalities; "time shift ratio", reflecting the frequency of temporal shifts; and "time shift magnitude", reflecting their size. We evaluated these metrics in verbally fluent adults with autism spectrum disorder (ASD; N = 35) and controls (N = 44). The proposed metrics did not differ significantly between diagnostic groups but showed associations with individual differences in communication and anxiety. Scaling factor was linked to trait anxiety in ASD, whereas time shift ratio was associated with communication characteristics across participants. The proposed metrics also showed little association with conventional vocal pitch variability, suggesting that they capture aspects of expressive organization not reflected by vocal variability alone. Our findings support alignment-derived metrics as a complementary approach for characterizing individual variability in facial-vocal expressive behavior.
ER -