Generative-augmented multimodal neuroaffective decoding for product evaluation: toward neurofeedback-driven human-in-the-loop co-design
- 1Department of Industrial Design, School of Mechanical Engineering and Automation, Northeastern University, Shenyang, China
Abstract
Background: Kansei engineering traditionally relies on self-reported evaluations of product impressions, which are susceptible to contextual influences, recall bias, and assessment timing. Multimodal psychophysiological measurements offer complementary insights into affective responses, but integrating heterogeneous signals with limited sample sizes remains challenging. This study proposes Generative-Augmented Multimodal Affective Fusion (GAMAF) to predict perceived attractiveness, innovativeness, and friendliness in product evaluation.
Methods: GAMAF integrates feature-space conditional Wasserstein generative adversarial learning with gradient penalty (cWGAN-GP), cross-modal attention, graph-based interactions, and adaptive modality weighting. Fifty participants evaluated three automobile cockpit designs, yielding 150 participant–stimulus trials after multimodal quality control. Electroencephalography (EEG), eye tracking (ET), and peripheral physiological signals were integrated and evaluated using nested participant-wise cross-validation.
Results: GAMAF achieved mean outer-fold R² values of 0.810, 0.850, and 0.840 for attractiveness, innovativeness, and friendliness, respectively, each with a standard deviation of 0.016. These values exceeded those of the Multimodal Transformer baseline (0.75, 0.78, and 0.76, respectively). Across the three targets, GAMAF reduced aggregate mean absolute error (MAE) and root mean squared error (RMSE) by approximately 29.6% and 29.7%, respectively. Ablation analyses supported the integrated architecture, while adaptive modality weights revealed target-dependent patterns.
Discussion: GAMAF demonstrates the potential of generative-augmented multimodal fusion for decoding Kansei-related product impressions. However, performance comparisons remain descriptive because baseline variability and participant-level uncertainty were not quantified. Further validation on unseen product designs and closed-loop design optimization is needed to establish its generalizability and practical utility.
Related
- Multimodal Physiological Decoding Reveals Individualized Arousal Dynamics in Closed-Loop Neurofeedback
- GAMPoint: Geometry-augmented vision transformer with a multimodal-ready interface for 3D point-cloud segmentation
- Synthetic-augmented multimodal deep learning fuses dual-angle RGB images and phenology to unlock genotype-informative canopy structural trait in wheat
- Programmable design of functional RNAs through multimodal generative learning
- AcupunctureBench: A Multimodal Benchmark for Evaluating Generative Models in the Acupuncture Domain
BibTeX
@article{deng2026generativeaugmented,
title = {{Generative-augmented multimodal neuroaffective decoding for product evaluation: toward neurofeedback-driven human-in-the-loop co-design}},
author = {Ming Deng},
journal = {Frontiers in Human Neuroscience},
year = {2026},
month = oct,
doi = {10.3389/fnhum.2026.1997654},
url = {https://doi.org/10.3389/fnhum.2026.1997654},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Generative-augmented multimodal neuroaffective decoding for product evaluation: toward neurofeedback-driven human-in-the-loop co-design
AU - Ming Deng
PY - 2026
DA - 2026/10/21
JO - Frontiers in Human Neuroscience
DO - 10.3389/fnhum.2026.1997654
UR - https://doi.org/10.3389/fnhum.2026.1997654
AB - Background: Kansei engineering traditionally relies on self-reported evaluations of product impressions, which are susceptible to contextual influences, recall bias, and assessment timing. Multimodal psychophysiological measurements offer complementary insights into affective responses, but integrating heterogeneous signals with limited sample sizes remains challenging. This study proposes Generative-Augmented Multimodal Affective Fusion (GAMAF) to predict perceived attractiveness, innovativeness, and friendliness in product evaluation. Methods: GAMAF integrates feature-space conditional Wasserstein generative adversarial learning with gradient penalty (cWGAN-GP), cross-modal attention, graph-based interactions, and adaptive modality weighting. Fifty participants evaluated three automobile cockpit designs, yielding 150 participant–stimulus trials after multimodal quality control. Electroencephalography (EEG), eye tracking (ET), and peripheral physiological signals were integrated and evaluated using nested participant-wise cross-validation. Results: GAMAF achieved mean outer-fold R² values of 0.810, 0.850, and 0.840 for attractiveness, innovativeness, and friendliness, respectively, each with a standard deviation of 0.016. These values exceeded those of the Multimodal Transformer baseline (0.75, 0.78, and 0.76, respectively). Across the three targets, GAMAF reduced aggregate mean absolute error (MAE) and root mean squared error (RMSE) by approximately 29.6% and 29.7%, respectively. Ablation analyses supported the integrated architecture, while adaptive modality weights revealed target-dependent patterns. Discussion: GAMAF demonstrates the potential of generative-augmented multimodal fusion for decoding Kansei-related product impressions. However, performance comparisons remain descriptive because baseline variability and participant-level uncertainty were not quantified. Further validation on unseen product designs and closed-loop design optimization is needed to establish its generalizability and practical utility.
SN - 1662-5161
ER -