Abstract
In order to solve the problems that existing wearable emotion monitoring methods have in continuously describing the stress assessment process and lack the ability of dynamic emotion regulation and feedback, this study proposes Multimodal Stress-aware Cross-modal Emotion Network (MS-CMENet). A multimodal continuous emotion dataset, including Electroencephalography (EEG), Electrocardiography (ECG), Electromyography (EMG), and music stimulation, is constructed from data collected from 240 participants. Then, the model’s performance is verified in a 40-minute stress-induction and music-intervention experiment. This study uses a unified time axis to dynamically model multimodal physiological signals; it realizes continuous tracking of pressure index [Formula: see text], valence state vector [Formula: see text], heart rate variability [Formula: see text], arousal state vector [Formula: see text], and frontal EEG Beta relative power [Formula: see text]. The results reveal that in the stress induction stage, [Formula: see text] increases from 0.25 to 0.68, while it decreases to 0.42 in the music intervention stage; [Formula: see text] drops from 50.20 milliseconds (ms) to 35.64 ms and then recovers to 40.80 ms. Frontal [Formula: see text] adds from 0.21 to 0.37 and then declines to 0.28. Repeated measures analysis of variance indicates that all indicators have significant stage main effects (p < 0.001); the pressure index shows the highest effect size (partial [Formula: see text]). The findings show that MS-CMENet can stably achieve multimodal emotion perception, dynamic stress assessment, and tracking of music intervention effects.