Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders

Salam Hamieh
  • Fonction : Auteur correspondant
  • PersonId : 1122569

Connectez-vous pour contacter l'auteur
Vincent Heiries
  • Fonction : Auteur
  • PersonId : 948855
Christelle Godin
  • Fonction : Auteur
  • PersonId : 859057

Résumé

Human stress detection is of great importance for monitoring mental health. The Multimodal Sentiment Analysis Challenge (MuSe) 2021 focuses on emotion, physiological-emotion, and stress recognition as well as sentiment classification by exploiting several modalities. In this paper, we present our solution for the Muse-Stress sub-challenge. The target of this sub-challenge is continuous prediction of arousal and valence for people under stressful conditions where text transcripts and audio and video recordings are provided. To this end, we utilize bidirectional Long Short-Term Memory (LSTM) and Gated Recurrent Unit networks (GRU) to explore high-level and low-level features from different modalities. We employ Concordance Correlation Coefficient (CCC) as a loss function and evaluation metric for our model. To improve the unimodal predictions, we add difficulty indicators of the data obtained by using Auto-Encoders. Finally, we perform late fusion on our unimodal predictions in addition to the difficulty indicators to obtain our final predictions. With this approach, we achieve CCC of 0.4278 and 0.5951 for arousal and valence respectively, our submission to MuSe 2021 ranks in the top three for arousal and fourth for valence.
Fichier principal
Vignette du fichier
Hamieh et al. - 2021 - Multi-modal Fusion for Continuous Emotion Recognit.pdf (1.28 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

cea-03517175 , version 1 (07-01-2022)

Licence

Paternité - Pas d'utilisation commerciale - Partage selon les Conditions Initiales

Identifiants

Citer

Salam Hamieh, Vincent Heiries, Hussein Al Osman, Christelle Godin. Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders. MM '21: ACM Multimedia Conference, Oct 2021, Virtual Event China, France. pp.21-27, ⟨10.1145/3475957.3484455⟩. ⟨cea-03517175⟩
91 Consultations
91 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More