Authors: Abirami A, Miranda Lakshmi T, Martin A
Abstract: Conversational emotion recognition is inherently challenging because an utterance's emotional label does not depend on wording alone — it is shaped jointly by content, delivery, and the dialogue history surrounding it. This work introduces a multimodal deep-learning pipeline that fuses textual, acoustic, and contextual signals to classify emotion at the utterance level within conversations. Linguistic meaning is captured through pretrained BERT embeddings, broad acoustic patterns through Wav2Vec2 embeddings, and explicit vocal-affect cues through handcrafted prosodic descriptors (pitch statistics and short-time energy). Once concatenated at the utterance level, these heterogeneous features are passed into a Bidirectional Gated Recurrent Unit (BiGRU) that captures dependencies across dialogue turns, with a fully connected head performing the final classification. Because the benchmark dataset's seven-class emotion taxonomy is heavily skewed, a weighted cross-entropy loss built from inverse class-frequency statistics is applied to offset this imbalance during training. On the MELD (Multimodal EmotionLines Dataset) benchmark, the proposed pipeline reaches 77.56% overall accuracy and a weighted F1-score of 0.804, with per-class ROC-AUC spanning 0.89 to 0.96. Confusion-matrix and error analysis reveal that common emotions such as neutral and joy are classified reliably, whereas rarer emotions such as fear and disgust remain substantially harder to recognize even with class-weighted training — underlining how persistent the data-imbalance problem remains in this domain and pointing toward future work on stronger imbalance-handling and context-modelling techniques.
DOI: https://doi.org/10.5281/zenodo.22204728
