Citation

Su, Yen Ding and Tong, Boon Tang and Lu, Cheng Kai and Abdul Razak, Normy Norfiza and Ahmad Fauzi, Mohammad Faizal and Khalid, Ahmad Shahrafidz (2025) Multimodal Emotion Recognition with Fusion of 1D and 2D Convolutional Neural Network. In: 2025 Multimedia University Engineering Conference, MECON 2025, 21 July 2025 - 23 July 2025, Cyberjaya, Malaysia.

Text 451.pdf - Published Version Restricted to Repository staff only Download (1MB)

Abstract

Emotion is one of the key components in daily life, serving multiple purposes in social interactions. The advancements in Artificial Intelligence (AI) had made emotion recognition possible with more accurateness. The AI model, specifically the Convolutional Neural Network (CNN), had presented superior performance in computer vision and image recognition tasks, leading to highly accurate Emotion Recognition (ER). However, the typical CNN architecture is mostly complex and hardly adaptable for real-time applications, especially when multi-input types are involved. To overcome this challenge, this study proposes a tailored lightweight Multimodal Emotion Recognition CNN architecture that accepts facial images and vocal features (i.e., mel-spectrogram in decibels) as input and is benchmarked on Surrey Audio-Visual Expressed Emotion (SAVEE) and Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). The SAVEE dataset contains recordings of 4 male actors with single level emotion intensity while RAVDESS contains recordings of 24 actors, balanced in gender, with two levels of emotion intensity. The lightweight CNN model achieved an on-par accuracy of 97.15% in SAVEE and 86.72% in RAVDESS while significantly simpler than state-of-the-arts MER models, making it a more feasible option for real-time emotion recognition on resource-constrained devices.

Item Type:	Conference or Workshop Item (Paper)
Uncontrolled Keywords:	Multimodal emotion recognition, convolutional neural network, lightweight
Subjects:	T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK4125-4399 Electric lighting
Divisions:	Faculty of Artificial Intelligence & Engineering (FAIE)
Depositing User:	Ms Suzilawati Abu Samah
Date Deposited:	19 Mar 2026 02:43
Last Modified:	19 Mar 2026 02:43
URII:	http://shdl.mmu.edu.my/id/eprint/15613

Downloads

Edit (login required)