Citation

Poh, Jia Qing and See, John Su Yang and El Gayar, Neamat and Wong, Lai Kuan (2024) Are You Paying Attention? Multimodal Linear Attention Transformers for Affect Prediction in Video Conversations. MRAC '24: Proceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing. pp. 15-23.

Text 3689092.3689409.pdf - Published Version Restricted to Repository staff only Download (2MB)

Abstract

The post-COVID-19 era has seen continual adoption and reliance on video-based communication, underscoring the need for unobtrusive affect recognition in digital interactions. This paper proposes an efficient multimodal approach to emotion recognition in video conversational scenarios, leveraging linear attention-based Transformer networks to process both visual and audio cues. We explore various linear attention mechanisms, comparing them with classical self-attention. Using the K-EmoCon dataset, we demonstrate that the proposed approach yields competitive performance in predicting the affective states of conversing persons while significantly improving memory efficiency. Our ablation studies reveal that carefully tuned simple fusion methods can match or exceed more complex approaches. This research contributes to developing more accessible and efficient multimodal emotion recognition systems for video-based conversations, with applications for enhancing remote communication and monitoring digital well-being in the post-pandemic era.

Item Type:	Article
Uncontrolled Keywords:	Multimodal transformers, linear attention, affect prediction, video conversations
Subjects:	T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK5101-6720 Telecommunication. Including telegraphy, telephone, radio, radar, television
Divisions:	Faculty of Computing and Informatics (FCI)
Depositing User:	Ms Nurul Iqtiani Ahmad
Date Deposited:	03 Jan 2025 03:04
Last Modified:	03 Jan 2025 03:04
URII:	http://shdl.mmu.edu.my/id/eprint/13271

Downloads

Edit (login required)