An enhanced integration of fuzzy measure and convolutional neural network for multimodal sentiment analysis

Multimodal Sentiment Analysis (MSA) is an evolving field in Natural Language Processing (NLP) that extends traditional Sentiment Analysis (SA) by analyzing opinions and emotions across multiple modalities, like video, rather than relying solely on text. The challenge of MSA lies in interpreting complex relationships within and across these diverse data types, referred to as intra- and inter-modality dynamics. While recent advances in machine learning (ML), particularly deep learning (DL), have improved MSA’s performance, handling the dynamics across multiple modalities remains a significant obstacle. A key limitation is the inability of current approaches, such as convolutional neural networks (CNNs), to fully understand and leverage these complex dynamics. The study addresses the challenge of understanding modality dynamics in MSA using a fuzzy strategic approach, particularly the Choquet fuzzy integral. The methodology involves using CNNs to extract features across multiple modalities and integrating a fuzzy integral strategy (Choquet) to enhance interpretability and feature ranking. Benchmark datasets—including RADVESS, SAVEE, MOSI, and Chest X-Ray—were utilized to evaluate the model. Key procedures focused on integrating CNN with the Choquet fuzzy integral for improved feature importance ranking and feature pattern interpretation. The study found that the Choquet fuzzy integral significantly enhances CNN’s ability to handle complex, crossmodal relationships, improving sentiment analysis accuracy. The fuzzy integral approach allowed for a 33% improvement in feature extraction capability and an 18% reduction in model parameters compared to baseline models, indicating enhanced performance and efficiency in handling MSA tasks. The study contributes to MSA by integrating the Choquet fuzzy integral with CNN-based models, offering a novel integration that enhances interpretability and feature ranking and effectively handles intra- and inter-modality dynamics. The findings suggest that the Choquet fuzzy integral technique can be effectively applied in NLP to enhance the performance of DL models, particularly in multimodal contexts, contributing practical insights for developing more robust sentiment analysis models in real-world applications