Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture
Analyzing sentiments using single-modal approaches, such as text or image analysis alone, frequently encounters significant limitations. These drawbacks include inadequate feature representation, an inability to capture the full complexity of emotional expressions, and challenges in handling diverse...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10758628/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850221048781340672 |
|---|---|
| author | P. Vasanthi V. Madhu Viswanatham |
| author_facet | P. Vasanthi V. Madhu Viswanatham |
| author_sort | P. Vasanthi |
| collection | DOAJ |
| description | Analyzing sentiments using single-modal approaches, such as text or image analysis alone, frequently encounters significant limitations. These drawbacks include inadequate feature representation, an inability to capture the full complexity of emotional expressions, and challenges in handling diverse and noisy data types. This underscores the urgent need for a more comprehensive approach, capable of integrating multiple types of data to offer a richer and more nuanced understanding of sentiment. To address these challenges, Multimodal Sentiment Analysis (MSA) has emerged as a crucial advancement. In this article, the Enhanced LinkNet (EnLNet)-based Federated Learning (FL) approach is proposed for MSA. This approach utilizes an EnLNet model within an FL framework to manage and process multimodal data, including text, signals, and images. The EnLNet model in the FL framework is an advanced model by encompassing modified encoder and decoder block, interpolation blocks and a modified activation function. The approach is structured in three stages: the initialization stage, where global parameters are established and shared among clients; the local training stage, where several local models independently handle preprocessing, feature extraction, and fusion of text (with Improved Aspect Term Extraction (IATE) and Term Frequency-Inverse Document Frequency (TF-IDF)), signals (using Modified Mel Frequency Cepstral Coefficient (MMFCC) and spectral features), and images (through Improved Active Appearance Model (IAAM) and Median Binary Pattern (MBP)) before training the EnLNet model; and the model aggregation stage, where updates from local models are collected and aggregated by the central server to refine the global model. This iterative process lasts until convergence or the extreme number of iterations is attained. The efficiency of this approach is validated through Accuracy, Precision, FNR, FPR and performance comparisons with state-of-the-art approaches, demonstrating its capacity to enhance MSA by successfully incorporating and processing diverse multimodal data. |
| format | Article |
| id | doaj-art-af7b06cd4dff40cd85bd19dd17dc2246 |
| institution | OA Journals |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-af7b06cd4dff40cd85bd19dd17dc22462025-08-20T02:06:50ZengIEEEIEEE Access2169-35362024-01-011217521817523910.1109/ACCESS.2024.350329010758628Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet ArchitectureP. Vasanthi0https://orcid.org/0009-0007-0676-9311V. Madhu Viswanatham1https://orcid.org/0000-0001-8355-7290School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, IndiaSchool of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, IndiaAnalyzing sentiments using single-modal approaches, such as text or image analysis alone, frequently encounters significant limitations. These drawbacks include inadequate feature representation, an inability to capture the full complexity of emotional expressions, and challenges in handling diverse and noisy data types. This underscores the urgent need for a more comprehensive approach, capable of integrating multiple types of data to offer a richer and more nuanced understanding of sentiment. To address these challenges, Multimodal Sentiment Analysis (MSA) has emerged as a crucial advancement. In this article, the Enhanced LinkNet (EnLNet)-based Federated Learning (FL) approach is proposed for MSA. This approach utilizes an EnLNet model within an FL framework to manage and process multimodal data, including text, signals, and images. The EnLNet model in the FL framework is an advanced model by encompassing modified encoder and decoder block, interpolation blocks and a modified activation function. The approach is structured in three stages: the initialization stage, where global parameters are established and shared among clients; the local training stage, where several local models independently handle preprocessing, feature extraction, and fusion of text (with Improved Aspect Term Extraction (IATE) and Term Frequency-Inverse Document Frequency (TF-IDF)), signals (using Modified Mel Frequency Cepstral Coefficient (MMFCC) and spectral features), and images (through Improved Active Appearance Model (IAAM) and Median Binary Pattern (MBP)) before training the EnLNet model; and the model aggregation stage, where updates from local models are collected and aggregated by the central server to refine the global model. This iterative process lasts until convergence or the extreme number of iterations is attained. The efficiency of this approach is validated through Accuracy, Precision, FNR, FPR and performance comparisons with state-of-the-art approaches, demonstrating its capacity to enhance MSA by successfully incorporating and processing diverse multimodal data.https://ieeexplore.ieee.org/document/10758628/Multimodal sentiment analysisIATEMMFCCIAAMEnLNet model |
| spellingShingle | P. Vasanthi V. Madhu Viswanatham Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture IEEE Access Multimodal sentiment analysis IATE MMFCC IAAM EnLNet model |
| title | Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture |
| title_full | Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture |
| title_fullStr | Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture |
| title_full_unstemmed | Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture |
| title_short | Federated Learning for Multimodal Sentiment Analysis: Advancing Global Models With an Enhanced LinkNet Architecture |
| title_sort | federated learning for multimodal sentiment analysis advancing global models with an enhanced linknet architecture |
| topic | Multimodal sentiment analysis IATE MMFCC IAAM EnLNet model |
| url | https://ieeexplore.ieee.org/document/10758628/ |
| work_keys_str_mv | AT pvasanthi federatedlearningformultimodalsentimentanalysisadvancingglobalmodelswithanenhancedlinknetarchitecture AT vmadhuviswanatham federatedlearningformultimodalsentimentanalysisadvancingglobalmodelswithanenhancedlinknetarchitecture |