Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models
Semantic communication is a promising technology for next-generation wireless networks. However, the out-of-distribution (OOD) problem, where a pre-trained machine learning (ML) model is applied to unseen tasks that are outside the distribution of its training data, may compromise the integrity of s...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Transactions on Machine Learning in Communications and Networking |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11113346/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849738008554635264 |
|---|---|
| author | Feifan Zhang Yuyang Du Kexin Chen Yulin Shao Soung Chang Liew |
| author_facet | Feifan Zhang Yuyang Du Kexin Chen Yulin Shao Soung Chang Liew |
| author_sort | Feifan Zhang |
| collection | DOAJ |
| description | Semantic communication is a promising technology for next-generation wireless networks. However, the out-of-distribution (OOD) problem, where a pre-trained machine learning (ML) model is applied to unseen tasks that are outside the distribution of its training data, may compromise the integrity of semantic compression. This paper explores the use of multi-modal large language models (MLLMs) to address the OOD issue in image semantic communication. We propose a novel “Plan A - Plan B” framework that leverages the broad knowledge and strong generalization ability of an MLLM to assist a conventional ML model when the latter encounters an OOD input in the semantic encoding process. Furthermore, we propose a Bayesian optimization scheme that reshapes the probability distribution of the MLLM’s inference process based on the contextual information of the image. The optimization scheme significantly enhances the MLLM’s performance in semantic compression by 1) filtering out irrelevant vocabulary in the original MLLM output; and 2) using contextual similarities between prospective answers of the MLLM and the background information as prior knowledge to modify the MLLM’s probability distribution during inference. Further, at the receiver side of the communication system, we put forth a “generate-criticize” framework that utilizes the cooperation of multiple MLLMs to enhance the reliability of image reconstruction. |
| format | Article |
| id | doaj-art-3f23e661038848ab8c36c331b42864dc |
| institution | DOAJ |
| issn | 2831-316X |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Transactions on Machine Learning in Communications and Networking |
| spelling | doaj-art-3f23e661038848ab8c36c331b42864dc2025-08-20T03:06:44ZengIEEEIEEE Transactions on Machine Learning in Communications and Networking2831-316X2025-01-013997101310.1109/TMLCN.2025.359584111113346Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language ModelsFeifan Zhang0https://orcid.org/0009-0009-4153-925XYuyang Du1https://orcid.org/0000-0001-9213-9875Kexin Chen2Yulin Shao3https://orcid.org/0000-0002-6300-3175Soung Chang Liew4https://orcid.org/0000-0001-7055-6483Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong, SAR, ChinaDepartment of Information Engineering, The Chinese University of Hong Kong, Hong Kong, SAR, ChinaDepartment of Computer Science Engineering, The Chinese University of Hong Kong, Hong Kong, SAR, ChinaDepartment of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, SAR, ChinaDepartment of Information Engineering, The Chinese University of Hong Kong, Hong Kong, SAR, ChinaSemantic communication is a promising technology for next-generation wireless networks. However, the out-of-distribution (OOD) problem, where a pre-trained machine learning (ML) model is applied to unseen tasks that are outside the distribution of its training data, may compromise the integrity of semantic compression. This paper explores the use of multi-modal large language models (MLLMs) to address the OOD issue in image semantic communication. We propose a novel “Plan A - Plan B” framework that leverages the broad knowledge and strong generalization ability of an MLLM to assist a conventional ML model when the latter encounters an OOD input in the semantic encoding process. Furthermore, we propose a Bayesian optimization scheme that reshapes the probability distribution of the MLLM’s inference process based on the contextual information of the image. The optimization scheme significantly enhances the MLLM’s performance in semantic compression by 1) filtering out irrelevant vocabulary in the original MLLM output; and 2) using contextual similarities between prospective answers of the MLLM and the background information as prior knowledge to modify the MLLM’s probability distribution during inference. Further, at the receiver side of the communication system, we put forth a “generate-criticize” framework that utilizes the cooperation of multiple MLLMs to enhance the reliability of image reconstruction.https://ieeexplore.ieee.org/document/11113346/Semantic communicationmulti-modal large language modelgenerative AIsout-of-distribution problem |
| spellingShingle | Feifan Zhang Yuyang Du Kexin Chen Yulin Shao Soung Chang Liew Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models IEEE Transactions on Machine Learning in Communications and Networking Semantic communication multi-modal large language model generative AIs out-of-distribution problem |
| title | Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models |
| title_full | Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models |
| title_fullStr | Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models |
| title_full_unstemmed | Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models |
| title_short | Out-of-Distribution in Image Semantic Communication: A Solution With Multimodal Large Language Models |
| title_sort | out of distribution in image semantic communication a solution with multimodal large language models |
| topic | Semantic communication multi-modal large language model generative AIs out-of-distribution problem |
| url | https://ieeexplore.ieee.org/document/11113346/ |
| work_keys_str_mv | AT feifanzhang outofdistributioninimagesemanticcommunicationasolutionwithmultimodallargelanguagemodels AT yuyangdu outofdistributioninimagesemanticcommunicationasolutionwithmultimodallargelanguagemodels AT kexinchen outofdistributioninimagesemanticcommunicationasolutionwithmultimodallargelanguagemodels AT yulinshao outofdistributioninimagesemanticcommunicationasolutionwithmultimodallargelanguagemodels AT soungchangliew outofdistributioninimagesemanticcommunicationasolutionwithmultimodallargelanguagemodels |