Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP
Digital media art has a wide application in the field of image caption generation. In digital media art exhibitions or online works displays, some complex image works may have multiple layers of meanings or abstract expressions, which can help viewers better understand the works. It can also serve a...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10979925/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849722212317134848 |
|---|---|
| author | Lu Gao Xiaofei Pang |
| author_facet | Lu Gao Xiaofei Pang |
| author_sort | Lu Gao |
| collection | DOAJ |
| description | Digital media art has a wide application in the field of image caption generation. In digital media art exhibitions or online works displays, some complex image works may have multiple layers of meanings or abstract expressions, which can help viewers better understand the works. It can also serve as another auxiliary element besides sound, collaborating with visual elements to provide a richer experience for the audience. The purpose of picture captioning is to provide textual descriptions that correlate to input images. The CLIP paradigm is highly versatile to resolve vision-text difficulties. In the field of picture description, the standard Transformer architecture has also exhibited good effects, which uses an image encoder and a text decoder. Large parameter numbers and the demand for further data preprocessing are still significant difficulties. In order to replace the fundamental features of conventional multi-modal fusion models, we propose a New Multi-modal Fusion Attention module (NMFA), which efficiently decreases parameter sizes and computational complexity in half. Expanding upon this, we propose the Transformer Fusion CLIP (TFC) model, which minimizes parameter sizes and processing demands while getting remarkable assessment scores. Additionally, we strengthen the mechanism for cumulative points and reward sequence length to encourage the construction of larger sequences. Finally, we combine the enhanced beam search technique to further train the TFC model. Results from our testing on the MSCOCO dataset reveal that we have not only greatly improved the efficiency of the TFC model but also speeded up its runtime by eight times and reduced model parameters by over 50%. |
| format | Article |
| id | doaj-art-024be486ed944325ba1fb219baf91fa8 |
| institution | DOAJ |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-024be486ed944325ba1fb219baf91fa82025-08-20T03:11:25ZengIEEEIEEE Access2169-35362025-01-0113758947591010.1109/ACCESS.2025.356568210979925Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIPLu Gao0https://orcid.org/0009-0000-2243-2822Xiaofei Pang1https://orcid.org/0009-0002-8850-4352School of Art Design and Creativity, Chengdu Textile College, Chengdu, Sichuan, ChinaSchool of Information Engineering, Bingtuan Xingxin Vocational and Technical College, Tiemenguan City, ChinaDigital media art has a wide application in the field of image caption generation. In digital media art exhibitions or online works displays, some complex image works may have multiple layers of meanings or abstract expressions, which can help viewers better understand the works. It can also serve as another auxiliary element besides sound, collaborating with visual elements to provide a richer experience for the audience. The purpose of picture captioning is to provide textual descriptions that correlate to input images. The CLIP paradigm is highly versatile to resolve vision-text difficulties. In the field of picture description, the standard Transformer architecture has also exhibited good effects, which uses an image encoder and a text decoder. Large parameter numbers and the demand for further data preprocessing are still significant difficulties. In order to replace the fundamental features of conventional multi-modal fusion models, we propose a New Multi-modal Fusion Attention module (NMFA), which efficiently decreases parameter sizes and computational complexity in half. Expanding upon this, we propose the Transformer Fusion CLIP (TFC) model, which minimizes parameter sizes and processing demands while getting remarkable assessment scores. Additionally, we strengthen the mechanism for cumulative points and reward sequence length to encourage the construction of larger sequences. Finally, we combine the enhanced beam search technique to further train the TFC model. Results from our testing on the MSCOCO dataset reveal that we have not only greatly improved the efficiency of the TFC model but also speeded up its runtime by eight times and reduced model parameters by over 50%.https://ieeexplore.ieee.org/document/10979925/Digital media artimage captioningbeam search algorithmtransformer fusion CLIP (TFC) modelnew multi-modal fusion attention module (NMFA) |
| spellingShingle | Lu Gao Xiaofei Pang Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP IEEE Access Digital media art image captioning beam search algorithm transformer fusion CLIP (TFC) model new multi-modal fusion attention module (NMFA) |
| title | Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP |
| title_full | Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP |
| title_fullStr | Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP |
| title_full_unstemmed | Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP |
| title_short | Research on Digital Media Art for Image Caption Generation Based on Integrated Transformer Models in CLIP |
| title_sort | research on digital media art for image caption generation based on integrated transformer models in clip |
| topic | Digital media art image captioning beam search algorithm transformer fusion CLIP (TFC) model new multi-modal fusion attention module (NMFA) |
| url | https://ieeexplore.ieee.org/document/10979925/ |
| work_keys_str_mv | AT lugao researchondigitalmediaartforimagecaptiongenerationbasedonintegratedtransformermodelsinclip AT xiaofeipang researchondigitalmediaartforimagecaptiongenerationbasedonintegratedtransformermodelsinclip |