Patch is enough: naturalistic adversarial patch against vision-language pre-training models
Abstract Visual language pre-training (VLP) models have demonstrated significant success in various domains, but they remain vulnerable to adversarial attacks. Addressing these adversarial vulnerabilities is crucial for enhancing security in multi-modal learning. Traditionally, adversarial methods t...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2024-12-01
|
| Series: | Visual Intelligence |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s44267-024-00066-7 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850109782623518720 |
|---|---|
| author | Dehong Kong Siyuan Liang Xiaopeng Zhu Yuansheng Zhong Wenqi Ren |
| author_facet | Dehong Kong Siyuan Liang Xiaopeng Zhu Yuansheng Zhong Wenqi Ren |
| author_sort | Dehong Kong |
| collection | DOAJ |
| description | Abstract Visual language pre-training (VLP) models have demonstrated significant success in various domains, but they remain vulnerable to adversarial attacks. Addressing these adversarial vulnerabilities is crucial for enhancing security in multi-modal learning. Traditionally, adversarial methods that target VLP models involve simultaneous perturbation of images and text. However, this approach faces significant challenges. First, adversarial perturbations often fail to translate effectively into real-world scenarios. Second, direct modifications to the text are conspicuously visible. To overcome these limitations, we propose a novel strategy that uses only image patches for attacks, thus preserving the integrity of the original text. Our method leverages prior knowledge from diffusion models to enhance the authenticity and naturalness of the perturbations. Moreover, to optimize patch placement and improve the effectiveness of our attacks, we utilize the cross-attention mechanism, which encapsulates inter-modal interactions by generating attention maps to guide strategic patch placement. Extensive experiments conducted in a white-box setting for image-to-text scenarios reveal that our proposed method significantly outperforms existing techniques, achieving a 100% attack success rate. |
| format | Article |
| id | doaj-art-c87c92f554ec49888574afaf3ebf4399 |
| institution | OA Journals |
| issn | 2731-9008 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Springer |
| record_format | Article |
| series | Visual Intelligence |
| spelling | doaj-art-c87c92f554ec49888574afaf3ebf43992025-08-20T02:37:58ZengSpringerVisual Intelligence2731-90082024-12-012111010.1007/s44267-024-00066-7Patch is enough: naturalistic adversarial patch against vision-language pre-training modelsDehong Kong0Siyuan Liang1Xiaopeng Zhu2Yuansheng Zhong3Wenqi Ren4School of Cyber Science and TechnologySchool of Computing, National University of SingaporeGuangdong Testing Institute of Product Quality SupervisionGuangdong Testing Institute of Product Quality SupervisionSchool of Cyber Science and TechnologyAbstract Visual language pre-training (VLP) models have demonstrated significant success in various domains, but they remain vulnerable to adversarial attacks. Addressing these adversarial vulnerabilities is crucial for enhancing security in multi-modal learning. Traditionally, adversarial methods that target VLP models involve simultaneous perturbation of images and text. However, this approach faces significant challenges. First, adversarial perturbations often fail to translate effectively into real-world scenarios. Second, direct modifications to the text are conspicuously visible. To overcome these limitations, we propose a novel strategy that uses only image patches for attacks, thus preserving the integrity of the original text. Our method leverages prior knowledge from diffusion models to enhance the authenticity and naturalness of the perturbations. Moreover, to optimize patch placement and improve the effectiveness of our attacks, we utilize the cross-attention mechanism, which encapsulates inter-modal interactions by generating attention maps to guide strategic patch placement. Extensive experiments conducted in a white-box setting for image-to-text scenarios reveal that our proposed method significantly outperforms existing techniques, achieving a 100% attack success rate.https://doi.org/10.1007/s44267-024-00066-7Adversarial PatchPhysical AttackDiffusion ModelNaturalistic |
| spellingShingle | Dehong Kong Siyuan Liang Xiaopeng Zhu Yuansheng Zhong Wenqi Ren Patch is enough: naturalistic adversarial patch against vision-language pre-training models Visual Intelligence Adversarial Patch Physical Attack Diffusion Model Naturalistic |
| title | Patch is enough: naturalistic adversarial patch against vision-language pre-training models |
| title_full | Patch is enough: naturalistic adversarial patch against vision-language pre-training models |
| title_fullStr | Patch is enough: naturalistic adversarial patch against vision-language pre-training models |
| title_full_unstemmed | Patch is enough: naturalistic adversarial patch against vision-language pre-training models |
| title_short | Patch is enough: naturalistic adversarial patch against vision-language pre-training models |
| title_sort | patch is enough naturalistic adversarial patch against vision language pre training models |
| topic | Adversarial Patch Physical Attack Diffusion Model Naturalistic |
| url | https://doi.org/10.1007/s44267-024-00066-7 |
| work_keys_str_mv | AT dehongkong patchisenoughnaturalisticadversarialpatchagainstvisionlanguagepretrainingmodels AT siyuanliang patchisenoughnaturalisticadversarialpatchagainstvisionlanguagepretrainingmodels AT xiaopengzhu patchisenoughnaturalisticadversarialpatchagainstvisionlanguagepretrainingmodels AT yuanshengzhong patchisenoughnaturalisticadversarialpatchagainstvisionlanguagepretrainingmodels AT wenqiren patchisenoughnaturalisticadversarialpatchagainstvisionlanguagepretrainingmodels |