Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy
Text-to-image generation is trending in the generative artificial intelligence (GenAI) field. Among open-sourced image generation projects, Stable Diffusion is the state-of-the-art. Many artists and service providers customize the diffusion model to generate featured high-quality images. However, th...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10971394/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849323785319088128 |
|---|---|
| author | Po-Chu Hsu Ziying Yu Shuhei Mise Hideaki Miyaji |
| author_facet | Po-Chu Hsu Ziying Yu Shuhei Mise Hideaki Miyaji |
| author_sort | Po-Chu Hsu |
| collection | DOAJ |
| description | Text-to-image generation is trending in the generative artificial intelligence (GenAI) field. Among open-sourced image generation projects, Stable Diffusion is the state-of-the-art. Many artists and service providers customize the diffusion model to generate featured high-quality images. However, there is no protection to the privacy of the input text prompt, output image, and customized model. Privacy is very important since it can increase users’ willingness to use the service and protect the service provider’s intellectual property. Existing privacy-preserving diffusion model require fully homomorphic encryption (FHE) to ensure its privacy and security. Nonetheless, FHE is very time-consuming and may reduce accuracy due to approximations and deteriorate image quality. In this research, we propose Privacy-Diffusion, a privacy-preserving diffusion framework without FHE. By utilizing the irreversible property of neural network layers and the property that the predicted noise in the diffusion process is a normalized Gaussian distribution. Our framework can be applied to all kinds of diffusion models to protect clients’ input text prompt and the generated image from being learned by the server, as well as customized models from being learned by the clients. Our protocol is secure and efficient. Compared with existing research, HE-diffusion, which spent 200% extra time and visible quality loss, our protocol can reach the same security level with only 19% extra time and has no quality loss. To the best of our knowledge, our Privacy-Diffusion is the first protocol that achieves this goal without using FHE and maintain the same high-quality image output as the original model. |
| format | Article |
| id | doaj-art-01d58bdd8534482bbc415026fa7c4dab |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-01d58bdd8534482bbc415026fa7c4dab2025-08-20T03:48:57ZengIEEEIEEE Access2169-35362025-01-0113751947520310.1109/ACCESS.2025.356256310971394Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential PrivacyPo-Chu Hsu0https://orcid.org/0000-0002-0017-1657Ziying Yu1https://orcid.org/0009-0003-3706-9765Shuhei Mise2https://orcid.org/0009-0009-4022-7350Hideaki Miyaji3https://orcid.org/0000-0002-4182-8141Animechain.ai Inc., Tokyo, JapanAmazon, Irvine, CA, USAAnimechain.ai Inc., Tokyo, JapanDepartment of Information Science and Engineering, Ritsumeikan University, Kusatsu, JapanText-to-image generation is trending in the generative artificial intelligence (GenAI) field. Among open-sourced image generation projects, Stable Diffusion is the state-of-the-art. Many artists and service providers customize the diffusion model to generate featured high-quality images. However, there is no protection to the privacy of the input text prompt, output image, and customized model. Privacy is very important since it can increase users’ willingness to use the service and protect the service provider’s intellectual property. Existing privacy-preserving diffusion model require fully homomorphic encryption (FHE) to ensure its privacy and security. Nonetheless, FHE is very time-consuming and may reduce accuracy due to approximations and deteriorate image quality. In this research, we propose Privacy-Diffusion, a privacy-preserving diffusion framework without FHE. By utilizing the irreversible property of neural network layers and the property that the predicted noise in the diffusion process is a normalized Gaussian distribution. Our framework can be applied to all kinds of diffusion models to protect clients’ input text prompt and the generated image from being learned by the server, as well as customized models from being learned by the clients. Our protocol is secure and efficient. Compared with existing research, HE-diffusion, which spent 200% extra time and visible quality loss, our protocol can reach the same security level with only 19% extra time and has no quality loss. To the best of our knowledge, our Privacy-Diffusion is the first protocol that achieves this goal without using FHE and maintain the same high-quality image output as the original model.https://ieeexplore.ieee.org/document/10971394/AI securityprivacy MLstable diffusiongenerative AI |
| spellingShingle | Po-Chu Hsu Ziying Yu Shuhei Mise Hideaki Miyaji Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy IEEE Access AI security privacy ML stable diffusion generative AI |
| title | Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy |
| title_full | Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy |
| title_fullStr | Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy |
| title_full_unstemmed | Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy |
| title_short | Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy |
| title_sort | privacy diffusion privacy preserving stable diffusion without fhe and differential privacy |
| topic | AI security privacy ML stable diffusion generative AI |
| url | https://ieeexplore.ieee.org/document/10971394/ |
| work_keys_str_mv | AT pochuhsu privacydiffusionprivacypreservingstablediffusionwithoutfheanddifferentialprivacy AT ziyingyu privacydiffusionprivacypreservingstablediffusionwithoutfheanddifferentialprivacy AT shuheimise privacydiffusionprivacypreservingstablediffusionwithoutfheanddifferentialprivacy AT hideakimiyaji privacydiffusionprivacypreservingstablediffusionwithoutfheanddifferentialprivacy |