Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study
We introduce a weakly supervised segmentation approach that leverages class activation maps and the Segment Anything Model to generate high-quality masks using only classification data. A pre-trained classifier produces class activation maps that, once thresholded, yield bounding boxes encapsulating...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-02-01
|
| Series: | Machine Learning and Knowledge Extraction |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2504-4990/7/1/22 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850204976643571712 |
|---|---|
| author | Hanna Borgli Håkon Kvale Stensland Pål Halvorsen |
| author_facet | Hanna Borgli Håkon Kvale Stensland Pål Halvorsen |
| author_sort | Hanna Borgli |
| collection | DOAJ |
| description | We introduce a weakly supervised segmentation approach that leverages class activation maps and the Segment Anything Model to generate high-quality masks using only classification data. A pre-trained classifier produces class activation maps that, once thresholded, yield bounding boxes encapsulating the regions of interest. These boxes prompt the SAM to generate detailed segmentation masks, which are then refined by selecting the best overlap with automatically generated masks from the foundational model using the intersection over union metric. In a polyp segmentation case study, our approach outperforms existing zero-shot and weakly supervised methods, achieving a mean intersection over union of 0.63. This method offers an efficient and general solution for image segmentation tasks where segmentation data are scarce. |
| format | Article |
| id | doaj-art-c457f5057fc34ad79512430879bf1b85 |
| institution | OA Journals |
| issn | 2504-4990 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Machine Learning and Knowledge Extraction |
| spelling | doaj-art-c457f5057fc34ad79512430879bf1b852025-08-20T02:11:11ZengMDPI AGMachine Learning and Knowledge Extraction2504-49902025-02-01712210.3390/make7010022Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case StudyHanna Borgli0Håkon Kvale Stensland1Pål Halvorsen2Department of High-Performance Computing, Simula Research Laboratory, 0164 Oslo, NorwayDepartment of High-Performance Computing, Simula Research Laboratory, 0164 Oslo, NorwayDepartment of Holistic Systems, SimulaMet, 0170 Oslo, NorwayWe introduce a weakly supervised segmentation approach that leverages class activation maps and the Segment Anything Model to generate high-quality masks using only classification data. A pre-trained classifier produces class activation maps that, once thresholded, yield bounding boxes encapsulating the regions of interest. These boxes prompt the SAM to generate detailed segmentation masks, which are then refined by selecting the best overlap with automatically generated masks from the foundational model using the intersection over union metric. In a polyp segmentation case study, our approach outperforms existing zero-shot and weakly supervised methods, achieving a mean intersection over union of 0.63. This method offers an efficient and general solution for image segmentation tasks where segmentation data are scarce.https://www.mdpi.com/2504-4990/7/1/22image segmentationclass activation mapsegment anythingannotation toolszero-shot learningweakly supervised semantic segmentation |
| spellingShingle | Hanna Borgli Håkon Kvale Stensland Pål Halvorsen Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study Machine Learning and Knowledge Extraction image segmentation class activation map segment anything annotation tools zero-shot learning weakly supervised semantic segmentation |
| title | Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study |
| title_full | Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study |
| title_fullStr | Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study |
| title_full_unstemmed | Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study |
| title_short | Automatic Prompt Generation Using Class Activation Maps for Foundational Models: A Polyp Segmentation Case Study |
| title_sort | automatic prompt generation using class activation maps for foundational models a polyp segmentation case study |
| topic | image segmentation class activation map segment anything annotation tools zero-shot learning weakly supervised semantic segmentation |
| url | https://www.mdpi.com/2504-4990/7/1/22 |
| work_keys_str_mv | AT hannaborgli automaticpromptgenerationusingclassactivationmapsforfoundationalmodelsapolypsegmentationcasestudy AT hakonkvalestensland automaticpromptgenerationusingclassactivationmapsforfoundationalmodelsapolypsegmentationcasestudy AT palhalvorsen automaticpromptgenerationusingclassactivationmapsforfoundationalmodelsapolypsegmentationcasestudy |