A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2
Accurate segmentation of cellular structures in whole slide images (WSIs) is essential for quantitative analysis in computational pathology. However, the complexity and scale of WSIs present significant challenges for conventional segmentation methods. In this study, we propose a novel hybrid deep l...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-06-01
|
| Series: | Bioengineering |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2306-5354/12/6/674 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849467425227014144 |
|---|---|
| author | Julius Bamwenda Mehmet Siraç Özerdem Orhan Ayyıldız Veysı Akpolat |
| author_facet | Julius Bamwenda Mehmet Siraç Özerdem Orhan Ayyıldız Veysı Akpolat |
| author_sort | Julius Bamwenda |
| collection | DOAJ |
| description | Accurate segmentation of cellular structures in whole slide images (WSIs) is essential for quantitative analysis in computational pathology. However, the complexity and scale of WSIs present significant challenges for conventional segmentation methods. In this study, we propose a novel hybrid deep learning framework that integrates three complementary approaches, YOLOv11, StarDist, and Segment Anything Model v2 (SAM2), to achieve robust and precise cell segmentation. The proposed pipeline utilizes YOLOv11 as an object detector to localize regions of interest, generating bounding boxes or preliminary masks that are subsequently used either as prompts to guide SAM2 or to filter segmentation outputs. StarDist is employed to model cell and nuclear boundaries with high geometric precision using star-convex polygon representations, which are particularly effective in densely packed cellular regions. The framework was evaluated on a unique WSI dataset comprising 256 × 256 image tiles annotated with high-resolution cell-level masks. Quantitative evaluations using the Dice coefficient, intersection over union (IoU), F1-score, precision, and recall demonstrated that the proposed method significantly outperformed individual baseline models. The integration of object detection and prompt-based segmentation led to enhanced boundary accuracy, improved localization, and greater robustness across varied tissue types. This work contributes a scalable and modular solution for advancing automated histopathological image analysis. |
| format | Article |
| id | doaj-art-2a23a7118729471fa3ac9f59dd42e7a4 |
| institution | Kabale University |
| issn | 2306-5354 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Bioengineering |
| spelling | doaj-art-2a23a7118729471fa3ac9f59dd42e7a42025-08-20T03:26:15ZengMDPI AGBioengineering2306-53542025-06-0112667410.3390/bioengineering12060674A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2Julius Bamwenda0Mehmet Siraç Özerdem1Orhan Ayyıldız2Veysı Akpolat3Engineering Faculty, Electrical & Electronics Engineering Department, Dicle University, 21280 Diyarbakır, TürkiyeEngineering Faculty, Electrical & Electronics Engineering Department, Dicle University, 21280 Diyarbakır, TürkiyeMedical Faculty, Department of Internal Medicine-Hematology, Dicle University, 21280 Diyarbakır, TürkiyeMedical Faculty, Department of Biophysics, Dicle University, 21280 Diyarbakır, TürkiyeAccurate segmentation of cellular structures in whole slide images (WSIs) is essential for quantitative analysis in computational pathology. However, the complexity and scale of WSIs present significant challenges for conventional segmentation methods. In this study, we propose a novel hybrid deep learning framework that integrates three complementary approaches, YOLOv11, StarDist, and Segment Anything Model v2 (SAM2), to achieve robust and precise cell segmentation. The proposed pipeline utilizes YOLOv11 as an object detector to localize regions of interest, generating bounding boxes or preliminary masks that are subsequently used either as prompts to guide SAM2 or to filter segmentation outputs. StarDist is employed to model cell and nuclear boundaries with high geometric precision using star-convex polygon representations, which are particularly effective in densely packed cellular regions. The framework was evaluated on a unique WSI dataset comprising 256 × 256 image tiles annotated with high-resolution cell-level masks. Quantitative evaluations using the Dice coefficient, intersection over union (IoU), F1-score, precision, and recall demonstrated that the proposed method significantly outperformed individual baseline models. The integration of object detection and prompt-based segmentation led to enhanced boundary accuracy, improved localization, and greater robustness across varied tissue types. This work contributes a scalable and modular solution for advancing automated histopathological image analysis.https://www.mdpi.com/2306-5354/12/6/674WSIcell segmentationSAM2deep learning |
| spellingShingle | Julius Bamwenda Mehmet Siraç Özerdem Orhan Ayyıldız Veysı Akpolat A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2 Bioengineering WSI cell segmentation SAM2 deep learning |
| title | A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2 |
| title_full | A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2 |
| title_fullStr | A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2 |
| title_full_unstemmed | A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2 |
| title_short | A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2 |
| title_sort | hybrid deep learning framework for accurate cell segmentation in whole slide images using yolov11 stardist and sam2 |
| topic | WSI cell segmentation SAM2 deep learning |
| url | https://www.mdpi.com/2306-5354/12/6/674 |
| work_keys_str_mv | AT juliusbamwenda ahybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT mehmetsiracozerdem ahybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT orhanayyıldız ahybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT veysıakpolat ahybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT juliusbamwenda hybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT mehmetsiracozerdem hybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT orhanayyıldız hybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 AT veysıakpolat hybriddeeplearningframeworkforaccuratecellsegmentationinwholeslideimagesusingyolov11stardistandsam2 |