Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images
Exploring the relationships between plant phenotypes and genetic information requires advanced phenotypic analysis techniques for precise characterization. However, the diversity and variability of plant morphology challenge existing methods, which often fail to generalize across species and require...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/25/2/526 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832587517526802432 |
|---|---|
| author | Yunlong Wang Zhiyong Zhang |
| author_facet | Yunlong Wang Zhiyong Zhang |
| author_sort | Yunlong Wang |
| collection | DOAJ |
| description | Exploring the relationships between plant phenotypes and genetic information requires advanced phenotypic analysis techniques for precise characterization. However, the diversity and variability of plant morphology challenge existing methods, which often fail to generalize across species and require extensive annotated data, especially for 3D datasets. This paper proposes a zero-shot 3D leaf instance segmentation method using RGB sensors. It extends the 2D segmentation model SAM (Segment Anything Model) to 3D through a multi-view strategy. RGB image sequences captured from multiple viewpoints are used to reconstruct 3D plant point clouds via multi-view stereo. HQ-SAM (High-Quality Segment Anything Model) segments leaves in 2D, and the segmentation is mapped to the 3D point cloud. An incremental fusion method based on confidence scores aggregates results from different views into a final output. Evaluated on a custom peanut seedling dataset, the method achieved point-level precision, recall, and F1 scores over 0.9 and object-level mIoU and precision above 0.75 under two IoU thresholds. The results show that the method achieves state-of-the-art segmentation quality while offering zero-shot capability and generalizability, demonstrating significant potential in plant phenotyping. |
| format | Article |
| id | doaj-art-a5402682a11740c2b589d4705b79b503 |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-a5402682a11740c2b589d4705b79b5032025-01-24T13:49:13ZengMDPI AGSensors1424-82202025-01-0125252610.3390/s25020526Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View ImagesYunlong Wang0Zhiyong Zhang1School of Electronic and Communication Engineering, Sun Yat-sen University, Shenzhen 518000, ChinaSchool of Electronic and Communication Engineering, Sun Yat-sen University, Shenzhen 518000, ChinaExploring the relationships between plant phenotypes and genetic information requires advanced phenotypic analysis techniques for precise characterization. However, the diversity and variability of plant morphology challenge existing methods, which often fail to generalize across species and require extensive annotated data, especially for 3D datasets. This paper proposes a zero-shot 3D leaf instance segmentation method using RGB sensors. It extends the 2D segmentation model SAM (Segment Anything Model) to 3D through a multi-view strategy. RGB image sequences captured from multiple viewpoints are used to reconstruct 3D plant point clouds via multi-view stereo. HQ-SAM (High-Quality Segment Anything Model) segments leaves in 2D, and the segmentation is mapped to the 3D point cloud. An incremental fusion method based on confidence scores aggregates results from different views into a final output. Evaluated on a custom peanut seedling dataset, the method achieved point-level precision, recall, and F1 scores over 0.9 and object-level mIoU and precision above 0.75 under two IoU thresholds. The results show that the method achieves state-of-the-art segmentation quality while offering zero-shot capability and generalizability, demonstrating significant potential in plant phenotyping.https://www.mdpi.com/1424-8220/25/2/526plant phenotypinginstance segmentationRGB sensorszero-shot segmentationmulti-view stereo |
| spellingShingle | Yunlong Wang Zhiyong Zhang Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images Sensors plant phenotyping instance segmentation RGB sensors zero-shot segmentation multi-view stereo |
| title | Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images |
| title_full | Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images |
| title_fullStr | Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images |
| title_full_unstemmed | Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images |
| title_short | Segment Any Leaf 3D: A Zero-Shot 3D Leaf Instance Segmentation Method Based on Multi-View Images |
| title_sort | segment any leaf 3d a zero shot 3d leaf instance segmentation method based on multi view images |
| topic | plant phenotyping instance segmentation RGB sensors zero-shot segmentation multi-view stereo |
| url | https://www.mdpi.com/1424-8220/25/2/526 |
| work_keys_str_mv | AT yunlongwang segmentanyleaf3dazeroshot3dleafinstancesegmentationmethodbasedonmultiviewimages AT zhiyongzhang segmentanyleaf3dazeroshot3dleafinstancesegmentationmethodbasedonmultiviewimages |