Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images
As the variety and quantity of goods in modern warehouse management continue to increase, optimizing space utilization and ensuring the safe and orderly storage of goods have become critical challenges. High-rise shelving systems are increasingly favored by enterprises, but long-term use, collisions...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10854213/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832540564343488512 |
|---|---|
| author | Hong Yan Jingjing Fan Yajun Liu |
| author_facet | Hong Yan Jingjing Fan Yajun Liu |
| author_sort | Hong Yan |
| collection | DOAJ |
| description | As the variety and quantity of goods in modern warehouse management continue to increase, optimizing space utilization and ensuring the safe and orderly storage of goods have become critical challenges. High-rise shelving systems are increasingly favored by enterprises, but long-term use, collisions with stacker cranes, and overloading can lead to structural deformation of the shelves. If these deformations are not detected and addressed in a timely manner, they may result in serious safety incidents and significant property damage. To address this issue, this study proposes a zero-shot shelf deformation detection method based on multimodal data fusion. The proposed approach integrates Micro-Electro-Mechanical Systems (MEMS) sensors and image data to establish a real-time monitoring and alert mechanism. Specifically, MEMS sensors are employed for real-time acquisition of shelf status, with threshold values set to trigger an initial alert mechanism. Simultaneously, cameras capture shelf images, and multiple You Only Look Once (YOLO) models are used to detect and classify critical components of the shelf, such as beams and columns. YOLOv11n is ultimately selected as the optimal model for detecting these structural elements. Based on the detected beams and columns, further feature extraction is performed, and the sensor data is fused with these features. A K-Means clustering algorithm is then applied to conduct the clustering analysis. To address the issue of a lack of negative samples in the dataset, the study employs oversampling techniques, including SMOTE, ADASYN, and Borderline-SMOTE, combined with machine learning models such as Random Forest and Gradient Boosting Decision Trees (GBDT). The experimental results demonstrate that both Random Forest and GBDT achieved precision, recall, and F1 scores exceeding 95%, confirming the effectiveness and accuracy of the proposed method in shelf deformation detection. The multimodal detection method proposed in this study not only improves the accuracy and real-time performance of shelf deformation detection but also provides strong technical support for the safety management of warehouse operations. |
| format | Article |
| id | doaj-art-d561f6ef350c41ed9bf2acb15a1dd924 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-d561f6ef350c41ed9bf2acb15a1dd9242025-02-05T00:01:08ZengIEEEIEEE Access2169-35362025-01-0113214862150210.1109/ACCESS.2025.353441110854213Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and ImagesHong Yan0Jingjing Fan1Yajun Liu2https://orcid.org/0000-0002-2698-9473Logistics Center, Zhangjiakou Cigarette Factory Company Ltd., Zhangjiakou, Hebei, ChinaInformation Engineering College, Hebei University of Architecture, Zhangjiakou, Hebei, ChinaInformation Engineering College, Hebei University of Architecture, Zhangjiakou, Hebei, ChinaAs the variety and quantity of goods in modern warehouse management continue to increase, optimizing space utilization and ensuring the safe and orderly storage of goods have become critical challenges. High-rise shelving systems are increasingly favored by enterprises, but long-term use, collisions with stacker cranes, and overloading can lead to structural deformation of the shelves. If these deformations are not detected and addressed in a timely manner, they may result in serious safety incidents and significant property damage. To address this issue, this study proposes a zero-shot shelf deformation detection method based on multimodal data fusion. The proposed approach integrates Micro-Electro-Mechanical Systems (MEMS) sensors and image data to establish a real-time monitoring and alert mechanism. Specifically, MEMS sensors are employed for real-time acquisition of shelf status, with threshold values set to trigger an initial alert mechanism. Simultaneously, cameras capture shelf images, and multiple You Only Look Once (YOLO) models are used to detect and classify critical components of the shelf, such as beams and columns. YOLOv11n is ultimately selected as the optimal model for detecting these structural elements. Based on the detected beams and columns, further feature extraction is performed, and the sensor data is fused with these features. A K-Means clustering algorithm is then applied to conduct the clustering analysis. To address the issue of a lack of negative samples in the dataset, the study employs oversampling techniques, including SMOTE, ADASYN, and Borderline-SMOTE, combined with machine learning models such as Random Forest and Gradient Boosting Decision Trees (GBDT). The experimental results demonstrate that both Random Forest and GBDT achieved precision, recall, and F1 scores exceeding 95%, confirming the effectiveness and accuracy of the proposed method in shelf deformation detection. The multimodal detection method proposed in this study not only improves the accuracy and real-time performance of shelf deformation detection but also provides strong technical support for the safety management of warehouse operations.https://ieeexplore.ieee.org/document/10854213/Shelf deformation detectionMEMS sensorsYOLO modelzero-shot learning |
| spellingShingle | Hong Yan Jingjing Fan Yajun Liu Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images IEEE Access Shelf deformation detection MEMS sensors YOLO model zero-shot learning |
| title | Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images |
| title_full | Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images |
| title_fullStr | Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images |
| title_full_unstemmed | Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images |
| title_short | Multimodal Zero-Shot Shelf Deformation Detection Based on MEMS Sensors and Images |
| title_sort | multimodal zero shot shelf deformation detection based on mems sensors and images |
| topic | Shelf deformation detection MEMS sensors YOLO model zero-shot learning |
| url | https://ieeexplore.ieee.org/document/10854213/ |
| work_keys_str_mv | AT hongyan multimodalzeroshotshelfdeformationdetectionbasedonmemssensorsandimages AT jingjingfan multimodalzeroshotshelfdeformationdetectionbasedonmemssensorsandimages AT yajunliu multimodalzeroshotshelfdeformationdetectionbasedonmemssensorsandimages |