Machine vision-based testing action recognition method for robotic testing of mobile application
The explosive growth and rapid version iteration of various mobile applications have brought enormous workloads to mobile application testing. Robotic testing methods can efficiently handle repetitive testing tasks, which can compensate for the accuracy of manual testing and improve the efficiency o...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2022-08-01
|
| Series: | International Journal of Distributed Sensor Networks |
| Online Access: | https://doi.org/10.1177/15501329221115375 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849435249025482752 |
|---|---|
| author | Tao Zhang Zhengqi Su Jing Cheng Feng Xue Shengyu Liu |
| author_facet | Tao Zhang Zhengqi Su Jing Cheng Feng Xue Shengyu Liu |
| author_sort | Tao Zhang |
| collection | DOAJ |
| description | The explosive growth and rapid version iteration of various mobile applications have brought enormous workloads to mobile application testing. Robotic testing methods can efficiently handle repetitive testing tasks, which can compensate for the accuracy of manual testing and improve the efficiency of testing work. Vision-based robotic testing identifies the types of test actions by analyzing expert test videos and generates expert imitation test cases. The mobile application expert imitation testing method uses machine learning algorithms to analyze the behavior of experts imitating test videos, generates test cases with high reliability and reusability, and drives robots to execute test cases. However, the difficulty of estimating multi-dimensional gestures in 2D images leads to complex algorithm steps, including tracking, detection, and recognition of dynamic gestures. Hence, this article focuses on the analysis and recognition of test actions in mobile application robot testing. Combined with the improved YOLOv5 algorithm and the ResNet-152 algorithm, a visual modeling method of mobile application test action based on machine vision is proposed. The precise localization of the hand is accomplished by injecting dynamic anchors, attention mechanism, and the weighted boxes fusion in the YOLOv5 algorithm. The improved algorithm recognition accuracy increased from 82.6% to 94.8%. By introducing the pyramid context awareness mechanism into the ResNet-152 algorithm, the accuracy of test action classification is improved. The accuracy of the test action classification was improved from 72.57% to 76.84%. Experiments show that this method can reduce the probability of multiple detections and missed detection of test actions, and improve the accuracy of test action recognition. |
| format | Article |
| id | doaj-art-4f121aeb71f042adbdc07c806978670b |
| institution | Kabale University |
| issn | 1550-1477 |
| language | English |
| publishDate | 2022-08-01 |
| publisher | Wiley |
| record_format | Article |
| series | International Journal of Distributed Sensor Networks |
| spelling | doaj-art-4f121aeb71f042adbdc07c806978670b2025-08-20T03:26:21ZengWileyInternational Journal of Distributed Sensor Networks1550-14772022-08-011810.1177/15501329221115375Machine vision-based testing action recognition method for robotic testing of mobile applicationTao Zhang0Zhengqi Su1Jing Cheng2Feng Xue3Shengyu Liu4School of Software, Northwestern Polytechnical University, Xi’an, ChinaSchool of Software, Northwestern Polytechnical University, Xi’an, ChinaSchool of Computer Science and Engineering, Xi’an Technological University, Xi’an, ChinaSchool of Software, Northwestern Polytechnical University, Xi’an, ChinaSchool of Software, Northwestern Polytechnical University, Xi’an, ChinaThe explosive growth and rapid version iteration of various mobile applications have brought enormous workloads to mobile application testing. Robotic testing methods can efficiently handle repetitive testing tasks, which can compensate for the accuracy of manual testing and improve the efficiency of testing work. Vision-based robotic testing identifies the types of test actions by analyzing expert test videos and generates expert imitation test cases. The mobile application expert imitation testing method uses machine learning algorithms to analyze the behavior of experts imitating test videos, generates test cases with high reliability and reusability, and drives robots to execute test cases. However, the difficulty of estimating multi-dimensional gestures in 2D images leads to complex algorithm steps, including tracking, detection, and recognition of dynamic gestures. Hence, this article focuses on the analysis and recognition of test actions in mobile application robot testing. Combined with the improved YOLOv5 algorithm and the ResNet-152 algorithm, a visual modeling method of mobile application test action based on machine vision is proposed. The precise localization of the hand is accomplished by injecting dynamic anchors, attention mechanism, and the weighted boxes fusion in the YOLOv5 algorithm. The improved algorithm recognition accuracy increased from 82.6% to 94.8%. By introducing the pyramid context awareness mechanism into the ResNet-152 algorithm, the accuracy of test action classification is improved. The accuracy of the test action classification was improved from 72.57% to 76.84%. Experiments show that this method can reduce the probability of multiple detections and missed detection of test actions, and improve the accuracy of test action recognition.https://doi.org/10.1177/15501329221115375 |
| spellingShingle | Tao Zhang Zhengqi Su Jing Cheng Feng Xue Shengyu Liu Machine vision-based testing action recognition method for robotic testing of mobile application International Journal of Distributed Sensor Networks |
| title | Machine vision-based testing action recognition method for robotic testing of mobile application |
| title_full | Machine vision-based testing action recognition method for robotic testing of mobile application |
| title_fullStr | Machine vision-based testing action recognition method for robotic testing of mobile application |
| title_full_unstemmed | Machine vision-based testing action recognition method for robotic testing of mobile application |
| title_short | Machine vision-based testing action recognition method for robotic testing of mobile application |
| title_sort | machine vision based testing action recognition method for robotic testing of mobile application |
| url | https://doi.org/10.1177/15501329221115375 |
| work_keys_str_mv | AT taozhang machinevisionbasedtestingactionrecognitionmethodforrobotictestingofmobileapplication AT zhengqisu machinevisionbasedtestingactionrecognitionmethodforrobotictestingofmobileapplication AT jingcheng machinevisionbasedtestingactionrecognitionmethodforrobotictestingofmobileapplication AT fengxue machinevisionbasedtestingactionrecognitionmethodforrobotictestingofmobileapplication AT shengyuliu machinevisionbasedtestingactionrecognitionmethodforrobotictestingofmobileapplication |