Research Article
Detection and Segmentation of Graphical Elements on GUIs for Mobile Apps Based on Deep Learning
2 downloads
@INPROCEEDINGS{10.1007/978-3-030-64214-3_13, author={Rui Hu and Mingang Chen and Lizhi Cai and Wenjie Chen}, title={Detection and Segmentation of Graphical Elements on GUIs for Mobile Apps Based on Deep Learning}, proceedings={Mobile Computing, Applications, and Services. 11th EAI International Conference, MobiCASE 2020, Shanghai, China, September 12, 2020, Proceedings}, proceedings_a={MOBICASE}, year={2020}, month={12}, keywords={Mobile apps GUI dataset Instance segmentation Mask R-CNN}, doi={10.1007/978-3-030-64214-3_13} }- Rui Hu
Mingang Chen
Lizhi Cai
Wenjie Chen
Year: 2020
Detection and Segmentation of Graphical Elements on GUIs for Mobile Apps Based on Deep Learning
MOBICASE
Springer
DOI: 10.1007/978-3-030-64214-3_13
Abstract
Recently, mobile devices are more popular than computers. However, mobile apps are not as thoroughly tested as desktop ones, especially for graphical user interface (GUI). In this paper, we study the detection and segmentation of graphical elements on GUIs for mobile apps based on deep learning. It is the preliminary work of GUI testing for mobile apps based on artificial intelligence. We create a dataset, which consists of 2,100 GUI screenshots (or pages) labeled with 42,156 graphic elements in 8 classes. Based on our dataset, we adopt Mask R-CNN to train the detection and segmentation of graphic elements on GUI screenshots. The experimental results show that the mAP value achieves 98%.
Copyright © 2020–2025 ICST
