Research Article
GUI2DSVec: Detecting Visual Design Smells Based on Semantic Embedding of GUI Images and Components
@INPROCEEDINGS{10.1007/978-3-031-63992-0_29, author={Bo Yang and Shanping Li}, title={GUI2DSVec: Detecting Visual Design Smells Based on Semantic Embedding of GUI Images and Components}, proceedings={Mobile and Ubiquitous Systems: Computing, Networking and Services. 20th EAI International Conference, MobiQuitous 2023, Melbourne, VIC, Australia, November 14--17, 2023, Proceedings, Part II}, proceedings_a={MOBIQUITOUS PART 2}, year={2024}, month={7}, keywords={Mobile GUI testing GUI representation Violation detection}, doi={10.1007/978-3-031-63992-0_29} }- Bo Yang
Shanping Li
Year: 2024
GUI2DSVec: Detecting Visual Design Smells Based on Semantic Embedding of GUI Images and Components
MOBIQUITOUS PART 2
Springer
DOI: 10.1007/978-3-031-63992-0_29
Abstract
Visual graphical user interface testing (VGT) can simulate end-user behavior on essentially any device, and data-driven VGT approaches can automatically discern certain visual design imperfections. The visual information representing GUI images and the semantic information of components are crucial for data-driven design defect detection methodologies. The GUI representation techniques in the existing VGT approaches are incapable of unifying multimodal semantic information such as text content, screen images, metadata, et al., but require subsequent manual execution and implementation. To address the dual issues, this paper presents GUI2DSVec (GUI to Design Smell Vector), a novel GUI representation method that automatically extracts semantic information from GUI images and components to embedding vectors without manual exertion. By representing image and component metadata into vector space, this study introduces an automated end-to-end mobile application design smell detection method based on GUI representation. Performance validation on 64,759 real Android UIs demonstrates GUI2DSVec’s efficiency in detecting visual design smells (accuracy@20 of 0.77). Additionally, the paper analyzes the impact of discrete representation modules and model variations on the performance of the smell detection method.
