Privacy Protection Against Shoulder Surfing in Mobile Environments

Smartphones and other mobile devices have seen an unprecedented rise in use among consumers. These devices are widely used in public locations where traditional computers could hardly be accessed. Although such ubiquitous computing is desirable for users, the use of mobile devices in public locations has led to rising privacy concerns. Malicious individuals can easily glean personal data from a mobile device screen by visual eavesdropping without a user’s knowledge. In this paper, we propose two schemes to identify and protect private user data displayed on mobile device screens in public environments. The first scheme considers generic mobile applications’ complex user interfaces as an image, and uses a deep, convolutional object detection network to automatically identify sensitive content displayed by mobile applications. Such content is then blurred against shoulder surfing attacks. To allow users to identify custom fields in applications that they think should be hidden, we introduce methods for dynamic sample generation and model retraining that only need users to provide a small number of seed samples. The second scheme focuses on web applications due to the popularity of the web platform, and automates the detection and blurring of sensitive web fields through HTML (HyperText Markup Language) parsing and CSS (Cascading Style Sheets) style modification as showcased via a Chromium-based browser extension. Evaluations show the effectiveness of our schemes.