Enabling fine-grained permissions for augmented reality applications with recognizers

TLDR: This work proposes a fine-grained permission system where applications request permissions at the granularity of recognizer objects, and builds a prototype on Windows that exposes nine recognizers to applications, including the Kinect skeleton tracker.

Abstract: Augmented reality (AR) applications sense the environment, then render virtual objects on human senses. Examples include smartphone applications that annotate storefronts with reviews and XBox Kinect games that show "avatars" mimicking human movements. No current OS has special support for such applications. As a result, permissions for AR applications are necessarily coarse-grained: applications must ask for access to raw sensor feeds, such as video and audio. These raw feeds expose significant additional information beyond what applications need, including sensitive information such as the user's location, face, or surroundings. Instead of exposing raw sensor data to applications directly, we introduce a new OS abstraction: the recognizer. A recognizer takes raw sensor data as input and exposes higher-level objects, such as a skeleton or a face, to applications. We propose a fine-grained permission system where applications request permissions at the granularity of recognizer objects. We analyze 87 shipping AR applications and find that a set of four core recognizers covers almost all current apps. We also introduce privacy goggles, a visualization of sensitive data exposed to an application. Surveys of 962 people establish a clear "privacy ordering" over recognizers and demonstrate that privacy goggles are effective at communicating application capabilities. We build a prototype on Windows that exposes nine recognizers to applications, including the Kinect skeleton tracker. Our prototype incurs negligible overhead for single applications, while improving performance of concurrent applications and enabling secure offloading of heavyweight recognizer computation.