Attention and engagement-awareness in the wild: A large-scale study with adaptive notifications

TLDR: The results show that in most cases delaying the notification delivery until an interruptible moment is detected is beneficial to users and results in significant reduction of user response time compared to delivering the notifications immediately.

Abstract: In today's advancing ubiquitous computing age, with its ever-increasing amount of information from various applications and services available for consumption, the management of people's attention has become very important. In particular, the high volume of notifications on mobile devices has become a major cause of interruption of users. There has been much research aimed at detecting the opportune moment to present such information to users with in a way that lowers the cognitive load or frustration. However, evaluation of such systems in the real-world production environment with real users and notifications, and evaluation on user's engagement to the presented notification beyond simple responsiveness have not been adequately studied. To the best of our knowledge, this study is the first to investigate user interruptibility and engagement using a real-world large-scale mobile application and real-world notifications consisting of actual news content. We equipped the Yahoo! JAPAN Android app, one of the most popular applications on the national market, with our mobile-sensing and machine-learning-based interruptibility estimation logic. We conducted a large-scale in-the-wild user study with more than 680,000 users for three weeks. The results show that in most cases delaying the notification delivery until an interruptible moment is detected is beneficial to users and results in significant reduction of user response time (49.7%) compared to delivering the notifications immediately. We also observed a higher number of notifications opened in our system as well as constant improvement in user engagement levels throughout the entire study period.