We designed and deployed automatic alt-text (AAT), a system that applies computer vision technology to identify faces, objects, and themes from photos to generate photo alt-text for screen reader users on Facebook. We designed our system through iterations of prototyping and in-lab user studies. Our lab test participants had a positive reaction to our system and an enhanced experience with Facebook photos. We also evaluated our system through a two-week field study as part of the Facebook iOS app for 9K VoiceOver users. We randomly assigned them into control and test groups and collected two weeks of activity data and their survey feedback. The test group reported that photos on Facebook were easier to interpret and more engaging, and found Facebook more useful in general. Our system demonstrates that artificial intelligence can be used to enhance the experience for visually impaired users on social networking sites (SNSs), while also revealing the challenges with designing automated assistive technology in a SNS context.

Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service

In this paper, we explore blind people's motivations, challenges, interactions, and experiences with visual content on Social Networking Services (SNSs). We present findings from an interview study of 11 individuals and a survey study of 60 individuals, all with little to no functional vision. Compared to sighted SNS users, our blind participants faced profound accessibility challenges, including the prevalence of photos without sufficient text descriptions. To overcome the challenges, they developed creative strategies, including using a variety of methods to access SNS features (e.g., opening the mobile site on a desktop browser), and inferring photo content from textual cues and social interactions. When strategies failed, participants reached out for help from trusted friends, or avoided certain features. We discuss our findings in the context of CSCW research and SNS accessibility as a design value. We highlight the social significance of photo interactions for blind people and suggest design practices.

How Blind People Interact with Visual Content on Social Networking Services

We describe the development and evaluation of a tool, iSonic, to assist users with visual impairment in exploring georeferenced data using coordinated maps and tables, augmented with nontextual sounds and speech output. Our in-depth case studies with 7 blind users during 42 hours of data collection, showed that iSonic enabled them to find facts and discover trends in georeferenced data, even in unfamiliar geographical contexts, without special devices. Our design was guided by an Action-by-Design-Component (ADC) framework, which was also applied to scatterplots to demonstrate its generalizability. Video and download is available at www.cs.umd.edu/hcil/iSonic/.

Data Sonification for Users with Visual Impairment: A Case Study with Georeferenced Data

Voice-activated personal assistants (VAPAs)--like Amazon Echo or Apple Siri--offer considerable promise to individuals who are blind due to widespread adoption of these non-visual interaction platforms. However, studies have yet to focus on the ways in which these technologies are used by individuals who are blind, along with whether barriers are encountered during the process of interaction. To address this gap, we interviewed fourteen legally-blind adults with experience of home and/or mobile-based VAPAs. While participants appreciated the access VAPAs provided to inaccessible applications and services, they faced challenges relating to the input, responses from VAPAs, and control of information presented. User behavior varied depending on the situation or context of the interaction. Implications for design are suggested to support inclusivity when interacting with VAPAs. These include accounting for privacy and situational factors in design, examining ways to support concerns over trust, and synchronizing presentation of visual and non-visual cues.

"Siri Talks at You": An Empirical Investigation of Voice-Activated Personal Assistant (VAPA) Usage by Individuals Who Are Blind

Information provision for challenging markets: the case of the accessibility requiring market in the context of tourism

In previous research, the computer frustrations of student and workplace users have been documented. However, the challenges faced by blind users on the Web have not been previously examined. In this study, 100 blind users, using time diaries, recorded their frustrations using the Web. The top causes of frustration reported were (a) page layout causing confusing screen reader feedback; (b) conflict between screen reader and application; (c) poorly designed/unlabeled forms; (d) no alt text for pictures; and (e) 3-way tie between misleading links, inaccessible PDF, and a screen reader crash. Most of the causes of frustration, such as inappropriate form and graphic labels and confusing page layout, are relatively simple to solve if Webmasters and Web designers focus on this effort. In addition, the more technically challenging frustrations, such as screen reader crashes and conflicts, need to be addressed by the screen reader developers. Blind users in this study were likely to repeatedly attempt to...

Chris Malarkey

Papers

What Frustrates Screen Reader Users on the Web: A Study of 100 Blind Users