Current image captioning methods are usually trained via maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

Improved Image Captioning via Policy Gradient optimization of SPIDEr

The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting. VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so are often poor quality, (2) questions are spoken and so are more conversational, and (3) often visual questions cannot be answered. Evaluation of modern algorithms for answering visual questions and deciding if a visual question is answerable reveals that VizWiz is a challenging dataset. We introduce this dataset to encourage a larger community to develop more generalized algorithms that can assist blind people.

/pdf/vizwiz-grand-challenge-answering-visual-questions-from-blind-1cdyju9vbk.pdf

VizWiz Grand Challenge: Answering Visual Questions from Blind People

Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

SynOptics Communications Inc.

April 24-29, 2004 Vienna, Austria

Chi 2004

We designed and deployed automatic alt-text (AAT), a system that applies computer vision technology to identify faces, objects, and themes from photos to generate photo alt-text for screen reader users on Facebook. We designed our system through iterations of prototyping and in-lab user studies. Our lab test participants had a positive reaction to our system and an enhanced experience with Facebook photos. We also evaluated our system through a two-week field study as part of the Facebook iOS app for 9K VoiceOver users. We randomly assigned them into control and test groups and collected two weeks of activity data and their survey feedback. The test group reported that photos on Facebook were easier to interpret and more engaging, and found Facebook more useful in general. Our system demonstrates that artificial intelligence can be used to enhance the experience for visually impaired users on social networking sites (SNSs), while also revealing the challenges with designing automated assistive technology in a SNS context.

Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service

Previous studies show that traditional usability evaluation methods can be problematic for collecting feedback on visual design [1]. Desirability studies have been used by usability practitioners to collect feedback on the affective response to interactive systems, but none focus on assessing visual design. We describe the Visual Design Card Sort (VDCS), a desirability research method for collecting visual design feedback from domain experts. We also present a case study where VDCS was used to collect visual design feedback from expert users of Computer-Aided Design (CAD) software. Additionally, we propose a set of analyses for looking at data from desirability studies from different perspectives. Future research is needed to validate some of the assumptions made in designing VDCS and to understand how findings from VDCS compare to other desirability research methods and traditional evaluation techniques.

/pdf/how-does-this-look-desirability-methods-for-evaluating-55be47a902.pdf

How Does This Look? Desirability Methods for Evaluating Visual Design

Informal analysis of many usability tests suggests that there is regularly one participant that is substantially slower than all the others. Moreover, such outliers are more extreme and more frequent than would be predicted by a normal distribution. We propose using a rational model to explain the outliers and the work described here begins to parameterise the model based on empirical data to provide accurate analyses of user performance. This prediction appears to be correct and the model begins to reflect the outlying performance. Moreover, by using an executable model, we believe that it could be used in future as an analytical tool to help designers improve usability for those users who are struggling the most.

There's always one!: modelling outlying user performance

Previous studies show that traditional usability evaluation methods can be problematic for collecting feedback on visual design [1]. Desirability studies have been used by usability practitioners to collect feedback on the affective response to interactive systems, but none allow end users to contribute feedback in the language of visual design experts. We describe how we adapted two traditional user research techniques (card sort, directed storytelling) to collect feedback on visual design. We then compare and contrast the kinds of data gathered from these methods with data gathered in a think-aloud exercise. A mixed-methods research strategy that includes methods adapted for visual design offers a path to engaging end users in a conversation that results in concise and actionable feedback for visual designers.

/pdf/desirability-methods-for-evaluating-visual-design-22vgk43f1y.pdf

Desirability Methods for Evaluating Visual Design

A electronic method for monitoring one or more destination ramps in a manufacturing enterprise which manufactures a number of products for purchase through a number of dealers. The method includes the steps of receiving ramp profile information for one or more destination ramps and daily activity information for the one or more destinations ramps; and calculating a carrying cost for each of the one or more destination ramps based on the daily activity information.

Julie Schiller

Papers

Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service

How Does This Look? Desirability Methods for Evaluating Visual Design

There's always one!: modelling outlying user performance

Desirability Methods for Evaluating Visual Design

Electronic method and system for monitoring destination ramp systems