Advances in artificial intelligence, sensors and big data management have far-reaching societal impacts. As these systems augment our everyday lives, it becomes increasing-ly important for people to understand them and remain in control. We investigate how HCI researchers can help to develop accountable systems by performing a literature analysis of 289 core papers on explanations and explaina-ble systems, as well as 12,412 citing papers. Using topic modeling, co-occurrence and network analysis, we mapped the research space from diverse domains, such as algorith-mic accountability, interpretable machine learning, context-awareness, cognitive psychology, and software learnability. We reveal fading and burgeoning trends in explainable systems, and identify domains that are closely connected or mostly isolated. The time is ripe for the HCI community to ensure that the powerful new autonomous systems have intelligible interfaces built-in. From our results, we propose several implications and directions for future research to-wards this goal.

https://dl.acm.org/doi/pdf/10.1145/3173574.3174156

Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda

It represents a universally applicable attitude and skill set everyone, not just computer scientists, would be eager to learn and use.

/pdf/computational-thinking-ji-suan-lun-de-si-kao-3qvv34ha3s.pdf

Computational Thinking 計算論的思考

Two parallel phenomena are gaining attention in human–computer interaction research: gamification and crowdsourcing Because crowdsourcing's success depends on a mass of motivated crowdsourcees, crowdsourcing platforms have increasingly been imbued with motivational design features borrowed from games; a practice often called gamification While the body of literature and knowledge of the phenomenon have begun to accumulate, we still lack a comprehensive and systematic understanding of conceptual foundations, knowledge of how gamification is used in crowdsourcing, and whether it is effective We first provide a conceptual framework for gamified crowdsourcing systems in order to understand and conceptualize the key aspects of the phenomenon The paper's main contributions are derived through a systematic literature review that investigates how gamification has been examined in different types of crowdsourcing in a variety of domains This meticulous mapping, which focuses on all aspects in our framework, enables us to infer what kinds of gamification efforts are effective in different crowdsourcing approaches as well as to point to a number of research gaps and lay out future research directions for gamified crowdsourcing systems Overall, the results indicate that gamification has been an effective approach for increasing crowdsourcing participation and the quality of the crowdsourced work; however, differences exist between different types of crowdsourcing: the research conducted in the context of crowdsourcing of homogenous tasks has most commonly used simple gamification implementations, such as points and leaderboards, whereas crowdsourcing implementations that seek diverse and creative contributions employ gamification with a richer set of mechanics

/pdf/gamified-crowdsourcing-conceptualization-literature-review-1p89vv9dmt.pdf

Gamified crowdsourcing: Conceptualization, literature review, and future agenda

We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.

RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records

This study investigates how different gamification implementations can increase crowdsourcees' motivation and participation in crowdsourcing (CS). To this end, we review empirical literature that has investigated the use of gamification in crowdsourcing settings. Overall, the results of the review indicate that gamification has been an effective approach for increasing crowdsourcing participation. When comparing crowdcreating, -solving, -processing and-rating CS approaches, the results show differences in the use of gamification across CS types. Crowdsourcing initiatives that provide more monotonous tasks most commonly used mere points and other simpler gamification implementations, whereas CS initiatives that seek for diverse and creative contributions have employed gamification in more manifold ways employing a richer set of mechanics. These findings provide insights for designers of gamified systems and further research on the topics of gamification and crowdsourcing.

/pdf/gamification-in-crowdsourcing-a-review-4ye8qg3t3z.pdf

Gamification in Crowdsourcing: A Review

/pdf/ctarcade-computational-thinking-with-games-in-school-age-2qq875ertl.pdf

CTArcade: Computational thinking with games in school age children

Topic modeling is a common tool for understanding large bodies of text, but is typically provided as a “take it or leave it” proposition. Incorporating human knowledge in unsupervised learning is a promising approach to create high-quality topic models. Existing interactive systems and modeling algorithms support a wide range of refinement operations to express feedback. However, these systems’ interactions are primarily driven by algorithmic convenience, ignoring users who may lack expertise in topic modeling. To better understand how non-expert users understand, assess, and refine topics, we conducted two user studies—an in-person interview study and an online crowdsourced study. These studies demonstrate a disconnect between what non-expert users want and the complex, low-level operations that current interactive systems support. In particular, our findings include: (1) analysis of how non-expert users perceive topic models; (2) characterization of primary refinement operations expected by non-expert users and ordered by relative preference; (3) further evidence of the benefits of supporting users in directly refining a topic model; (4) design implications for future human-in-the-loop topic modeling interfaces.

/pdf/the-human-touch-how-non-expert-users-perceive-interpret-and-2b3cesrwnm.pdf

The human touch: How non-expert users perceive, interpret, and fix topic models

This paper examines the relationship between motivational design and its longitudinal effects on crowdsourcing systems. In the context of a company internal web site that crowdsources the identification of Twitter accounts owned by company employees, we designed and investigated the effects of various motivational features including individual / social achievements and gamification. Our 6-month experiment with 437 users allowed us to compare the features in terms of both quantity and quality of the work produced by participants over time. While we found that gamification can increase workers’ motivation overall, the combination of motivational features also matters. Specifically, gamified social achievement is the best performing design over a longer period of time. Mixing individual and social achievements turns out to be less effective and can even encourage users to game the system.

https://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/viewFile/6118/6373

Experiments on Motivational Feedback for Crowdsourced Workers

Probabilistic topic models are important tools for indexing, summarizing, and analyzing large document collections by their themes.  However, promoting end-user understanding of topics remains an open research problem.  We compare labels generated by users given four topic visualization techniques—word lists, word lists with bars, word clouds, and network graphs—against each other and against automatically generated labels.  Our basis of comparison is participant ratings of how well labels describe documents from the topic. Our study has two phases: a labeling phase where participants label visualized topics and a validation phase where different participants select which labels best describe the topics' documents.  Although all visualizations produce similar quality labels, simple visualizations such as word lists allow participants to quickly understand topics, while complex visualizations take longer but expose multi-word expressions that simpler visualizations obscure.  Automatic labels lag behind user-created labels, but our dataset of manually labeled topics highlights linguistic patterns (e.g., hypernyms, phrases) that can be used to improve automatic topic labeling algorithms.

/pdf/evaluating-visual-representations-for-topic-understanding-3o82etv84o.pdf

Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Labels

In this paper we describe CTArcade, a web application framework that seeks to engage users through game play resulting in the improvement of computational thinking (CT) skills. Our formative study indicates that CT skills are employed when children are asked to define strategies of common games such as Connect Four. In CTArcade, users can train their own virtual characters while playing games with it. Trained characters then play matches against other virtual characters. Based on reviewing the matches played, users can improve their game character. A basic usability evaluation was performed on the system, which helped to define plans for improving CTArcade and assessing its design goals.

/pdf/ctarcade-learning-computational-thinking-while-training-40pxg3su26.pdf

Tak Yeon Lee

Papers

CTArcade: Computational thinking with games in school age children

The human touch: How non-expert users perceive, interpret, and fix topic models

Experiments on Motivational Feedback for Crowdsourced Workers

Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Labels

CTArcade: learning computational thinking while training virtual characters through game play