Showing papers by "Michael S. Bernstein published in 2018"

Iris: A Conversational Agent for Complex Tasks

Abstract: Today, most conversational agents are limited to simple tasks supported by standalone commands, such as getting directions or scheduling an appointment. To support more complex tasks, agents must be able to generalize from and combine the commands they already understand. This paper presents a new approach to designing conversational agents inspired by linguistic theory, where agents can execute complex requests interactively by combining commands through nested conversations. We demonstrate this approach in Iris, an agent that can perform open-ended data science tasks such as lexical analysis and predictive modeling. To power Iris, we have created a domain-specific language that transforms Python functions into combinable automata and regulates their combinations through a type system. Running a user study to examine the strengths and limitations of our approach, we find that data scientists completed a modeling task 2.6 times faster with Iris than with Jupyter Notebook.

Referring Relationships

Abstract: Images are not simply sets of objects: each image represents a web of interconnected relationships. These relationships between entities carry semantic meaning and help a viewer differentiate between instances of an entity. For example, in an image of a soccer match, there may be multiple persons present, but each participates in different relationships: one is kicking the ball, and the other is guarding the goal. In this paper, we formulate the task of utilizing these "referring relationships" to disambiguate between entities of the same category. We introduce an iterative model that localizes the two entities in the referring relationship, conditioned on one another. We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another. We demonstrate that our model can not only outperform existing approaches on three datasets - CLEVR, VRD and Visual Genome - but also that it produces visually meaningful predicate shifts, as an instance of interpretable neural networks. Finally, we show that by modelling predicates as attention shifts, we can even localize entities in the absence of their category, allowing our model to find completely unseen categories.

Referring Relationships

Abstract: Images are not simply sets of objects: each image represents a web of interconnected relationships. These relationships between entities carry semantic meaning and help a viewer differentiate between instances of an entity. For example, in an image of a soccer match, there may be multiple persons present, but each participates in different relationships: one is kicking the ball, and the other is guarding the goal. In this paper, we formulate the task of utilizing these "referring relationships" to disambiguate between entities of the same category. We introduce an iterative model that localizes the two entities in the referring relationship, conditioned on one another. We formulate the cyclic condition between the entities in a relationship by modelling predicates that connect the entities as shifts in attention from one entity to another. We demonstrate that our model can not only outperform existing approaches on three datasets --- CLEVR, VRD and Visual Genome --- but also that it produces visually meaningful predicate shifts, as an instance of interpretable neural networks. Finally, we show that by modelling predicates as attention shifts, we can even localize entities in the absence of their category, allowing our model to find completely unseen categories.

Rotating Online Behavior Change Interventions Increases Effectiveness But Also Increases Attrition

Abstract: Behavior change systems help people manage their time online and achieve many other goals. These systems typically consist of a single static intervention, such as a timer or site blocker, to persuade users to behave in ways consistent with their stated goals. However, static interventions decline in effectiveness over time as users begin to ignore them. In this paper, we compare the effectiveness of static interventions to a rotation strategy, where users experience different interventions over time. We built and deployed a browser extension called HabitLab, which features many interventions that the user can enable across social media and other web sites to control their time spent browsing. We ran three in-the-wild field experiments on HabitLab to compare static interventions to rotated interventions. We found that rotating between interventions increased effectiveness as measured by time on site, but also increased attrition: more users uninstalled HabitLab. To minimize attrition, we introduced a just-in-time information design about rotation. This design reduced attrition rates by half. With this research, we suggest that interaction design, paired with rotation of behavior change interventions, can help users gain control of their online habits.

In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures

Abstract: Team structures---roles, norms, and interaction patterns---define how teams work. HCI researchers have theorized ideal team structures and built systems nudging teams towards them, such as those increasing turn-taking, deliberation, and knowledge distribution. However, organizational behavior research argues against the existence of universally ideal structures. Teams are diverse and excel under different structures: while one team might flourish under hierarchical leadership and a critical culture, another will flounder. In this paper, we present DreamTeam: a system that explores a large space of possible team structures to identify effective structures for each team based on observable feedback. To avoid overwhelming teams with too many changes, DreamTeam introduces multi-armed bandits with temporal constraints: an algorithm that manages the timing of exploration--exploitation trade-offs across multiple bandits simultaneously. A field experiment demonstrated that DreamTeam teams outperformed self-managing teams by 38%, manager-led teams by 46%, and teams with unconstrained bandits by 41%. This research advances computation as a powerful partner in establishing effective teamwork.

Hive: Collective Design Through Network Rotation

Abstract: Collectives gather online around challenges they face, but frequently fail to envision shared outcomes to act on together. Prior work has developed systems for improving collective ideation and design by exposing people to each others' ideas and encouraging them to intermix those ideas. However, organizational behavior research has demonstrated that intermixing ideas does not result in meaningful engagement with those ideas. In this paper, we introduce a new class of collective design system that intermixes people instead of ideas: instead of receiving mere exposure to others' ideas, participants engage deeply with other members of the collective who represent those ideas, increasing engagement and influence. We thus present Hive: a system that organizes a collective into small teams, then intermixes people by rotating team membership over time. At a technical level, Hive must balance two competing forces: (1) networks are better at connecting diverse perspectives when network efficiency is high, but (2) moving people diminishes tie strength within teams. Hive balances these two needs through network rotation: an optimization algorithm that computes who should move where, and when. A controlled study compared network rotation to alternative rotation systems which maximize only tie strength or network efficiency, finding that network rotation produced higher-rated proposals. Hive has been deployed by Mozilla for a real-world open design drive to improve Firefox accessibility.

Ink: Increasing Worker Agency to Reduce Friction in Hiring Crowd Workers

Abstract: The web affords connections by which end-users can receive paid, expert help—such as programming, design, and writing—to reach their goals. While a number of online marketplaces have emerged to facilitate such connections, most end-users do not approach a market to hire an expert when faced with a challenge. To reduce friction in hiring from peer-to-peer expert crowd work markets, we propose Ink, a system that crowd workers can use to showcase their services by embedding tasks inside web tutorials—a common destination for users with information needs. Workers have agency to define and manage tasks, through which users can request their help to review or execute each step of the tutorial, for example, to give feedback on a paper outline, perform a statistical analysis, or host a practice programming interview. In a public deployment, over 25,000 pageviews led 168 tutorial readers to pay crowd workers for their services, most of whom had not previously hired from crowdsourcing marketplaces. A field experiment showed that users were more likely to hire crowd experts when the task was embedded inside the tutorial rather than when they were redirected to the same worker’s Upwork profile to hire them. Qualitative analysis of interviews showed that Ink framed hiring expert crowd workers within users’ well-established information seeking habits and gave workers more control over their work.

Shared Autonomy for an Interactive AI System

Abstract: Across many domains, interactive systems either make decisions for us autonomously or yield decision-making authority to us and play a supporting role. However, many settings, such as those in education or the workplace, benefit from sharing this autonomy between the user and the system, and thus from a system that adapts to them over time. In this paper, we pursue two primary research questions: (1) How do we design interfaces to share autonomy between the user and the system? (2) How does shared autonomy alter a user"s perception of a system? We present SharedKeys, an interactive shared autonomy system for piano instruction that plays different video segments of a piece for students to emulate and practice. Underlying our approach to shared autonomy is a mixed-observability Markov decision process that estimates a user"s desired autonomy level based on her performance and attentiveness. Pilot studies revealed that students sharing autonomy with the system learned more quickly and perceived the system as more intelligent.

Engagement Learning: Expanding Visual Knowledge by Engaging Online Participants

Abstract: Most artificial intelligence (AI) systems to date have focused entirely on performance, and rarely if at all on their social interactions with people and how to balance the AIs' goals against their human collaborators'. Learning quickly from interactions with people poses both social challenges and is unresolved technically. In this paper, we introduce engagement learning: a training approach that learns to trade off what the AI needs---the knowledge value of a label to the AI---against what people are interested to engage with---the engagement value of the label. We realize our goal with ELIA (Engagement Learning Interaction Agent), a conversational AI agent who's goal is to learn new facts about the visual world by asking engaging questions of people about the photos they upload to social media. Our current deployment of ELIA on Instagram receives a response rate of 26%.