Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this market have been assessed on dimensions that are universally relevant to understanding whether, why, and when they should be recruited as research participants. We discuss the characteristics of MTurk as a participant pool for psychology and other social sciences, highlighting the traits of the MTurk samples, why people become MTurk workers and research participants, and how data quality on MTurk compares to that from other pools and depends on controllable and uncontrollable factors.

/pdf/inside-the-turk-understanding-mechanical-turk-as-a-33lzlsoyyp.pdf

Inside the Turk Understanding Mechanical Turk as a Participant Pool

where A represents the magnetic vector potential, is an integral of the hydromagnetic equations. This -integral made it possible to formulate a variational principle for the force-free magnetic fields. The integral expresses the fact that motions cannot transform a given field in an entirely arbitrary different field, if the conductivity of the medium isconsidered infinite. In this paper we shall show that the full set of hydromagnetic equations admit five more integrals, besides the energy integral, if dissipative processes are absent. These integrals, as we shall presently verify, are I2 =fbHvdV, (2)

Proceedings of the National Academy of Sciences

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

/pdf/evaluating-amazon-s-mechanical-turk-as-a-tool-for-1acmhyqzqj.pdf

Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

Prolific.ac—A subject pool for online experiments

Economic games are used to investigate the cognitive mechanisms underlying cooperative behaviour, and show that intuition supports cooperation in social dilemmas, whereas reflection can undermine these cooperative impulses. Many people are willing to make sacrifices for the common good, but little is known about the cognitive mechanisms that underlie such cooperative behaviour. In economic experiments subjects often contribute cooperatively against what rational self-interest should dictate. This study uses a series of ten varied experimental designs, including both one-shot and repeated games, to establish whether we are intuitively predisposed to cooperate or to act selfishly. And it seems our gut response is to cooperate — but given more time to think the logic of self-interest undermines collective action and we become less generous. Cooperation is central to human social behaviour1,2,3,4,5,6,7,8,9. However, choosing to cooperate requires individuals to incur a personal cost to benefit others. Here we explore the cognitive basis of cooperative decision-making in humans using a dual-process framework10,11,12,13,14,15,16,17,18. We ask whether people are predisposed towards selfishness, behaving cooperatively only through active self-control; or whether they are intuitively cooperative, with reflection and prospective reasoning favouring ‘rational’ self-interest. To investigate this issue, we perform ten studies using economic games. We find that across a range of experimental designs, subjects who reach their decisions more quickly are more cooperative. Furthermore, forcing subjects to decide quickly increases contributions, whereas instructing them to reflect and forcing them to decide slowly decreases contributions. Finally, an induction that primes subjects to trust their intuitions increases contributions compared with an induction that promotes greater reflection. To explain these results, we propose that cooperation is intuitive because cooperative heuristics are developed in daily life where cooperation is typically advantageous. We then validate predictions generated by this proposed mechanism. Our results provide convergent evidence that intuition supports cooperation in social dilemmas, and that reflection can undermine these cooperative impulses.

/pdf/spontaneous-giving-and-calculated-greed-10q2b8pxf2.pdf

Spontaneous giving and calculated greed

The Simple-Meta agent uses machine learning to select the negotiation strategy that is predicted to be most successful based on structural features of the domain.

The Simple-Meta Agent

Many environments in which people and computer agents interact involve deploying resources to accomplish tasks and satisfy goals. This paper investigates the way that the context in which decisions are made affects the behavior of people and the performance of computer agents that interact with people in such environments. It presents experiments that measured negotiation behavior in two different types of settings. One setting was a task context that made explicit the relationships among goals, (sub)tasks and resources. The other setting was a completely abstract context in which only the payoffs for the decision choices were listed. Results show that people are more helpful, less selfish, and less competitive when making decisions in task contexts than when making them in completely abstract contexts. Further, their overall performance was better in task contexts. A predictive computational model that was trained on data obtained in the task context outperformed a model that was trained under the abstract context. These results indicate that taking context into account is essential for the design of computer agents that will interact well with people.

/pdf/the-influence-of-task-contexts-on-the-decision-making-of-4bih3afp68.pdf

The influence of task contexts on the decision-making of humans and computers

Natural Intelligence is based not only on conscious procedural anddeclarative knowledge, but also on knowledge that is inferred fromobserving the actions of others. This knowledge is tacit, in that theprocess of its acquisition remains unspecified. However, tacitknowledge is an accepted guide of behavior, especiallyin unfamiliar contexts. In situations where knowledge is lacking,animals act on these beliefs without explicitly reasoning about theworld or fully considering the consequences of their actions. Thispaper provides a computational model of behavior in which tacitknowledge plays a crucial role. We model how knowledge arises fromobserving different types of agents, each of whom reacts differentlyto the behaviors of others in an unfamiliar context. Agents'interaction in this context is described using directed graphs. Weshow how a set of observations guide agents' knowledge and behaviorgiven different states of the world.

/pdf/a-model-of-tacit-knowledge-and-action-6l3tuchqnv.pdf

A Model of Tacit Knowledge and Action

This paper presents preliminary results of our work with a major financial company, where we try to use methods of plan recognition in order to investigate the interactions of a costumer with the company's online interface. In this paper, we present the first steps of integrating a plan recognition algorithm in a real-world application for detecting and analyzing the interactions of a costumer. It uses a novel approach for plan recognition from bare-bone UI data, which reasons about the plan library at the lowest recognition level in order to define the relevancy of actions in our domain, and then uses it to perform plan recognition. 
We present preliminary results of inference on three different use-cases modeled by domain experts from the company, and show that this approach manages to decrease the overload of information required from an analyst to evaluate a costumer's session - whether this is a malicious or benign session, whether the intended tasks were completed, and if not - what actions are expected next.

Session Analysis using Plan Recognition.

Goal Recognition Design (GRD) is the problem of designing a domain in a way that will allow easy identification of agents’ goals. This work extends the original GRD problem to the Plan Recognition Design (PRD) problem which is the task of designing a domain using plan libraries in order to facilitate fast identification of an agent’s plan. While GRD can help to explain faster which goal the agent is trying to achieve, PRD can help in faster understanding of how the agent is going to achieve its goal. we define a new measure that quantifies the worst-case distinctiveness of a given planning domain, propose a method to reduce it in a given domain and show the reduction of this new measure in three domains from the literature.

Ya'akov Gal

Papers

The Simple-Meta Agent

The influence of task contexts on the decision-making of humans and computers

A Model of Tacit Knowledge and Action

Session Analysis using Plan Recognition.

Plan Recognition Design.