PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments
TL;DR: This article reviews PsyToolkit, a free web-based service designed for setting up, running, and analyzing online questionnaires and reaction-time (RT) experiments.
Abstract: This article reviews PsyToolkit, a free web-based service designed for setting up, running, and analyzing online questionnaires and reaction-time (RT) experiments. It comes with extensive documenta...
Citations
More filters
TL;DR: The Gorilla Experiment Builder (gorilla.sc) is presented, a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities.
Abstract: Behavioral researchers are increasingly conducting their studies online, to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web browsers can present problems for consistent timing-an important issue with reaction-time-sensitive measures. For example, to ensure accuracy and test-retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioral research, as well as how well they address the issues of usability and timing. We then present the Gorilla Experiment Builder (gorilla.sc), a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities. To demonstrate the platform's aptitude for accessible, reliable, and scalable research, we administered a task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in both schools and public engagement events), equipment (participant's own computer, computer supplied by the researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the "conflict network" effect in all these populations, demonstrating the platform's capability to run reaction-time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions and some future features of the platform.
540 citations
TL;DR: The Gorilla Experiment Builder (gorilla.sc) is presented, a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues, and make reliable online experimentation open and accessible to a wider range of technical abilities.
Abstract: Behavioural researchers are increasingly conducting their studies online to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web-browsers can present problems for consistent timing -- an important issue with reaction time-sensitive measures. For example, to ensure accuracy and test-retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioural research, and how well they address the issues of usability and timing. We then present The Gorilla Experiment Builder (gorilla.sc) a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues, and make reliable online experimentation open and accessible to a wider range of technical abilities. In order to demonstrate the platform's aptitude for accessible, reliable and scalable research, we administered the task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in schools and public engagement events), equipment (own computers, computer supplied by researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task, taken from the Attentional Networks task (Rueda et al., 2004). We replicated the 'conflict network' effect in all these populations, demonstrating the platform's capability to run reaction time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions, and some future features of the platform.
249 citations
Cites methods from "PsyToolkit: A Novel Web-Based Metho..."
...One recent example of this is the online implementation of Psy-Toolkit (Stoet, 2017), where users can create, host and run experiments on a managed web server and interface – however, there is still a requirement to write out the experiment in code – representing another access limitation....
[...]
TL;DR: Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science and who is in the crowd and who can be reached by the average laboratory is reviewed.
150 citations
Additional excerpts
...org which allows online or offline development [95,96], and Flash-based scriptingRT [97]....
[...]
TL;DR: In this paper, the authors assessed whether individual differences in pathogen disgust sensitivity, social anxiety and generalized social trust predicted judgments of trustworthiness, desired social distance and perceptions of sickness of target faces wearing surgical masks.
97 citations
Cites methods from "PsyToolkit: A Novel Web-Based Metho..."
...Participants performed the experiment online, using the opensource Psytoolkit software (Stoet, 2017)....
[...]
TL;DR: In this paper, the authors carried out an online study across the Italian population during the first pandemic peak, where participants were shown static facial expressions (Angry, Happy and Neutral) covered by a sanitary mask or by a scarf.
Abstract: The COVID-19 pandemic has dramatically changed the nature of our social interactions. In order to understand how protective equipment and distancing measures influence the ability to comprehend others' emotions and, thus, to effectively interact with others, we carried out an online study across the Italian population during the first pandemic peak. Participants were shown static facial expressions (Angry, Happy and Neutral) covered by a sanitary mask or by a scarf. They were asked to evaluate the expressed emotions as well as to assess the degree to which one would adopt physical and social distancing measures for each stimulus. Results demonstrate that, despite the covering of the lower-face, participants correctly recognized the facial expressions of emotions with a polarizing effect on emotional valence ratings found in females. Noticeably, while females' ratings for physical and social distancing were driven by the emotional content of the stimuli, males were influenced by the "covered" condition. The results also show the impact of the pandemic on anxiety and fear experienced by participants. Taken together, our results offer novel insights on the impact of the COVID-19 pandemic on social interactions, providing a deeper understanding of the way people react to different kinds of protective face covering.
84 citations
References
More filters
15,981 citations
TL;DR: Correlational analysis revealed that anger is the bridge between both physical and verbal aggression and hostility and the need to assess not only overall aggression but also its individual components.
Abstract: A new questionnaire on aggression was constructed Replicated factor analyses yielded 4 scales: Physical Aggression, Verbal Aggression, Anger, and Hostility Correlational analysis revealed that anger is the bridge between both physical and verbal aggression and hostility The scales showed internal consistency and stability over time Men scored slightly higher on Verbal Aggression and Hostility and much higher on Physical Aggression There was no sex difference for Anger The various scales correlated differently with various personality traits Scale scores correlated with peer nominations of the various kinds of aggression These findings suggest the need to assess not only overall aggression but also its individual components
5,337 citations
"PsyToolkit: A Novel Web-Based Metho..." refers background in this paper
...…Useful Depression Outcome Scale, Zimmerman, Chelminski, McGlinchey, & Posternak, 2008), negative traits (e.g., the Buss–Perry Aggression Scale, Buss & Perry, 1992; the Problematic Internet Use Questionnaire, Demetrovics, Szeredi, & Rózsa, 2008), and handedness (which is practical in…...
[...]
...There are numerous scales for different subdisciplines of psychology, including scales for measuring the Big Five personality traits (e.g., the short Big Five Inventory [BFI-S], Lang et al., 2011), mental health issues (e.g., Clinically Useful Depression Outcome Scale, Zimmerman, Chelminski, McGlinchey, & Posternak, 2008), negative traits (e.g., the Buss–Perry Aggression Scale, Buss & Perry, 1992; the Problematic Internet Use Questionnaire, Demetrovics, Szeredi, & Rózsa, 2008), and handedness (which is practical in combination with certain RT experiments; Veale, 2014)....
[...]
TL;DR: In this article, task-set reconfiguration was investigated in 5 experiments and on every 4th trial in a final experiment, where the tasks were to classify either the digit member of a pair of characters as even/odd or the letter member as consonant/vowel.
Abstract: In an investigation of task-set reconfiguration, participants switched between 2 tasks on every 2nd trial in 5 experiments and on every 4th trial in a final experiment. The tasks were to classify either the digit member of a pair of characters as even/odd or the letter member as consonant/vowel. As the response-stimulus interval increased up to 0.6 s, the substantial cost to performance of this predictable task-switch fell: Participants could partially reconfigure in advance of the stimulus. However, even with 1.2 s available for preparation, a large asymptotic reaction time (RT) cost remained, but only on the 1st trial of the new task. This is attributed to a component of reconfiguration triggered exogenously, i. e., only by a task-relevant stimulus. That stimuli evoke associated task-sets also explains why RT and switch costs increased when the stimulus included a character associated with the currently irrelevant task. © 1995 American Psychological Association.
2,896 citations
"PsyToolkit: A Novel Web-Based Metho..." refers methods in this paper
...…code of the Cognitive Flexibility Scale (Martin & Rubin, 1995) or the Webexec (Buchanan et al., 2010), and then embed a task-switching experiment (e.g., using the alternative runs task-switching paradigm, Rogers & Monsell, 1995) to relate self-reported measures of executive control with RT data....
[...]
TL;DR: OpenSesame is a graphical experiment builder for the social sciences that features a comprehensive and intuitive graphical user interface and supports Python scripting for complex tasks.
Abstract: In the present article, we introduce OpenSesame, a graphical experiment builder for the social sciences. OpenSesame is free, open-source, and cross-platform. It features a comprehensive and intuitive graphical user interface and supports Python scripting for complex tasks. Additional functionality, such as support for eyetrackers, input devices, and video playback, is available through plug-ins. OpenSesame can be used in combination with existing software for creating experiments.
1,666 citations
1,285 citations
"PsyToolkit: A Novel Web-Based Metho..." refers methods in this paper
...Currently, the library includes more than 20 different paradigms, including popular paradigms such as the Stroop (1935) task, the N-back task (Kirchner, 1958), and visual search (Treisman, 1977)....
[...]
...rently, the library includes more than 20 different paradigms, including popular paradigms such as the Stroop (1935) task, the N-back task (Kirchner, 1958), and visual search (Treisman, 1977)....
[...]