Showing papers on "Task analysis published in 2014"

Findings of the 2014 Workshop on Statistical Machine Translation

Abstract: This paper presents the results of the WMT14 shared tasks, which included a standard news translation task, a separate medical translation task, a task for run-time estimation of machine translation quality, and a metrics task. This year, 143 machine translation systems from 23 institutions were submitted to the ten translation directions in the standard translation task. An additional 6 anonymized systems were included, and were then evaluated both automatically and manually. The quality estimation task had four subtasks, with a total of 10 teams, submitting 57 entries

The CoNLL-2014 Shared Task on Grammatical Error Correction

Abstract: The CoNLL-2014 shared task was devoted to grammatical error correction of all error types. In this paper, we give the task definition, present the data sets, and describe the evaluation metric and scorer used in the shared task. We also give an overview of the various approaches adopted by the participating teams, and present the evaluation results. Compared to the CoNLL2013 shared task, we have introduced the following changes in CoNLL-2014: (1) A participating system is expected to detect and correct grammatical errors of all types, instead of just the five error types in CoNLL-2013; (2) The evaluation metric was changed from F1 to F0.5, to emphasize precision over recall; and (3) We have two human annotators who independently annotated the test essays, compared to just one human annotator in CoNLL-2013.

Improving Training Effectiveness in Work Organizations

Abstract: Contents: E.A. Fleishman, Foreword. Preface. J.K. Ford, Advances in Training Research and Practice: An Historical Perspective. Part I:Integrating Advances in Cognitive Science With Training Assessment, Design, and Evaluation Constructs. W. Rogers, T. Maurer, E. Salas A. Fisk, Task Analysis and Cognitive Theory: Controlled and Automatic Processing Task Analytic Methodology. M. Coovert, J.P. Craiger, Performance Modeling for Training Effectiveness. T. Goldsmith, K. Kraiger, Structural Knowledge Assessment and Training Evaluation. Part II:Viewing Learning as a Continuous Process: Implications for Training, Socialization, and Employee Development. T. Baldwin, R.J. Magjuka, Organizational Context and Training Effectiveness. G.T. Chao, Unstructured Training and Development: The Role of Organizational Socialization. R.A. Noe, S.L. Wilk, E.J. Mullen, J.E. Wanek, Employee Development: Construct Validation Issues. Part III:Understanding How Context Affects Training Evaluation and Transfer. J.E. Mathieu, J.W. Martineau, Individual and Situational Influences in Training Motivation. G.M. Alliger, S. Katzman, Reconsidering Training Evaluation: Heterogeneity of Variance as a Training Effect. S.W.J. Kozlowski, E. Salas, An Organizational Systems Approach for the Implementation and Transfer of Training. Part IV:Applied Psychology and Training: Methods for Improving the Linkage Between Training Research and Practice. E. Salas, J.A. Cannon-Bowers, E.L. Blickensderfer, Enhancing Reciprocity Between Training Theory and Training Practice: Principles, Guidelines, and Specifications. K.J. Klein, R.S. Ralls, An Exploration of the Unintended Organizational Consequences of Technology Training: Implications of Practice for Training Theory and Research. Part V:Final Observations. E. Salas, J.A. Cannon-Bowers, S.W.J. Kozlowski, The Science and Practice of Training--Current Trends and Emerging Themes.

The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment.

Abstract: Computer-based assessment can provide new insights into behavioral processes of task completion that cannot be uncovered by paper-based instruments. Time presents a major characteristic of the task completion process. Psychologically, time on task has 2 different interpretations, suggesting opposing associations with task outcome: Spending more time may be positively related to the outcome as the task is completed more carefully. However, the relation may be negative if working more fluently, and thus faster, reflects higher skill level. Using a dual processing theory framework, the present study argues that the validity of each assumption is dependent on the relative degree of controlled versus routine cognitive processing required by a task, as well as a person’s acquired skill. A total of 1,020 persons ages 16 to 65 years participated in the German field test of the Programme for the International Assessment of Adult Competencies. Test takers completed computer-based reading and problem solving tasks. As revealed by linear mixed models, in problem solving, which required controlled processing, the time on task effect was positive and increased with task difficulty. In reading tasks, which required more routine processing, the time on task effect was negative and the more negative, the easier a task was. In problem solving, the positive time on task effect decreased with increasing skill level. In reading, the negative time on task effect increased with increasing skill level. These heterogeneous effects suggest that time on task has no uniform interpretation but is a function of task difficulty and individual skill.

SemEval-2015 Task 14: Analysis of Clinical Text

Abstract: This paper describes the SemEval-2014, Task 7 on the Analysis of Clinical Text and presents the evaluation results. It focused on two subtasks: (i) identification (Task A) and (ii) normalization (Task B) of diseases and disorders in clinical reports as annotated in the Shared Annotated Resources (ShARe) 1 corpus. This task was a follow-up to the ShARe/CLEF eHealth 2013 shared task, subtasks 1a and 1b, 2 but using a larger test set. A total of 21 teams competed in Task A, and 18 of those also participated in Task B. For Task A, the best system had a strict F1-score of 81.3, with a precision of 84.3 and recall of 78.6. For Task B, the same group had the best strict accuracy of 74.1. The organizers have made the text corpora, annotations, and evaluation tools available for future research and development at the shared task website. 3

The hare and the tortoise: emphasizing speed can change the evidence used to make decisions.

Abstract: Decision-makers effortlessly balance the need for urgency against the need for caution. Theoretical and neurophysiological accounts have explained this tradeoffsolely in terms of the quantity of evidence required to trigger a decision (the "threshold"). This explanation has also been used as a benchmark test for evaluating new models of decision making, but the explanation itself has not been carefully tested against data. We rigorously test the assumption that emphasizing decision speed versus decision accuracy selectively influences only decision thresholds. In data from a new brightness discrimination experiment we found that emphasizing decision speed over decision accuracy not only decreases the amount of evidence required for a decision but also decreases the quality of information being accumulated during the decision process. This result was consistent for 2 leading decision-making models and in a model-free test. We also found the same model-based results in archival data from a lexical decision task (reported by Wagenmakers, Ratcliff, Gomez, & McKoon, 2008) and new data from a recognition memory task. We discuss implications for theoretical development and applications.

Task-based learning and teaching in China: Secondary school teachers’ beliefs and practices:

Abstract: While much has been written about task-based language teaching (TBLT), research examining teachers’ understandings of what TBLT means remains limited. This article explores the understandings of TB...

Peers increase adolescent risk taking even when the probabilities of negative outcomes are known.

Abstract: The majority of adolescent risk taking occurs in the presence of peers, and recent research suggests that the presence of peers may alter how the potential rewards and costs of a decision are valuated or perceived. The current study further explores this notion by investigating how peer observation affects adolescent risk taking when the information necessary to make an informed decision is explicitly provided. We used a novel probabilistic gambling task in which participants decided whether to play or pass on a series of offers for which the reward and loss outcome probabilities were made explicit. Adolescent participants completed the task either alone or under the belief that they were being observed by an unknown peer in a neighboring room. Participants who believed a peer was observing them chose to gamble more often than participants who completed the task alone, and this effect was most evident for decisions with a greater probability of loss. These results suggest that the presence of peers can increase risk taking among adolescents even when specific information regarding the likelihood of positive and negative outcomes is provided. The findings expand our understanding of how peers influence adolescent decision making and have important implications regarding the value of educational programs aimed at reducing risky behaviors during adolescence.

The Relationship Between Media Multitasking and Executive Function in Early Adolescents

Abstract: The increasing prevalence of media multitasking among adolescents is concerning because it may be negatively related to goal-directed behavior. This study investigated the relationship between media multitasking and executive function in 523 early adolescents (aged 11-15; 48% girls). The three central components of executive functions (i.e., working memory, shifting, and inhibition) were measured using self-reports and standardized performance-based tasks (Digit Span, Eriksen Flankers task, Dots-Triangles task). Findings show that adolescents who media multitask more frequently reported having more problems in the three domains of executive function in their everyday lives. Media multitasking was not related to the performance on the Digit Span and Dots-Triangles task. Adolescents who media multitasked more frequently tended to be better in ignoring irrelevant distractions in the Eriksen Flankers task. Overall, results suggest that media multitasking is negatively related to executive function in everyday life.

A Study of Learners' Perceptions of Online Intercultural Exchange through Web 2.0 Technologies.

Abstract: This paper reports a Spanish-American telecollaborative project through which students used Twitter, blogs and podcasts for intercultural exchange over the course of one semester. The paper outlines the methodology for the project including pedagogical objectives, task design, selection of web tools and implementation. Using qualitative and quantitative data collection, the study explored how the application of Web 2.0 facilitated cross-cultural communication. How the use of digital technology affected the way in which the students viewed intercultural learning and peer feedback was examined. The findings showed that students viewed the online exchange as a superb venue for intercultural communication with native speakers. Through social engagements, students not only gained cultural knowledge but also became more aware of their own beliefs and attitudes toward their own culture. In addition, discussions on topics of tangible and intangible cultures afforded the opportunity to raise students’ awareness of cultural norms and practices. Peer feedback helped learners increase lexical knowledge, prevent language fossilization, and acquire native-sounding discourse. The study suggests that allocating sufficient time to complete each task and making personal commitment to online contributions are essential to successful intercultural exchanges.

A taxonomy of crowdsourcing based on task complexity

Abstract: Although a great many different crowdsourcing approaches are available to those seeking to accomplish individual or organizational tasks, little research attention has yet been given to characterizing how those approaches might be based on task characteristics. To that end, we conducted an extensive review of the crowdsourcing landscape, including a look at what types of taxonomies are currently available. Our review found that no taxonomy explored the multidimensional nature of task complexity. This paper develops a taxonomy whose specific intent is the classification of approaches in terms of the types of tasks for which they are best suited. To develop this task-based taxonomy, we followed an iterative approach that considered over 100 well-known examples of crowdsourcing. The taxonomy considers three dimensions of task complexity: (a) task structure - is the task well-defined, or does it require a more open-ended solution; (2) task interdependence - can the task be solved by an individual, or does it require a community of problem solvers; and (3) task commitment - what level of commitment is expected from crowd members? Based on this taxonomy, we identify seven categories of crowdsourcing and discuss prototypical examples of each approach. Furnished with such an understanding, one should be able to determine which crowdsourcing approach is most suitable for a particular task situation.

Using wikis to promote collaborative EFL writing

Abstract: Zeliha Aydin, Ozyegin University Senem Yildiz, Bogazici University This study focuses on the use of wikis in collaborative writing projects in foreign language learning classrooms. A total of 34 intermediate level university students learning English as a foreign language (EFL) were asked to accomplish three different wiki-based collaborative writing tasks, (argumentative, informative and decision-making) working in groups of four. Student wiki pages were then analyzed to investigate the role of task type in the number of self and peer-corrections as well as form-related and meaning-related changes. In addition, focus-group interviews and questionnaires were conducted to find out how students would describe their overall experience with the integration of a wikibased collaborative writing project in their foreign language learning process. The results revealed that the argumentative task promoted more peer-corrections than the informative and decision-making tasks. In addition, the informative task yielded more self-corrections than the argumentative and decision-making tasks. Furthermore, the use of wiki-based collaborative writing tasks led to the accurate use of grammatical structures 94% of the time. The results of the study also suggest that students paid more attention to meaning rather than form regardless of the task type. Finally, students had positive experiences using wikis in foreign language writing, and they believed that their writing performance had improved.

On the application of motivation theory to human factors/ergonomics: Motivational design principles for human–technology interaction.

Abstract: Objective: Motivation is a driving force in human‐ technology interaction. This paper represents an effort to (a) describe a theoretical model of motivation in human technology interaction, (b) provide design principles and guidelines based on this theory, and (c) describe a sequence of steps for the evaluation of motivational factors in human‐technology interaction. Background: Motivation theory has been relatively neglected in human factors/ergonomics (HF/E). In both research and practice, the (implicit) assumption has been that the operator is already motivated or that motivation is an organizational concern and beyond the purview of HF/E. However, technology can induce task-related boredom (e.g., automation) that can be stressful and also increase system vulnerability to performance failures. Method: A theoretical model of motivation in human‐technology interaction is proposed, based on extension of the self-determination theory of motivation to HF/E. This model provides the basis for both future research and for development of practical recommendations for design. Results: General principles and guidelines for motivational design are described as well as a sequence of steps for the design process. Conclusion: Human motivation is an important concern for HF/E research and practice. Procedures in the design of both simple and complex technologies can, and should, include the evaluation of motivational characteristics of the task, interface, or system. In addition, researchers should investigate these factors in specific human‐technology domains. Application: The theory, principles, and guidelines described here can be incorporated into existing techniques for task analysis and for interface and system design.

A task is a task is a task: putting complex span, n-back, and other working memory indicators in psychometric context.

Abstract: Based on a meta-analysis, Redick and Lindsey (2013) found that complex span and n-back tasks show an average correlation of r = 0.20, and concluded that “complex span and n-back tasks cannot be used interchangeably as working memory measures in research applications” (p. 1102). Here, we comment on this conclusion from a psychometric perspective. In addition to construct variance, performance on a test contains measurement error, task-specific variance, and paradigm-specific variance. Hence, low correlations among dissimilar indicators do not provide strong evidence for the existence, or absence, of a construct common to both indicators. One way to arrive at such evidence is to fit hierarchical latent factors that model task-specific, paradigm-specific, and construct variance. We report analyses for 101 younger and 103 older adults who worked on nine different working memory tasks. The data are consistent with a hierarchical model of working memory, according to which both complex span and n-back tasks are valid indicators of working memory. The working memory factor predicts 71% of the variance in a factor of reasoning among younger adults (83% for among older adults). When the working memory factor was restricted to any possible triplet of working memory tasks, the correlation between working memory and reasoning was inversely related to the average magnitude of the correlations among the indicators, indicating that more highly intercorrelated indicators may provide poorer coverage of the construct space. We stress the need to go beyond specific tasks and paradigms when studying higher-order cognitive constructs, such as working memory.

Morphological and Syntactic Awareness in Poor Comprehenders Another Piece of the Puzzle

Abstract: Poor comprehenders have intact word-reading skills but struggle specifically with understanding what they read. We investigated whether two metalinguistic skills, morphological and syntactic awareness, are specifically related to poor reading comprehension by including separate and combined measures of each. We identified poor comprehenders (n = 15) and average comprehenders (n = 15) in Grade 4 who were matched on word-reading accuracy and speed, vocabulary, nonverbal cognitive ability, and age. The two groups performed comparably on a morphological awareness task that involved both morphological and syntactic cues. However, poor comprehenders performed less well than average comprehenders on a derivational word analogy task in which there was no additional syntactic information, thus tapping only morphological awareness, and also less well on a syntactic awareness task, in which there were no morphological manipulations. Our task and participant-selection process ruled out key nonmetalinguistic sources of influence on these tasks. These findings suggest that the relationships among reading comprehension, morphological awareness, and syntactic awareness depend on the tasks used to measure the latter two. Future research needs to identify precisely in which ways these metalinguistic difficulties connect to challenges with reading comprehension.

The Effects of Task Complexity and Input Frequency on the Acquisition of the Past Counterfactual Construction Through Recasts

Abstract: This investigation examined two techniques that may help learners focus on second language constructions when recasts are provided during meaning-based communicative activities: altering the cognitive complexity of tasks and manipulating the input frequency distributions of target constructions. We first independently assessed the validity of our cognitive task complexity manipulation by means of expert judgments, dual-task methodology, and eye tracking. Next, in our main study, we employed a pretest-posttest design with two treatment sessions. The participants were 51 adult ESL learners, randomly assigned to one of four comparison groups or a control group. All comparison groups received recasts but differed as to whether they carried out simple tasks with lower reasoning demands or complex tasks with higher reasoning demands, and whether they received skewed or balanced input of the linguistic target, the past counterfactual construction. An oral production test and two written receptive tests were utilized to measure changes in participants’ knowledge. Our results revealed no effects for the input frequency manipulations, but participants achieved higher oral production gains under the simple task condition.

Robot deictics: how gesture and context shape referential communication

Abstract: As robots collaborate with humans in increasingly diverse environments, they will need to effectively refer to objects of joint interest and adapt their references to various physical, environmental, and task conditions. Humans use a broad range of deictic gestures—gestures that direct attention to collocated objects, persons, or spaces—that include pointing, touching, and exhibiting to help their listeners understand their references. These gestures offer varying levels of support under different conditions, making some gestures more or less suitable for different settings. While these gestures offer a rich space for designing communicative behaviors for robots, a better understanding of how different deictic gestures affect communication under different conditions is critical for achieving effective human-robot interaction. In this paper, we seek to build such an understanding by implementing six deictic gestures on a humanlike robot and evaluating their communicative effectiveness in six diverse settings that represent physical, environmental, and task conditions under which robots are expected to employ deictic communication. Our results show that gestures which come into physical contact with the object offer the highest overall communicative accuracy and that specific settings benefit from the use of particular types of gestures. Our results highlight the rich design space for deictic gestures and inform how robots might adapt their gestures to the specific physical, environmental, and task conditions. Categories and Subject Descriptors H.1.2 [Models and Principles]: User/Machine Systems— human factors, software psychology; H.5.2 [Information Interfaces and Presentation]: User Interfaces— evaluation/methodology, usercentered design. General Terms Design, Human Factors

A multifaceted approach to investigating pre-task planning effects on paired oral test performance

Abstract: Despite the growing popularity of paired format speaking assessments, the effects of pre-task planning time on performance in these formats are not yet well understood. For example, some studies have revealed the benefits of planning but others have not. Using a multifaceted approach including analysis of the process of speaking performance, the aim of this paper is to investigate the effect of pre-task planning in a paired format. Data were collected from 32 students who carried out two decision-making tasks in pairs, under planned and unplanned conditions. The study used analyses of rating scores, discourse analytic measures, and conversation analysis (CA) of test-taker discourse to gain insight into co-constructing processes. A post-test questionnaire was also administered to understand the participants’ perceptions toward planned and unplanned interactions. The results from rating scores and discourse analytic measures revealed that planning had limited effect on performance, and analysis of the quest...

Effect of Alignment on L2 Written Production

Abstract: This article aims to uncover how L2 writing is affected by alignment, a socio-cognitive process involving dynamic coordination and adaptation. For this, two studies were conducted. Study 1 required two groups of 24 learners of English as a foreign language (EFL) to continue in English two stories with their endings removed, both of which had a Chinese version and an English version. Each group continued one story following the reading of its Chinese version and the other following the reading of its English version. Study 2 asked another two groups of 24 EFL learners to write, without an input text, on two topics meant to activate their contextual knowledge. One topic was related to an English film and the other to an L1-based anecdotal experience. Results showed that (i) alignment manifested itself in the continuation when the learners performed the English-version task, as they used more lexical items in the original story and thus committed significantly fewer errors in comparison with their performance on the Chinese-version task, and (ii) a task that activated the learners’ L1 contextual knowledge incurred significantly more L1 transfer errors due to self-alignment.

Untangling search task complexity and difficulty in the context of interactive information retrieval studies

Abstract: Purpose – One core element of interactive information retrieval (IIR) experiments is the assignment of search tasks. The purpose of this paper is to provide an analytical review of current practice in developing those search tasks to test, observe or control task complexity and difficulty. Design/methodology/approach – Over 100 prior studies of IIR were examined in terms of how each defined task complexity and/or difficulty (or related concepts) and subsequently interpreted those concepts in the development of the assigned search tasks. Findings – Search task complexity is found to include three dimensions: multiplicity of subtasks or steps, multiplicity of facets, and indeterminability. Search task difficulty is based on an interaction between the search task and the attributes of the searcher or the attributes of the search situation. The paper highlights the anomalies in our use of these two concepts, concluding with suggestions for future methodological research related to search task complexity and d...

The Effects of Complexity, Accuracy, and Fluency on Communicative Adequacy in Oral Task Performance

Abstract: Communicative adequacy is a key construct in second language research, as the primary goal of most language learners is to communicate successfully in real-world situations. Nevertheless, little is known about what linguistic features contribute to communicatively adequate speech. This study fills this gap by investigating the extent to which complexity, accuracy, and fluency (CAF) predict adequacy, and whether proficiency and task type moderate these relationships. In all, 20 native speakers and 80 second language users from four proficiency levels performed five tasks. Speech samples were rated for adequacy and coded for a range of CAF indices. Filled pause frequency, a feature of breakdown fluency, emerged as the strongest predictor of adequacy. Predictors with significant but smaller effects included indices of all three CAF dimensions: linguistic complexity (lexical diversity, overall syntactic complexity, syntactic complexity by subordination, and frequency of conjoined clauses), accuracy (general accuracy and accuracy of connectors), and fluency (silent pause frequency and speed fluency). For advanced speakers, incidence of false starts also emerged as predicting communicatively adequate speech. Task type did not influence the link between linguistic features and adequacy.

Encouraging innovation in a modern foreign language initial teacher education programme: What do beginning teachers make of task-based language teaching?

Abstract: Task-based language teaching (TBLT) is an innovative learner-centred and experiential approach to modern foreign language (MFL) teaching and learning that is not without controversy in the secondary MFL classroom. This article considers one secondary-level initial teacher education programme in New Zealand in which, following school curriculum reform, TBLT has become a specific focus. Drawing on aspects of participants' critical reflections as part of the year-long programme, the article explores participants' developing understandings of TBLT and its suitability for New Zealand's secondary schools. It was found that beginning teachers were generally positive about TBLT. They also perceived several challenges to its successful implementation. Reflections after experiences in schools revealed that local school contexts, including the levels of understanding of, and receptivity to, innovation among more experienced colleagues, made a difference to successful task utilisation. The article draws some conclusi...

Context-sensitive adjustment of cognitive control in dual-task performance.

Abstract: Performing 2 highly similar tasks at the same time requires an adaptive regulation of cognitive control to shield prioritized primary task processing from between-task (cross-talk) interference caused by secondary task processing. In the present study, the authors investigated how implicitly and explicitly delivered information promotes the flexible online adjustment of task shielding in dual-task performance. Context-specific implicit activation of cognitive control was implemented by location-dependent manipulations of the likelihood of between-task interference (i.e., locations containing high vs. low proportions of between-task interference trials). Following practice, between-task interference was reduced in a subsequent test session for locations associated with high (compared to locations with low) task-shielding demands, indicating that the cognitive system can register and utilize implicit context features (Experiments 1 and 2). In Experiment 3, cues were used that provided additional explicit information. Whereas cues validly indicating the interference level in the next trial failed to further optimize context-specific task shielding, cues indicating the location of subsequent stimulus presentation resulted in an instant adjustment of task shielding already in the first part of the experiment. Results highlight the role of implicit and explicit information for context-sensitive adjustments of cognitive control in dual-task performance.

Optimizing task recommendations in context-aware mobile crowdsourcing

Abstract: A “Context-Aware Crowdsourced Task Optimizer” provides various processes to optimize task recommendations for workers in mobile crowdsourcing scenarios by automatically identifying and recommending bundles of tasks compatible with workers' contexts (e.g., worker history, present or expected locations, travel paths, working hours, skill sets, capabilities of worker's mobile computing devices, etc.). The Context-Aware Crowdsourced Task Optimizer bundles tasks to both maximize expected numbers of completed tasks and to dynamically price tasks to maximize the system's utility, which is a function of task values and task completion rates. Advantageously, the resulting task identification and recommendation process incentivizes individual workers to perform more tasks in a shorter time period, thereby helping tasks to complete faster, even with smaller budgets. While such optimization problems are NP-hard, the Context-Aware Crowdsourced Task Optimizer exploits monotonicity and submodularity of various objective functions to provide computationally feasible task identification and recommendation algorithms with tight optimality bounds.

Does listening to preferred music improve reading comprehension performance

Abstract: This article was published in Applied Cognitive Psychology on 06 January 2014 (online), available at http://dx.doi.org/10.1002/acp.2994