Qualitative Interviewing: The Art of Hearing Data

Background
In fields such as medicine, psychology, and education, systematic reviews of the literature critically appraise and summarize research to inform policy and practice. We argue that now is an appropriate time in the development of the field of engineering education to both support systematic reviews and benefit from them. More reviews of prior work conducted more systematically would help advance the field by lowering the barrier for both researchers and practitioners to access the literature, enabling more objective critique of past efforts, identifying gaps, and proposing new directions for research.

Purpose
The purpose of this article is to introduce the methodology of systematic reviews to the field of engineering education and to adapt existing resources on systematic reviews to engineering education and other developing interdisciplinary fields.

Scope/Method
This article is primarily a narrative review of the literature on conducting systematic reviews. Methods are adapted to engineering education and similar developing interdisciplinary fields. To offer concrete, pertinent examples, we also conducted a systematic review of systematic review articles published on engineering education topics since 1990. Fourteen exemplars are presented in this article and used to illustrate systematic review procedures.

Conclusions
Systematic reviews can benefit the field of engineering education by synthesizing prior work, by better informing practice, and by identifying important new directions for research. Engineering education researchers should consider including systematic reviews in their repertoire of methodologies.

Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields

Context Global Software Development (GSD) is the process whereby software is developed by different teams located in various parts of the globe. One of the major reasons for GSD project failure is that a number of organizations endorse global development prior to understanding project management challenges for the global activity. Objective The objective of this paper is to identify the challenges, from the client and vendor perspectives, which can undermine the successful management of GSD projects. Method We followed a two-phase approach: we first identified the challenges via a Systematic Literature Review (SLR) and then the identified challenges were validated using a questionnaire-based survey. Results Through both approaches, we identified 19 challenges important to the success of GSD project management. A comparison of the challenges identified in client and vendor organizations indicates that there are more similarities than differences between the challenges. Our results show a positive correlation between the ranks obtained from the SLR and the questionnaire ((r s (19) = 0.102), p = 0.679). The results of t -test (i.e., t = 0.299, p = 0.768 > 0.05) show that there is no significant difference between the findings of SLR and questionnaire. Conclusions GSD organizations should try to address the identified challenges when managing their global software development activities to increase the probability of project success.

Challenges of project management in global software development: A client-vendor analysis

Context There is extensive interest in global software development (GSD) which has led to a large number of papers reporting on GSD. A number of systematic literature reviews (SLRs) have attempted to aggregate information from individual studies. Objective: We wish to investigate GSD SLR research with a focus on discovering what research has been conducted in the area and to determine if the SLRs furnish appropriate risk and risk mitigation advice to provide guidance to organizations involved with GSD. Method: We performed a broad automated search to identify GSD SLRs. Data extracted from each study included: (1) authors, their affiliation and publishing venue, (2) SLR quality, (3) research focus, (4) GSD risks, (5) risk mitigation strategies and, (6) for each SLR the number of primary studies reporting each risk and risk mitigation strategy. Results: We found a total of 37 papers reporting 24 unique GSD SLR studies. Major GSD topics covered include: (1) organizational environment, (2) project execution, (3) project planning and control and (4) project scope and requirements. We extracted 85 risks and 77 risk mitigation advice items and categorized them under four major headings: outsourcing rationale, software development, human resources, and project management. The largest group of risks was related to project management. GSD outsourcing rationale risks ranked highest in terms of primary study support but in many cases these risks were only identified by a single SLR. Conclusions: The focus of the GSD SLRs we identified is mapping the research rather than providing evidence-based guidance to industry. Empirical support for the majority of risks identified is moderate to low, both in terms of the number of SLRs identifying the risk, and in the number of primary studies providing empirical support. Risk mitigation advice is also limited, and empirical support for these items is low.

Risks and risk mitigation in global software development: A tertiary study

Background

Systematic review refers to a growing set of research methodologies intended to critically appraise and synthesize research to inform policy and practice. Systematic review methods, widely used in fields such as medicine, psychology, and education, are being applied in engineering education. Analysis of current systematic review practices in engineering education can contribute to continuous improvement of an important set of synthesis methodologies.



Purpose

To promote continuous improvement in systematic reviews, researchers are currently developing criteria to evaluate the quality of systematic review methodologies. The purpose of this article is to identify prior systematic reviews in areas of interest to the engineering education community, use these criteria to critique these systematic reviews, and make recommendations for improving use and quality of systematic review methods in engineering education research.



Scope/Method

Using systematic review search methodologies, we identified 12 conference papers and 37 journal articles that are systematic reviews published since 1990. Two reviewers coded each by topic, education level, quantitative or qualitative sources and synthesis methods, and 18 quantitative measures of review quality. Interrater reliability was calculated using kappa as described by Landis and Koch, which ranged from 0.43 (moderate) to 0.80 (almost perfect) with overall agreement of 0.64 (substantial).



Conclusions

We found that the 49 identified systematic reviews in engineering education are particularly strong in stating their purposes, rationale, and inclusion criteria. They can be improved by explaining reasons for excluding studies, appraising quality of included studies, engaging multiple coders in evaluation procedures, stating their limitations, and, in the case of meta-analyses, describing statistical methods in greater detail.

What is the state of the art of systematic reviewin engineering education

In recent years, the evolution of Global Software Development (GSD) has grown both rapidly and significantly, and although the efficiency of this new type of development has been proven, some challenging issues must still be confronted. Of all these, our research line is focused on designing the specific training that members of virtual teams must receive. Universities and companies therefore need to design training schemas to deal with the specifics of GSD, which are principally related to communication difficulties and time and cultural differences. In this work we present the findings of a Systematic Literature Review in the field of GSD training and teaching. Our intention is twofold: on the one hand we wish to discover the existing strategies and proposals available up to the present day, and on the other hand we wish to identify the open challenges, that will be helpful for practitioners and researchers in the future.

Preparing Students and Engineers for Global Software Development: A Systematic Review

SUMMARY
This paper tackles the need for universities and practitioners to train their students and engineers in the new challenges that global software development entails, which are principally related to communication, collaboration and cultural differences. Teaching the necessary skills requires practical experience. However, training in educational environments is difficult, and some challenging issues must still be confronted. We have focused our work on the development of a virtual training environment that can simulate global software development scenarios involving virtual agents (VAs) from different cultures. The VAs interact with learners, who use typical communication tools to solve predefined problems. This environment considers common problems caused by distance and cultural differences when using English as a means of communication. It allows learners to train at any time, because the VAs are always available, and it also permits them to play different roles in the various stages of the project. In this paper, we depict the design and development of the tool, as well as an initial evaluation. Copyright © 2011 John Wiley & Sons, Ltd.

Cultural and linguistic problems in GSD: a simulator to train engineers in these issues

Global Software Development (GSD) is an emerging paradigm in which development teams are geographically distributed throughout different countries whilst working on the same projects. Lower costs, greater access to a skilled workforce, the leveraging of time-zone effectiveness and a closer proximity to the market are some of the proven advantages that have contributed to this recent expansion. However, GSD also entails certain drawbacks caused mainly by distance and socio-cultural differences. Traditional face-to-face meetings are no longer common, thus making communication, coordination and collaboration more complex. Moreover, interactions frequently involve members who use non-native languages and whose customs and behavior may be misinterpreted. Companies consequently require their workforce to have a balanced set of socio-technical skills more than ever before. Students and practitioners therefore need to acquire not only new technical skills, but also the ability to collaborate and interact effectively in a variety of geographic, cultural and linguistic settings. This kind of training is not currently common and companies often complain about their new recruits? lack of preparation in GSD related skills. However, companies do not always have the resources to provide these kinds of training programs, as the teaching of this kind of skills requires appropriate tools, methods and course materials. Moreover, training these skills is not easy at universities since it necessitates providing trainees with real experiences that it may be difficult to reproduce. Some universities organize joint courses through collaboration with distant universities so that students can benefit from a wide set of knowledge and experiences, but the coordination problems of these approaches make them hard to apply. One way in which to avoid the problems outlined above, i.e., a lack of resources, coordination complexities, and poor levels of realism, is by using simulation, which has been used in various areas of Software Engineering training such as project management. However, although simulation has the potential to enable independent and controlled training, no significant proposals exist in the field of GSD training. The research question that guides this work is therefore: ?How can we model a GSD educational framework that replicates real-world settings?? This thesis presents a framework (VENTURE) for interactive GSD training which is based on simulation. The framework simulates real world interactions that occur in GSD which are appropriate for use in both industrial and academic settings. In order to simulate GSD settings, trainees take on a specific role and interact with Virtual Agents, characterized by a specific culture. A special kind of Virtual Agent, called a Virtual Guide, directs the trainees and provides feedback when they interact in an inappropriate manner. These training scenarios take place by means of a chat simulator for synchronous communication training and through an email simulator for asynchronous communication training. The feedback provided by the Virtual Guide is automatically used to assess the trainees? performance in real time following a defined assessment method. The courses are provided via an e-learning environment that allows students to access the training sessions at any time. Moreover, a scenario designer has been implemented to support the design of new simulated interactions. New simulations will build on predefined GSD problems, including cultural and linguistic rules that represent typical interaction challenges in real settings. In order to design GSD scenarios, realistic low-level GSD problems that arise during the day-to-day work are required. However, accessing these problems or scenarios is difficult, mainly because companies are not always willing to share their knowledge and experience. In order to tackle this challenge, the framework additionally considers a Web based knowledgebase of GSD patterns and scenarios. The GSD community can, through this open social network, share their experiences, problems and scenarios in an organized manner. Simulations may therefore be designed by applying the data collected in this collaborative knowledgebase. The VENTURE framework was created by following a mixed methodology that combines both qualitative and quantitative methods. An initial systematic literature review on GSD served to focus the topic on the field of GSD education, after which a second systematic literature review was conducted in the field of GSD education. Having studied the problems of the existing educational methods, a prototype of the framework was built and evaluated by experts, leading to a first version of the framework. The principal components of VENTURE (the chat simulator and the scenario designer) were then evaluated by a group of experts. These experts in GSD and e-learning examined the design and executed a brief training simulation in order to analyze the tool?s potential to provide GSD training, along with the tool?s usability. The results of the survey, which were analyzed by means of a thematic analysis procedure, were mainly positive. Both evaluations served to confirm that VENTURE has the potential to fill the gap in GSD virtual team training. Moreover, indications on where improvements could be made served to develop a new version of the framework. A Field Study of the new version of the framework was conducted in order to evaluate whether learning was achieved after participation on a real GSD course based on two training simulations. This evaluation involved potential users of the framework (students, trainers, researchers) interacting with VENTURE and completing two training scenarios. A rigorous evaluation of the user experience was conducted using qualitative and quantitative data collection. Several questionnaires (including pre and post questionnaires) were administered at different stages of the process and the interactions were also tracked for reasons of analysis in order to uncover any significant differences after the simulation process took place. The results revealed that participants, after going through the evaluation process, acquired new knowledge and capabilities that would allow them to perform better in GSD settings. The participants also agreed that they were satisfied with both the learning outcomes and the user experience. This thesis provides two main contributions: the first of these is a training framework designed to take advantage of Virtual Agents in order to design accurate simulations of GSD interactions. Part of this training framework is also to automatically assess the student?s progress. An invention disclosure has been produced, and commercialization activities are currently being addressed with regard to this contribution. The second contribution is the introduction of a new pattern model with which to acquire real world GSD problems that can be used with pedagogical purposes. Future work will consider the population of the GSD pattern based knowledgebase, along with improvements that could allow a more intelligent interaction of the Virtual Agents and facilitate the design of the simulations.

A framework for interaction training in Global Software Development

Sources of Global Software Development (GSD) information, such as academic literature, often focus on high-level issues rather than on specific problems. Researchers tend to generalize problems and solutions, however, practitioners and instructors frequently need to identify real low-level scenarios and patterns in an effort to study specific problems and their solutions. We propose a method for collecting and defining GSD scenarios and related patterns. Scenarios depicting events that happen in certain GSD contexts associated with communication, coordination are central to this method. In this paper we show how problems and solutions extracted from these events can lead to the definition of patterns. Patterns describe generalized information that can be re-used in similar contexts. To facilitate knowledge sharing, we have integrated this pattern model into a GSD Community Web intended to promote collaboration between industry and academia. News, resources and discussion forums on GSD topics are also available through this website.

Towards a Global Software Development Community Web: Identifying Patterns and Scenarios

In Global Software Development (GSD) the human factor is one of the main assets for the companies. Their efficiency in communication and collaboration, as well as their knowledge of the processes applied in GSD, can lead the companies to be more competitive.

Participants require knowing the customs and culture of other participants. Moreover they need to improve their social and interpersonal competencies such as: negotiation, teamwork, conflict resolution, time management, leadership, and communication skills using a common language.

In this paper we present a simulation-based approach for training GSD with which users can train by interacting with Virtual Agents which play a role in the development process. These Virtual Agents textually interact with users by means of a chat by simulating being people with different personalities, experiences, skills and culture.

The lessons learned in a feasibility study carried out with a group of practitioners and PhD students are also analyzed in this paper.

Miguel J. Monasor

Papers

Preparing Students and Engineers for Global Software Development: A Systematic Review

Cultural and linguistic problems in GSD: a simulator to train engineers in these issues

A framework for interaction training in Global Software Development

Towards a Global Software Development Community Web: Identifying Patterns and Scenarios

Providing training in GSD by using a virtual environment