The issue of whether IS positivist researchers were validating their instruments sufficiently was initially raised fifteen years ago. Rigor in IS research is still one of the critical scientific issues facing the field. Without solid validation of the instruments that are used to gather data on which findings and interpretations are based, the very scientific basis of the profession is threatened. This study builds on four prior retrospectives of IS research that conclude that IS positivist researchers continue to face major barriers in instrument, statistical, and other forms of validation. It goes beyond these studies by offering analyses of the state-of-the-art of research validities and deriving specific heuristics for research practice in the validities. Some of these heuristics will, no doubt, be controversial. But we believe that it is time for the IS academic profession to bring such issues into the open for community debate. This article is a first step in that direction. Based on our interpretation of the importance of a long list of validities, this paper suggests heuristics for reinvigorating the quest for validation in IS research via content/construct validity, reliability, manipulation validity, and statistical conclusion validity. New guidelines for validation and new research directions are offered.

/pdf/validation-guidelines-for-is-positivist-research-5fusz79fbw.pdf

Validation guidelines for is positivist research

Internet technologies are having a significant impact on the learning industry. For-profit organizations and traditional institutions of higher education have developed and are using web-based courses, but little is known about their effectiveness compared to traditional classroom education. Our work focuses on the effectiveness of a web-based virtual learning environment (VLE) in the context of basic information technology skills training.

This article provides three main contributions. First, it introduces and defines the concept of VLE, discussing how a VLE differs from the traditional classroom and differentiating it from the related, but narrower, concept of computer aided instruction (CAI). Second, it presents a framework of VLE effectiveness, grounded in the technology-mediated learning literature, which frames the VLE research domain, and addresses the relationship between the main constructs. Finally, it focuses on one essential VLE design variable, learner control, and compares a web-based VLE to a traditional classroom through a longitudinal experimental design.

Our results indicate that, in the context of IT basic skills training in undergraduate education, there are no significant differences in performance between students enrolled in the two environments. However, the VLE leads to higher reported computer self-efficacy, while participants report being less satisfied with the learning process.

/pdf/web-based-virtual-learning-environments-a-research-1e0yna0ug3.pdf

Web-Based virtual learning environments: a research framekwork and a preliminary assessment of effectiveness in basic IT skills training

With the growing reliance on computerized systems and increasing rapidity of the introduction of new technologies, user acceptance of technology continues to be an important issue. Drawing upon recent findings in information systems, human computer interaction, and social psychology, the present research extends the technology acceptance model by incorporating the motivation variables of self-efficacy, enjoyment, and learning goal orientation in order to predict the use of Web-based information systems. One hundred nine subjects participated in the study, which was conducted in a field setting with the Blackboard system, a Web-based class management system. A survey was administered after a 2-week trial period and the actual use of the system was recorded by the Blackboard system over 8 weeks. The results largely support the proposed model, highlighting the important roles of self-efficacy, enjoyment, and learning goal orientation in determining the actual use of the system. Practical implications of the results are provided.

Predicting the use of web-based information systems: self-efficacy, enjoyment, learning goal orientation, and the technology acceptance model

In information systems, most research on knowledge management assumes that knowledge has positive implications for organizations. However, knowledge is a double-edged sword: while too little might result in expensive mistakes, too much might result in unwanted accountability. The purpose of this paper is to highlight the lack of attention paid to the unintended consequences of managing organizational knowledge and thereby to broaden the scope of IS-based knowledge management research. To this end, this paper analyzes the IS literature on knowledge management. Using a framework developed by Deetz (1996), research articles published between 1990 and 2000 in six IS journals are classified into one of four scientific discourses. These discourses are the normative, the interpretive, the critical, and the dialogic. For each of these discourses, we identify the research focus, the metaphors of knowledge, the theoretical foundations, and the implications apparent in the articles representing it. The metaphors of knowledge that emerge from this analysis are knowledge as object, asset, mind, commodity, and discipline. Furthermore, we present a paper that is exemplary of each discourse. Our objective with this analysis is to raise IS researchers' awareness of the potential and the implications of the different discourses in the study of knowledge and knowledge management.

Studying knowledge management in information systems research: discourses and theoretical assumptions

Computer skills are key to organizational performance, and past research indicates that behavior modeling is a highly effective form of computer skill training. The present research develops and tests a new theoretical model of the underlyingobservational learning processes by which modeling-based training interventions influence computer task performance. Observational learning processes are represented as a second-order construct with four dimensions (attention, retention, production, and motivation). New measures for these dimensions were developed and shown to have strong psychometric properties. The proposed model controls for two pretraining individual differences (motivation to learn and self-efficacy) and specifies the relationships among three training outcomes (declarative knowledge, post-training self-efficacy, and task performance). The model was tested using PLS on data from an experiment ( N = 95) on computer spreadsheet training. As hypothesized, observational learning processes significantly influenced training outcomes. A representative modeling-based training intervention (retention enhancement) significantly improved task performance through its specific effects on the retention processes dimension of observational learning. The new model provides a more complete theoretical account of the mechanisms by which modeling-based interventions affect training outcomes, which should enable future research to systematically evaluate the effectiveness of a wide range of modeling-based training interventions. Further, the new instruments can be used by practitioners to refine ongoing training programs.

Developing and Validating an Observational Learning Model of Computer Software Training and Skill Acquisition

This study compares traditional and nontraditional training techniques with regard to computer related training. Its purpose was to determine which training methods could best be utilized in computer related training to maximize a trainee's retention of material and transfer of learning. A field experiment was conducted using two hundred members of active duty U.S. Naval Construction Battalion as subjects. Evaluation of trainees included a pre-training screening, post-training evaluation immediately after training, and a follow-up session four weeks after the post-training session utilizing previously validated instruments. Training treatments included instruction lecture, exploration independent study, and a nontraditional technique---behavior modeling an enhanced combination of the other two methods. Performance outcomes were operationalized using hands-on task performance and comprehension of the computer system as dependent variables. End-user satisfaction with the computer system was also measured. Two covariates, cognitive ability and system use, were also introduced into the study. The use of hands-on training methods, especially behavior modeling, resulted in superior retention of knowledge, transfer of learning, and end-user satisfaction. Cognitive ability failed to be a good predictor of trainee success but a connection was established between training methodology, system use, and end-user satisfaction.

The Relationship of Information System Training Methods and Cognitive Ability to End-User Satisfaction, Comprehension, and Skill Transfer: A Longitudinal Field Study

In this study, major MIS research institutions are identified and ranked based on their contributions to quality MIS publication outlets. Over 10,000 pages of research generated over the past decade (1982-1991) are credited to voer 190 institutions. The top 50 institutions are ranked. The method followed is similar to other studies conducted in well-established research areas in business, such as finance, marketing and economics. The results provide objective and useful information to the profession.

An assessment of institutional research productivity in MIS

Information Technologyâ€™s (IT) link to the strategy of the firm has been explored in Information System literature, but the application of IT to the strategy of the international firm has received limited attention. This paper addresses the topic by exploring the use of IT in conjunction with a popular framework, the Integration- Responsiveness (IR) framework, of international business strategy. The frameworkâ€™s three subgroups; global integration, multifocal, and local responsiveness are discussed. Specific applications of IT are then examined with reference to the dimensions that have been used to operationalize the IR framework. They are complex innovation, marketing differentiation, breadth, and conservative cost control. The paper concludes with implications for practice and research.

Strategic Use of Information Technology in International Business: A Framework for Information Technology Application

A major goal of information systems is to make computers easier to use and of more benefit to users, thereby having more satisfied users. This becomes increasingly important as businesses and organizations find themselves with growing numbers of inexperienced users and expanding use of computers in the work place. Information Systems departments confronted with this problem have been tasked with developing more effective training programs and have sought factors through which an individual's success can be predicted. The purpose of this dissertation is to investigate different training methods and their relation to the individual user's cognitive ability, comprehension, skill transfer, and end user satisfaction.
A field experiment was conducted that compared three training techniques: (1) Lecture (Instruction-based) training presents general rules and requires users to apply them to specific examples. (2) Independent (Exploration-based) training requires users to independently learn a computer system by reasoning from specific examples. (3) Behavior Modeling--a non-traditional training technique--is a combination of the exploration and instruction methods which concentrates on the idea of observing and doing while following a role model. Cognitive ability is to be used as a covariate across training treatment groups and to determine its role as a predictor of training success.
The field experiment was conducted utilizing two Navy Construction Battalions that are currently implementing a new computerized Logistics System. Novice computer users were randomly trained utilizing one of the three training methods. Upon completion of each training session the trainees were tested for retention of general and procedural knowledge. Testing consisted of both traditional paper-and-pencil as well as direct computer interaction material. Four weeks after the initial training sessions, the trainees were retested to determine the level of skill transfer and long term retention. At that time, trainees were polled as to their satisfaction with the computer system.
Results of the study indicate that the behavior modeling technique led to higher levels of retention and skill transfer. This was reflected in better performance in far-transfer (complex) tasks and near-transfer (simple) tasks and resulted in more satisfied users. Trainees in the exploration treatment performed better and were more satisfied than the instruction trainees indicating the need to include a hands-on component in computer related training. Cognitive ability failed to perform as a predictor of training success.

The relationship of information system training methods and cognitive ability to end user satisfaction, comprehension and skill transfer: a field experiment

Steven J. Simon

Papers

The Relationship of Information System Training Methods and Cognitive Ability to End-User Satisfaction, Comprehension, and Skill Transfer: A Longitudinal Field Study

An assessment of institutional research productivity in MIS

Strategic Use of Information Technology in International Business: A Framework for Information Technology Application

The relationship of information system training methods and cognitive ability to end user satisfaction, comprehension and skill transfer: a field experiment