The State of Research on Situational Judgment Tests: A Content Analysis and Directions for Future Research
TL;DR: Situational judgment tests (SJTs) have become popular selection methods, with 59 empirical studies having been published since 1990 as discussed by the authors, and a review is organized around a single question: What are the current practices in SJT research? Using this question as a foundation, the content analysis on three significant theoretical and practical themes: (a) SJT development, scoring methods, and uses; (b) types of reliability estimates reported for SJTs, and (c) attributes that enhance or reduce internal consistency reliability.
Abstract: Situational judgment tests (SJTs) have become popular selection methods, with 59 empirical studies having been published since 1990. In contrast to prior narrative reviews or meta-analyses, this study develops (a) a comprehensive structure of SJT features, or “attributes,” (b) uses this structure to quantitatively and qualitatively summarize existing research in a content analysis, and then (c) uses the content analysis to generate directions for future research. The review is organized around a single question: What are the current practices in SJT research? Using this question as a foundation, we focus the content analysis on three significant theoretical and practical themes: (a) SJT development, scoring methods, and uses; (2) types of reliability estimates reported for SJTs, and (3) attributes that enhance or reduce internal consistency reliability.
Citations
More filters
TL;DR: In this paper, the authors examined the dynamic properties of turnover rates, including the changing quality and quantity of the human capital resources that depart, the changing turnover dispersion (i.e., how distributed turnover events are over time), and the changing quantity of replacement hires.
Abstract: This study builds from context-emergent turnover theory to examine the dynamic properties of turnover rates, including: (a) the changing quality and quantity of the human capital resources that depart, (b) the changing turnover dispersion (i.e., how distributed turnover events are over time), and (c) the changing quality and quantity of replacement hires. We examine these properties using data drawn from a sample of retail employees nested within stores of a prominent U.S. retail chain over five quarters, and show that the turnover rate (level) is conceptually and empirically distinct from turnover rate change, and that the two interact with each other to influence changes in unit performance. We also find that the relationship between turnover rate change and change in unit performance is moderated by both the quality of those who leave as well as turnover dispersion. Overall, we contribute to turnover rate, staffing, and human capital resource literatures by testing core context-emergent turnover theory propositions to show when, why, and how turnover rate change and replacement hires, as part of a holistic human capital resource system, influence unit performance
91 citations
TL;DR: This review discusses various conceptualizations of social-emotional skills, demonstrates their overlap with related constructs such as emotional intelligence and the Big Five personality dimensions, and proposes an integrative set of social/emotional skill domains that has been developed recently.
Abstract: The development and promotion of social-emotional skills in childhood and adolescence contributes to subsequent well-being and positive life outcomes. However, the assessment of these skills is associated with conceptual and methodological challenges. This review discusses how social-emotional skill measurement in youth could be improved in terms of skills' conceptualization and classification, and in terms of assessment techniques and methodologies. The first part of the review discusses various conceptualizations of social-emotional skills, demonstrates their overlap with related constructs such as emotional intelligence and the Big Five personality dimensions, and proposes an integrative set of social-emotional skill domains that has been developed recently. Next, methodological approaches that are innovative and may improve social-emotional assessments are presented, illustrated by concrete examples. We discuss how these innovations could advance social-emotional assessments, and demonstrate links to similar issues in related fields. We conclude the review by providing several concrete assessment recommendations that follow from this discussion. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
88 citations
Cites background from "The State of Research on Situationa..."
...…of SJT scenarios should be inductively driven—based on critical incidents as provided by subject matter experts—, or deductively driven—based on theoretical frameworks—(Campion et al., 2014), the study by Klassen and his colleagues suggests that an integration of both should be the ultimate goal....
[...]
...While researchers do not unanimously agree on whether the development of SJT scenarios should be inductively driven—based on critical incidents as provided by subject matter experts—, or deductively driven—based on theoretical frameworks—(Campion et al., 2014), the study by Klassen and his colleagues suggests that an integration of both should be the ultimate goal....
[...]
TL;DR: In this article, the authors systematically review, meta-analyze, and critically discuss the literature on selection, optimization, and compensation (SOC) strategy use at work and outline directions for future research and practice, concluding that SOC strategy use can mediate and moderate relationships of person and contextual antecedents with work outcomes.
Abstract: Over the past two decades, the selection, optimization, and compensation (SOC) model has been applied in the work context to investigate antecedents and outcomes of employees’ use of action regulation strategies. We systematically review, meta-analyze, and critically discuss the literature on SOC strategy use at work and outline directions for future research and practice. The systematic review illustrates the breadth of constructs that have been studied in relation to SOC strategy use, and that SOC strategy use can mediate and moderate relationships of person and contextual antecedents with work outcomes. Results of the meta-analysis show that SOC strategy use is positively related to age (rc= .04), job autonomy (rc= .17), self-reported job performance (rc= .23), non-self-reported job performance (rc= .21), job satisfaction (rc= .25), and job engagement (rc= .38), whereas SOC strategy use is not significantly related to job tenure, job demands, and job strain. Overall, our fi ndings underline the importance of the SOC model for the work context, and they also suggest that its measurement and reporting standards need to be improved to become a reliable guide for future research and organizational practice. Copyright © 2016 John Wiley & Sons, Ltd.
73 citations
TL;DR: The authors examined what education professionals in four countries (England, Finland, Malawi, and Oman) deemed as the critical (i.e., most important for effective teaching) non-cognitive attributes of novice teachers.
70 citations
Cites background from "The State of Research on Situationa..."
...These attributes, once identified and defined, form the basis for SJT development, typically using a critical incident approach to generate test content that assesses the non-cognitive attributes under consideration (Campion et al., 2014)....
[...]
TL;DR: In this article, the authors present a comprehensive review of existing frameworks, definitions, and assessments of civic-related constructs from approximately 30 projects relevant to higher education, and include a discussion of the challenges related to assessment design and implementation.
Abstract: Civic learning is increasingly recognized as important by the higher education and workforce communities. The development of high-quality assessments that can be used to evaluate students' civic learning during the college years has become a priority. This paper presents a comprehensive review of existing frameworks, definitions, and assessments of civic-related constructs from approximately 30 projects relevant to higher education, and includes a discussion of the challenges related to assessment design and implementation. Synthesizing information from the review, we propose an assessment framework to guide the design of a next-generation assessment of individuals' civic learning that takes advantage of recent advances in assessment methods. The definition identifies 2 key domains within civic learning: civic competency and civic engagement. Civic competency encompasses 3 areas (civic knowledge; analytic skills; and participatory and involvement skills), and civic engagement also captures 3 areas (motivations, attitudes, and efficacy; democratic norms and values; and participation and activities). We discuss item formats and task types that would ensure fair and reliable scoring for the assessment. The review of definitions of civic learning and its components developed by organizations, the proposed assessment framework, and assessment considerations presented here have potential benefits for a range of higher education institutions. This includes institutions that currently have students engaged in relevant curricular or cocurricular activities and also institutions that would find assessments of civic competency and engagement helpful in program development or in evaluating students' accomplishments.
66 citations
References
More filters
TL;DR: In this paper, a general formula (α) of which a special case is the Kuder-Richardson coefficient of equivalence is shown to be the mean of all split-half coefficients resulting from different splittings of a test, therefore an estimate of the correlation between two random samples of items from a universe of items like those in the test.
Abstract: A general formula (α) of which a special case is the Kuder-Richardson coefficient of equivalence is shown to be the mean of all split-half coefficients resulting from different splittings of a test. α is therefore an estimate of the correlation between two random samples of items from a universe of items like those in the test. α is found to be an appropriate index of equivalence and, except for very short tests, of the first-factor concentration in the test. Tests divisible into distinct subtests should be so divided before using the formula. The index
$$\bar r_{ij} $$
, derived from α, is shown to be an index of inter-item homogeneity. Comparison is made to the Guttman and Loevinger approaches. Parallel split coefficients are shown to be unnecessary for tests of common types. In designing tests, maximum interpretability of scores is obtained by increasing the first-factor concentration in any separately-scored subtest and avoiding substantial group-factor clusters within a subtest. Scalability is not a requisite.
37,235 citations
"The State of Research on Situationa..." refers background in this paper
...In reality, the magnitude of coefficient alpha is more a function of the level of item-specific variance (or item equivalence; see Cronbach, 1951)....
[...]
Book•
01 Nov 1980
TL;DR: In his book Culture's Consequences, Geert Hofstede proposed four dimensions on which the differences among national cultures can be understood: Individualism, Power Distance, Uncertainty Avoidance and Masculinity as mentioned in this paper.
Abstract: In his bestselling book Culture's Consequences, Geert Hofstede proposed four dimensions on which the differences among national cultures can be understood: Individualism, Power Distance, Uncertainty Avoidance and Masculinity. This volume comprises the first in-depth discussion of the masculinity dimension and how it can help us to understand differences among cultures. The book begins with a general explanation of the masculinity dimension, and discusses how it illuminates broad features of different cultures. The following parts apply the dimension more specifically to gender (and gender identity), sexuality (and sexual behaviour) and religion, probably the most influential variable of all. Hofstede closes the book with a synthesizing statement about cultural values as they are linked to sexuality, gender and religion.
19,826 citations
TL;DR: A review of the Social Sciences Citations Index for the literature from 1966 to 1990 revealed that Cronbach's (1951) article had been cited approximately 60 times per year and in a total of 278 different journals.
Abstract: Coefficient alpha (Cronbach, 1951) is certainly one of the most important and pervasive statistics in research involving test construction and use. A review of the Social Sciences Citations Index for the literature from 1966 to 1990 revealed that Cronbach's (1951) article had been cited approximately 60 times per year and in a total of 278 different journals. In addition to the areas of psychology in which one may expect to see alpha used, such as educational, industrial, social, clinical, child, community, and abnormal psychology, this list of journals included representatives from experimental psychology, sociology, statistics, medicine, counseling, nursing, economics, political science, criminology, gerontology, broadcasting, anthropology, and accounting. In spite of its widespread use, however, there is some confusion as to the true meaning and proper interpretation of the statistic. In this article I address this confusion in two ways. First, a theoretical discussion of alpha is presented. This includes some of the many statements that have been made about alpha and an attempt to integrate these statements. Second, I take a more practical approach in which the interpretation of alpha is examined by observing the changes in alpha as the number of items and interitem correlations are manipulated.
7,137 citations
"The State of Research on Situationa..." refers background in this paper
...First, it is a commonly held belief that coefficient alpha is a measure of first-factor saturation (Cortina, 1993)....
[...]
...Indeed, as Cortina (1993) stated, “the finer the distinction that needs to be made, the better the reliability must be” (p....
[...]
...First, it is a commonly held belief that coefficient alpha is a measure of first-factor saturation (Cortina, 1993). As shown and discussed in detail by Green, Lissitz, and Mulaik (1977), this does not have to be the case....
[...]
...…of SJTs may not be the principal factor in determining the magnitude of coefficient alpha; this point is made clear in discussions involving the relationship between alpha (interrelatedness of items) and homogeneity (dimensionality; e.g., Cortina, 1993; Green et al., 1977; Schmitt, 1996)....
[...]
...Internal consistency is a function of the interrelatedness of items, whereas homogeneity relates to dimensionality (Cortina, 1993; Schmitt, 1996)....
[...]
01 Jan 1980
4,406 citations
Book•
01 Jan 1993TL;DR: The adaptive decision maker: a look backward and a look forward Appendix Footnotes Bibliography.
Abstract: Preface 1. Adaptive decision behaviour: an introduction 2. Contingencies in decision making 3. Deciding how to decide: an effort/accuracy framework 4. Studying contingent decisions: an integrated methodology 5. Constructive processes in decision making 6. Why may adaptivity fail? 7. Improving decisions and other practical matters 8. The adaptive decision maker: a look backward and a look forward Appendix Footnotes Bibliography.
4,056 citations
"The State of Research on Situationa..." refers background in this paper
...Although these narrative reviews and meta-analyses are important, there are still many important but unanswered questions about SJTs....
[...]
...It seems likely that both national culture (e.g., Hofstede, 1980; House, Hanges, Javidan, Dorfman, & Gupta, 2004) and organizational culture (see Ployhart & Schneider, 2012) can influence the interpretation, response, and scoring of SJTs....
[...]
...A related issue concerns the internal consistency reliability of SJTs....
[...]
...Although valuable, it is time for researchers to expand their focus and begin pursuing other SJT research questions: questions that involve new theoretical approaches, questions that look at new developmental methodologies, and questions that broaden the nomological network of SJTs....
[...]
...Schmitt, Oswald, Kim, Gillespie, and Ramsay (2004) presented findings that highlight the relationship between respondent perceptions of relevance and fairness on SJTs....
[...]