Definition: To what extent does the study allow us to draw conclusions about a causal effect between two or more constructs? Issues: Selection, maturation, history, mortality, testing, regression towrd the mean, selection by maturation, treatment by mortality, treatment by testing, measured treatment variables Increase: Eliminate the threats, above all do experimental manipulations, random assignment, and counterbalancing.

Reliability and validity

We present a targeted review of recent developments and advances in digital selection procedures (DSPs) with particular attention to advances in internet-based techniques. By reviewing the emergence of DSPs in selection research and practice, we highlight five main categories of methods (online applications, online psychometric testing, digital interviews, gamified assessment and social media). We discuss the evidence base for each of these DSP groups, focusing on construct and criterion validity, and applicant reactions to their use in organizations. Based on the findings of our review, we present a critique of the evidence base for DSPs in industrial, work and organizational psychology and set out an agenda for advancing research. We identify pressing gaps in our understanding of DSPs, and ten key questions to be answered. Given that DSPs are likely to depart further from traditional non-digital selection procedures in the future, a theme in this agenda is the need to establish a distinct and specific literature on DSPs, and to do so at a pace that reflects the speed of the underlying technological advancement. In concluding, we, therefore, issue a call to action for selection researchers in work and organizational psychology to commence a new and rigorous multidisciplinary programme of scientific study of DSPs.

/pdf/personnel-selection-in-the-digital-age-a-review-of-validity-46epcwqayz.pdf

Personnel selection in the digital age: a review of validity and applicant reactions, and future research challenges

Context Historically, Situational Judgement Tests (SJTs) have been widely used for personnel selection. Their use in medical selection in Europe is growing with plans for further expansion into North America and Australasia in an attempt to measure and select on ‘non-academic’ personal attributes. However, there is a lack of clarity regarding what such tests actually measure and how they should be designed, scored and implemented within the medical and health education selection process. In particular, the theoretical basis from which such tests are developed will determine the scoring options available, influencing their psychometric properties and, ultimately, their validity. Objective The aim of this article is to create an awareness of the previous theory and practice that has informed SJT development. We describe the emerging interest in the use of the SJT format to measure specific constructs (e.g. ‘resilience’, ‘dependability’ etc.), drawing on the tradition of ‘individual differences’ psychology. We compare and contrast this newer ‘construct-driven’ method with the traditional, pragmatic approach to SJT creation, often employed by organisational psychologists. Making reference to measurement theory, we highlight how the anticipated psychometric properties of traditional versus construct-driven SJTs are likely to differ. Conclusions Compared to traditional SJTs, construct-driven SJTs have a strong theoretical basis, are uni- rather than multidimensional, and may behave more like personality selfreport instruments. Emerging evidence also suggests that construct-driven SJTs have comparable predictive validity for workplace performance, although they may be more prone to ‘faking’ effects. It is possible that construct-driven approaches prove more appropriate at early stages of medical selection, where candidates have little or no healthcare work experience. Conversely, traditional SJTs may be more suitable for 3 specialty recruitment, where a range of hypothetical workplace scenarios can be sampled in assessments.

/pdf/situational-judgement-tests-for-selection-traditional-vs-2af7loa7c2.pdf

Situational judgement tests for selection: Traditional vs construct-driven approaches.

Recent theorizing and empirical evidence suggesting that Situational Judgment Tests (SJTs) are more context-independent than previously thought has sparked a debate about the role of situation descriptions in SJTs. To contribute to this debate and add to our understanding of how SJTs work, this article conceptually embeds SJT performance in a situation construal model and examines the effects of situation descriptions on the construct saturation and predictive validity of SJT scores, as well as on applicant perceptions. Across two studies (N = 1,092 and 578) and different SJTs, personality and cognitive ability were equally important determinants of SJT performance regardless of whether situation descriptions were presented or omitted. The effects of removing situation descriptions on the criterion-related validity of SJT scores differed depending on the breadth of the criteria. For predicting global job performance criteria (in-role performance and organizational citizenship behavior), SJT validity was not significantly affected, whereas it decreased for predicting more specific criteria (interpersonal adaptability, efficacy for teamwork). Finally, the effects of omitting situation descriptions in SJTs on applicant perceptions were either negligible or small. Implications for SJT theory, research, and design are discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=7431&context=lkcsb_research

The role of Situations in Situational Judgment Tests: Effects on construct saturation, predictive validity, and applicant perceptions.

Situational Judgment Tests (SJTs) are criterion valid low fidelity measures that have gained much popularity as predictors of job performance. A broad variety of SJTs have been studied, but SJTs measuring personality are still rare. Personality traits such as Conscientiousness are valid predictors of many educational, work and life-related outcomes and SJTs are less prone to faking than classical self-report measurements. We developed an SJT measure of Dependability, a core facet of Conscientiousness, by gathering critical incidents in semi-structured interviews using the construct definition of Dependability as a prompt. We examined the psychometric properties of the newly developed SJTs across two studies (N = 546 general population; N = 440 sales professionals). The internal validity of the SJTs was examined by correlating the SJT scores with related self-report measures of Dependability and Conscientiousness, as well as testing the unidimensionality of the measure with CFA. Additionally, we specified a bi-factor model of SJT, self-report and behavioral checklist measures of Dependability accounting for common and specific measurement variance. External validity was examined by correlating the SJT scale and specific factor with work-related outcomes. The results show that the Dependability SJTs with an expert based scoring procedure were psychometrically sound and correlated moderately to highly with traditional self-report measures of Dependability and Conscientiousness. However, a large proportion of SJT variance cannot be accounted for by personality alone. This supports the notion that SJTs measure general domain knowledge about the effectiveness of personality-related behaviors. We conclude that SJT measures of personality can be a promising addition to classical self-report assessments and can be used in a wide variety of applications beyond measurement and selection, for instance as formative assessments of personality.

/pdf/situational-judgment-tests-as-a-method-for-measuring-35nqasnw3d.pdf

Situational Judgment Tests as a method for measuring personality: Development and validity evidence for a test of Dependability.

Despite their popularity and capacity to predict performance, there is no clear consensus on the internal measurement characteristics of situational judgment tests (SJTs). Contemporary propositions in the literature focus on treating SJTs as methods, as measures of dimensions, or as measures of situational responses. However, empirical evidence relating to the internal structure of SJT scores is lacking. Using generalizability theory, we decomposed multiple sources of variance for three different SJTs used with different samples of job candidates (N1 = 2,320; N2 =989; N3 = 7,934). Results consistently indicated that (a) the vast majority of reliable observed score variance reflected SJT-specific candidate main effects, analogous to a general judgment factor and that (b) the contribution of dimensions and situations to reliable SJT variance was, in relative terms, negligible. These findings do not align neatly with any of the proposals in the contemporary literature; however they do suggest an internal structure for SJTs.

/pdf/the-internal-structure-of-situational-judgement-tests-29qvn1xcm8.pdf

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

To promote diversity in organizations it is important to have accurate knowledge about subgroup differences associated with selection procedures. However, current estimates of subgroup differences in situational judgment tests (SJTs) are overwhelm‐ingly based on range‐restricted incumbent samples that are downwardly biased. This study provides much‐needed applicant level estimates of SJT subgroup differences (N = 37,530). As a key finding, Black‐White differences (d = 0.66) were higher than in incumbent samples (d = 0.38). Overall, sex differences were small. Females scored higher for management jobs (d = −0.13) and males scored higher for administrative jobs (d = 0.15). By analyzing applicant samples that do not suffer from range restric‐tion, this study adds knowledge about subgroup differences in SJTs.

/pdf/subgroup-differences-in-situational-judgment-test-scores-4b72ijmdmh.pdf

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

The following paper is focused on neurodiversity and disability in assessment, an area we believe needs significantly greater attention by researchers, organisations and practitioners. Currently there is very limited research on this topic and, as a result, guidance around neurodiversity in recruitment is generally unclear. The aim of this paper is to provide some insights into the area and share the results of an evaluation of 452,933 candidates who completed a Situational Judgement Test (SJT) in a recruitment context. We hope that the evidence from this study will help increase our understanding of the topics of SJTs, disability, neurodiversity, and assessment.

Ali Shalfrooshan

Papers

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

Neurodiversity, disability and assessment: Evaluation of the impact of SJTs during recruitment

Are assessment centres robot proof?