When data are collected via anonymous Internet surveys, particularly under conditions of obligatory participation (such as with student samples), data quality can be a concern. However, little guidance exists in the published literature regarding techniques for detecting careless responses. Previously several potential approaches have been suggested for identifying careless respondents via indices computed from the data, yet almost no prior work has examined the relationships among these indicators or the types of data patterns identified by each. In 2 studies, we examined several methods for identifying careless responses, including (a) special items designed to detect careless response, (b) response consistency indices formed from responses to typical survey items, (c) multivariate outlier analysis, (d) response time, and (e) self-reported diligence. Results indicated that there are two distinct patterns of careless response (random and nonrandom) and that different indices are needed to identify these different response patterns. We also found that approximately 10%–12% of undergraduates completing a lengthy survey for course credit were identified as careless responders. In Study 2, we simulated data with known random response patterns to determine the efficacy of several indicators of careless response. We found that the nature of the data strongly influenced the efficacy of the indices to identify careless responses. Recommendations include using identified rather than anonymous responses, incorporating instructed response items before data collection, as well as computing consistency indices and multivariate outlier analysis to ensure high-quality data.

http://ubc-emotionlab.ca/wp-content/uploads/2012/06/Meade-Craig-2012-Careless-Responding-in-Survey-Data.pdf

Identifying careless responses in survey data.

Responses provided by unmotivated survey participants in a careless, haphazard, or random fashion can threaten the quality of data in psychological and organizational research. The purpose of this study was to summarize existing approaches to detect insufficient effort responding (IER) to low-stakes surveys and to comprehensively evaluate these approaches. In an experiment (Study 1) and a nonexperimental survey (Study 2), 725 undergraduates responded to a personality survey online. Study 1 examined the presentation of warnings to respondents as a means of deterrence and showed the relative effectiveness of four indices for detecting IE responses: response time, long string, psychometric antonyms, and individual reliability coefficients. Study 2 demonstrated that the detection indices measured the same underlying construct and showed the improvement of psychometric properties (item interrelatedness, facet dimensionality, and factor structure) after removing IE respondents identified by each index. Three approaches (response time, psychometric antonyms, and individual reliability) with high specificity and moderate sensitivity were recommended as candidates for future application in survey research. The identification of effective IER indices may help researchers ensure the quality of their low-stake survey data. This study is a first attempt to comprehensively evaluate IER detection methods using both experimental and nonexperimental designs. Results from both studies corroborated each other in suggesting the three more effective approaches. This study also provided convergent validity evidence regarding various indices for IER.

http://jasonhuangatwork.com/papers/Huang_et_al_2012_JBP-IE_Responding.pdf

Detecting and Deterring Insufficient Effort Responding to Surveys

Best Practices in Exploratory Factor Analysis

Caring about carelessness: Participant inattention and its effects on research.

Etiology of Posttraumatic States Symptomatology and Phenomenology Critical Issues in Trauma-Relevant Assessment Assessment of Traumatic Events Diagnostic Interviews General Objective and Projective Measures Trauma-Specific Objective Measures Putting it All Together Assessment of Pre-Trauma Functioning Traumatic Events Moderating Influences Posttraumatic Response.

Psychological assessment of adult posttraumatic states

This study investigated the validity of F, back F (Fb), and variable response inconsistency (VRIN) indices of the revised Minnesota Multiphasic Personality Inventory (MMPI-2) for detection of self-reported partially random responding in several samples, including college students, community volunteers, and job applicants. Substantial numbers of subjects admitted to providing some random responses, with 29%-60% of the various samples acknowledging an average of 12-38 random responses in their MMPI-2 answers. For all groups, with the expected exception of job applicants, the appropriate MMPI-2 validity indices were reliably and positively correlated with self-estimates of random responses

MMPI-2 Random Responding Indices: Validation Using a Self-Report Methodology.

This study explored the effects of both random responding and malingering on the validity scales of the MMPI-2 in an analog design using 4 groups of college students. One group completed the entire answer sheet randomly, a second group was instructed to malinger a moderate psychological disturbance, a third group was asked to simulate a severe disturbance, and a fourth group was given standard test instructions

Lene H. Larsen

Papers

MMPI-2 Random Responding Indices: Validation Using a Self-Report Methodology.

Sensitivity of MMPI-2 validity scales to random responding and malingering.