Interest in the problem of method biases has a long history in the behavioral sciences. Despite this, a comprehensive summary of the potential sources of method biases and how to control for them does not exist. Therefore, the purpose of this article is to examine the extent to which method biases influence behavioral research results, identify potential sources of method biases, discuss the cognitive processes through which method biases influence responses to measures, evaluate the many different procedural and statistical techniques that can be used to control method biases, and provide recommendations for how to select appropriate procedural and statistical remedies for different types of research settings.

/pdf/common-method-biases-in-behavioral-research-a-critical-53ocysaemj.pdf

Common method biases in behavioral research: a critical review of the literature and recommended remedies.

Despite the concern that has been expressed about potential method biases, and the pervasiveness of research settings with the potential to produce them, there is disagreement about whether they really are a problem for researchers in the behavioral sciences. Therefore, the purpose of this review is to explore the current state of knowledge about method biases. First, we explore the meaning of the terms “method” and “method bias” and then we examine whether method biases influence all measures equally. Next, we review the evidence of the effects that method biases have on individual measures and on the covariation between different constructs. Following this, we evaluate the procedural and statistical remedies that have been used to control method biases and provide recommendations for minimizing method bias.

Sources of Method Bias in Social Science Research and Recommendations on How to Control It

Prior to the twentieth century, theories of knowledge were inherently perceptual. Since then, developments in logic, statis- tics, and programming languages have inspired amodal theories that rest on principles fundamentally different from those underlying perception. In addition, perceptual approaches have become widely viewed as untenable because they are assumed to implement record- ing systems, not conceptual systems. A perceptual theory of knowledge is developed here in the context of current cognitive science and neuroscience. During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas. Later, in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols. The stor- age and reactivation of perceptual symbols operates at the level of perceptual components - not at the level of holistic perceptual expe- riences. Through the use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory (e.g., individual memories of green, purr, hot). As memories of the same component become organized around a com- mon frame, they implement a simulator that produces limitless simulations of the component (e.g., simulations of purr). Not only do such simulators develop for aspects of sensory experience, they also develop for aspects of proprioception (e.g., lift, run) and introspec- tion (e.g., compare, memory, happy, hungry). Once established, these simulators implement a basic conceptual system that represents types, supports categorization, and produces categorical inferences. These simulators further support productivity, propositions, and ab- stract concepts, thereby implementing a fully functional conceptual system. Productivity results from integrating simulators combinato- rially and recursively to produce complex simulations. Propositions result from binding simulators to perceived individuals to represent type-token relations. Abstract concepts are grounded in complex simulations of combined physical and introspective events. Thus, a per- ceptual theory of knowledge can implement a fully functional conceptual system while avoiding problems associated with amodal sym- bol systems. Implications for cognition, neuroscience, evolution, development, and artificial intelligence are explored.

/pdf/perceptual-symbol-systems-4spsk97r61.pdf

Perceptual symbol systems.

The classic survey design reference, updated for the digital ageFor over two decades, Dillman's classic text on survey design has aided both students and professionals in effectively planning and conducting mail, telephone, and, more recently, Internet surveys. The new edition is thoroughly updated and revised, and covers all aspects of survey research. It features expanded coverage of mobile phones, tablets, and the use of do-it-yourself surveys, and Dillman's unique Tailored Design Method is also thoroughly explained. This invaluable resource is crucial for any researcher seeking to increase response rates and obtain high-quality feedback from survey questions. Consistent with current emphasis on the visual and aural, the new edition is complemented by copious examples within the text and accompanying website.This heavily revised Fourth Edition includes:Strategies and tactics for determining the needs of a given survey, how to design it, and how to effectively administer itHow and when to use mail, telephone, and Internet surveys to maximum advantageProven techniques to increase response ratesGuidance on how to obtain high-quality feedback from mail, electronic, and other self-administered surveysDirection on how to construct effective questionnaires, including considerations of layoutThe effects of sponsorship on the response rates of surveysUse of capabilities provided by newly mass-used media: interactivity, presentation of aural and visual stimuli.The Fourth Edition reintroduces the telephoneincluding coordinating land and mobile.Grounded in the best research, the book offers practical how-to guidelines and detailed examples for practitioners and students alike.

/pdf/internet-mail-and-mixed-mode-surveys-the-tailored-design-3tlkuxua87.pdf

Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method

1. Introduction and Overview. 2. Detecting Influential Observations and Outliers. 3. Detecting and Assessing Collinearity. 4. Applications and Remedies. 5. Research Issues and Directions for Extensions. Bibliography. Author Index. Subject Index.

Regression Diagnostics: Identifying Influential Data and Sources of Collinearity

1. Introduction 2. Respondents' understanding of survey questions 3. The role of memory in survey responding 4. Answering questions about date and durations 5. Attitude questions 6. Factual judgments and numerical estimates 7. Attitude judgments and context effects 8. Mapping and formatting 9. Survey reporting of sensitive topics 10. Mode of data collection 11. Impact of the application of cognitive models to survey measurement.

The Psychology of Survey Response

Psychologists have worried about the distortions introduced into standardized personality measures by social desirability bias. Survey researchers have had similar concerns about the accuracy of survey reports about such topics as illicit drug use, abortion, and sexual behavior. The article reviews the research done by survey methodologists on reporting errors in surveys on sensitive topics, noting parallels and differences from the psychological literature on social desirability. The findings from the survey studies suggest that misreporting about sensitive topics is quite common and that it is largely situational. The extent of misreporting depends on whether the respondent has anything embarrassing to report and on design features of the survey. The survey evidence also indicates that misreporting on sensitive topics is a more or less motivated process in which respondents edit the information they report to avoid embarrassing themselves in the presence of an interviewer or to avoid repercussions from third parties.

https://www.learnlab.org/research/wiki/images/a/a8/Tourangeau_SensitiveQuestions.pdf

Sensitive questions in surveys.

This study compared three methods of collecting sur- vey data about sexual behaviors and other sensitive topics: com- puter-assisted personal interviewing (CAPI), computer-assisted self-administered interviewing (CASI), and audio computer- assisted self-administered interviewing (ACASI). Interviews were conducted with an area probability sample of more than 300 adults in Cook County, Illinois. The experiment also compared open and closed questions about the number of sex partners and varied the context in which the sex partner items were embed- ded. The three mode groups did not differ in response rates, but the mode of data collection did affect the level of reporting of sensitive behaviors: both forms of self-administration tended to reduce the disparity between men and women in the number of sex partners reported. Self-administration, especially via ACASI, also increased the proportion of respondents admitting that they had used illicit drugs. In addition, when the closed answer op- tions emphasized the low end of the distribution, fewer sex part- ners were reported than when the options emphasized the high end of the distribution; responses to the open-ended versions of the sex partner items generally fell between responses to the two closed versions.

Asking sensitive questions the impact of data collection mode, question format, and question context

Although it is well established that self-administered ques- tionnaires tend to yield fewer reports in the socially desirable direction than do interviewer-administered questionnaires, less is known about whether different modes of self-administration vary in their effects on socially desirable responding In addition, most mode comparison stud- ies lack validation data and thus cannot separate the effects of differ- ential nonresponse bias from the effects of differences in measurement error This paper uses survey and record data to examine mode effects on the reporting of potentially sensitive information by a sample of re- cent university graduates Respondents were randomly assigned to one of three modes of data collection—conventional computer-assisted tele- phone interviewing (CATI), interactive voice recognition (IVR), and the Web—and were asked about both desirable and undesirable attributes of their academic experiences University records were used to evaluate the accuracy of the answers and to examine differences in nonresponse bias by mode Web administration increased the level of reporting of sensi- tive information and reporting accuracy relative to conventional CATI, with IVR intermediate between the other two modes Both mode of data collection and the actual status of the respondent influenced whether respondents found an item sensitive

/pdf/social-desirability-bias-in-cati-ivr-and-web-surveys-the-1p7csrjrtz.pdf

Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity

We begin this article with the assumption that attitudes are best understood as structures in longterm memory, and we look at the implications of this view for the response process in attitude surveys. More specifically, we assert that an answer to an attitude question is the product of a fourstage process. Respondents first interpret the attitude question, determining what attitude the question is about. They then retrieve relevant beliefs and feelings. Next, they apply these beliefs and feelings in rendering the appropriate judgment. Finally, they use this judgment to select a response. All four of the component processes can be affected by prior items. The prior items can provide a framework for interpreting later questions and can also make some responses appear to be redundant with earlier answers. The prior items can prime some beliefs, making them more accessible to the retrieval process. The prior items can suggest a norm or standard of comparison for making the judgment. Finally, the prior items can create consistency pressures or pressures to appear moderate. Because of the multiple processes involved, context effects are difficult to predict and sometimes difficult to replicate. We attempt to sort out when context is likely to affect later responses and include a list of the variables that affect the size and direction of the effects of context.

Roger Tourangeau

Papers

The Psychology of Survey Response

Sensitive questions in surveys.

Asking sensitive questions the impact of data collection mode, question format, and question context

Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity

Cognitive Processes Underlying Context Effects in Attitude Measurement