The author suggests that we apply recent research knowledge to improve our conceptualization, measures, and methodology for studying the effects of teachers’ professional development on teachers and students. She makes the case that there is a research consensus to support the use of a set of core features and a common conceptual framework in professional development impact studies. She urges us to move away from automatic biases either for or against observation, interviews, or surveys in such studies. She argues that the use of a common conceptual framework would elevate the quality of professional development studies and subsequently the general understanding of how best to shape and implement teacher learning opportunities for the maximum benefit of both teachers and students.

/pdf/improving-impact-studies-of-teachers-professional-3pzi4k3euo.pdf

Improving Impact Studies of Teachers’ Professional Development: Toward Better Conceptualizations and Measures

Statistical Analysis with Missing Data

Psychologists have worried about the distortions introduced into standardized personality measures by social desirability bias. Survey researchers have had similar concerns about the accuracy of survey reports about such topics as illicit drug use, abortion, and sexual behavior. The article reviews the research done by survey methodologists on reporting errors in surveys on sensitive topics, noting parallels and differences from the psychological literature on social desirability. The findings from the survey studies suggest that misreporting about sensitive topics is quite common and that it is largely situational. The extent of misreporting depends on whether the respondent has anything embarrassing to report and on design features of the survey. The survey evidence also indicates that misreporting on sensitive topics is a more or less motivated process in which respondents edit the information they report to avoid embarrassing themselves in the presence of an interviewer or to avoid repercussions from third parties.

https://www.learnlab.org/research/wiki/images/a/a8/Tourangeau_SensitiveQuestions.pdf

Sensitive questions in surveys.

Many surveys of the U.S. household population are experiencing higher refusal rates. Nonresponse can, but need not, induce nonresponse bias in survey estimates. Recent empirical findings illustrate cases when the linkage between nonresponse rates and nonresponse biases is absent. Despite this, professional standards continue to urge high response rates. Statistical expressions of nonresponse bias can be translated into causal models to guide hypotheses about when nonresponse. causes bias. Alternative designs to measure nonresponse bias exist, providing different but incomplete information about the nature of the bias. A synthesis of research studies estimating nonresponse bias shows the bias often present. A logical question at this moment in history is what advantage probability sample surveys have if they suffer from high nonresponse rates. Since postsurvey adjustment for nonresponse requires auxiliary variables, the answer depends on the nature of the design and the quality of the auxiliary variables.

Nonresponse Rates and Nonresponse Bias in Household Surveys

Statistical Analysis With Missing Data (2nd ed.) (Book)

Summary At first sight, web surveys seem to be an interesting and attractive means of data collection. They providesimple,cheap,andfastaccesstoalargegroupofpotentialrespondents.However,websurveys are not without methodological problems. Specific groups in the populations are under-represented because they have less access to Internet. Furthermore, recruitment of respondents is often based on self-selection. Both under-coverage and self-selection may lead to biased estimates. This paper describes these methodological problems. It also explores the effect of various correction techniques (adjustment weighting and use of reference surveys). This all leads to the question whether properly design web surveys can be used for data collection. The paper attempts to answer this question. It concludes that under-coverage problems may solve itself in the future, but that self-selection leads to unreliable survey outcomes.

/pdf/selection-bias-in-web-surveys-47465metbs.pdf

Selection Bias in Web Surveys

Many survey organizations use the response rate as an indicator for the quality of survey data. As a consequence, a variety of measures are implemented to reduce non-response or to maintain response at an acceptable level. However, the response rate is not necessarily a good indicator of non-response bias. A higher response rate does not imply smaller non-response bias. What matters is how the composition of the response differs from the composition of the sample as a whole. This paper describes the concept of R-indicators to assess potential differences between the sample and the response. Such indicators may facilitate analysis of survey response over time, between various fieldwork strategies or data collection modes. Some practical examples are given.

Indicators for the representativeness of survey response

The History and Development of Computer Assisted Survey Information Collection Methods (M. Couper & W. Nicholls). Diffusion of Technolgical Innovation: Computer Assisted Data Collection in the U.K. (M. Collins, et al.). Integrating CASIC into Existing Designs and Organizations: A Survey of the Field (R. Groves & R. Tortora). Development and Implementation of CASIC in Government Statistical Agencies (C. Clark, et al.). Organization Effects of CATI in Small to Medium Survey Centers (J. Tarnai, et al.). A Systematic Approach to Instrument Development in CAI (S. Kinsey & D. Jewell). Producing CAI Instruments for a Program of Surveys (M. Pierzchala & T. Manners). The Individual Interview (D. Sikkel). Incorporating Experiments into Computer Assisted Surveys (T. Piazza & P. Sniderman). Use of Computer Assisted Interviewing in Longitudinal Surveys (A. Brown, et al.). The Future of Data Editing (J. Bethlehem & F. van de Pol). Automated Coding of Survey Data (H. Speizer & P. Buckley). Automated Management of Survey Data: An Overview (W. Connett). CAPI Survey Management Systems: Case Management on Laptops (L. Hofman & J. Gray). Automated Call Scheduling: Current Systems and Practices (T. Edwards, et al.). Getting From There to Here: Electronic Data Communications in Field Surveys (J. Smith, et al.). Training Field Interviewers to Use Computers: Past, Present, and Future Trends (M. Wojcik & E. Hunt). Evaluating Interviewer Use of CAPI Navigation Features (S. Sperry, et al.). Mode, Behavior, and Data Recording Error (J. Lepkowski, et al.). Computerized Self-Administered Questionnaires (M. Ramos, et al.). Ten Years of Interviewing without Interviewers: The Telepanel (W. Saris). Collecting Sensitive Information with Different Modes of Data Collection (R. Tourangeau & T. Smith). Automated Self-Interviewing and the Survey Measurement of Sensitive Behaviors (C. Turner, et al.). Response Rates, Data Quality, and Cost Feasibility for Optically Scannable Mail Surveys by Small Research Centers (D. Dillman & K. Miller). Scanning and Optical Character Recognition in Survey Organizations (E. Blom & L. Lyberg). Pen CASIC: Design and Usability (J. Bosley, et al.). Business Surveys of the Future: The World Wide Web as a Data Collection Methodology (R. Clayton & G. Werking). Current and Future Technolgoy Utilization in European Market Research (B. Blyth). The CASIC Future (R. Baker). References. Glossary of CASIC Acronyms. Index.

Computer Assisted Survey Information Collection

Preface. 1. The Survey Process. 1.1. About Surveys. 1.2. A Survey, Step-by-Step. 1.3. Some History of Survey Research. 1.4. This Book. 1.5. Samplonia. Exercises. 2. Basic Concepts. 2.1. The Survey Objectives. 2.2. The Target Population. 2.3. The Sampling Frame. 2.4. Sampling. 2.5. Estimation. Exercises. 3. Questionnaire Design. 3.1. The Questionnaire. 3.2. Factual and Nonfactual Questions. 3.3. The Question Text. 3.4. Answer Types. 3.5. Question Order. 3.6. Questionnaire Testing. Exercises. 4. Single Sampling Designs. 4.1. Simple Random Sampling. 4.2. Systematic Sampling. 4.3. Unequal Probability Sampling. 4.4. Systematic Sampling with Unequal Probabilities. Exercises. 5. Composite Sampling Designs. 5.1. Stratified Sampling. 5.2. Cluster Sampling. 5.3. Two-Stage Sampling. 5.4. Two-Dimensional Sampling. Exercises. 6. Estimators. 6.1. Use of Auxiliary Information. 6.2. A Descriptive Model. 6.3. The Direct Estimator. 6.4. The Ratio Estimator. 6.5. The Regression Estimator. 6.6. The Poststratification Estimator. Exercises. 7. Data Collection. 7.1. Traditional Data Collection. 7.2. Computer-Assisted Interviewing. 7.3. Mixed-Mode Data Collection. 7.4. Electronic Questionnaires. 7.5. Data Collection with Blaise. Exercises. 8. The Quality of the Results. 8.1. Errors in Surveys. 8.2. Detection and Correction of Errors. 8.3. Imputation Techniques. 8.4. Data Editing Strategies. Exercises. 9. The Nonresponse Problem. 9.1. Nonresponse. 9.2. Response Rates. 9.3. Models for Nonresponse. 9.4. Analysis of Nonresponse. 9.5. Nonresponse Correction Techniques. Exercises. 10. Weighting Adjustment. 10.1. Introduction. 10.2. Poststratification. 10.3. Linear Weighting. 10.4. Multiplicative Weighting. 10.5. Calibration Estimation. 10.6. Other Weighting Issues. 10.7. Use of Propensity Scores. 10.8. A Practical Example. Exercises. 11. Online Surveys. 11.1. The Popularity of Online Research. 11.2. Errors in Online Surveys. 11.3. The Theoretical Framework. 11.4. Correction by Adjustment Weighting. 11.5. Correction Using a Reference Survey. 11.6. Sampling the Non-Internet Population. 11.7. Propensity Weighting. 11.8. Simulating the Effects of Undercoverage. 11.9. Simulating the Effects of Self-Selection. 11.10. About the Use of Online Surveys. Exercises. 12. Analysis and Publication. 12.1. About Data Analysis. 12.2. The Analysis of Dirty Data. 12.3. Preparing a Survey Report. 12.4. Use of Graphs. Exercises. 13. Statistical Disclosure Control. 13.1. Introduction. 13.2. The Basic Disclosure Problem. 13.3. The Concept of Uniqueness. 13.4. Disclosure Scenarios. 13.5. Models for the Disclosure Risk. 13.6. Practical Disclosure Protection. Exercises. References. Index.

Applied Survey Methods: A Statistical Perspective

This volume presents an all-inclusive guide to the problem of nonresponse in household surveys, providing an overview of the theory while also describing practical implications. The book begins with a general overview of the nonresponse problem, outlining existing sources of error and guidelines for calculating response rates according to various international standards. The theoretical content of survey sampling is also explored and presented in a chapter along with a discussion on nonresponse on survey estimates and missing data mechanisms. Subsequent chapters provide guidance on reducing and analyzing nonresponse, the concept of representativity, weighting adjustment techniques, and approaches for obtaining additional information from nonrespondents. Each chapter organizes the presented information in uniform components. Following an introduction, a description of the key theory and formulae is presented. Next, the discussed theory is applied to a real data set and accompanied with vivid descriptions. Example sets are spread throughout each chapter and a summary provides a brief overview of main points and concepts. Real data sets are used throughout the book and additional data sets available on the book's related Web site can be easily worked with using the Stata or SPSS software packages.

Jelke Bethlehem

Papers

Selection Bias in Web Surveys

Indicators for the representativeness of survey response

Computer Assisted Survey Information Collection

Applied Survey Methods: A Statistical Perspective

Handbook of Nonresponse in Household Surveys