Showing papers in "Perspectives on Psychological Science in 2014"

Beyond Power Calculations Assessing Type S (Sign) and Type M (Magnitude) Errors

Abstract: Statistical power analysis provides the conventional approach to assess error rates when designing a research study. However, power analysis is flawed in that a narrow emphasis on statistical significance is placed as the primary focus of study design. In noisy, small-sample settings, statistically significant results can often be misleading. To help researchers address this problem in the context of their own studies, we recommend design calculations in which (a) the probability of an estimate being in the wrong direction (Type S [sign] error) and (b) the factor by which the magnitude of an effect might be overestimated (Type M [magnitude] error or exaggeration ratio) are estimated. We illustrate with examples from recent published research and discuss the largest challenge in a design calculation: coming up with reasonable estimates of plausible effect sizes based on external information.

The Value of Direct Replication

Abstract: Reproducibility is the cornerstone of science. If an effect is reliable, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. Two of the articles in this special section question the value of direct replication by other laboratories. In this commentary, I discuss the problematic implications of some of their assumptions and argue that direct replication by multiple laboratories is the only way to verify the reliability of an effect.

p-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results

Abstract: Journals tend to publish only statistically significant evidence, creating a scientific record that markedly overstates the size of effects. We provide a new tool that corrects for this bias without requiring access to nonsignificant results. It capitalizes on the fact that the distribution of significant p values, p-curve, is a function of the true underlying effect. Researchers armed only with sample sizes and test results of the published findings can correct for publication bias. We validate the technique with simulations and by reanalyzing data from the Many-Labs Replication project. We demonstrate that p-curve can arrive at conclusions opposite that of existing tools by reanalyzing the meta-analysis of the “choice overload” literature.

The Alleged Crisis and the Illusion of Exact Replication

Abstract: There has been increasing criticism of the way psychologists conduct and analyze studies. These critiques as well as failures to replicate several high-profile studies have been used as justification to proclaim a “replication crisis” in psychology. Psychologists are encouraged to conduct more “exact” replications of published studies to assess the reproducibility of psychological research. This article argues that the alleged “crisis of replicability” is primarily due to an epistemological misunderstanding that emphasizes the phenomenon instead of its underlying mechanisms. As a consequence, a replicated phenomenon may not serve as a rigorous test of a theoretical hypothesis because identical operationalizations of variables in studies conducted at different times and with different subject populations might test different theoretical constructs. Therefore, we propose that for meaningful replications, attempts at reinstating the original circumstances are not sufficient. Instead, replicators must ascertain that conditions are realized that reflect the theoretical variable(s) manipulated (and/or measured) in the original study.

The Origins of Neuroticism.

Abstract: In this article, we provide a fresh perspective on the developmental origins of neuroticism--a dimension of temperament marked by elevated stress reactivity resulting in the frequent experience of negative emotions. This negative affectivity is accompanied by a pervasive perception that the world is a dangerous and threatening place, along with beliefs about one's inability to manage or cope with challenging events. Historically, neuroticism has been viewed as a stable, genetically based trait. However, recent understanding of ongoing gene-environment interactions that occur throughout the life span suggests there may be a more complex and dynamic etiology. Thus, the purpose of this article is to offer a theory for understanding the development of neuroticism that integrates genetic, neurobiological, and environmental contributions to this trait. Given the strong correlation between neuroticism and the development of negative health outcomes--most notably, the full range of anxiety and mood disorders--an enhanced understanding of how neuroticism originates has implications for the treatment and prevention of a broad range of pathologies and, perhaps, even for the prevention of neuroticism itself.

Do People Have Insight Into Their Abilities? A Metasynthesis:

Abstract: Having insight into one’s abilities is essential, yet it remains unclear whether people generally perceive their skills accurately or inaccurately. In the present analysis, we examined the overall correspondence between self-evaluations of ability (e.g., academic ability, intelligence, language competence, medical skills, sports ability, and vocational skills) and objective performance measures (e.g., standardized test scores, grades, and supervisor evaluations) across 22 meta-analyses, in addition to considering factors that moderate this relationship. Although individual meta-analytic effects ranged from .09 to .63, the mean correlation between ability self-evaluations and performance outcomes across meta-analyses was moderate (M = .29, SD = .11). Further, the relation was stronger when self-evaluations were specific to a given domain rather than broad and when performance tasks were objective, familiar, or low in complexity. Taken together, these findings indicate that people have only moderate insight into their abilities but also underscore the contextual factors that enable accurate self-perception of ability.

Continuously Cumulating Meta-Analysis and Replicability

Abstract: The current crisis in scientific psychology about whether our findings are irreproducible was presaged years ago by Tversky and Kahneman (1971), who noted that even sophisticated researchers believe in the fallacious Law of Small Numbers—erroneous intuitions about how imprecisely sample data reflect population phenomena. Combined with the low power of most current work, this often leads to the use of misleading criteria about whether an effect has replicated. Rosenthal (1990) suggested more appropriate criteria, here labeled the continuously cumulating metaanalytic (CCMA) approach. For example, a CCMA analysis on a replication attempt that does not reach significance might nonetheless provide more, not less, evidence that the effect is real. Alternatively, measures of heterogeneity might show that two studies that differ in whether they are significant might have only trivially different effect sizes. We present a nontechnical introduction to the CCMA framework (referencing relevant software), and then explain how it can be used to address aspects of replicability or more generally to assess quantitative evidence from numerous studies. We then present some examples and simulation results using the CCMA approach that show how the combination of evidence can yield improved results over the consideration of single studies.

Priming, Replication, and the Hardest Science

Abstract: Concerns have been raised recently about the replicability of behavioral priming effects, and calls have been issued to identify priming methodologies with effects that can be obtained in any context and with any population. I argue that such expectations are misguided and inconsistent with evolutionary understandings of the brain as a computational organ. Rather, we should expect priming effects to be highly sensitive to variations in experimental features and subject populations. Such variation does not make priming effects frivolous or capricious but instead can be predicted a priori. However, absent theories specifying the precise contingencies that lead to such variation, failures to replicate another researcher's findings will necessarily be ambiguous with respect to the inferences that can be made. Priming research is not yet at the stage where such theories exist, and therefore failures are uninformative at the current time. Ultimately, priming researchers themselves must provide direct replications of their own effects; researchers have been deficient in meeting this responsibility and have contributed to the current state of confusion. The recommendations issued in this article reflect concerns both with the practice of priming researchers and with the inappropriate expectations of researchers who have failed to replicate others' priming effects.

The Neuroscience of Intergroup Relations An Integrative Review

Abstract: We review emerging research on the psychological and biological factors that underlie social group formation, cooperation, and conflict in humans. Our aim is to integrate the intergroup neuroscience literature with classic theories of group processes and intergroup relations in an effort to move beyond merely describing the effects of specific social out-groups on the brain and behavior. Instead, we emphasize the underlying psychological processes that govern intergroup interactions more generally: forming and updating our representations of “us” and “them” via social identification and functional relations between groups. This approach highlights the dynamic nature of social identity and the context-dependent nature of intergroup relations. We argue that this theoretical integration can help reconcile seemingly discrepant findings in the literature, provide organizational principles for understanding the core elements of intergroup dynamics, and highlight several exciting directions for future research a...

Submentalizing: I Am Not Really Reading Your Mind.

Abstract: The nativist view of mentalizing—the view that humans have an inherent capacity to think about the mental states of others—has been recently reinvigorated by reports that adults and infants automat...

Friends or Foes: Is Empathy Necessary for Moral Behavior?

Abstract: In the past decade, a flurry of empirical and theoretical research on morality and empathy has taken place, and interest and usage in the media and the public arena have increased. At times, in both popular culture and academia, morality and empathy are used interchangeably, and quite often the latter is considered to play a foundational role for the former. In this article, we argue that although there is a relationship between morality and empathy, it is not as straightforward as apparent at first glance. Moreover, it is critical to distinguish among the different facets of empathy (emotional sharing, empathic concern, and perspective taking), as each uniquely influences moral cognition and predicts differential outcomes in moral behavior. Empirical evidence and theories from evolutionary biology as well as developmental, behavioral, and affective and social neuroscience are comprehensively integrated in support of this argument. The wealth of findings illustrates a complex and equivocal relationship between morality and empathy. The key to understanding such relations is to be more precise on the concepts being used and, perhaps, abandoning the muddy concept of empathy.

Sailing From the Seas of Chaos Into the Corridor of Stability: Practical Recommendations to Increase the Informational Value of Studies

Abstract: Recent events have led psychologists to acknowledge that the inherent uncertainty encapsulated in an inductive science is amplified by problematic research practices. In this article, we provide a practical introduction to recently developed statistical tools that can be used to deal with these uncertainties when performing and evaluating research. In Part 1, we discuss the importance of accurate and stable effect size estimates as well as how to design studies to reach a corridor of stability around effect size estimates. In Part 2, we explain how, given uncertain effect size estimates, well-powered studies can be designed with sequential analyses. In Part 3, we (a) explain what p values convey about the likelihood that an effect is true, (b) illustrate how the v statistic can be used to evaluate the accuracy of individual studies, and (c) show how the evidential value of multiple studies can be examined with a p-curve analysis. We end by discussing the consequences of incorporating our recommendations in terms of a reduced quantity, but increased quality, of the research output. We hope that the practical recommendations discussed in this article will provide researchers with the tools to make important steps toward a psychological science that allows researchers to differentiate among all possible truths on the basis of their likelihood.

A Sex-Positive Framework for Research on Adolescent Sexuality:

Abstract: In this article, I propose a sex-positive framework for research on adolescent sexuality in which I consider consensual sexual activities in adolescence as developmentally normative and potentially healthy. The sex-positive framework is contrasted with the predominant "risk" perspective that presumes that abstinence from sexual activity is the ideal behavioral outcome for teenagers. Evidence from longitudinal and behavioral genetic studies indicates that engaging in sexual intercourse in adolescence does not typically cause worse psychological functioning. The relationship context of sexual experience may be a critical moderator of its psychological impact. Moreover, cross-cultural data on adolescents' contraception usage, unintended pregnancy, and sexually transmitted infections suggest that, despite the unacceptably high rate of negative health consequences among U.S. teenagers, adolescents can have the developmental capacity to regulate the health risks inherent in sexual activity. Understanding adolescent sexuality can be fostered by considering sexual well-being, a multidimensional construct that incorporates an adolescent's sexual self-efficacy, sexual self-esteem, feelings of sexual pleasure and satisfaction, and freedom from pain and negative affect regarding sexuality. New research is necessary to understand the development of adolescent sexual well-being, including its normative age trends, its reciprocal links with sexual behavior, and its impact on psychological and physical health.

Selective Engagement of Cognitive Resources: Motivational Influences on Older Adults' Cognitive Functioning.

Abstract: In this article, I present a framework for understanding the impact of aging-related declines in cognitive resources on functioning. I make the assumption that aging is associated with an increase in the costs of cognitive engagement, as reflected in both the effort required to achieve a specific level of task performance and the associated depletion or fatigue effects. I further argue that these costs result in older adults being increasingly selective in the engagement of cognitive resources in response to these declines. This selectivity is reflected in (a) a reduction in the intrinsic motivation to engage in cognitively demanding activities, which, in part, accounts for general reductions in engagement in such activities, and (b) greater sensitivity to the self-related implications of a given task. Both processes are adaptive if viewed in terms of resource conservation, but the former may also be maladaptive to the extent that it results in older adults restricting participation in cognitively demanding activities that could ultimately benefit cognitive health. I review supportive research and make the general case for the importance of considering motivational factors in understanding aging effects on cognitive functioning.

How Much (More) Should CEOs Make? A Universal Desire for More Equal Pay

Abstract: Do people from different countries and different backgrounds have similar preferences for how much more the rich should earn than the poor? Using survey data from 40 countries (N = 55,238), we compare respondents' estimates of the wages of people in different occupations-chief executive officers, cabinet ministers, and unskilled workers-to their ideals for what those wages should be. We show that ideal pay gaps between skilled and unskilled workers are significantly smaller than estimated pay gaps and that there is consensus across countries, socioeconomic status, and political beliefs. Moreover, data from 16 countries reveals that people dramatically underestimate actual pay inequality. In the United States-where underestimation was particularly pronounced-the actual pay ratio of CEOs to unskilled workers (354:1) far exceeded the estimated ratio (30:1), which in turn far exceeded the ideal ratio (7:1). In sum, respondents underestimate actual pay gaps, and their ideal pay gaps are even further from reality than those underestimates.

Banishing the Control Homunculi in Studies of Action Control and Behavior Change

Abstract: For centuries, human self-control has fascinated scientists and nonscientists alike. Current theories often attribute it to an executive control system. But even though executive control receives a great deal of attention across disciplines, most aspects of it are still poorly understood. Many theories rely on an ill-defined set of “homunculi” doing jobs like “response inhibition” or “updating” without explaining how they do so. Furthermore, it is not always appreciated that control takes place across different timescales. These two issues hamper major advances. Here we focus on the mechanistic basis for the executive control of actions. We propose that at the most basic level, action control depends on three cognitive processes: signal detection, action selection, and action execution. These processes are modulated via error-correction or outcome-evaluation mechanisms, preparation, and task rules maintained in working and long-term memory. We also consider how executive control of actions becomes automatized with practice and how people develop a control network. Finally, we discuss how the application of this unified framework in clinical domains can increase our understanding of control deficits and provide a theoretical basis for the development of novel behavioral change interventions.

Registered Replication Report: Schooler and Engstler-Schooler (1990)

Abstract: Trying to remember something now typically improves your ability to remember it later. However, after watching a video of a simulated bank robbery, participants who verbally described the robber were 25% worse at identifying the robber in a lineup than were participants who instead listed U.S. states and capitals—this has been termed the “verbal overshadowing” effect (Schooler & Engstler-Schooler, 1990). More recent studies suggested that this effect might be substantially smaller than first reported. Given uncertainty about the effect size, the influence of this finding in the memory literature, and its practical importance for police procedures, we conducted two collections of preregistered direct replications (RRR1 and RRR2) that differed only in the order of the description task and a filler task. In RRR1, when the description task immediately followed the robbery, participants who provided a description were 4% less likely to select the robber than were those in the control condition. In RRR2, when the description was delayed by 20 min, they were 16% less likely to select the robber. These findings reveal a robust verbal overshadowing effect that is strongly influenced by the relative timing of the tasks. The discussion considers further implications of these replications for our understanding of verbal overshadowing.

Expectations for Replications Are Yours Realistic

Abstract: Failures to replicate published psychological research findings have contributed to a “crisis of confidence.” Several reasons for these failures have been proposed, the most notable being questiona...

Why Ineffective Psychotherapies Appear to Work: A Taxonomy of Causes of Spurious Therapeutic Effectiveness

Abstract: The past 40 years have generated numerous insights regarding errors in human reasoning. Arguably, clinical practice is the domain of applied psychology in which acknowledging and mitigating these errors is most crucial. We address one such set of errors here, namely, the tendency of some psychologists and other mental health professionals to assume that they can rely on informal clinical observations to infer whether treatments are effective. We delineate four broad, underlying cognitive impediments to accurately evaluating improvement in psychotherapy-naive realism, confirmation bias, illusory causation, and the illusion of control. We then describe 26 causes of spurious therapeutic effectiveness (CSTEs), organized into a taxonomy of three overarching categories: (a) the perception of client change in its actual absence, (b) misinterpretations of actual client change stemming from extratherapeutic factors, and (c) misinterpretations of actual client change stemming from nonspecific treatment factors. These inferential errors can lead clinicians, clients, and researchers to misperceive useless or even harmful psychotherapies as effective. We (a) examine how methodological safeguards help to control for different CSTEs, (b) delineate fruitful directions for research on CSTEs, and (c) consider the implications of CSTEs for everyday clinical practice. An enhanced appreciation of the inferential problems posed by CSTEs may narrow the science-practice gap and foster a heightened appreciation of the need for the methodological safeguards afforded by evidence-based practice.

Deny, Distance, or Dismantle? How White Americans Manage a Privileged Identity

Abstract: Social scientists have traditionally argued that whiteness-the attribute of being recognized and treated as a White person in society-is powerful because it is invisible. On this view, members of the racially dominant group have the unique luxury of rarely noticing their race or the privileges it confers. This article challenges this "invisibility thesis," arguing that Whites frequently regard themselves as racial actors. We further argue that whiteness defines a problematic social identity that confronts Whites with 2 psychological threats: the possibility that their accomplishments in life were not fully earned (meritocratic threat) and the association with a group that benefits from unfair social advantages (group-image threat). We theorize that Whites manage their racial identity to dispel these threats. According to our deny, distance, or dismantle (3D) model of White identity management, dominant-group members have three strategies at their disposal: deny the existence of privilege, distance their own self-concepts from the White category, or strive to dismantle systems of privilege. Whereas denial and distancing promote insensitivity and inaction with respect to racial inequality, dismantling reduces threat by relinquishing privileges. We suggest that interventions aimed at reducing inequality should attempt to leverage dismantling as a strategy of White identity management.

Understanding How Prior Knowledge Influences Memory in Older Adults.

Abstract: Older adults have a harder time than younger adults remembering specific events and experiences (episodic memory), whereas the ability to use one’s general knowledge either improves or remains stable over the life span. Our focus is on the sometimes overlooked but critical possibility that this intact general knowledge can facilitate older adults’ episodic memory performance. After reviewing literature that shows how prior knowledge can support remembering in aging as well as lead it astray, we consider open questions including whether prior knowledge is used only to fill in the gaps after a memory failure and when older adults might need to be instructed to apply their prior knowledge. Overall, we situate our claims within theories of cognitive aging, arguing that prior knowledge is a key factor in understanding older adults’ memory performance, with the potential to serve as a compensatory mechanism.

Which Personality Attributes Are Most Important in the Workplace

Abstract: Employees face a variety of work demands that place a premium on personal attributes, such as the degree to which they can be depended on to work independently, deal with stress, and interact positively with coworkers and customers. We examine evidence for the importance of these personality attributes using research strategies intended to answer three fundamental questions, including (a) how well does employees’ standing on these attributes predict job performance?, (b) what types of attributes do employers seek to evaluate in interviews when considering applicants?, and (c) what types of attributes are rated as important for performance in a broad sampling of occupations across the U.S. economy? We summarize and integrate results from these three strategies using the Big Five personality dimensions as our organizing framework. Our findings indicate that personal attributes related to Conscientiousness and Agreeableness are important for success across many jobs, spanning across low to high levels of job...

Why Are Childhood Family Factors Associated With Timing of Maturation? A Role for Internal Prediction.

Abstract: Children, particularly girls, who experience early familial adversity tend to go on to reach sexual maturity relatively early. This feature of adolescent development is believed to be an evolved strategy that arose because individuals with genes that caused them to mature relatively early under certain conditions left behind more descendants than those who did not. However, although much has been done to uncover the psychological and physiological mechanisms underlying this process, less attention has been paid to the evolutionary reasons behind why it might be advantageous. It has previously been suggested that this strategy evolved because early familial adversity accurately indicated later environmental adversity, under which conditions early reproduction would likely maximize evolutionary fitness. In this article, we contrast this “external prediction” model with an alternative explanation, which builds on the existing explanation and is mutually compatible with it but also distinct from it. We argue that accelerated development is advantageous because early adversity detrimentally affects the individual’s body, increasing later morbidity and mortality; individuals may adapt to this internal setback by accelerating their development. Unlike the external prediction model, this “internal prediction” relies not on temporal environmental continuity but on long-term effects of early circumstances on the body.

Safeguard Power as a Protection Against Imprecise Power Estimates.

Abstract: An essential first step in planning a confirmatory or a replication study is to determine the sample size necessary to draw statistically reliable inferences using power analysis. A key problem, however, is that what is available is the sample-size estimate of the effect size, and its use can lead to severely underpowered studies when the effect size is overestimated. As a potential remedy, we introduce safeguard power analysis, which uses the uncertainty in the estimate of the effect size to achieve a better likelihood of correctly identifying the population effect size. Using a lower-bound estimate of the effect size, in turn, allows researchers to calculate a sample size for a replication study that helps protect it from being underpowered. We show that in most common instances, compared with nominal power, safeguard power is higher whereas standard power is lower. We additionally recommend the use of safeguard power analysis to evaluate the strength of the evidence provided by the original study.

Statistics Anxiety Update: Refining the Construct and Recommendations for a New Research Agenda:

Abstract: Appreciation of the importance of statistics literacy for citizens of a democracy has resulted in an increasing number of degree programs making statistics courses mandatory for university students. Unfortunately, empirical evidence suggests that students in nonmathematical disciplines (e.g., social sciences) regard statistics courses as the most anxiety-inducing course in their degree programs. Although a literature review exists for statistics anxiety, it was done more than a decade ago, and newer studies have since added findings for consideration. In this article, we provide a current review of the statistics anxiety literature. Specifically, related variables, definitions, and measures of statistics anxiety are reviewed with the goal of refining the statistics anxiety construct. Antecedents, effects, and interventions of statistics anxiety are also reviewed to provide recommendations for statistics instructors and for a new research agenda.

An Introduction to Registered Replication Reports at Perspectives on Psychological Science

Abstract: Much of science, including psychology, consists of eliciting, measuring, and documenting effects. In psychology, these effects are used to test and support theories about how the mind works or why people behave the way they do. Yet, for several reasons, the first report of an effect in the published literature rarely provides enough evidence to draw firm conclusions about the actual size of the effect. First, the sample sizes typically used in psychology studies are not large enough to measure an effect with precision (Marszalek, Barber, Kohlhart, & Holmes, 2012). Second, scientific publishing favors statistically significant, novel results over inconclusive or negative ones (Fanelli, 2012). If discrepant results rarely enter the literature, then published studies will, on average, overestimate the true size of the effects they report. And third, just by chance, some published studies will be false positives (Ioannidis, 2005). Uncertainty about the true size of important effects hampers psychological theorizing. Moreover, single studies provide little evidence for the robustness of an effect across the population to which it is assumed to generalize. This issue of Perspectives on Psychological Science includes the first example of a new type of journal article, one designed to provide a more definitive measure of the size and reliability of important effects: the Registered Replication Report (RRR; see Simons & Holcombe, 2014). RRRs compile a set of studies from a variety of laboratories that all followed an identical, vetted protocol designed to reproduce the original method and finding as closely as possible. By combining the resources of multiple labs, RRRs provide the ingredients for a meta-analysis that can authoritatively establish the size and reliability of an effect.

Let’s Put Our Money Where Our Mouth Is: If Authors Are to Change Their Ways, Reviewers (and Editors) Must Change With Them

Abstract: A number of scholars recently have argued for fundamental changes in the way psychological scientists conduct and report research. The behavior of researchers is influenced partially by incentive structures built into the manuscript evaluation system, and change in researcher practices will necessitate a change in the way journal reviewers evaluate manuscripts. This article outlines specific recommendations for reviewers that are designed to facilitate open data reporting and to encourage researchers to disseminate the most generative and replicable studies. These recommendations include changing the way reviewers respond to imperfections in empirical data, focusing less on individual tests of statistical significance and more on meta-analyses, being more open to null findings and failures to replicate previous research, and attending carefully to the theoretical contribution of a manuscript in addition to its methodological rigor. The article also calls for greater training and guidance for reviewers so that they can evaluate research in a manner that encourages open reporting and ultimately strengthens our science.

Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning

Abstract: Mental disorders are prevalent and can lead to significant impairment. Some progress has been made toward establishing treatments; however, effect sizes are small to moderate, gains may not persist, and many patients derive no benefit. Our goal is to highlight the potential for empirically supported psychosocial treatments to be improved by incorporating insights from cognitive psychology and research on education. Our central question is: If it were possible to improve memory for the content of sessions of psychosocial treatments, would outcome substantially improve? We leverage insights from scientific knowledge on learning and memory to derive strategies for transdiagnostic and transtreatment cognitive support interventions. These strategies can be applied within and between sessions and to interventions delivered via computer, the Internet, and text message. Additional novel pathways to improving memory include improving sleep, engaging in exercise, and using imagery. Given that memory processes change across the lifespan, services to children and older adults may benefit from different types and amounts of cognitive support.

Toward an Integrative Theory of Psychological Defense

Abstract: According to theories of “psychological defense,” humans are motivated to protect themselves against various types of psychological threat, including death awareness, uncertainty, and other inherently anxiety-provoking experiences. Protective mechanisms include strengthening close relationships; maintaining appraisals of self-worth, accomplishment, and agency; and cultivating meaningful views of the world. Thus, defensiveness theories incorporate research from many areas of psychology (e.g., information-processing biases, attitudes, and interpersonal and intergroup relations), to help explain why people think, feel, and act in the diverse ways that they do. Currently, the study of psychological defense is hindered by contradictory empirical results and a proliferation of theories that make very similar predictions. This article examines a cross-section of defensiveness theories and research, highlighting conclusions that can be drawn and areas where conceptual and research problems linger. It suggests tha...

You Cannot Step Into the Same River Twice When Power Analyses Are Optimistic

Abstract: Statistical power depends on the size of the effect of interest. However, effect sizes are rarely fixed in psychological research: Study design choices, such as the operationalization of the dependent variable or the treatment manipulation, the social context, the subject pool, or the time of day, typically cause systematic variation in the effect size. Ignoring this between-study variation, as standard power formulae do, results in assessments of power that are too optimistic. Consequently, when researchers attempting replication set sample sizes using these formulae, their studies will be underpowered and will thus fail at a greater than expected rate. We illustrate this with both hypothetical examples and data on several well-studied phenomena in psychology. We provide formulae that account for between-study variation and suggest that researchers set sample sizes with respect to our generally more conservative formulae. Our formulae generalize to settings in which there are multiple effects of interest. We also introduce an easy-to-use website that implements our approach to setting sample sizes. Finally, we conclude with recommendations for quantifying between-study variation.