Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and identified 15 meta-analyses on the same topics as multiple-laboratory replications. We find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost three times as large as replication effect sizes. We also implement three methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results. Kvarven, Stromland and Johannesson compare meta-analyses to multiple-laboratory replication projects and find that meta-analyses overestimate effect sizes by a factor of almost three. Commonly used methods of adjusting for publication bias do not substantively improve results.

Comparing meta-analyses and preregistered multiple-laboratory replication projects

One-shot anonymous unselfishness in economic games is commonly explained by social preferences, which assume that people care about the monetary pay-offs of others. However, during the last 10 years, research has shown that different types of unselfish behaviour, including cooperation, altruism, truth-telling, altruistic punishment and trustworthiness are in fact better explained by preferences for following one's own personal norms-internal standards about what is right or wrong in a given situation. Beyond better organizing various forms of unselfish behaviour, this moral preference hypothesis has recently also been used to increase charitable donations, simply by means of interventions that make the morality of an action salient. Here we review experimental and theoretical work dedicated to this rapidly growing field of research, and in doing so we outline mathematical foundations for moral preferences that can be used in future models to better understand selfless human actions and to adjust policies accordingly. These foundations can also be used by artificial intelligence to better navigate the complex landscape of human morality.

https://royalsocietypublishing.org/doi/pdf/10.1098/rsif.2020.0880

Mathematical foundations of moral preferences.

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. 

https://pure.rug.nl/ws/files/256853210/annurev_psych_020821_114157.pdf

Replicability, Robustness, and Reproducibility in Psychological Science

Sustaining large-scale public goods requires individuals to make environmentally friendly decisions today to benefit future generations1–6 Recent research suggests that second-order normative beliefs are more powerful predictors of behaviour than first-order personal beliefs7,8 We explored the role that second-order normative beliefs—the belief that community members think that saving energy helps the environment—play in curbing energy use We first analysed a data set of 211 independent, randomized controlled trials conducted in 27 US states by Opower, a company that uses comparative information about energy consumption to reduce household energy usage (pooled N = 16,198,595) Building off the finding that the energy savings varied between 081% and 255% across states, we matched this energy use data with a survey that we conducted of over 2,000 individuals in those same states on their first-order personal and second-order normative beliefs We found that second-order normative beliefs predicted energy savings but first-order personal beliefs did not A subsequent pre-registered experiment provides causal evidence for the role of second-order normative beliefs in predicting energy conservation above first-order personal beliefs Our results suggest that second-order normative beliefs play a critical role in promoting energy conservation and have important implications for policymakers concerned with curbing the detrimental consequences of climate change Analyses of data from 211 independent, randomized controlled trials (N = 16,198,595) show that second-order normative beliefs—community members’ belief that saving energy helps the environment—play a critical role in promoting energy conservation

/pdf/the-critical-role-of-second-order-normative-beliefs-in-3u5605oadr.pdf

The critical role of second-order normative beliefs in predicting energy conservation

http://mariaprecalde.com/Recalde_Riedl_Vesterlund_2015.pdf

Error-Prone Inference from Response Time: The Case of Intuitive Generosity in Public-Good Games

In an anonymous 4-person economic game, participants contributed more money to a common project (i.e., cooperated) when required to decide quickly than when forced to delay their decision (Rand, Greene & Nowak, 2012), a pattern consistent with the social heuristics hypothesis proposed by Rand and colleagues. The results of studies using time pressure have been mixed, with some replication attempts observing similar patterns (e.g., Rand et al., 2014) and others observing null effects (e.g., Tinghog et al., 2013; Verkoeijen & Bouwmeester, 2014). This Registered Replication Report (RRR) assessed the size and variability of the effect of time pressure on cooperative decisions by combining 21 separate, preregistered replications of the critical conditions from Study 7 of the original article (Rand et al., 2012). The primary planned analysis used data from all participants who were randomly assigned to conditions and who met the protocol inclusion criteria (an intent-to-treat approach that included the 65.9% of participants in the time-pressure condition and 7.5% in the forced-delay condition who did not adhere to the time constraints), and we observed a difference in contributions of −0.37 percentage points compared with an 8.6 percentage point difference calculated from the original data. Analyzing the data as the original article did, including data only for participants who complied with the time constraints, the RRR observed a 10.37 percentage point difference in contributions compared with a 15.31 percentage point difference in the original study. In combination, the results of the intent-to-treat analysis and the compliant-only analysis are consistent with the presence of selection biases and the absence of a causal effect of time pressure on cooperation.

/pdf/registered-replication-report-rand-greene-and-nowak-2012-156jxp1x46.pdf

C. Mauro

Papers

Registered Replication Report : Rand, Greene, and Nowak (2012)