At what sample size do correlations stabilize

TL;DR: In this article, the authors used Monte-Carlo simulations to determine the critical sample size from which on the magnitude of a correlation can be expected to be stable, which depends on the effect size, the width of the corridor of stability, and the requested confidence that the trajectory does not leave this corridor any more.

About: This article is published in Journal of Research in Personality.The article was published on 2013-10-01 and is currently open access. It has received 1302 citations till now. The article focuses on the topics: Sample size determination & Sample (statistics).

Summary

1. Introduction

• Many researchers might have observed that the magnitude of a correlation is pretty unstable in small samples, as the following empirical example demonstrates.
• Multiple questionnaire scales have been administered in an open online study (Schönbrodt & Gerstenberg, 2012 ; Study 3).
• From a visual inspection, the trajectory did not stabilize up to a sample size of around 150.
• Data have not been rearranged -it is simply the order how participants dropped into the study.

2. Definition and Operationalization of Stability

• With these definitions, the answer to the research question can be formulated more precisely:.
• The authors are interested in the critical sample size POS crit , from which value on the estimate of a correlation does not leave the COS w with a confidence of 80% (90%, 95%).

3. Method and Results

• To explore the impact of these deviations on the POS crit values, the authors used four real world data sets provided by T. Micceri 2 as marginal distributions and imposed the specified population correlations (Ruscio & Kaczetow, 2008) .
• The authors constrained their analysis to typical non-normal distributions found in psychology (i.e., some skewness and somewhat heavier tails) 3 .
• Results for these variables were comparable to the Gaussian simulated data set.
• In non-normal distributions the POS had a median increase of 1.7% compared to the normal case (i.e., on average the correlations stabilized slightly later), and 90% of the differences between the nonnormal and normal POS were smaller than 6%.

4. Discussion

• If Table 1 should be boiled down to simple answers, one can ask what effect size typically can be expected in personality.
• In a meta-meta-analysis summarizing 322 metaanalyses with more than 25'000 published studies in the field of personality and social psychology, Richard, Bond, and Stokes-Zoota (2003) report that the average published effect is r = .21, less than 25% of all meta-analytic effects sizes are greater than .30, and only 5.28% of all effects are greater than .50.
• Further let's assume that a confidence level of 80% is requested (a level that is typically used for statistical power analyses), and only small effect sizes (w < .10) are considered as acceptable fluctuations.
• Of course, what is a meaningful or expected correlation can vary depending on the research context and questions.
• This would reduce the necessary sample size.

Figures

Table 1: The critical points of stability (POScrit) for different widths (w) of the corridor of stability (COS), different levels of confidence, and different ρs.

Figure 1: Actual (thick black line) and bootstrapped (thin gray lines) trajectories of a correlation. The dotted curved lines show the 95% confidence interval for the final correlation of r = .26 at each n. Dashed lines show the ± .1 corridor of stability (COS) around the final correlation. The point of stability (POS) is at n = 161. After that sample size the actual trajectory does not leave the COS.

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "At what sample size do correlations stabilize?" ?

In this report the authors use Monte-Carlo simulations to determine the critical sample size from which on the magnitude of a correlation can be expected to be stable.