(Open Access) Meaningful Effects in the Adolescent Brain Cognitive Development Study (2020) | Anthony Steven Dick

NeuroImage 239 (2021) 118262

Contents lists available at ScienceDirect

NeuroImage

journal homepage: www.elsevier.com/locate/neuroimage

Meaningful associations in the adolescent brain cognitive development

study

Anthony Steven Dick

, Daniel A. Lopez

, Ashley L. Watts

, Steven Heeringa

, Chase Reuter

Hauke Bartsch

, Chun Chieh Fan

, David N. Kennedy

, Clare Palmer

, Andrew Marshall

Frank Haist

, Samuel Hawes

, Thomas E. Nichols

, Deanna M. Barch

, Terry L. Jernigan

Hugh Garavan

, Steven Grant

, Vani Pariyadath

, Elizabeth Homan

, Michael Neale

Elizabeth A. Stuart

, Martin P. Paulus

, Kenneth J. Sher

, Wesley K. Thompson

e , g , ∗

Department of Psychology and Center for Children and Families, Florida International University, Miami, FL, United States

Division of Epidemiology, Department of Public Health Sciences, University of Rochester Medical Center, Rochester, NY 14642, United States

Department of Psychology, University of Missouri, MO, United States

Institute for Social Research, University of Michigan, Ann Arbor, MI 48109, United States

Herbert Wertheim School of Public Health and Human Longevity Science, University of California, San Diego, La Jolla, CA 92093, United States

Mohn Medical Imaging and Visualization Center, Department of Radiology, Haukeland University Hospital, Bergen, Norway

Population Neuroscience and Genetics Lab, University of California, San Diego, La Jolla, CA 92093, United States

Department of Psychiatry, University of Massachusetts Medical School, MA United States, 01604

Center for Human Development, University of California, San Diego, La Jolla, CA 92093, United States

Children’s Hospital Los Angeles, and the Department of Pediatrics, University of Southern California, Los Angeles, CA, United States

Department of Radiology, University of California, San Diego, La Jolla, CA 92093, United States

Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuﬃeld Department of Population Health, University of Oxford, Oxford, United

Kingdom

Departments of Psychological & Brain Sciences, Psychiatry and Radiology, Washington University, St. Louis, MO 63130, United States

Department of Psychiatry, University of Vermont, Burlington, VT, 05405, United States

Behavioral and Cognitive Neuroscience Branch, Division of Neuroscience and Behavior, National Institute on Drug Abuse, National Institutes of Health, Department of

Health and Human Services, Bethesda, MD, United States

National Institute on Drug Abuse, National Institutes of Health, Department of Heatlh and Human Services, Bethesda, MD, United States

Department of Psychiatry, Virginia Commonwealth University, Richmond, VA 23298, United States

Department of Biostatistics, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States

Laureate Institute for Brain Research, Tulsa, OK, United States

          

Keywords:

Adolescent brain cognitive development study

Population neuroscience

Genetics

Hypothesis testing

Reproducibility

Covariate Adjustments

Eect Sizes

       

The Adolescent Brain Cognitive Development (ABCD) Study is the largest single-cohort prospective longitudinal

study of neurodevelopment and children’s health in the United States. A cohort of n = 11,880 children aged 9–10

years (and their parents/guardians) were recruited across 22 sites and are being followed with in-person visits on

an annual basis for at least 10 years. The study approximates the US population on several key sociodemographic

variables, including sex, race, ethnicity, household income, and parental education. Data collected include assess-

ments of health, mental health, substance use, culture and environment and neurocognition, as well as geocoded

exposures, structural and functional magnetic resonance imaging (MRI), and whole-genome genotyping. Here, we

describe the ABCD Study aims and design, as well as issues surrounding estimation of meaningful associations

using its data, including population inferences, hypothesis testing, power and precision, control of covariates,

interpretation of associations, and recommended best practices for reproducible research, analytical procedures

and reporting of results.

1. Introduction

The Adolescent Brain Cognitive Development

(ABCD) Study is

the largest single-cohort long-term longitudinal study of neurodevelop-

∗

Corresponding author.

E-mail address: wkthompson@health.ucsd.edu (W.K. Thompson).

ment and child and adolescent health in the United States. The study

was conceived and initiated by the United States’ National Institutes

of Health (NIH), with funding beginning on September 30, 2015. The

ABCD Study® collects observational data to characterize US population

trait distributions and to assess how biological, psychological, and en-

vironmental factors (including interpersonal, institutional, cultural, and

physical environments) can relate to how individuals live and develop

https://doi.org/10.1016/j.neuroimage.2021.118262 .

Received 13 January 2021; Received in revised form 7 May 2021; Accepted 10 June 2021

Available online 18 June 2021.

( http://creativecommons.org/licenses/by-nc-nd/4.0/ )

A.S. Dick, D.A. Lopez, A.L. Watts et al. NeuroImage 239 (2021) 118262

in today’s society. From the outset, the NIH and ABCD scientic inves-

tigators were motivated to develop a baseline sample that reected the

sociodemographic variation present in the US population of 9–10 year-

old children, and to follow them longitudinally through adolescence and

into early adulthood.

The ABCD Study was designed to address some of the most im-

portant public health questions facing today’s children and adolescents

( Volkow et al., 2018 ). These questions include identifying factors lead-

ing to the initiation and consumption patterns of psychoactive sub-

stances, substance-related problems, and substance use disorders as well

as their subsequent impact on the brain, neurocognition, health, and

mental health over the course of adolescence and into early adulthood.

More broadly, a large epidemiologically informed longitudinal study be-

ginning in childhood and continuing on through early adulthood will

provide a wealth of unique data on normative development, as well as

environmental and biological factors associated with variation in devel-

opmental trajectories. This broader perspective has led to the involve-

ment of multiple NIH Institutes that are stakeholders in the range of

health outcomes targeted in the ABCD design. (Information regarding

funding agencies, recruitment sites, investigators, and project organiza-

tion can be obtained at https://abcdstudy.org ).

Population representativeness, or more precisely, absence of uncor-

rected selection bias in the subject pool, is important in achieving exter-

nal validity, i.e., the ability to generalize specic results of the study to

US society at large. As described below, the ABCD Study attempted to

match the diverse US population of 9–10 year-old children on key de-

mographic characteristics. However, even with a largely representative

sample, failure to account for key confounders can aect internal valid-

ity, i.e., the degree to which observed associations accurately reect the

eects of underlying causal mechanisms. Moreover, it is crucial that the

study collects a rich array of variables that may act as moderators or

mediators, including biological and environmental variables, in order

to aid in identifying potentially causal pathways of interest, to quan-

tify individualized risk for (or resilience to) poor outcomes, and to in-

form public policy decisions. External and internal validity also depend

on assessing the impact of random and systematic measurement error,

implementing analytical methods that incorporate relevant aspects of

study design, and emphasizing robust and replicable estimation of asso-

ciations.

The ABCD Study primary aims are given in the Supplementary Ma-

terials (SM) Section S.1. We describe the study design and outline an-

alytic strategies to address the primary study aims, including worked

examples, with emphasis on approaches that incorporate relevant as-

pects of study design ( Section 2 : Study Design; Section 3 : Population

Weighting). We emphasize the impact of sample size on the precision of

association estimates and thoughtful control of covariates in the context

of the large-scale population neuroscience data produced by the ABCD

Study ( Section 4 : Hypothesis Testing and Power; Section 5 : Eect Sizes;

Section 6 : Control and Confounding Variables), and we briey outline

state-of-the-eld recommendations for promoting reproducible science

(Section SM.5) and best practices for statistical analyses and reporting

of results using the ABCD Study data (Section SM.6). For readability,

more technical subject matter is also largely left to SM sections.

2. Study design

The ABCD Study is a prospective longitudinal cohort study of US chil-

dren born between 2006 and 2008. A total cohort of 𝑛 = 11 , 880 children

aged 9–10 years at baseline (and their parents/guardians) was recruited

from 22 sites (with one site no longer active) and are being followed for

at least ten years. Eligible children were recruited from the household

populations in dened catchment areas for each of the study sites dur-

ing the roughly two-year period beginning September 2016 and ending

in October 2018.

2.1. Recruitment

Within study sites, consenting parents and assenting children were

primarily recruited through a probability sample of public and private

schools augmented to a smaller extent by special recruitment through

summer camp programs and community volunteers. ABCD employed

a probability sampling strategy to identify schools within the catch-

ment areas as the primary method for contacting and recruiting eligible

children and their parents. This method has been used in other large

national studies (e.g., Monitoring the Future ( Bachman et al., 2011 );

the Add Health Study ( Chantala and Tabor, 1999 ); the National Co-

morbidity Replication-Adolescent Supplement ( Conway et al., 2016 );

and the National Education Longitudinal Studies ( Ingels et al., 1990 )).

Twins at four “twin-hub ”sites were recruited from birth registries (see

( Garavan et al., 2018 ; Iacono et al., 2017 ) for participant recruitment

details). A minority of participants were recruited through non-school-

based community outreach and word-of-mouth referrals.

2.2. Inclusion criteria

Across recruitment sites, inclusion criteria consisted of being in the

required age range and able to provide informed consent (parents) and

assent (child). Exclusions were minimal and were limited to lack of En-

glish language prociency in the children, the presence of severe sen-

sory, intellectual, medical or neurological issues that would impact the

validity of collected data or the child’s ability to comply with the pro-

tocol, and contraindications to MRI scanning ( Garavan et al., 2018 ).

Parents must be uent in either English or Spanish.

2.3. Measures

Measures collected in the ABCD Study include a neurocognitive bat-

tery ( Luciana et al., 2018 ; Thompson et al., 2019 ), mental and physical

health assessments ( Barch et al., 2018 ), measures of culture and envi-

ronment ( Zucker et al., 2018 ), biospecimens ( Uban et al., 2018 ), struc-

tural and functional brain imaging ( Casey et al., 2018 ; Hagler et al.,

2018 ), geolocation-based environmental exposure data, wearables and

mobile technology ( Bagot et al., 2018 ), and whole genome genotyping

( Loughnan et al., 2020 ). Many of these measures are collected at in-

person annual visits, with brain imaging collected at baseline and at

every other year going forward. A limited number of assessments are

collected in semi-annual telephone interviews between in-person visits.

Data are publicly released on an annual basis through the NIMH Data

Archive (NDA, https://nda.nih.gov/abcd ). Fig. 1 graphically displays

the measures that have been collected as part of the ABCD NDA 3.0.

Release. Fig. 2 depicts the planned data collection and release schedule

over the initial 10 years of the study.

2.4. Sociodemographics

ABCD sample baseline demographics (from NDA Release 2.0.1,

which contains data from 𝑛 = 11 , 879 subjects) are presented in Table 1 ,

along with a comparison to the corresponding statistics from the Amer-

ican Community Survey (ACS). The ACS is a large probability sample

survey of US households conducted annually by the US Bureau of Census

and provides a benchmark for selected demographic and socioeconomic

characteristics of US children aged 9–10 years. The 2011–2015 ACS Pub-

lic Use Microsample (PUMS) le provides data on over 8000,000 sample

US households. Included in this ve-year national sample of households

are 376,370 individual observations for children aged 9–10 and their

households.

With some minor dierences, the unweighted distributions for the

ABCD baseline sample closely match the ACS-based national estimates

for demographic characteristics including age, sex, and household size.

A.S. Dick, D.A. Lopez, A.L. Watts et al. NeuroImage 239 (2021) 118262

Fig. 1. ABCD Study Assessments for NDA 2.0.1

Release Data.

The general concordance of the samples can be attributed in large part

to three factors: 1) the inherent demographic diversity across the ABCD

study sites; 2) stratication (by race/ethnicity) in the probability sam-

pling of schools within sites; and 3) demographic controls employed

in the recruitment by site teams. Likewise, the unweighted percent-

ages of ABCD children for the most prevalent race/ethnicity categories

are an approximate match to the ACS estimates for US children age

9 and 10. Collectively, children of Asian, American Indian/Alaska Na-

tive (AIAN) and Native Hawaiian/Pacic Islander (NHPI) ancestry are

under-represented in the unweighted ABCD data (3.2%) compared with

ACS national estimates (5.9%). This outcome, which primarily aects

ABCD’s sample of Asian children, may be due in part to dierences in

how the parent/caregiver of the child reports multiple race/ethnicity

ancestry in ABCD and the ACS.

Population inferences

The ABCD recruitment eort worked very hard to maintain similar-

ity of the ABCD sample and the US population with respect to sex and

race/ethnicity of the children in the study. The predominantly proba-

bility sampling methodology for recruiting children within each study

site was intended to randomize over confounding factors that were not

explicitly controlled (or subsequently reected in the population weight-

ing). Nevertheless, school consent and parental consent were strong

forces that certainly may have altered the eectiveness of the random-

ization over these uncontrolled confounders.

3.1. Population weighting

The purpose of population weighting is to control for specic

sources of selection bias and restore unbiasedness to descriptive and

analytical estimates of the population characteristics and relationships

( Heeringa et al., 2017 ). Briey, construction of the population weights

required identication of a key set of demographic and socioeconomic

variables for the children and their households that are measured in

both the ABCD Study and in the ACS household interviews. For the

ABCD eligible children, the common variables include 1) age; 2) sex;

and 3) race/ethnicity. For the child’s household, additional variables

include: 4) family income; 5) family type (married parents, single par-

ent); 6) household size 7) parents’ work force status (family type by

parent employment status); 8) Census Region. A multiple logistic re-

gression model using these variables was then t to the concatenated

ACS and ABCD data to predict study membership. The construction

of the population weights for the ABCD Study is described in detail

in Heeringa and Berglund (2020) ( Heeringa and Berglund, 2020 ). R

scripts for computing the ABCD population weights and for applying

them in analyses are available at https://github.com/ABCD-STUDY/

abcd _ acs _ raked _ propensity . The population weights are available in the

NDA data releases 2.0.1 and 3.0.

3.2. Recommendations

Heeringa and Berglund (2020) ( Heeringa and Berglund, 2020 )

present regression analyses with and without using the population

weights. Although it is important not to over-generalize from a small

set of comparisons to all possible analyses of the ABCD data, the re-

sults described therein lead to recommendations for researchers who

are analyzing the ABCD baseline data. First, unweighted analysis may

result in biased estimates of descriptive population statistics. The po-

tential for bias in unweighted estimates from the ABCD data is strongest

when the variable of interest is highly correlated with socioeconomic

variables including family income, family type and parental work force

A.S. Dick, D.A. Lopez, A.L. Watts et al. NeuroImage 239 (2021) 118262

Fig. 2. ABCD Data Collection and NDA Release Schedule.

participation. Second, for regression models of the ABCD baseline data,

an unweighted analysis using mixed-eects models (e.g., site, family,

individual) is the preferred choice. Presently, there is no empirical ev-

idence from comparative analyses that methods for multi-level weight-

ing ( Rabe-Hesketh and Skrondal, 2006 ) will improve the accuracy or

precision of the model t, although additional research on this topic is

ongoing.

3.3. Example: Application to baseline brain volumes

As a demonstration of the implications of the weighting strategy em-

ployed in the ABCD Study, weighted and unweighted means and stan-

dard errors for ABCD baseline brain morphometry - volumes of cortical

Desikan parcels ( Desikan et al., 2006 ) - are presented in Table 2 . Missing

observations were rst imputed using the R package mice ( van Buuren

and Groothuis-Oudshoorn, 2011 ) before applying weights to the com-

pleted sample. Dierences between unweighted and weighted means are

quite small in the baseline sample in this case. As longitudinal MRI data

become available in ABCD (starting with the second post-baseline an-

nual follow-up visit), population-valid mean trajectories of brain-related

outcomes will also be computable using a similar population weighting

scheme, also allowing for characterization of variation of trajectories

from the population mean.

Hypothesis testing and power

Developing an operational approach to evaluate the meaningfulness

of research ndings has been a subject of consistent debate throughout

the history of statistics (

Stigler, 1986 ). Even with the continued eorts

to synthesize systems of statistical inference ( Efron and Hastie, 2016 ),

the resolution of this issue is unlikely to occur any time soon. Most neu-

roscientists continue to work within the context of the classical frequen-

tist null-hypothesis signicance testing (NHST) paradigm ( Efron, 1998 ;

Lehmann, 1993 ), although non-frequentist approaches (e.g. Bayesian,

machine learning prediction ( Efron, 2013 ; Efron, 2020 )) are increas-

ingly common and may be more appropriate for large datasets like the

ABCD Study.

Despite growing enthusiasm for these alternatives, p-values continue

to be important data points in the presentation of results in the be-

havioral and neurosciences. The NHST p-value “…is the probability

under a specied statistical model that a statistical summary of the

data…would be equal to or more extreme than its observed value ”

(

Wasserstein and Lazar, 2016 ). The utility of NHST and the arbitrariness

of the 0 . 05 signicance threshold has been debated extensively ( Gelman,

2018 ; Wasserstein and Lazar, 2016 ; Nickerson, 2000 ; Harlow et al.,

2013 ). While we will not relitigate these issues here, we will at-

tempt to address how best to present statistical evidence that lever-

ages the ABCD Study’s large sample size (aecting statistical power),

population sampling frame, and rich longitudinal assessment proto-

col to enable meaningful and valid insights into child and adolescent

neurodevelopment.

4.1. Power

Statistical power in the NHST framework is dened as the probabil-

ity of rejecting a false null hypothesis. Power is determined by three

A.S. Dick, D.A. Lopez, A.L. Watts et al. NeuroImage 239 (2021) 118262

Table 1

ABCD Baseline and ACS 2011–2015 Demographic Characteristics.

Characteristic Category ABCD ( n = 11,879) ACS 2011–2015

% N %

Population Total 100 8211,605 100

Age 9 52.3 4074,807 49.6

10 47.8 4136,798 50.4

Sex Male 52.2 4205,925 51.2

Female 47.8 4005,860 48.8

Race/Ethnicity NH White 52.2 4305,552 52.4

NH Black 15.1 1101,297 13.4

Hispanic 20.4 1973,827 24.0

Asian, AIAN, NHPI 3.2 487,673 5.9

Multiple 9.2 343,256 4.2

Family Income <

$25k 16.1 1762,415 21.5

$25k - $49k 15.1 1784,747 21.7

$50k - $74k 14.0 1397,641 17.0

$75k - $99k 14.1 1023,127 12.5

$100k - $199k 29.5 1685,036 20.5

$200k + 11.2 558,639 6.8

Family Type Married Parents 73.4 5426,131 66.1

Other Family Type 26.6 2785,474 33.9

Parent Employment Married, 2

in LF 50.2 3353,572 40.8

Married, 1 in LF 21.9 1949,288 23.7

Married, 0 in LF 1.3 156,807 1.9

Single, in LF 21.1 2174,365 26.5

Single, Not in LF 5.4 577,573 7.0

Region Northeast 16.9 1336,183 16.3

Midwest 20.4 1775,723 21.6

South 28.3 3117,158 38.0

West 34.4 1982,541 24.1

Household Size

2 to 3 17.3 1522,216 18.5

4 33.5 2751,942 33.5

5 24.9 2085,666 25.4

6 14.0 1025,285 12.5

7 + 10.3 826,496 10.1

LF = labor force.

ACS = American Community Survey.

factors: 1) the signicance level 𝛼; 2) the magnitude of the population

parameter; and 3) the accuracy (precision and bias) of the model es-

timates. Increasing power while maintaining a specied Type I error

rate depends largely on obtaining more precise association parameter

estimates from improved study designs, more ecient statistical meth-

ods, and, importantly, increasing sample size ( Rothman et al., 2008 ;

Button et al., 2013 ; Hong and Park, 2012 ).

The ABCD Study has a large sample compared to typical neurode-

velopmental studies, so much so that one might expect even very small

associations to be statistically signicant. In our experience, not all as-

sociations in the ABCD Study are guaranteed to have small p-values. For

example, a recent study attempting to replicate the often-cited bilingual

executive function advantage failed to nd evidence for the advantage in

the rst data release (NDA 1.0) of the ABCD Study ( 𝑛 = 4524 ) ( Dick et al.,

2019 ).

Nevertheless, even very small associations are well-powered in the

ABCD Study. Fig. 3 displays power curves as a function of sample size

for dierent values of absolute Pearson correlations |𝑟 |. The dashed line

in Fig. 3 indicates the full ABCD baseline sample size of 𝑛 = 11 , 880 .

As can be seen, Pearson correlations |r| = 0.04 and above have power

> 0 . 99 at 𝛼 = 0 . 05 . Simply rejecting a null hypothesis without reporting

on other aspects of the study design and statistical analyses (includ-

ing discussion of plausible alternative explanatory models and threats

to validity), as well as the observed magnitude of associations, is unin-

formative, perhaps particularly so in the context of very well-powered

studies ( Abadie, 2020 ).

5. Eﬀect sizes

Because p-values may be less informative in the context of very well-

powered studies like ABCD, eect sizes become important data points

in determining the importance of the ndings. Eect sizes quantify

relationships between two or more variables, e.g., correlation coe-

cients, proportion of variance explained ( 𝑅

), Cohen’s 𝑑, relative risk,

number needed to treat, and so forth ( Cohen, 1988 ; Kraemer, 1992 ;

Rosenthal et al., 2000 ), with one variable often thought of as indepen-

dent (exposure) and the other dependent (outcome) ( Rothman et al.,

2008 ). Eect sizes are independent of sample size, e.g., t-tests and p-

values are not eect sizes; however, the precision of eect size estima-

tors depend on sample size as described earlier. Consensus best practice

recommendations are that eect size point estimates be accompanied

by intervals to illustrate the precision of the estimate and the conse-

quent range of plausible values indicated by the data ( Wasserstein and

Lazar, 2016 ). Table 3 presents a number of commonly used eect size

metrics ( Kirk, 1996 ; Fidler et al., 2004 ). We wish to avoid being overly

prescriptive for which of these eect sizes to employ in ABCD applica-

tions, as researchers should think carefully about the intended use of

their analyses and pick an eect size metric that addresses their partic-

ular research question.

5.1. Small eﬀects

As much as the choice of which eect size statistic to report is driven

by context, the interpretation of the practical utility of the observed ef-

fect size is even more so. While small p-values do not imply that reported

eects are inherently substantive, “small ”eect sizes might have prac-

tical or even clinical signicance in the right context ( Rosenthal et al.,

2000 ).

As described in SM Section S.2, known problems of publication

bias and incentives for researchers to nd signicant associations

( Button et al., 2013 ; Simonsohn et al., 2014 ) combined with the pre-

dominantly small sample sizes of most prior neurodevelopmental stud-

Meaningful Effects in the Adolescent Brain Cognitive Development Study

Figures

Citations

R. A. Fisher in the 21st Century

New insights into the dynamic development of the cerebral cortex in childhood and adolescence: Integrating macro- and microstructural MRI findings.

Development of white matter microstructure and executive functions during childhood and adolescence: a review of diffusion MRI studies

Understanding Vulnerability and Adaptation in Early Brain Development using Network Neuroscience.

A history of previous childbirths is linked to women's white matter brain age in midlife and older age.

References

mice: Multivariate Imputation by Chained Equations in R

An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest.

Power failure: why small sample size undermines the reliability of neuroscience

Statistical Power Analysis

Why Most Published Research Findings Are False

Related Papers (5)

The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites.

Recruiting the ABCD sample: Design considerations and procedures

Image processing and analysis methods for the Adolescent Brain Cognitive Development Study.

Demographic, physical and mental health assessments in the adolescent brain and cognitive development study: Rationale and description

Measuring the thickness of the human cerebral cortex from magnetic resonance images

Frequently Asked Questions (1)

Q1. What are the contributions in "Meaningful associations in the adolescent brain cognitive development study" ?