False alarm? A comprehensive reanalysis of "Evidence that psychopathology symptom networks have limited replicability" by Forbes, Wright, Markon, and Krueger (2017).

TL;DR: An assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples and a comprehensive reanalysis of the data led to results directly opposed to Forbes et al.

Abstract: Forbes, Wright, Markon, and Krueger (2017) stated that "psychopathology networks have limited replicability" (p. 1011) and that "popular network analysis methods produce unreliable results" (p. 1011). These conclusions are based on an assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples; in addition, Forbes et al. analyzed the stability of the network models within the samples using split-halves. Our reanalysis of the same data with the same methods led to results directly opposed to theirs: All network models replicated very well across the two data sets and across the split-halves. We trace the differences between Forbes et al.'s results and our own to the fact that they did not appear to accurately implement all network models and used debatable metrics to assess replicability. In particular, they deviated from existing estimation routines for relative importance networks, did not acknowledge the fact that the skip structure used in the interviews strongly distorted correlations between symptoms, and incorrectly assumed that network structures and metrics should be the same not only across the different samples but also across the different network models used. In addition to a comprehensive reanalysis of the data, we end with a discussion of best practices concerning future research into the replicability of psychometric networks. (PsycINFO Database Record

Summary

Introduction

• A second issue that the authors consider to qualify as a statistical inaccuracy concerns FWMK’s use of a distorted tetrachoric correlation matrix, which underlies both their factor analyses and their association networks.
• FWMK’s abstract presents, as a main result, that “only 13-21% of the edges were consistently estimated across these networks”.
• Analysis code should naturally always be available, as it is needed to replicate and verify reported analyses – the current report illustrates how important this is – and the authors commend FWMK for sharing their code.

Estimating network structures

• E.g., the authors obtained 35 edges in the NCS DAG network often.
• Co-morbid obsessive-compulsive disorder and depression: a Bayesian net work approach.

Replicating errors in Forbes

• Shape of curve and placement of nodes might differ using different qgraph v ersions Error 2: Implausible correlation matrix due to imputation method.
• Establish correlation matrices using different methods for handling missing data.
• This shows that the main problem lies in the imputation of zeroes and not in the fact that the nearest positive definite matrix is used.

Stability Analyses

• These codes use bootnet to establish stability assessments of the Ising model, default="IsingFit") # Bootstraps ran on 24-core supercomputer.
• Table B1. Summary of split-half comparisons for the NCS-R data.
• This table matches the analysis reported in Table 3 of Forbes et al. (2017).
• In addition to the metrics discussed by FMWK (see their Table 2 for detailed explanations), the table reports Pearson correlations between network parameters in the two samples (all > .9), and replication statistics for censored and uncensored relative importance networks as implemented in accordance with Robinaugh et al. (2014).

In addition to the metrics discussed by FMWK (see their Table 2 for detailed explanations), the table reports Pearson correlations between network parameters in the two samples (all > .9), and replication statistics for censored and uncensored relative importance networks as implemented in accordance with Robinaugh et al. (2014).

• Ising Model estimated on NSMHWB data, also known as Right.
• Black boxes represent significant differences and gray boxes represent non-significant differences.
• Given the similarity of the datasets and networks, this is surprising.
• In NCS, the edge 6— 14 is slightly stronger than 4—18, leading shortest paths between the two clusters (on which betweenness centrality is estimated) to more consistently (irrespective of the particular participants included in the sample) go through nodes 6 and 14.

Figures

Figure 1. Network structures estimated with the Ising model (left panels), censored relative importance networks (middle panels), and Directed Acyclic Graphs (DAGs; right panels) for the NCS-R (top panels) and NSMHWB (bottom panels) data.

Table 1. Replication results of comparing the networks for the NCS-R and NSMHWB data. In addition to the metrics discussed by Forbes et al. (see their Table 2 for detailed explanations), the table reports Pearson correlations between network parameters in the two samples (all > .9), replication statistics for censored and uncensored relative importance networks as implemented in accordance with Robinaugh et al. (2014), and most central nodes for different centrality measures. See Table 1 of Forbes et al. (2017) for node abbreviations.

Figure 2. Relative importance networks (estimated on the NCS-R data) using normalized lmg (left, as used by Robinaugh et al. 2014; 118 edges) and non-normalized lmg (middle and right). Red edges in the middle panel (99 edges) indicate edges that are removed by the thresholding rule used by Forbes et al. (2017), and the right panel shows the network they reported (31 edges).

Figure 4. The absolute percentage change in edge weights across datasets relative to the size of the edge weights (left panel). This panel shows that smaller edge weights show larger changes expressed as a percentage of the original weight. The right panel shows that these changes are mostly irrelevant: the strong linear relation (r=.95) between edge weights in the NCS-R and NSMHWB data (right panel) is unaffected by the parameter changes.

Figure 3. The effect of the imputation strategy used by FWMK on the network structure of the NCS-R data. The figure shows how imputation alters the tetrachoric correlation between depression symptoms, and the resulting networks. The correlations shown represent actual NCS-R correlations between four non-skip items (weight problems, sleep problems, psychomotor problems, and fatigue) before and after imputation.

Full text

UvA-DARE is a service provided by the library of the University of Amsterdam (http
s
://dare.uva.nl)
UvA-DARE (Digital Academic Repository)
False alarm? A comprehensive reanalysis of "evidence that psychopathology
symptom networks have limited replicability" by Forbes, Wright, Markon, and
Krueger (2017)
Borsboom, D.; Fried, E.I.; Epskamp, S.; Waldorp, L.J.; van Borkulo, C.D.; van der Maas,
H.L.J.; Cramer, A.O.J.
DOI
10.1037/abn0000306
Publication date
2017
Document Version
Submitted manuscript
Published in
Journal of Abnormal Psychology
Link to publication
Citation for published version (APA):
Borsboom, D., Fried, E. I., Epskamp, S., Waldorp, L. J., van Borkulo, C. D., van der Maas, H.
L. J., & Cramer, A. O. J. (2017). False alarm? A comprehensive reanalysis of "evidence that
psychopathology symptom networks have limited replicability" by Forbes, Wright, Markon,
and Krueger (2017).
Journal of Abnormal Psychology
,
126
(7), 989-999.
https://doi.org/10.1037/abn0000306
General rights
It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)
and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open
content license (like Creative Commons).
Disclaimer/Complaints regulations
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please
let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material
inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter
to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You
will be contacted as soon as possible.
Download date:10 Aug 2022

Seediscussions,stats,andauthorprofilesforthispublicationat:https://www.researchgate.net/publication/320885559
Falsealarm?Acomprehensivereanalysisof
"Evidencethatpsychopathologysymptom
networkshavelimitedre....
ArticleinJournalofAbnormalPsychology·October2017
DOI:10.1037/abn0000306
CITATIONS
8
READS
361
7authors,including:
Someoftheauthorsofthispublicationarealsoworkingontheserelatedprojects:
PersonalizedNetworkModelinginPsychopathology:TheImportanceofContemporaneousand
TemporalConnectionsViewproject
Psychosis:TowardsaDynamicalSystemsApproachViewproject
DennyBorsboom
UniversityofAmsterdam
195PUBLICATIONS8,138CITATIONS
SEEPROFILE
EikoFried
UniversityofAmsterdam
70PUBLICATIONS804CITATIONS
SEEPROFILE
SachaEpskamp
UniversityofAmsterdam
49PUBLICATIONS2,183CITATIONS
SEEPROFILE
LourensWaldorp
UniversityofAmsterdam
97PUBLICATIONS2,169CITATIONS
SEEPROFILE
AllcontentfollowingthispagewasuploadedbyEikoFriedon12November2017.
Theuserhasrequestedenhancementofthedownloadedfile.

1
False alarm?
A comprehensive reanalysis of “Evidence that psychopathology symptom networks have
limited replicability” by Forbes, Wright, Markon, and Krueger.
Denny Borsboom
Eiko I. Fried
Sacha Epskamp
Lourens J. Waldorp
Claudia D. van Borkulo
Han L. J. Van der Maas
University of Amsterdam
Angélique O. J. Cramer
Tilburg University
Word count:
Main text: 7061
Abstract: 223

2
Author note
We would like to thank Richard McNally, Jeroen Vermunt, Claudi Bockting, and Helma van
den Berg for their comments on an earlier draft of this paper, and Ria Hoekstra for her help in
gathering and processing data used in this manuscript. Denny Borsboom, Eiko Fried, Claudia
van Borkulo, and Lourens Waldorp are supported by European Research Council
Consolidator Grant no. 647209. Angélique Cramer is supported by Veni grant no. 451-14-
002 awarded by the Netherlands Organisation for Scientific Research (NWO).
Correspondence concerning this article should be addressed to Denny Borsboom, Department
of Psychology, University of Amsterdam, Nieuwe Achtergracht 129-B, 1018 WT
Amsterdam, The Netherlands, email: dennyborsboom@gmail.com.

3
Abstract
Forbes, Wright, Markon, and Krueger (2017) state that “psychopathology networks have
limited replicability” and that “popular network analysis methods produce unreliable results”.
These conclusions are based on an assessment of the replicability of four different network
models for symptoms of major depression and generalized anxiety across two samples; in
addition, Forbes et al. (2017) analyze the stability of the network models within the samples
using split-halves. Our re-analysis of the same data with the same methods led to results
directly opposed to those of Forbes et al. (2017): All network models replicate very well
across the two datasets and across the split-halves. We trace the differences between Forbes
et al.’s (2017) results and our own to the fact that they did not appear to accurately implement
all network models, and used debatable metrics to assess replicability. In particular, Forbes et
al. (2017) deviate from existing estimation routines for relative importance networks, do not
acknowledge the fact that the skip-structure used in the interviews strongly distorted
correlations between symptoms, and incorrectly assume that network structures and metrics
should not only be expected to be the same across the different samples, but also across the
different network models used. In addition to a comprehensive re-analysis of the data, we end
with a discussion of best practices concerning future research into the replicability of
psychometric networks.

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "False alarm? a comprehensive reanalysis of "Evidence that psychopathology symptom networks have limited re..." ?

Borsboom, D., Fried, E. I., Epskamp, S., Waldorp, L. J., van Borkulo, C. D., van der Maas, H. L. this paper, and Cramer, A. J. ( 2017 ). False alarm ? A comprehensive reanalysis of `` evidence that psychopathology symptom networks have limited replicability '' by Forbes, Wright, Markon, and Krueger. 

Q2. what is the censored value of a network?

Relative importance networks (uncensored) DAGsFirst half Second half First half Second half First half Second half First half Second half Network characteristics Connectivity (% of possible) 46.7% (45.1-48.4) 47.1% (44.4- 49.7) 38.6% (37.9- 39.2) 38.6% (37.9- 38.9) 100% (100-100) 100% (100-100) 17% (16.3-19) 17.3% (15.7-18.3) Density (as in Forbes et al.) 1.14 (1.11-1.17) 1.12 (1.08-1.19) 0.13 (0.13-0.13) 0.13 (0.13-0.13) 0.06 (0.06-0.06) 0.06 (0.06-0.06) 

Q3. What is the way to run the DAG analysis?

<- boot.strength(DataNCScat, R = 1000, algorithm = "hc", algor ithm.args = list(restart = 5, perturb = 10), debug = TRUE) # Edges with strength > 0.85: DAG_NCS <- amat(averaged.network(bnlearnRes_NCS, threshold = 0.85)) Step 4: Run relative importance networks using Robinaugh et al.'s (2014) procedure, using normalized lmg. 

Q4. what is the problem with the relimp_NCS_uncensored?

This shows that the main problem lies in the imputation of zeroes and not in the fact that the nearest positive definite matrix is used.