On unreplicable inferences in psychopathology symptom networks and the importance of unreliable parameter estimates

We recently wrote an article comparing the conclusions that followed from two different approaches to quantifying the reliability and replicability of psychopathology symptom networks. Two commentaries on the article have raised five core criticisms, which are addressed in this response with supporting evidence. 1) We did not over-generalise about the replicability of symptom networks, but rather focused on interpreting the contradictory conclusions of the two sets of methods we examined. 2) We closely followed established recommendations when estimating and interpreting the networks. 3) We also closely followed the relevant tutorials, and used examples interpreted by experts in the field, to interpret the bootnet and NetworkComparisonTest results. 4) It is possible for statistical control to increase reliability, but that does not appear to be the case here. 5) Distinguishing between statistically significant versus substantive differences makes it clear that the differences between the networks affect the inferences we would make about symptom-level relationships (i.e., the basis of the purported utility of symptom networks). Ultimately, there is an important point of agreement between our article and the commentaries: All of these applied examples of cross-sectional symptom networks are demonstrating unreliable parameter estimates. While the commentaries propose that the resulting differences between networks are not genuine or meaningful because they are not statistically significant, we propose that the unreplicable inferences about the symptom-level relationships of interest fundamentally undermine the utility of the symptom networks.

On Formalizing Theoretical Expectations: Bayesian Testing of Central Structures in Psychological Networks

Network theory has emerged as a popular framework for conceptualizing psychological constructs and mental disorders. Initially, network analysis was motivated in part by the thought that it can be used for hypothesis generation. Although the customary approach for network modeling is inherently exploratory, we argue that there is untapped potential for confirmatory hypothesis testing. In this work, we bring to fruition the potential of Gaussian graphical models for generating testable hypotheses. This is accomplished by merging exploratory and confirmatory analyses into a cohesive framework built around Bayesian hypothesis testing of partial correlations. We first present a motivating example based on a customary, exploratory analysis, where it is made clear how information encoded by the conditional (in)dependence structure can be used to formulate hypotheses. Building upon this foundation, we then provide several empirical examples that unify exploratory and confirmatory testing in psychopathology symptom networks. In particular, we (1) estimate exploratory graphs; (2) derive hypotheses based on the most central structures; and (3) test those hypotheses in a confirmatory setting. Our confirmatory results uncovered an intricate web of relations, including an order to edge weights within comorbidity networks. This illuminates the rich and informative inferences that can be drawn with the proposed approach. We conclude with recommendations for applied researchers, in addition to discussing how our methodology answers recent calls to begin developing formal models related to the conditional (in)dependence structure of psychological networks.

M24. NETWORK STRUCTURE OF PSYCHOPATHOLOGY SYMPTOMS IN A COMMUNITY SAMPLE OF YOUTH

Background The field of network psychometrics has developed into a promising alternative to the common cause theory and depicts mental health disorders as arising from the interactions between individual symptoms. Currently, major depressive disorder and post-traumatic stress disorder have been the two main disorders studied using network models. In this study, we aimed to examine the network structure of psychopathology symptoms in a community youth sample to detect the most influential symptoms. We also identify influential bridge symptoms which may lead to comorbidities. Methods The sample (n = 2875) was taken from the Philadelphia Neurodevelopmental Cohort and comprised of youth between the ages of 11–21. 112 variables corresponding to 17 psychopathology symptom groups were used to build the network model. We estimated the network structure using a mixed graphical model. Edges were estimated using a pairwise weighted adjacency matrix with EBIC regularization at a default gamma level of 0.25. The relative influence of each node was determined using predictability and centrality measurements including node strength, closeness, and betweenness. A network was similarly created to detect the most influential bridge symptoms using community clusters. Results The network generated from 17 psychopathology symptom domains (comprising ADD, agoraphobia, conduct disorder, depression, generalized anxiety disorder, mania, OCD, ODD, panic disorder, phobia, psychosis, PTSD, general probes, separation anxiety, psychosis prodromal symptoms, social anxiety and suicide) had several distinct cluster regions and two independent psychosis prodromal symptom nodes. No negative associations were observed in the network. The strongest edge regression coefficient (1.593) was detected between a general screening probe asking whether the subject had received previous treatment and a psychosis variable related to hallucination. An OCD item eliciting subject’s fear over accidentally doing something bad had the greatest average centrality measurement (2.317) followed closely by a conduct disorder item eliciting if the subject had ever threatened someone (2.254). Two depression items - irritability (2.228) and depressive mood (1.825) had the largest average bridge centrality values. History of inpatient treatment (0.997), fear of traveling in a car (0.989) and compulsive checking (0.989) had the largest predictability values, suggesting they could potentially be effective intervention targets. Discussion OCD and conduct disorder symptoms had the largest centrality values and are influential symptoms that could potentially be used to more effectively screen youth for mental health disorders. Depression symptoms had the largest bridge centrality values and should be targeted to prevent comorbidity of associated symptoms. Understanding psychopathology symptom networks could potentially lead to greater insights for prevention and individualizing treatments.

Riskier Tests of the Validity of the Bifactor Model of Psychopathology

We advanced several “riskier tests” of the validity of bifactor models of psychopathology, which included that the general and specific psychopathology factors should be reliable and well represented by their respective indicators and that including a general factor should improve on the correlated factor model’s external validity. We compared bifactor and correlated factors models of psychopathology using data from a community sample of youth ( N = 2,498) whose parents provided ratings on psychopathology and theoretically relevant external criteria (i.e., personality, aggression, antisociality). Bifactor models tended to yield either general or specific factors that were unstable and difficult to interpret. The general factor appeared to reflect a differentially weighted amalgam of psychopathology rather than a liability for psychopathology broadly construed. With rare exceptions, bifactor models did not explain additional variance in first-order psychopathology symptom dimensions or external criteria compared with correlated factors models. Together, our findings call into question the validity of bifactor models of psychopathology and the p factor more broadly.

Riskier Tests of the Validity of Bifactor Models of Psychopathology

We advanced several “riskier tests” of the validity of bifactor models of psychopathology, which included that the general and specific factors should be reliable and well-represented by their indicators, and that including a general factor should improve the correlated factor model’s external validity. We compared bifactor and correlated factors models using data from a community sample of youth (N=2498) whose parents provided ratings on psychopathology and external criteria (i.e., temperament, aggression, antisociality). Bifactor models tended to yield either general or specific factors that were unstable and difficult to interpret. The general factor appeared to reflect a differentially-weighted amalgam of psychopathology rather than a liability for psychopathology broadly construed. With rare exceptions, bifactor models did not explain additional variance in psychopathology symptom dimensions or external criteria compared with correlated factors models. Together, our findings call into question the validity of bifactor models of psychopathology, and the p-factor more broadly.

Problems with centrality measures in psychopathology symptom networks: Why network psychometrics cannot escape psychometric theory

Understanding patterns of symptom co-occurrence is one of the most difficult challenges in psychopathology research. Do symptoms co-occur because of a latent factor, or might they directly and causally influence one another? Motivated by such questions, there has been a surge of interest in network analyses that emphasize the putatively direct relationships among symptoms. In this critical paper, we highlight conceptual and statistical problems with using centrality measures in cross-sectional networks. In particular, common network analyses assume that there are no unmodeled latent variables that confound symptom co-occurrence. In simulations that include latent variables, we demonstrate that closeness and betweenness are vulnerable to spurious covariance among symptoms that connect subgraphs (e.g., diagnoses). Furthermore, strength is redundant with factor loading in several cases. Finally, if a symptom reflects multiple latent causes, centrality metrics reflect a weighted combination, undermining their interpretability in empirical data. Our results suggest that it is essential for network psychometric approaches to examine the evidence for latent variables prior to analyzing or interpreting symptom centrality. Failing to do so risks identifying spurious relationships or failing to detect causally important effects. Altogether, centrality measures do not provide solid ground for understanding the structure of psychopathology when latent confounding exists.