Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

BMC Medicine

Research Article

Ji Chen¹  Chao Li²  Xuerong Liu³  Zhengzhi Feng³  Kuan Miao³  Xingmei Gu³  Zhiyi Chen⁴  Bowen Hu⁵  Xin Dai⁵  Simon B. Eickhoff⁶  Yancheng Tang⁷  Artemiy Leonov⁸  Hu Chuan-Peng⁹  Zhibing Xiao^1,10  Benjamin Becker^1,11 

[1] Department of Psychology and Behavioral Sciences, Zhejiang University, Hangzhou, China;Department of Psychiatry, The Fourth Affiliated Hospital, Zhejiang University School of Medicine, Yiwu, Zhejiang, China;Department of Radiology, The Third Affiliated Hospital, Sun Yat-Sen University, Guangdong, China;Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China;Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China;Faculty of Psychology, Southwest University, Chongqing, China;Faculty of Psychology, Southwest University, Chongqing, China;Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf, Düsseldorf, Germany;School of Business and Management, Shanghai International Studies University, Shanghai, China;School of Psychology, Clark University, Worcester, MA, USA;School of Psychology, Nanjing Normal University, Nanjing, China;State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China;The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People’s Hospital, Chengdu, China;The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Laboratory for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, China;

关键词: Psychiatric machine learning;    Diagnostic classification;    Meta-analysis;    Neuroimaging;    Sampling inequalities;   

DOI  :  10.1186/s12916-023-02941-4

 received in 2023-02-10, accepted in 2023-06-13,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundThe development of machine learning models for aiding in the diagnosis of mental disorder is recognized as a significant breakthrough in the field of psychiatry. However, clinical practice of such models remains a challenge, with poor generalizability being a major limitation.MethodsHere, we conducted a pre-registered meta-research assessment on neuroimaging-based models in the psychiatric literature, quantitatively examining global and regional sampling issues over recent decades, from a view that has been relatively underexplored. A total of 476 studies (n = 118,137) were included in the current assessment. Based on these findings, we built a comprehensive 5-star rating system to quantitatively evaluate the quality of existing machine learning models for psychiatric diagnoses.ResultsA global sampling inequality in these models was revealed quantitatively (sampling Gini coefficient (G) = 0.81, p < .01), varying across different countries (regions) (e.g., China, G = 0.47; the USA, G = 0.58; Germany, G = 0.78; the UK, G = 0.87). Furthermore, the severity of this sampling inequality was significantly predicted by national economic levels (β =  − 2.75, p < .001, R2adj = 0.40; r =  − .84, 95% CI: − .41 to − .97), and was plausibly predictable for model performance, with higher sampling inequality for reporting higher classification accuracy. Further analyses showed that lack of independent testing (84.24% of models, 95% CI: 81.0–87.5%), improper cross-validation (51.68% of models, 95% CI: 47.2–56.2%), and poor technical transparency (87.8% of models, 95% CI: 84.9–90.8%)/availability (80.88% of models, 95% CI: 77.3–84.4%) are prevailing in current diagnostic classifiers despite improvements over time. Relating to these observations, model performances were found decreased in studies with independent cross-country sampling validations (all p < .001, BF10 > 15). In light of this, we proposed a purpose-built quantitative assessment checklist, which demonstrated that the overall ratings of these models increased by publication year but were negatively associated with model performance.ConclusionsTogether, improving sampling economic equality and hence the quality of machine learning models may be a crucial facet to plausibly translating neuroimaging-based diagnostic classifiers into clinical practice.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

41116_2023_38_Article_IEq167.gif

41116_2023_38_Article_IEq165.gif

41116_2023_38_Article_IEq149.gif

41116_2023_38_Article_IEq137.gif

41116_2023_38_Article_IEq132.gif

41116_2023_38_Article_IEq125.gif

41116_2023_38_Article_IEq116.gif

41116_2023_38_Article_IEq108.gif

41116_2023_38_Article_IEq101.gif