Assessing the transportability of clinical prediction models for cognitive impairment using causal models

BMC Medical Research Methodology

Research

Jana Fehr¹  Stefan Konigorski²  Tobias Kurth³  Marco Piccininni⁴ 

[1] Digital Engineering Faculty, University of Potsdam, Potsdam, Germany;Digital Health and Machine Learning, Hasso-Plattner-Institute, Potsdam, Germany;Digital Engineering Faculty, University of Potsdam, Potsdam, Germany;Digital Health and Machine Learning, Hasso-Plattner-Institute, Potsdam, Germany;Icahn School of Medicine at Mount Sinai, Hasso Plattner Institute for Digital Health at Mount Sinai, New York, NY, USA;Institute of Public Health, Charité – Universitätsmedizin Berlin, Berlin, Germany;Institute of Public Health, Charité – Universitätsmedizin Berlin, Berlin, Germany;Center for Stroke Research Berlin, Charité – Universitätsmedizin Berlin, Berlin, Germany;

关键词: Alzheimer’s Disease;    Clinical risk prediction;    DAG;    Causality;    Transportability;   

DOI  :  10.1186/s12874-023-02003-6

 received in 2022-08-05, accepted in 2023-07-27,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundMachine learning models promise to support diagnostic predictions, but may not perform well in new settings. Selecting the best model for a new setting without available data is challenging. We aimed to investigate the transportability by calibration and discrimination of prediction models for cognitive impairment in simulated external settings with different distributions of demographic and clinical characteristics.MethodsWe mapped and quantified relationships between variables associated with cognitive impairment using causal graphs, structural equation models, and data from the ADNI study. These estimates were then used to generate datasets and evaluate prediction models with different sets of predictors. We measured transportability to external settings under guided interventions on age, APOE ε4, and tau-protein, using performance differences between internal and external settings measured by calibration metrics and area under the receiver operating curve (AUC).ResultsCalibration differences indicated that models predicting with causes of the outcome were more transportable than those predicting with consequences. AUC differences indicated inconsistent trends of transportability between the different external settings. Models predicting with consequences tended to show higher AUC in the external settings compared to internal settings, while models predicting with parents or all variables showed similar AUC.ConclusionsWe demonstrated with a practical prediction task example that predicting with causes of the outcome results in better transportability compared to anti-causal predictions when considering calibration differences. We conclude that calibration performance is crucial when assessing model transportability to external settings.

【 授权许可】

CC BY   
© BioMed Central Ltd., part of Springer Nature 2023

【 图 表 】

13690_2023_1151_Article_IEq7.gif

Fig. 3

Fig. 2

Fig. 1

Fig. 3