External validation of existing dementia prediction models on observational health data

BMC Medical Research Methodology

Research

Egill A. Fridgeirsson¹  Luis H. John¹  Jan A. Kors¹  Peter R. Rijnbeek¹  Jenna M. Reps² 

[1] Department of Medical Informatics, Erasmus University Medical Center, Dr. Molewaterplein 40, 3015 GD, Rotterdam, The Netherlands;Janssen Research and Development, 1125 Trenton Harbourton Rd, 08560, Titusville, NJ, USA;

关键词: Patient-level prediction;    Prognostic model;    External validation;    Transportability;    Dementia;    Alzheimer;   

DOI  :  10.1186/s12874-022-01793-5

 received in 2022-06-09, accepted in 2022-11-15,  发布年份 2022

来源: Springer

【 摘 要 】

BackgroundMany dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article.In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records.MethodsWe identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters’ Dementia Risk Score, Mehta’s RxDx-Dementia Risk Index, and Nori’s ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models.ResultsWe reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67–0.76 c-statistic). The Mehta model (development c-statistic: 0.81) transported well to some of the external databases (0.69–0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62–0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard.ConclusionWe observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.

【 授权许可】

CC BY   
© The Author(s) 2022

【 图 表 】

41408_2022_764_Article_IEq24.gif

41408_2022_764_Article_IEq23.gif

41408_2022_764_Article_IEq22.gif

41408_2022_764_Article_IEq21.gif

41408_2022_764_Article_IEq20.gif

41408_2022_764_Article_IEq19.gif

Fig. 1

41408_2022_764_Article_IEq17.gif

41408_2022_764_Article_IEq15.gif