Comparison of discrimination and calibration performance of ECG-based machine learning models for prediction of new-onset atrial fibrillation

BMC Medical Research Methodology

Research

Andrea Di Lenarda¹  Arjuna Scagnetto¹  Chiara Cappelletto¹  Giovanni Baj²  Luca Bortolussi²  Giulia Barbati³  Ilaria Gandin³ 

[1] Cardiovascular Center, Territorial Specialistic Department, University Hospital and Health Services of Trieste, Trieste, Italy;Department of Mathematics and Geosciences, University of Trieste, Trieste, Italy;Department of Medical Sciences, Biostatistics Unit, University of Trieste, Trieste, Italy;

关键词: Atrial fibrillation;    Prediction;    Calibration;    Machine learning;    Deep learning;   

DOI  :  10.1186/s12874-023-01989-3

 received in 2023-01-24, accepted in 2023-07-13,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundMachine learning (ML) methods to build prediction models starting from electrocardiogram (ECG) signals are an emerging research field. The aim of the present study is to investigate the performances of two ML approaches based on ECGs for the prediction of new-onset atrial fibrillation (AF), in terms of discrimination, calibration and sample size dependence.MethodsWe trained two models to predict new-onset AF: a convolutional neural network (CNN), that takes as input the raw ECG signals, and an eXtreme Gradient Boosting model (XGB), that uses the signal’s extracted features. A penalized logistic regression model (LR) was used as a benchmark. Discrimination was evaluated with the area under the ROC curve, while calibration with the integrated calibration index. We investigated the dependence of models’ performances on the sample size and on class imbalance corrections introduced with random under-sampling.ResultsCNN's discrimination was the most affected by the sample size, outperforming XGB and LR only around n = 10.000 observations. Calibration showed only a small dependence on the sample size for all the models considered.Balancing the training set with random undersampling did not improve discrimination in any of the models. Instead, the main effect of imbalance corrections was to worsen the models’ calibration (for CNN, integrated calibration index from 0.014 [0.01, 0.018] to 0.17 [0.16, 0.19]).The sample size emerged as a fundamental point for developing the CNN model, especially in terms of discrimination (AUC = 0.75 [0.73, 0.77] when n = 10.000, AUC = 0.80 [0.79, 0.81] when n = 150.000). The effect of the sample size on the other two models was weaker. Imbalance corrections led to poorly calibrated models, for all the approaches considered, reducing the clinical utility of the models.ConclusionsOur results suggest that the choice of approach in the analysis of ECG should be based on the amount of data available, preferring more standard models for small datasets. Moreover, imbalance correction methods should be avoided when developing clinical prediction models, where calibration is crucial.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 3

13570_2023_282_Article_IEq7.gif

Fig. 3