Statistical models versus machine learning for competing risks: development and validation of prognostic models

BMC Medical Research Methodology

Research

Hein Putter¹  Saskia Litière²  Georgios Kantidakis³  Marta Fiocco⁴ 

[1] Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA, Leiden, The Netherlands;Department of Statistics, European Organisation for Research and Treatment of Cancer (EORTC) Headquarters, Ave E. Mounier 83/11, 1200, Brussels, Belgium;Mathematical Institute (MI) Leiden University, Niels Bohrweg 1, 2333 CA, Leiden, The Netherlands;Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA, Leiden, The Netherlands;Department of Statistics, European Organisation for Research and Treatment of Cancer (EORTC) Headquarters, Ave E. Mounier 83/11, 1200, Brussels, Belgium;Mathematical Institute (MI) Leiden University, Niels Bohrweg 1, 2333 CA, Leiden, The Netherlands;Department of Biomedical Data Sciences, Section Medical Statistics, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA, Leiden, The Netherlands;Trial and Data Center, Princess Máxima Center for pediatric oncology (PMC), Heidelberglaan 25, 3584 CS, Utrecht, the Netherlands;

关键词: Artificial neural networks;    Competing risks;    Predictive performance;    Random survival forests;    Regression models;    Supervised machine learning;    Survival analysis;   

DOI  :  10.1186/s12874-023-01866-z

 received in 2022-09-15, accepted in 2023-02-13,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundIn health research, several chronic diseases are susceptible to competing risks (CRs). Initially, statistical models (SM) were developed to estimate the cumulative incidence of an event in the presence of CRs. As recently there is a growing interest in applying machine learning (ML) for clinical prediction, these techniques have also been extended to model CRs but literature is limited. Here, our aim is to investigate the potential role of ML versus SM for CRs within non-complex data (small/medium sample size, low dimensional setting).MethodsA dataset with 3826 retrospectively collected patients with extremity soft-tissue sarcoma (eSTS) and nine predictors is used to evaluate model-predictive performance in terms of discrimination and calibration. Two SM (cause-specific Cox, Fine-Gray) and three ML techniques are compared for CRs in a simple clinical setting. ML models include an original partial logistic artificial neural network for CRs (PLANNCR original), a PLANNCR with novel specifications in terms of architecture (PLANNCR extended), and a random survival forest for CRs (RSFCR). The clinical endpoint is the time in years between surgery and disease progression (event of interest) or death (competing event). Time points of interest are 2, 5, and 10 years.ResultsBased on the original eSTS data, 100 bootstrapped training datasets are drawn. Performance of the final models is assessed on validation data (left out samples) by employing as measures the Brier score and the Area Under the Curve (AUC) with CRs. Miscalibration (absolute accuracy error) is also estimated. Results show that the ML models are able to reach a comparable performance versus the SM at 2, 5, and 10 years regarding both Brier score and AUC (95% confidence intervals overlapped). However, the SM are frequently better calibrated.ConclusionsOverall, ML techniques are less practical as they require substantial implementation time (data preprocessing, hyperparameter tuning, computational intensity), whereas regression methods can perform well without the additional workload of model training. As such, for non-complex real life survival data, these techniques should only be applied complementary to SM as exploratory tools of model’s performance. More attention to model calibration is urgently needed.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

40517_2023_248_Article_IEq15.gif

40517_2023_248_Article_IEq13.gif

40517_2023_248_Article_IEq11.gif

40517_2023_248_Article_IEq10.gif

40517_2023_248_Article_IEq9.gif

40517_2023_248_Article_IEq8.gif

40517_2023_248_Article_IEq7.gif

40517_2023_248_Article_IEq6.gif

40517_2023_248_Article_IEq5.gif

40517_2023_248_Article_IEq4.gif

40517_2023_248_Article_IEq3.gif

40517_2023_248_Article_IEq2.gif

40517_2023_248_Article_IEq1.gif

Fig. 9

Fig. 1

Fig. 12

12936_2023_4464_Article_IEq13.gif

Fig. 2

Fig. 5

Fig. 11

Fig. 1

Fig. 10

40854_2023_460_Article_IEq15.gif

12936_2023_4464_Article_IEq7.gif

Fig. 9

Fig. 4

Fig. 8

Fig. 1

Fig. 7

Fig. 3

Fig. 13

Fig. 7

Fig. 6

Fig. 5

Fig. 1

Fig. 1

Fig. 1

Fig.1

Fig. 2

13690_2023_1046_Article_IEq7.gif

Fig. 3

Fig. 5

Fig. 1

Fig. 4

Fig. 2

Fig. 6

Fig. 7

12936_2023_4470_Article_IEq1.gif

Fig. 6

Fig. 1

Fig. 2

Fig. 4

Fig. 3

Fig. 2

Fig. 2

Fig. 5

Fig. 5

Fig. 2

13690_2023_1029_Article_IEq21.gif

13690_2023_1029_Article_IEq5.gif

Fig. 7