Accounting for uncertainty in training data to improve machine learning performance in predicting new disease activity in early multiple sclerosis

Frontiers in Neurology

Neurology

Luanne M. Metz¹  Shannon Kolind²  David K.B. Li²  Anthony Traboulsee³  Robert Carruthers³  Roger C. Tam⁴  Maryam Tayyab⁵ 

[1] Cumming School of Medicine, University of Calgary, Calgary, AB, Canada;Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada;Division of Neurology, Department of Medicine, University of British Columbia, Vancouver, BC, Canada;Division of Neurology, Department of Medicine, University of British Columbia, Vancouver, BC, Canada;School of Biomedical Engineering, University of British Columbia, Vancouver, BC, Canada;Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada;Division of Neurology, Department of Medicine, University of British Columbia, Vancouver, BC, Canada;School of Biomedical Engineering, University of British Columbia, Vancouver, BC, Canada;Division of Neurology, Department of Medicine, University of British Columbia, Vancouver, BC, Canada;

关键词: multiple sclerosis;    machine learning;    MRI;    label uncertainty;    clinical prediction;    missing labels;    noisy labels;   

DOI  :  10.3389/fneur.2023.1165267

 received in 2023-02-13, accepted in 2023-05-09,  发布年份 2023

来源: Frontiers

【 摘 要 】

IntroductionMachine learning (ML) has great potential for using health data to predict clinical outcomes in individual patients. Missing data are a common challenge in training ML algorithms, such as when subjects withdraw from a clinical study, leaving some samples with missing outcome labels. In this study, we have compared three ML models to determine whether accounting for label uncertainty can improve a model’s predictions.MethodsWe used a dataset from a completed phase-III clinical trial that evaluated the efficacy of minocycline for delaying the conversion from clinically isolated syndrome to multiple sclerosis (MS), using the McDonald 2005 diagnostic criteria. There were a total of 142 participants, and at the 2-year follow-up 81 had converted to MS, 29 remained stable, and 32 had uncertain outcomes. In a stratified 7-fold cross-validation, we trained three random forest (RF) ML models using MRI volumetric features and clinical variables to predict the conversion outcome, which represented new disease activity within 2 years of a first clinical demyelinating event. One RF was trained using subjects with the uncertain labels excluded (RFexclude), another RF was trained using the entire dataset but with assumed labels for the uncertain group (RFnaive), and a third, a probabilistic RF (PRF, a type of RF that can model label uncertainty) was trained on the entire dataset, with probabilistic labels assigned to the uncertain group.ResultsProbabilistic random forest outperformed both the RF models with the highest AUC (0.76, compared to 0.69 for RFexclude and 0.71 for RFnaive) and F1-score (86.6% compared to 82.6% for RFexclude and 76.8% for RFnaive).ConclusionMachine learning algorithms capable of modeling label uncertainty can improve predictive performance in datasets in which a substantial number of subjects have unknown outcomes.

【 授权许可】

Unknown   
Copyright © 2023 Tayyab, Metz, Li, Kolind, Carruthers, Traboulsee and Tam.