Assessment of emerging pretraining strategies in interpretable multimodal deep learning for cancer prognostication

BioData Mining

Research

Ji-Qing Chen¹  Brock C. Christensen²  Lucas A. Salas³  Louis J. Vaickus⁴  Ze Zhang⁵  Joshua J. Levy⁶  Anish Suvarna⁷  Zarif L. Azher⁷ 

[1]Cancer Biology Graduate Program, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[2]Program in Quantitative Biomedical Sciences, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[3]Department of Epidemiology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[4]Department of Epidemiology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[5]Department of Molecular and Systems Biology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[6]Department of Community and Family Medicine, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[7]Department of Epidemiology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[8]Department of Molecular and Systems Biology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[9]Integrative Neuroscience at Dartmouth (IND) Graduate Program, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[10]Emerging Diagnostic and Investigative Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Health, Lebanon, NH, USA

[11]Program in Quantitative Biomedical Sciences, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[12]Department of Epidemiology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[13]Program in Quantitative Biomedical Sciences, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[14]Department of Epidemiology, Dartmouth College Geisel School of Medicine, Hanover, NH, USA

[15]Emerging Diagnostic and Investigative Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Health, Lebanon, NH, USA

[16]Department of Dermatology, Dartmouth Health, Lebanon, NH, USA

[17]Thomas Jefferson High School for Science and Technology, Alexandria, VA, USA

关键词: Multimodal;    Survival;    DNA methylation;    Whole slide images;    Gene expression;    Machine learning;    Graph neural networks;    Tumor infiltrating lymphocytes;   

DOI  :  10.1186/s13040-023-00338-w

 received in 2022-11-22, accepted in 2023-07-05,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundDeep learning models can infer cancer patient prognosis from molecular and anatomic pathology information. Recent studies that leveraged information from complementary multimodal data improved prognostication, further illustrating the potential utility of such methods. However, current approaches: 1) do not comprehensively leverage biological and histomorphological relationships and 2) make use of emerging strategies to “pretrain” models (i.e., train models on a slightly orthogonal dataset/modeling objective) which may aid prognostication by reducing the amount of information required for achieving optimal performance. In addition, model interpretation is crucial for facilitating the clinical adoption of deep learning methods by fostering practitioner understanding and trust in the technology.MethodsHere, we develop an interpretable multimodal modeling framework that combines DNA methylation, gene expression, and histopathology (i.e., tissue slides) data, and we compare performance of crossmodal pretraining, contrastive learning, and transfer learning versus the standard procedure.ResultsOur models outperform the existing state-of-the-art method (average 11.54% C-index increase), and baseline clinically driven models (average 11.7% C-index increase). Model interpretations elucidate consideration of biologically meaningful factors in making prognosis predictions.DiscussionOur results demonstrate that the selection of pretraining strategies is crucial for obtaining highly accurate prognostication models, even more so than devising an innovative model architecture, and further emphasize the all-important role of the tumor microenvironment on disease progression.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

12888_2023_5113_Article_IEq7.gif

40517_2023_261_Article_IEq35.gif

40517_2023_261_Article_IEq17.gif

40517_2023_261_Article_IEq5.gif

40517_2023_261_Article_IEq2.gif

Fig. 1

12974_2023_2864_Article_IEq1.gif

13068_2023_2361_Article_IEq16.gif