Automated Multi-View Multi-Modal Assessment of COVID-19 Patients Using Reciprocal Attention and Biomedical Transform

Frontiers in Public Health

article

Yanhan Li¹  Hongyun Zhao²  Tian Gan⁴  Yang Liu⁵  Lian Zou¹  Ting Xu⁴  Xuan Chen⁶  Cien Fan¹  Meng Wu⁴ 

[1] Electronic Information School, Wuhan University;Department of Gastroenterology, The Second Affiliated Hospital of Chongqing Medical University;Chongqing Key Laboratory of Ultrasound Molecular Imaging, The Second Affiliated Hospital of Chongqing Medical University;Department of Ultrasound, Zhongnan Hospital of Wuhan University;School of Economics and Management, Wuhan University;Beijing Genomics Institute ,(BGI) Research

关键词: COVID-19;    deep learning;    multi-view;    multi-modal;    computer aided diagnosis;   

DOI  :  10.3389/fpubh.2022.886958

学科分类：社会科学、人文和艺术（综合）

来源: Frontiers

【 摘 要 】

Automated severity assessment of coronavirus disease 2019 (COVID-19) patients can help rationally allocate medical resources and improve patients' survival rates. The existing methods conduct severity assessment tasks mainly on a unitary modal and single view, which is appropriate to exclude potential interactive information. To tackle the problem, in this paper, we propose a multi-view multi-modal model to automatically assess the severity of COVID-19 patients based on deep learning. The proposed model receives multi-view ultrasound images and biomedical indices of patients and generates comprehensive features for assessment tasks. Also, we propose a reciprocal attention module to acquire the underlying interactions between multi-view ultrasound data. Moreover, we propose biomedical transform module to integrate biomedical data with ultrasound data to produce multi-modal features. The proposed model is trained and tested on compound datasets, and it yields 92.75% for accuracy and 80.95% for recall, which is the best performance compared to other state-of-the-art methods. Further ablation experiments and discussions conformably indicate the feasibility and advancement of the proposed model.

【 授权许可】

CC BY