VANet: a medical image fusion model based on attention mechanism to assist disease diagnosis

BMC Bioinformatics

Research

Tiehu Fan¹  Xiaohan Hu²  Xiongfei Li³  Kai Guo³ 

[1] College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China;Department of Radiology, The First Hospital of Jilin University, Changchun, China;Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, China;College of Computer Science and Technology, Jilin University, Changchun, China;

关键词: Medical image;    Medical image fusion;    Attention mechanism;    Contextual information;    Multi scale feature extraction;   

DOI  :  10.1186/s12859-022-05072-4

 received in 2022-05-06, accepted in 2022-11-22,  发布年份 2022

来源: Springer

【 摘 要 】

BackgroundToday’s biomedical imaging technology has been able to present the morphological structure or functional metabolic information of organisms at different scale levels, such as organ, tissue, cell, molecule and gene. However, different imaging modes have different application scope, advantages and disadvantages. In order to improve the role of medical image in disease diagnosis, the fusion of biomedical image information at different imaging modes and scales has become an important research direction in medical image. Traditional medical image fusion methods are all designed to measure the activity level and fusion rules. They are lack of mining the context features of different modes of image, which leads to the obstruction of improving the quality of fused images.MethodIn this paper, an attention-multiscale network medical image fusion model based on contextual features is proposed. The model selects five backbone modules in the VGG-16 network to build encoders to obtain the contextual features of medical images. It builds the attention mechanism branch to complete the fusion of global contextual features and designs the residual multiscale detail processing branch to complete the fusion of local contextual features. Finally, it completes the cascade reconstruction of features by the decoder to obtain the fused image.ResultsTen sets of images related to five diseases are selected from the AANLIB database to validate the VANet model. Structural images are derived from MR images with high resolution and functional images are derived from SPECT and PET images that are good at describing organ blood flow levels and tissue metabolism. Fusion experiments are performed on twelve fusion algorithms including the VANet model. The model selects eight metrics from different aspects to build a fusion quality evaluation system to complete the performance evaluation of the fused images. Friedman’s test and the post-hoc Nemenyi test are introduced to conduct professional statistical tests to demonstrate the superiority of VANet model.ConclusionsThe VANet model completely captures and fuses the texture details and color information of the source images. From the fusion results, the metabolism and structural information of the model are well expressed and there is no interference of color information on the structure and texture; in terms of the objective evaluation system, the metric value of the VANet model is generally higher than that of other methods.; in terms of efficiency, the time consumption of the model is acceptable; in terms of scalability, the model is not affected by the input order of source images and can be extended to tri-modal fusion.

【 授权许可】

CC BY   
© The Author(s) 2022

【 图 表 】

Fig. 1

Fig. 2

Fig. 6

Fig. 8

Fig. 1

Fig. 2

Fig. 6

Fig. 2

Fig. 1

13731_2022_257_Article_IEq4.gif

Fig. 3

13731_2022_257_Article_IEq2.gif

Fig. 3

Fig. 1

Fig. 6

Fig. 5

40708_2022_178_Article_IEq64.gif

Fig. 4

Fig. 2

Fig. 1

Fig. 2

Fig. 2

Fig. 1

Fig. 9

Fig. 3

Fig. 2

Fig. 4

Fig. 2

Fig. 2

Fig. 5

Fig. 1 (abstract P46).

Fig. 1

Fig. 1