An adaptive multi-modal hybrid model for classifying thyroid nodules by combining ultrasound and infrared thermal images

BMC Bioinformatics

Research

Meng Wu¹  Ziling Wu¹  Zhaohui Cai²  Na Zhang²  Wensi Duan²  Juan Liu²  Yu Jin² 

[1] Department of Ultrasound, Zhongnan Hospital, Wuhan University, 430072, Wuhan, China;Institute of Artificial Intelligence, School of Computer Science, Wuhan University, 430072, Wuhan, China;

关键词: Multi-modal learning;    Adaptive weight;    Thyroid nodule;    Infrared thermal image;    Ultrasound image;   

DOI  :  10.1186/s12859-023-05446-2

 received in 2022-11-18, accepted in 2023-08-15,  发布年份 2023

来源: Springer

【 摘 要 】

BackgroundTwo types of non-invasive, radiation-free, and inexpensive imaging technologies that are widely employed in medical applications are ultrasound (US) and infrared thermography (IRT). The ultrasound image obtained by ultrasound imaging primarily expresses the size, shape, contour boundary, echo, and other morphological information of the lesion, while the infrared thermal image obtained by infrared thermography imaging primarily describes its thermodynamic function information. Although distinguishing between benign and malignant thyroid nodules requires both morphological and functional information, present deep learning models are only based on US images, making it possible that some malignant nodules with insignificant morphological changes but significant functional changes will go undetected.ResultsGiven the US and IRT images present thyroid nodules through distinct modalities, we proposed an Adaptive multi-modal Hybrid (AmmH) classification model that can leverage the amalgamation of these two image types to achieve superior classification performance. The AmmH approach involves the construction of a hybrid single-modal encoder module for each modal data, which facilitates the extraction of both local and global features by integrating a CNN module and a Transformer module. The extracted features from the two modalities are then weighted adaptively using an adaptive modality-weight generation network and fused using an adaptive cross-modal encoder module. The fused features are subsequently utilized for the classification of thyroid nodules through the use of MLP. On the collected dataset, our AmmH model respectively achieved 97.17% and 97.38% of F1 and F2 scores, which significantly outperformed the single-modal models. The results of four ablation experiments further show the superiority of our proposed method.ConclusionsThe proposed multi-modal model extracts features from various modal images, thereby enhancing the comprehensiveness of thyroid nodules descriptions. The adaptive modality-weight generation network enables adaptive attention to different modalities, facilitating the fusion of features using adaptive weights through the adaptive cross-modal encoder. Consequently, the model has demonstrated promising classification performance, indicating its potential as a non-invasive, radiation-free, and cost-effective screening tool for distinguishing between benign and malignant thyroid nodules. The source code is available at https://github.com/wuliZN2020/AmmH.

【 授权许可】

CC BY   
© BioMed Central Ltd., part of Springer Nature 2023

【 图 表 】

12888_2023_5012_Article_IEq5.gif

Fig. 7

12888_2023_5012_Article_IEq4.gif

12888_2023_5012_Article_IEq3.gif

Fig. 1

12888_2023_5012_Article_IEq2.gif

12888_2023_5113_Article_IEq7.gif

12888_2023_5113_Article_IEq6.gif

12888_2023_5113_Article_IEq5.gif

40517_2023_261_Article_IEq28.gif

Fig. 1

Fig. 2

40517_2023_261_Article_IEq24.gif

40517_2023_261_Article_IEq45.gif

Fig. 3

40517_2023_261_Article_IEq31.gif

40517_2023_261_Article_IEq30.gif

13690_2023_1154_Article_IEq10.gif

13690_2023_1154_Article_IEq17.gif

13690_2023_1154_Article_IEq8.gif

13690_2023_1154_Article_IEq7.gif

13690_2023_1154_Article_IEq6.gif

13690_2023_1154_Article_IEq5.gif

Fig. 2

13068_2023_2361_Article_IEq16.gif

Fig. 5

42490_2023_74_Article_IEq59.gif

Fig. 3

Fig. 1

42490_2023_74_Article_IEq38.gif

42490_2023_74_Article_IEq25.gif

42490_2023_74_Article_IEq24.gif

42490_2023_74_Article_IEq23.gif

Fig. 6

42490_2023_74_Article_IEq3.gif

Fig. 3

Fig. 7

40798_2023_622_Article_IEq21.gif

Fig. 9

40798_2023_622_Article_IEq4.gif