A Spatiotemporal Agent for Robust Multimodal Registration

IEEE Access

Xin Wang¹  Qi Song¹  Kunlin Cao¹  Youbing Yin¹  Ziwei Luo²  Jing Hu²  Xi Wu² 

[1] CuraCloud Corporation, Seattle, WA, USA;Department of Computer Science, Chengdu University of Information Technology, Chengdu, China;

关键词: Medical image;    reinforcement learning;    multimodal registration;    actor-critic;    convolutional GRU;   

DOI  :  10.1109/ACCESS.2020.2989150

来源: DOAJ

【 摘 要 】

Multimodal image registration is a crucial step for a variety of medical applications to provide complementary information from the combination of various data sources. Conventional image registration methods aim at finding a suited similarity metric as well as a descriptive image feature, which is quite challenging due to the high diversity of tissue appearance across modalities. In this paper, we present a novel approach to register images via an asynchronously trained reinforcement learning agent automatically. Within this approach, convolutional gated recurrent units (ConvGRU) is incorporated after stacked convolutional layers to extract both spatial and temporal features of the neighboring frames and implicitly learn the similarity metric. Moreover, we propose a customized reward function driven by fixed points error (FPE) to guide the agent to the correct registration direction. A Monte Carlo rollout strategy is also leveraged to perform a look-ahead inference to the elimination of jitter in the test stage. Evaluation is performed on paired CT and MR images from patients diagnosed as nasopharyngeal carcinoma. The results demonstrate that our method achieves state-of-the-art performance in medical image registration.

【 授权许可】

Unknown