Image Captioning with multi-level similarity-guided semantic matching

Visual Informatics

Weizhi Nie¹  Jiesi Li²  Ning Xu²  Shenyuan Zhang² 

[1] Corresponding author.;The School of Electrical and Information Engineering, Tianjin University, China;

关键词: Image Captioning;    Cross-modal semantic matching;    Reinforcement learning;   

DOI  :  

来源: DOAJ

【 摘 要 】

Image Captioning is a cross-modal task that needs to automatically generate coherent natural sentences to describe the image contents. Due to the large gap between vision and language modalities, most of the existing methods have the problem of inaccurate semantic matching between images and generated captions. To solve the problem, this paper proposes a novel multi-level similarity-guided semantic matching method for image captioning, which can fuse local and global semantic similarities to learn the latent semantic correlation between images and generated captions. Specifically, we extract the semantic units containing fine-grained semantic information of images and generated captions, respectively. Based on the comparison of the semantic units, we design a local semantic similarity evaluation mechanism. Meanwhile, we employ the CIDEr score to characterize the global semantic similarity. The local and global two-level similarities are finally fused using the reinforcement learning theory, to guide the model optimization to obtain better semantic matching. The quantitative and qualitative experiments on large-scale MSCOCO dataset illustrate the superiority of the proposed method, which can achieve fine-grained semantic matching of images and generated captions.

【 授权许可】

Unknown