Integrating information theory and adversarial learning for cross-modal retrieval

PATTERN RECOGNITION

卷:117

Article

Chen, Wei¹  Liu, Yu²  Bakker, Erwin M.¹  Lew, Michael S.¹ 

[1] Leiden Univ, LIACS, NL-2333 CA Leiden, Netherlands

[2] Katholieke Univ Leuven, ESAT PSI, B-3001 Heverlee, Belgium

关键词: Cross-modal retrieval;    Shannon information theory;    Adversarial learning;    Modality uncertainty;    Data imbalance;   

DOI  :  10.1016/j.patcog.2021.107983

来源: Elsevier

【 摘 要 】

Accurately matching visual and textual data in cross-modal retrieval has been widely studied in the multimedia community. To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon information theory and adversarial learning. In terms of the heterogeneity gap, we integrate modality classification and information entropy maximization adversarially. For this purpose, a modality classifier (as a discriminator) is built to distinguish the text and image modalities according to their different statistical properties. This discriminator uses its output probabilities to compute Shannon information entropy, which measures the uncertainty of the modality classification it performs. Moreover, feature encoders (as a generator) project uni-modal features into a commonly shared space and attempt to fool the discriminator by maximizing its output information entropy. Thus, maximizing information entropy gradually reduces the distribution discrepancy of cross-modal features, thereby achieving a domain confusion state where the discriminator cannot classify two modalities confidently. To reduce the semantic gap, Kullback-Leibler (KL) divergence and bi-directional triplet loss are used to associate the intra- and inter-modality similarity between features in the shared space. Furthermore, a regularization term based on KL-divergence with temperature scaling is used to calibrate the biased label classifier caused by the data imbalance issue. Extensive experiments with four deep models on four benchmarks are conducted to demonstrate the effectiveness of the proposed approach.& nbsp; (c) 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ )

【 授权许可】

Free