Cross-modal discriminant adversarial network

PATTERN RECOGNITION

卷:112

Article

Hu, Peng^1,2  Peng, Xi¹  Zhu, Hongyuan²  Lin, Jie²  Zhen, Liangli³  Wang, Wei¹  Peng, Dezhong^1,4,5 

[1] Sichuan Univ, Coll Comp Sci, Chengdu 610065, Peoples R China

[2] Agcy Sci Technol & Res, Inst Infocomm Res, Singapore, Singapore

[3] Agcy Sci Technol & Res, Inst High Performance Comp, Singapore, Singapore

[4] Shenzhen Peng Cheng Lab, Shenzhen 518052, Peoples R China

[5] Southwest Univ, Coll Comp & Informat Sci, Chongqing 400715, Peoples R China

关键词: Adversarial learning;    Cross-modal representation learning;    Cross-modal retrieval;    Discriminant adversarial network;    Cross-modal discriminant mechanism;    Latent common space;   

DOI  :  10.1016/j.patcog.2020.107734

来源: Elsevier

【 摘 要 】

Cross-modal retrieval aims at retrieving relevant points across different modalities, such as retrieving images via texts. One key challenge of cross-modal retrieval is narrowing the heterogeneous gap across diverse modalities. To overcome this challenge, we propose a novel method termed as Cross-modal discriminant Adversarial Network (CAN). Taking bi-modal data as a showcase, CAN consists of two parallel modality-specific generators, two modality-specific discriminators, and a Cross-modal Discriminant Mechanism (CDM). To be specific, the generators project diverse modalities into a latent cross-modal discriminant space. Meanwhile, the discriminators compete against the generators to alleviate the heterogeneous discrepancy in this space, i.e., the generators try to generate unified features to confuse the discriminators, and the discriminators aim to classify the generated results. To further remove the redundancy and preserve the discrimination, we propose CDM to project the generated results into a single common space, accompanying with a novel eigenvalue-based loss. Thanks to the eigenvalue-based loss, CDM could push as much discriminative power as possible into all latent directions. To demonstrate the effectiveness of our CAN, comprehensive experiments are conducted on four multimedia datasets comparing with 15 state-of-the-art approaches. (C) 2020 Elsevier Ltd. All rights reserved.Y

【 授权许可】

Free