Unsupervised image translation with distributional semantics awareness

Computational Visual Media

Research Article

Yin Yang¹  He Wang²  Zhexi Peng³  Tianjia Shao³  Yanlin Weng³ 

[1] School of Computing, Clemson University, Clemson, USA;School of Computing, University of Leeds, Leeds, UK;State Key Lab of CAD&CG, Zhejiang University, 310058, Hangzhou, China;

关键词: generative adversarial networks (GANs);    manifold alignment;    unsupervised learning;    image-to-image translation;    distributional semantics;   

DOI  :  10.1007/s41095-022-0295-3

 received in 2022-03-01, accepted in 2022-05-15,  发布年份 2022

来源: Springer

【 摘 要 】

Unsupervised image translation (UIT) studies the mapping between two image domains. Since such mappings are under-constrained, existing research has pursued various desirable properties such as distributional matching or two-way consistency. In this paper, we re-examine UIT from a new perspective: distributional semantics consistency, based on the observation that data variations contain semantics, e.g., shoes varying in colors. Further, the semantics can be multi-dimensional, e.g., shoes also varying in style, functionality, etc. Given two image domains, matching these semantic dimensions during UIT will produce mappings with explicable correspondences, which has not been investigated previously. We propose distributional semantics mapping (DSM), the first UIT method which explicitly matches semantics between two domains. We show that distributional semantics has been rarely considered within and beyond UIT, even though it is a common problem in deep learning. We evaluate DSM on several benchmark datasets, demonstrating its general ability to capture distributional semantics. Extensive comparisons show that DSM not only produces explicable mappings, but also improves image quality in general.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

41116_2023_36_Article_IEq201.gif