Backpropagation-Based Decoding for Multimodal Machine Translation

Frontiers in Artificial Intelligence

Artificial Intelligence

Franck Dernoncourt¹  Leticia Pinto-Alva²  Ziyan Yang³  Vicente Ordonez⁴ 

[1] Adobe Research, San José, CA, United States;Department of Computer Science, Universidad Católica San Pablo, Arequipa, Perú;Department of Computer Science, University of Virginia, Charlottesville, VA, United States;Department of Computer Science, University of Virginia, Charlottesville, VA, United States;Department of Computer Science, Rice University, Houston, TX, United States;

关键词: vision and language;    multimodal machine translation;    backpropagation-based decoding;    feedback-propagation;    multimodal machine learning;    computer vision;    natural language processing;   

DOI  :  10.3389/frai.2021.736722

 received in 2021-07-05, accepted in 2021-11-03,  发布年份 2022

来源: Frontiers

【 摘 要 】

People are able to describe images using thousands of languages, but languages share only one visual world. The aim of this work is to use the learned intermediate visual representations from a deep convolutional neural network to transfer information across languages for which paired data is not available in any form. Our work proposes using backpropagation-based decoding coupled with transformer-based multilingual-multimodal language models in order to obtain translations between any languages used during training. We particularly show the capabilities of this approach in the translation of German-Japanese and Japanese-German sentence pairs, given a training data of images freely associated with text in English, German, and Japanese but for which no single image contains annotations in both Japanese and German. Moreover, we demonstrate that our approach is also generally useful in the multilingual image captioning task when sentences in a second language are available at test time. The results of our method also compare favorably in the Multi30k dataset against recently proposed methods that are also aiming to leverage images as an intermediate source of translations.

【 授权许可】

Unknown   
Copyright © 2022 Yang, Pinto-Alva, Dernoncourt and Ordonez.