Iacer Calixto - Transfer Talk - 14th August 2015

Video Category: 
Transfer Talk
Iacer Calixto

Title: Towards Incorporating Visual  Information in Machine Translation

Supervisor: Prof. Qun Liu & Prof. Nick Campbell

Abstract: In this transfer report we propose new models for incorporating visual information into the standard Machine Translation pipeline in both Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) frameworks. We discuss which automatic metrics and also which corpora to use for training and evaluating our models. We propose different experiments for doing that: using information retrieval techniques, external linked data resources and also multimodal embeddings. We also propose a new multilingual, multimodal embedding model which we evaluate as part of our work. In SMT, we wish to incorporate information derived from images for reranking n-best lists retrieved by a baseline SMT system, as dynamic features in a log-linear model and a combination of both. In NMT, we conjecture a way to augment the encoder–decoder framework and make it a multimodal encoder–decoder framework.