A novel framework for retrieval and interactive visualization of multimodal data

ELCVIA Electronic Letters on Computer Vision and Image Analysis

Ilias Kalamaras²  Dimitrios Tzovaras¹  Athanasios Mademlis¹  Sotiris Malassiotis¹ 

[1] Information Technologies Institute, Centre for Research and Technology Hellas;Imperial College London,Information Technologies Institute, Centre for Research and Technology Hellas

关键词: Multimodal Interaction and Multimedia;    Multimedia analysis;    Indexing;    retrieval;    Interaction and Visualization;   

DOI  :  

学科分类：计算机科学（综合）

来源: ELCVIA

【 摘 要 】

With the abundance of multimedia in web databases and the increasing user need for content of many modalities, such as images, sounds, etc., new methods for retrieval and visualization of multimodal media are required. In this paper, novel techniques for retrieval and visualization of multimodal data, i.e. documents consisting of many modalities, are proposed. A novel cross-modal retrieval framework is presented, in which the results of several unimodal retrieval systems are fused into a single multimodal list by the introduction of a cross-modal distance. For the presentation of the retrieved results, a multimodal visualization framework is also proposed, which extends existing unimodal similarity-based visualization methods for multimodal data. The similarity measure between two multimodal objects is defined as the weighted sum of unimodal similarities, with the weights determined via an interactive user feedback scheme. Experimental results show that the cross-modal framework outperforms unimodal and other multimodal approaches while the visualization framework enhances existing visualization methods by efficiently exploiting multimodality and user feedback.

【 授权许可】

Unknown