Towards Automated Semantic Explainability of Multimedia Feature Graphs

Information

Paul Mc Kevitt¹  Stefan Wagenpfeil²  Matthias Hemmje² 

[1] Academy for International Science & Research (AISR), Londonderry BT48 7ER, UK;Faculty of Mathematics and Computer Science, University of Hagen, D-58097 Hagen, Germany;

关键词: indexing;    retrieval;    explainability;    semantic;    multimedia;    feature graph;   

DOI  :  10.3390/info12120502

来源: DOAJ

【 摘 要 】

Multimedia feature graphs are employed to represent features of images, video, audio, or text. Various techniques exist to extract such features from multimedia objects. In this paper, we describe the extension of such a feature graph to represent the meaning of such multimedia features and introduce a formal context-free PS-grammar (Phrase Structure grammar) to automatically generate human-understandable natural language expressions based on such features. To achieve this, we define a semantic extension to syntactic multimedia feature graphs and introduce a set of production rules for phrases of natural language English expressions. This explainability, which is founded on a semantic model provides the opportunity to represent any multimedia feature in a human-readable and human-understandable form, which largely closes the gap between the technical representation of such features and their semantics. We show how this explainability can be formally defined and demonstrate the corresponding implementation based on our generic multimedia analysis framework. Furthermore, we show how this semantic extension can be employed to increase the effectiveness in precision and recall experiments.

【 授权许可】

Unknown