Perceptually-guided deep neural networks for ego-action prediction: Object grasping

PATTERN RECOGNITION

卷:88

Article

Gonzalez-Diaz, Ivan¹  Benois-Pineau, Jenny²  Domenger, Jean-Philippe²  Cattaert, Daniel³  de Rugy, Aymar³ 

[1] Univ Carlos III Madrid, Dept Signal Theory & Commun, Madrid 28911, Spain

[2] Univ Bordeaux, CNRS, UMR 5800, LaBRI, F-33405 Talence, France

[3] Univ Bordeaux, CNRS, UMR 5287, INCIA, F-33076 Bordeaux, France

关键词: Human perception;    Grasping action prediction;    Weakly supervised active object detection;   

DOI  :  10.1016/j.patcog.2018.11.013

来源: Elsevier

【 摘 要 】

We tackle the problem of predicting a grasping action in ego-centric video for the assistance to upper limb amputees. Our work is based on paradigms of neuroscience that state that human gaze expresses intention and anticipates actions. In our scenario, human gaze fixations are recorded by a glass-worn eye-tracker and then used to predict the grasping actions. We have studied two aspects of the problem: which object from a given taxonomy will be grasped, and when is the moment to trigger the grasping action. To recognize objects, we using gaze to guide Convolutional Neural Networks (CNN) to focus on an object-to-grasp area. However, the acquired sequence of fixations is noisy due to saccades toward distractors and visual fatigue, and gaze is not always reliably directed toward the object-of-interest. To deal with this challenge, we use video-level annotations indicating the object to be grasped and a weak loss in Deep CNNs. To detect a moment when a person will take an object we take advantage of the predictive power of Long-Short Term Memory networks to analyze gaze and visual dynamics. Results show that our method achieves better performance than other approaches on a real-life dataset. (C) 2018 Elsevier Ltd. All rights reserved.

【 授权许可】

Free