TTPP: Temporal Transformer with Progressive Prediction for efficient action anticipation

NEUROCOMPUTING

卷:438

Article

Wang, Wen^1,3  Peng, Xiaojiang^3,4,5  Su, Yanzhou²  Qiao, Yu³  Cheng, Jian² 

[1] Univ Elect Sci & Technol China, Signal & Informat Proc, Chengdu 611731, Sichuan, Peoples R China

[2] Univ Elect Sci & Technol China, Sch Informat & Commun Engn, Chengdu 611731, Sichuan, Peoples R China

[3] Chinese Acad Sci, Shenzhen Inst Adv Technol, SIAT SenseTime Joint Lab, ShenZhen Key Lab Comp Vis & Pattern Recognit, Shenzhen, Peoples R China

[4] Shenzhen Inst Artificial Intelligence & Robot Soc, SIAT Branch, Shenzhen, Peoples R China

[5] Shenzhen Technol Univ, Shenzhen, Peoples R China

关键词: Action anticipation;    Transformer;    Progressive prediction;    Encoder-decoder;   

DOI  :  10.1016/j.neucom.2021.01.087

来源: Elsevier

【 摘 要 】

Video action anticipation aims to predict future action categories from observed frames. Current state-ofthe-art approaches mainly resort to recurrent neural networks to encode history information into hidden states, and predict future actions from the hidden representations. It is well known that the recurrent pipeline is inefficient in capturing long-term information which may limit its performance in predication task. To address this problem, this paper proposes a simple yet efficient Temporal Transformer with Progressive Prediction (TTPP) framework, which repurposes a Transformer-style architecture to aggregate observed features, and then leverages a light-weight network to progressively predict future features and actions. Specifically, predicted features along with predicted probabilities are accumulated into the inputs of subsequent prediction. We evaluate our approach on three action datasets, namely TVSeries, THUMOS-14, and TV-Human-Interaction. Additionally we also conduct a comprehensive study for several popular aggregation and prediction strategies. Extensive results show that TTPP not only outperforms the state-of-the-art methods but also more efficient. (c) 2021 Elsevier B.V. All rights reserved.

【 授权许可】

Free