SKELTER: unsupervised skeleton action denoising and recognition using transformers

Frontiers in Computer Science

Computer Science

Alessio Del Bue¹  Cigdem Beyan²  Giancarlo Paoletti³ 

[1] Pattern Analysis and Computer Vision Research Line, Fondazione Istituto Italiano di Tecnologia, Genoa, Italy;Pattern Analysis and Computer Vision Research Line, Fondazione Istituto Italiano di Tecnologia, Genoa, Italy;Department of Information Engineering and Computer Science, University of Trento, Trento, Italy;Pattern Analysis and Computer Vision Research Line, Fondazione Istituto Italiano di Tecnologia, Genoa, Italy;Electrical, Electronics, and Telecommunication Engineering and Naval Architecture Department, University of Genoa, Genoa, Italy;

关键词: action recognition;    autoencoder;    transformers;    full-body movement;    skeletal data;    unsupervised feature learning;   

DOI  :  10.3389/fcomp.2023.1203901

 received in 2023-04-11, accepted in 2023-07-31,  发布年份 2023

来源: Frontiers

【 摘 要 】

Unsupervised Human Action Recognition (U-HAR) methods currently leverage large-scale datasets of human poses to solve this challenging problem. As most of the approaches are dedicated to reaching the best recognition accuracies, no attention has been put into analyzing the resilience of such methods given perturbed data, a likely occurrence in real in-the-wild testing scenarios. Our first contribution is to systematically validate the decrease in performance of current U-HAR state-of-the-art using perturbed or altered data (e.g., obtained by removing some skeletal joints, rotating the entire pose, and injecting geometrical aberrations). Then, we propose a novel framework based on a transformer encoder–decoder with remarkable de-noising capabilities to counter such perturbations effectively. Moreover, we also present additional losses to have robust representations against rotation variances and provide temporal motion consistency. Our model, SKELTER, shows limited drops in performance when skeleton noise is present compared with previous approaches, favoring its use in challenging in-the-wild settings.

【 授权许可】

Unknown   
Copyright © 2023 Paoletti, Beyan and Del Bue.