Lagrangian motion magnification with double sparse optical flow decomposition

Frontiers in Applied Mathematics and Statistics

Applied Mathematics and Statistics

Cosmas Heiss¹  Gabriele Steidl¹  Philipp Flotho²  Daniel J. Strauss² 

[1] Institute of Mathematics, TU Berlin, Berlin, Germany;Systems Neuroscience and Neurotechnology Unit, Neurocenter, Faculty of Medicine, School of Engineering, htw saar and Center for Digital Neurotechnologies Saar (CDNS), Saarland University, Saarbrücken, Germany;

关键词: motion magnification;    optical flow;    microexpression;    Lagrangian motion magnification;    sparse PCA;   

DOI  :  10.3389/fams.2023.1164491

 received in 2023-02-12, accepted in 2023-08-18,  发布年份 2023

来源: Frontiers

【 摘 要 】

Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore, motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely, via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this study, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is 3-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for the OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.

【 授权许可】

Unknown   
Copyright © 2023 Flotho, Heiss, Steidl and Strauss.