Joint 3D facial shape reconstruction and texture completion from a single image

Computational Visual Media

Zhelun Wu¹  Yu Qiao¹  Xiaojiang Peng¹  Xiaoxing Zeng² 

[1] Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China;Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China;University of Chinese Academy of Sciences, Beijing, China;

关键词: 3D face reconstruction;    U;    pose invariant face recognition;    deep learning;   

DOI  :  10.1007/s41095-021-0238-4

来源: Springer

【 摘 要 】

Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks. However, current reconstruction methods often perform improperly in self-occluded regions and can lead to inaccurate correspondences between a 2D input image and a 3D face template, hindering use in real applications. To address these problems, we propose a deep shape reconstruction and texture completion network, SRTC-Net, which jointly reconstructs 3D facial geometry and completes texture with correspondences from a single input face image. In SRTC-Net, we leverage the geometric cues from completed 3D texture to reconstruct detailed structures of 3D shapes. The SRTC-Net pipeline has three stages. The first introduces a correspondence network to identify pixel-wise correspondence between the input 2D image and a 3D template model, and transfers the input 2D image to a U-V texture map. Then we complete the invisible and occluded areas in the U-V texture map using an inpainting network. To get the 3D facial geometries, we predict coarse shape (U-V position maps) from the segmented face from the correspondence network using a shape network, and then refine the 3D coarse shape by regressing the U-V displacement map from the completed U-V texture map in a pixel-to-pixel way. We examine our methods on 3D reconstruction tasks as well as face frontalization and pose invariant face recognition tasks, using both in-the-lab datasets (MICC, MultiPIE) and in-the-wild datasets (CFP). The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture; they outperform or are comparable to the state-of-the-art.

【 授权许可】

CC BY