Evaluation and improvement of HMD-based and RGB-based hand tracking solutions in VR

Frontiers in Virtual Reality

Virtual Reality

Daniel Scherzer¹  Iana Podkosova²  Hannes Kaufmann²  Dennis Reimer³ 

[1] Faculty for Electrical Engineering and Computer Science, Ravensburg-Weingarten University, Weingarten, Germany;Faculty of Informatics, Research Unit Virtual and Augmented Reality, TU Wien, Vienna, Austria;Faculty of Informatics, Research Unit Virtual and Augmented Reality, TU Wien, Vienna, Austria;Faculty for Electrical Engineering and Computer Science, Ravensburg-Weingarten University, Weingarten, Germany;

关键词: virtual reality;    hand tracking;    colocation;    RGB tracking;    mediapipe;   

DOI  :  10.3389/frvir.2023.1169313

 received in 2023-02-19, accepted in 2023-07-10,  发布年份 2023

来源: Frontiers

【 摘 要 】

Hand tracking has become a state-of-the-art technology in the modern generation of consumer VR devices. However, off-the-shelf solutions do not support hand detection for more than two hands at the same time at distances beyond arm’s length. The possibility to track multiple hands at larger distances would be beneficial for colocated multi-user VR scenarios, allowing user-worn devices to track the hands of other users and therefore reducing motion artifacts caused by hand tracking loss. With the global focus of enabling natural hand interactions in colocated multi-user VR, we propose an RGB image input-based hand tracking method, built upon the MediaPipe framework, that can track multiple hands at once at distances of up to 3 m. We compared our method’s accuracy to that of Oculus Quest and Leap Motion, at different distances from the tracking device and in static and dynamic settings. The results of our evaluation show that our method provides only slightly less accurate results than Oculus Quest or Leap motion in the near range (with median errors below 1.75 cm at distances below 75 cm); at larger distances, its accuracy remains stable (with a median error of 4.7 cm at the distance of 2.75 m) while Leap Motion and Oculus Quest either loose tracking or produce very inaccurate results. Taking into account the broad choice of suitable hardware (any RGB camera) and the ease of setup, our method can be directly applied to colocated multi-user VR scenarios.

【 授权许可】

Unknown   
Copyright © 2023 Reimer, Podkosova, Scherzer and Kaufmann.