A non-contact interactive system for multimodal surgical robots based on LeapMotion and visual tags

Frontiers in Neuroscience

Neuroscience

Jie Wang¹  Xinkang Zhang¹  Xinrong Chen¹  Xiaokun Dai¹  Shu Shen² 

[1] Academy for Engineering and Technology, Fudan University, Shanghai, China;Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, China;Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks, Nanjing University of Posts and Telecommunications, Nanjing, China;

关键词: surgical robot;    human-computer interaction;    LeapMotion;    Aruco;    non-contact;   

DOI  :  10.3389/fnins.2023.1287053

 received in 2023-09-01, accepted in 2023-10-03,  发布年份 2023

来源: Frontiers

【 摘 要 】

In recent years, the integration of robots in minimally invasive surgery has gained significant traction in clinical practice. However, conventional contact-based human-computer interaction poses the risk of bacterial infection, significantly limiting the role of robots in surgery. To address this limitation, we propose an innovative interaction method rooted in gestures and visual tags, allowing surgeons to control and fine-tune surgical robots without physical contact with the environment. By encoding the six gestures collected using LeapMotion, we can effectively control the surgical robot in a non-contact manner. Moreover, utilizing Aruco technology, we have accurately identified the 3D spatial position of the visual label, and developed 12 fine-tuning operations to refine surgical instruments. To evaluate the applicability of our proposed system in surgery, we designed a relevant experimental setup. In the experiment, we achieved enough precision. These results demonstrate that our system meets the clinical standard, providing doctors with a non-contact and flexible means of interacting with robots during surgery.

【 授权许可】

Unknown   
Copyright © 2023 Zhang, Wang, Dai, Shen and Chen.