Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study

BMC Medical Imaging

Tsuyoshi Takato²  Kazuto Hoshi²  Takeyoshi Dohi⁴  Ken Masamune⁵  Hongen Liao¹  Huy Hoang Tran³  Hideyuki Suenaga² 

[1] Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China;Department of Oral-Maxillofacial Surgery, Dentistry and Orthodontics, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo ku, Tokyo 113 8656, Japan;Department of Mechano-Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan;Department of Mechanical Engineering, School of Engineering, Tokyo Denki University, Tokyo, Japan;Faculty of Advanced Technology and Surgery, Institute of Advanced Biomedical Engineering and Science, Tokyo Women’s Medical University, Tokyo, Japan

关键词: Three-dimensional image;    Stereo vision;    Markerless registration;    Integral videography;    Augmented reality;   

Others  :  1233373
DOI  :  10.1186/s12880-015-0089-5

 received in 2015-04-23, accepted in 2015-10-09,  发布年份 2015

【 摘 要 】

Background

This study evaluated the use of an augmented reality navigation system that provides a markerless registration system using stereo vision in oral and maxillofacial surgery.

Method

A feasibility study was performed on a subject, wherein a stereo camera was used for tracking and markerless registration. The computed tomography data obtained from the volunteer was used to create an integral videography image and a 3-dimensional rapid prototype model of the jaw. The overlay of the subject’s anatomic site and its 3D-IV image were displayed in real space using a 3D-AR display. Extraction of characteristic points and teeth matching were done using parallax images from two stereo cameras for patient-image registration.

Results

Accurate registration of the volunteer’s anatomy with IV stereoscopic images via image matching was done using the fully automated markerless system, which recognized the incisal edges of the teeth and captured information pertaining to their position with an average target registration error of < 1 mm. These 3D-CT images were then displayed in real space with high accuracy using AR. Even when the viewing position was changed, the 3D images could be observed as if they were floating in real space without using special glasses.

Conclusion

Teeth were successfully used for registration via 3D image (contour) matching. This system, without using references or fiducial markers, displayed 3D-CT images in real space with high accuracy. The system provided real-time markerless registration and 3D image matching via stereo vision, which, combined with AR, could have significant clinical applications.

【 授权许可】

   
2015 Suenaga et al.

【 图 表 】

Fig. 1.

Fig. 2.

Fig. 3.

Fig. 4.

Fig. 5.

【 参考文献 】

• [1]Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, Lira IH et al.. A novel, inexpensive method of image coregistration for applications in image-guided surgery using augmented reality. Neurosurgery. 2007; 60:366-371.
• [2]Sielhorst T, Bichlmeier C, Heining SM, Navab N. Depth perception--a major issue in medical AR: evaluation study by twenty surgeons. Med Image Comput Comput Assist Interv. 2006; 9:364-372.
• [3]Kang X, Azizian M, Wilson E, Wu K, Martin AD, Kane TD. Stereoscopic augmented reality for laparoscopic surgery. Surg Endosc. 2014; 28:2227-2235.
• [4]Volonte F, Pugin F, Bucher P, Sugimoto M, Ratib O, Morel P. Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. J Hepatobiliary Pancreat Sci. 2011; 18:506-509.
• [5]Fritz J, Thainual P, Ungi T, Flammang AJ, Cho NB, Fichtinger G et al.. Augmented reality visualization with image overlay for MRI-guided intervention: accuracy for lumbar spinal procedures with a 1.5-T MRI system. AJR Am J Roentgenol. 2012; 198:W266-W273.
• [6]Mahvash M, Besharati TL. A novel augmented reality system of image projection for image-guided neurosurgery. Acta Neurochir. 2013; 155:943-947.
• [7]Puerto-Souza G, Cadeddu J, Mariottini GL. Toward Long-term and Accurate Augmented-Reality for Monocular Endoscopic Videos. IEEE Trans Biomed Eng. 2014; 61:2609-2620.
• [8]Kersten-Oertel M, Chen SS, Drouin S, Sinclair DS, Collins DL. Augmented reality visualization for guidance in neurovascular surgery. Stud Health Technol Inform. 2012; 173:225-229.
• [9]Matsumoto T, Kubo S, Muratsu H, Tsumura N, Ishida K, Matsushita T et al.. Differing prosthetic alignment and femoral component sizing between 2 computer-assisted CT-free navigation systems in TKA. Orthopedics. 2011; 34:e860-e865.
• [10]Yokoyama Y, Abe N, Fujiwara K, Suzuki M, Nakajima Y, Sugita N et al.. A new navigation system for minimally invasive total knee arthroplasty. Acta Med Okayama. 2013; 67:351-358.
• [11]Suzuki N, Hattori A, Iimura J, Otori N, Onda S, Okamoto T et al.. Development of AR Surgical Navigation Systems for Multiple Surgical Regions. Stud Health Technol Inform. 2014; 196:404-408.
• [12]Badiali G, Ferrari V, Cutolo F, Freschi C, Caramella D, Bianchi A et al.. Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J Craniomaxillofac Surg. 2014; 42:1970-1976.
• [13]Nijmeh AD, Goodger NM, Hawkes D, Edwards PJ, McGurk M. Image-guided navigation in oral and maxillofacial surgery. Br J Oral Maxillofac Surg. 2005; 43:294-302.
• [14]Wang J, Suenaga H, Yang L, Liao H, Kobayashi E, Takato T et al.. Real-time marker-free patient registration and image-based navigation using stereo vision for dental surgery. LNCS. 2013; 8090:9-18.
• [15]Wang J, Suenaga H, Hoshi K, Yang L, Kobayashi E, Sakuma I et al.. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE Trans Biomed Eng. 2014; 61:1295-1304.
• [16]Lin YK, Yau HT, Wang IC, Zheng C, Chung KH. A Novel Dental Implant Guided Surgery Based on Integration of Surgical Template and Augmented Reality. Clin Implant Dent Relat Res. 2013.
• [17]Kang SH, Kim MK, Kim JH, Park HK, Lee SH, Park W. The validity of marker registration for an optimal integration method in mandibular navigation surgery. J Oral Maxillofac Surg. 2013; 71:366-375.
• [18]Sun Y, Luebbers HT, Agbaje JO, Schepers S, Vrielinck L, Lambrichts I et al.. Validation of anatomical landmarks-based registration for image-guided surgery: an in-vitro study. J Craniomaxillofac Surg. 2013; 41:522-526.
• [19]Liao H, Hata N, Nakajima S, Iwahara M, Sakuma I, Dohi T. Surgical navigation by autostereoscopic image overlay of integral videography. IEEE Trans Inf Technol Biomed. 2004; 8:114-121.
• [20]Liao H, Ishihara H, Tran HH, Masamune K, Sakuma I, Dohi T. Precision-guided surgical navigation system using laser guidance and 3D autostereoscopic image overlay. Comput Med Imaging Graph. 2010; 34:46-54.
• [21]Tran HH, Matsumiya K, Masamune K, Sakuma I, Dohi T, Liao H. Interactive 3-D navigation system for image-guided surgery. Int J Virtual Real. 2009; 8:9-16.
• [22]Suenaga H, Hoang Tran H, Liao H, Masamune K, Dohi T, Hoshi K et al.. Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study. Int J Oral Sci. 2013; 5:98-102.
• [23]Wang J, Suenaga H, Liao H, Hoshi K, Yang L, Kobayashi E et al.. Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation. Comput Med Imaging Graph. 2015; 40:147-159.
• [24]Widmann G, Stoffner R, Bale R. Errors and error management in image-guided craniomaxillofacial surgery. Oral Surg Oral Med Oral Pathol Oral Radiol Endod. 2009; 107:701-715.
• [25]Noh H, Nabha W, Cho JH, Hwang HS. Registration accuracy in the integration of laser-scanned dental images into maxillofacial cone-beam computed tomography images. Am J Orthod Dentofacial Orthop. 2011; 140:585-591.
• [26]Fitzpatrick JM, West JB. The Distribution of Target Registration Error in Rigid-Body Point-Based Registration. IEEE Trans Med Imaging. 2001; 20:917-927.
• [27]Casap N, Wexler A, Eliashar R. Computerized navigation for surgery of the lower jaw: comparison of 2 navigation systems. J Oral Maxillofac Surg. 2008; 66:1467-1475.
• [28]Eggers G, Kress B, Muhling J. Fully automated registration of intraoperative computed tomography image data for image-guided craniofacial surgery. J Oral Maxillofac Surg. 2008; 66:1754-1760.
• [29]Zhu M, Chai G, Zhang Y, Ma X, Gan J. Registration strategy using occlusal splint based on augmented reality for mandibular angle oblique split osteotomy. J Craniofac Surg. 2011; 22:1806-1809.
• [30]Kang SH, Kim MK, Kim JH, Park HK, Park W. Marker-free registration for the accurate integration of CT images and the subject's anatomy during navigation surgery of the maxillary sinus. Dentomaxillofac Radiol. 2012; 41:679-685.
• [31]Bouchard C, Magill JC, Nikonovskiy V, Byl M, Murphy BA, Kaban LB et al.. Osteomark: a surgical navigation system for oral and maxillofacial surgery. Int J Oral Maxillofac Surg. 2012; 41:265-270.
• [32]Marmulla R, Luth T, Muhling J, Hassfeld S. Markerless laser registration in image-guided oral and maxillofacial surgery. J Oral Maxillofac Surg. 2004; 62:845-851.
• [33]Shamir RR, Joskowicz L. Geometrical analysis of registration errors in point-based rigid-body registration using invariants. Med Image Anal. 2011; 15:85-95.
• [34]Khadem R, Yeh CC, Sadeghi-Tehrani M, Bax MR, Johnson JA, Welch JN. Comparative tracking error analysis of five different optical tracking systems. Comput Aided Surg. 2000; 5:98-107.