Advanced VR Research Centre

Gesture Control Interaction System For Surgeons in Operating Theatres

Summary

Currently the interaction with most of the complex systems relies on the physical contact with the input mechanisms of the devices used. Usually this is achieved through a particular manipulation with fingers on the keyboard or sensor screen surface of the device. Consequently, this introduces limitations arising from the need to be close to the device to be able to get the desired output, to have both hands empty to hold and to input successfully the commands as well as the need to unavoidably touch dirty surfaces.

In medicine the ability of touch – free motion sensing input technology is particularly useful, where it can reduce the risk of contamination and will be beneficial to both patients and their caretakers. NUI can be used in several applications. For example, manipulation and view of x-ray and MRI imagery with the use of simple gesture control, taking notes of important information by writing in the air and saving the document on the hard drive, and the use of gesture control to perform complex surgical procedures through robotic mechanisms.

Surgeons clearly will benefit from touch – free control, since it allows to avoid interaction with non-sterile surfaces of the devices in use and to reduce the risk of infection. If the surgeon needs to access the computer, he can ask an assistant to help him, which can lead to reduced concentration while giving instructions of navigation through the massive amount of data or he needs to scrub out that can lead to prolonged operation sessions and possible further health complications of the patients.

Vascular surgeons often can avoid performing open-heart surgery by placing stents to open clogged arteries or to support aortas with an aneurysm. But proper placement of the stent requires a great deal of precision, and medical imaging plays a big role in enabling that precision. With the use of Kinect sensor, large display and specially designed application a surgeon can manipulate the image by moving or zooming it to see the details. This all excludes any surface touching and helps the surgeon to determine the placement of the stent more quickly.

As a surgeon, being able to manipulate information in a touch-free way while operating will have enormous benefits. Breaking the flow of the operation to get someone to do it for you not only distracts from performing a delicate surgery, but it’s also time-consuming.

It is worth to mention that the evolution of robotic arms that could potentially help the surgeon to conduct the surgery is proceeding slowly and the progress is not so certain. Precise hand tracking, eye tracking and gesture control in general could increase the performance of the surgeon and the effectiveness as well as quality of the operation by accessing the areas that are usually hard to reach. For example, neurosurgeons would benefit from being able to go deep into the brain with minimal amount of brain damage possible. They can thus get better imaging ability to view the tumor.

This project exploits the 3D tracking capability of the Microsoft Xbox Kinect's powerful ability to track gesture based input without any form of glove based input devices - relying entirely on visual tracking of the user's hands. By using Kinect's sensor depth camera it is possible accurately track hand gesture inputs and effective take control over a computer based system.

Implementation

Further details will be supplied in due course. The following images show the prototype system.

 Figure 1: Basic configuration showing gesture input via the Kinect.

Figure 2: Initial Kinect Gesture activated keyboard. Please note the keyboard layout - there is a rest section in the centre to allow improve tracking accuracy.

Figure 3: This image shows a CT scan being manipulated. THe user can manipulate the image, change its size, orientation etc.