This article gives a brief introduction to Kinect sensor and explains how to install OpenCV libraries in Ubuntu operating system. The article also covers some useful commands at the end for connecting Kinect to your computer. It can serve as a base for developing complex computer vision applications using Kinect for Ubuntu operating system.
Kinect is a motion-sensing device developed by Microsoft for Xbox 360 video game console. Though initially invented for gaming, people have begun using it for different purposes. With Kinect you can control a television set without remote control, a computer without keyboard, a mouse icon or touch-screen and games without any controller in your hand.
Kinect (see Fig. 1) has an array of sensors and specialised devices to pre-process the information received. Communication between Kinect and a game console, or Linux, is through a single USB cable. Its main features include:
Gesture recognition. Kinect can recognise gestures like hand movement based on inputs from an RGB camera and depth sensor.
Speech recognition. It can recognise spoken words and convert them into text, although accuracy strictly depends on the dictionary used. Input is from a microphone array.
The main components of Kinect are RGB camera, depth sensor and microphone array. The depth sensor combines an IR laser projector with a monochrome CMOS sensor to get 3D video data. Besides these, there is a motor to tilt the sensor array up and down for the best view of the scene, and an accelerometer to sense position.
To get the best out of Kinect, you should be familiar with two terms: natural user interface (NUI) and machine learning. NUI refers to close interaction between a user and the computer. It includes controlling the computer through gestures, or the computer recognising the user’s voice/face. Microsoft Surface, multi-touch and Kinect are a few examples of NUI.
Machine learning, according to Wikipedia, “…a branch of artificial intelligence, is a scientific discipline concerned with the design and development of algorithms that allow computers to evolve behaviour based on empirical data, such as from sensor data.”
Kinect projects. Among some amazing things/projects that can be done with Kinect include gesture-controlled robot (EFY, January 2013), robot operating system (ROS), Kinect-controlled computers based on user gestures and/or speech recognition, scanning of 3D objects (Kinect enables robots to map 3D objects, resulting in detailed and precise models of people), medical applications like gesture-based control of surgical tools, and in education where writing and calculator applications have already been developed using it.
Programming with Kinect
For programming with Kinect, we need open source computer vision (OpenCV)—a library for real-time computer vision. We also need libfreenect library that can take the data (such as colour, depth and other information) from Kinect and convert it into a format that is readable by OpenCV library for further processing and developing complex algorithms like face recognition.
Installing OpenCV. For installing OpenCV in Ubuntu 12.04, use the following commands:
Download Opencv-2.4.7.tar.gz from here and extract it to home folder.
At the end of the file, add the following lines and save it:
And then configure dynamic linker run-time bindings:
OpenCV-2.4.7 has been installed. Open another terminal to check some of the samples:
To check any sample, go to that folder and run the sample. For example, in the ‘c’ folder, we have contours, so to run it, enter the command given below:
You will get the output image as shown in Fig. 2.
Installing libfreenect. To install libfreenect, we need to first install some prerequisites for building it:
Now download the clone from ‘github’:
Change directory to ‘libfreenect’
Make a directory named ‘build’ and change current directory to ‘build’
CMake is used to control the software compilation process with simple platform and compiler-independent configuration files. CMake generates native make files and workspaces that can be used in the compiler environment of your choice.