Sixth Sense technology, based on the interaction of the physical world with digital information, has the potential of bringing about a revolution in the field of technology and paving the way for many more great innovations. Revealing the true meaning of this technology and further explaining its various applications and implementations in the real world is the primary focus of this article.
Sixth Sense technology uses a wearable gesture interface to augment the physical world. It allows us to make use of hand gestures for a particular set of tasks. The wearable devices used act as a gestural interface and integrates the real world with digital information.
Sixth Sense technology has the potential to bring about a digital revolution across the globe. The fact that it can be utilised in multiple ways, like performing basic actions, locating points on a map and calling someone directly without picking up the phone, asserts its importance.
Gesture recognition is a part of augmented reality (AR), which forms one of the primary basis of Sixth Sense technology. It is aimed at interpreting human gestures through program codes and mathematical algorithms.
The first person to make a device based on this concept was Steve Mann. He developed a wearable computing device in the 1990s when he was a student at Media Lab. He implemented the concept through the means of a neck-worn projector accompanied by a camera system.
The concept was further developed by Pranav Mistry, a research associate and PhD candidate at Massachusetts Institute of Technology, who is currently head of Think Tank team and vice president of research at Samsung Research America.
Working and design of the device
Many devices have been developed based on Sixth Sense technology. One such example is Wear Ur World (WUW) device. This prototype device gathers data from the user’s environment, answers questions and fulfills tasks using the Internet as an information bank and then presents the result back to the user by means of a display mechanism.
Simple computer-based vision-oriented codes and gesture identification and interpretation technologies are used to evaluate inputs received from the individual using the device in the form of images or gestures produced by the user’s hands or face. Application of this device includes every prominent sector, and it performs a variety of tasks.
Pranav Mistry was one of the first few individuals to develop a working prototype of this device, which had such components as camera, projector, camera-sensitive coloured markers, device for interpretation of signals and gestures, plane mirror and sound output device (Fig. 2). The projector and camera were connected to the primary computing device either using external wires or a wireless connection. The device, in turn, was connected to a 4G or 3G network, Wi-Fi network or any other reliable and efficient source of the Internet.
The entire system could be worn by the user around neck. The plane mirror served as an attachment to front part of the projector. The main function of the mirror was to reflect projections of the information on the available surface. The setup was not very expensive and cost around US$ 350.
The device was given the name Wear Ur World (WUW) due to its design and process of functioning. It involves the principle of gesture recognition to understand the gestures captured by the input device with the help of computer visualisation algorithms.
Nowadays, WUW is simply referred to as Sixth Sense technology. Pranav Mistry has published a beta software version of this device written in C++ and C# languages, which is a part of Google Code project, and has also commissioned project name WUW.
According to Mistry, Sixth Sense is a wearable gestural interface that digitally alters the physical world and lets the user use natural hand signals and indications to interact with digital information. Using a camera and a small projector affixed atop support, Sixth Sense serves as the user’s eyes and visually alters any surface or object the user is interacting with, and uses it for displaying output. Sixth Sense aims at providing an unprecedented level of freedom in the sphere of technology.
Working mechanism of this device involves multiple components working in unison. The camera serves as the input device for the gestures made by the user’s hand. The user’s fingertips are enveloped by different colour markers that are sensitive to the camera.
The camera consists of sensors to understand the motion and dynamics of gestures. Images of objects and gestures, which have been streamed, are identified as commands by the computing device and interpreted using the principles of coordinates and location of objects on the screen.
The device then works as per instructions and may send queries to the Internet, if needed. It then returns the information required by the user. This information may be projected via the output device while the plane mirror reflects it onto the desired surface or object. These commands may be followed by another instruction.
Concepts and technology behind the device
Different technological concepts are implemented through WUW. Some of these include ubiquitous computing, AR, gesture recognition, computer vision and radio frequency identification (RFID).
Ubiquitous computing is a form of computing that involves greater use of computers and technological devices to such an extent that they start affecting all actions and tasks performed by individuals. Sixth Sense technology may pave the way for this form of technology.
Developing an ubiquitous computing framework, while maintaining confidentiality of consumers at the same time, is indeed a monumental task for engineers of today. WUW device lives up to this principle as it provides flawless information about various different subjects.
AR involves interaction of digital information and visual displays with the physical world through phones and smart devices. This concept is implemented by the device by integrating the real-world environment with information and visuals.
The process of recognising human gestures is known as signal or gesture recognition, and the computer interface using this method is called gesture-recognition technology. This technology is a much better and more convenient substitute for text and graphical user interfaces as it greatly saves time and provides greater levels of freedom and interactions.
Applications and uses
WUW has a very wide range of applications that are especially relevant in today’s world. Some of these are:
Making a call
Using this technology, there is no need to physically take out the phone to make a call. Using Sixth Sense, one can call another person in a swift and convenient manner. On using this application, the number pad is displayed on any surface the user wants. When the user positions fingers on displayed numbers, the device automatically understands the numbers that the user is signalling and dials the number (Fig. 4).
Searching the map
This function can be used to plot a course, search for places, locate the most viable paths and provide assistance while navigation, to name a few.
Drawing application
Using this device, one can express artistic skills anywhere, any time. The associate application draws and projects images on available surfaces. Drawings saved in device memory can be retrieved any time.
Taking a picture
The camera is used to take an image of the scene the user is looking at. When four fingertips of the user are combined to form a rectangular frame, the camera understands it as a frame-capturing gesture and performs the appropriate task (Fig. 5).
Clock
Using clock application, one can check the time. When the user puts wrist in front of the camera, a digital watch appears.
Multimedia reading experience
The user can program the device to project related videos onto newspaper articles.
Getting product information
Product information can be obtained by Sixth Sense technology because it uses image-recognition or image-maker technology. As soon as the user picks up a product off the shelf, the system scans the product and feeds information such as product specification, user ratings, reviews and other relevant information to the user.
Checking emails
The user can check emails by gesturing @ sign. To open the application, the user has to draw @ in the air. The device will understand the gesture and open the application. Mails can be synchronised with the device in a way that the user can view old mails even in offline mode. This application uses the Internet to search, receive and send new, received mails.
Getting flight updates. Once the user scans the boarding pass, the system feeds relevant flight information like its on-time status or any change in the gate entry point.
Advantages and challenges
As the device is small, it is portable. It supports multiple touches and multi-user communication. Tasks can be carried out through the user’s motion and gestures. This, to a great extent, also saves the user’s time. Data can also be obtained from the Internet by the user.
The device can have a wide variety of applications for different consumers depending on what they imagine and want. This project involves an open source code, which further helps individuals develop their own applications that are workable on this device. WUW can perform tasks of various other devices by itself and, hence,
end up being economical.
Finding an adequate vision-based gesture-recognition algorithm has always been the most demanding part of this device. There are many obstacles involved in the working of the device with precision and usefulness of the gesture-recognition software. There are also limitations on the equipment used. Programming code is very specific to the particular device, and there may be trouble in executing it on another device.
Methods of further development
Searching for alternatives for colour markers would be one of the ways to further improve this device. Enhancing the device so that it can be utilised in such diverse sectors as gaming, education and entertainment will generate a greater number of consumers. For example, an extra speaker can be added to elevate the experience.
An application that can convert gestures into sound could be developed. And this application, in turn, may be used to help people with certain disabilities communicate more effectively with others.
The problem with vision algorithms for gesture recognition can be rectified by integrating artificial intelligence (AI) with the device. This would help recognise human signals much more efficiently, and errors in misinterpretation of gestures would be greatly reduced.
Conclusion
Sixth Sense technology is the science of the future and has the potential for ushering in a new era in the field of technology. This technology could truly change the way humans interact with the real world and give everyone complete awareness about their environment. Sixth Sense prototype implements several applications that demonstrate its utility, feasibility and practicability. There is no doubt that this technology will continue to grow and may very well become the next big thing.
Dr Rajiv Kumar Singh holds a PhD in electronics engineering from IIT (BHU), Varanasi, and is currently employed with Institute of Engineering & Technology (IET), Lucknow as assistant professor and is also acting as associate dean (PG and research), AKTU, Lucknow.
Taru Meshram is an aspiring engineer who has a keen interest in the fields of science and technology. He worked on this project for more than a year at Institute of Engineering & Technology (IET), Lucknow.
this device isnt new there was the original T E D … that initiated the T E D talks