It is circa 2020, a tele-mentoring session is in progress, wherein a veteran surgeon in New Delhi is step-by-step guiding a less experienced surgeon thousands of kilometres away in London. Both surgeons share the view of the surgical field and control of the robotic system and can talk using microphones. Your brain tumor has to be removed via surgery immediately. However, no senior surgeon is available in the London branch of the hospital. Tele-immersion allows dissection without the need of the senior surgeon even touching the tumor. The operation is successful and your life is saved.
A few weeks later, you schedule a very important meeting with your business associates in Bengaluru. However, you have visitors coming in from New York to sign a mega business deal the same day. Without any stress, you go to a room called holodeck. There, inside a simulated environment, using information technology you contact your business associates. You are able to conduct the meeting with them almost as if you are in Bengaluru. You even manage to shake hands with holographic images of your business associates. With a smile on your face you proceed to receive your guests from New York.
What is tele-immersion all about
Electronic visualisation laboratory (EVL), University of Illinois, defines tele-immersion as the union of networked virtual reality and video in the context of significant computing and data mining.
Tele-immersion is a technology to be implemented with the Internet2 that will enable users in different geographic locations to come together in a simulated holographic environment to interact. To users, it would appear as if they are actually interacting, communicating and meeting each other in person at a common location. In a tele-immersive environment, computers recognise the presence and movements of individuals as well as physical and virtual objects. These can then track living and non-living objects and project them in a realistic manner across many geographic locations. Holographic environments are generated by advanced computers that work at computing speeds thousands of times faster than PCs we use every day.
Three simple steps required to construct a holographic environment are:
1. The computer recognises the presence and movements of people and objects.
2. Then, it tracks those images.
3. Finally, it projects those images on a stereo-immersive surface.
The 3D reconstruction for tele-immersion is facilitated by projecting images on a stereo-immersive surface using IT. Two or more cameras take rapid sequential shots of the same object, continuously performing distance calculations and projecting those into the computer-simulated environment, to replicate real-time movement. By mixing cameras and the Internet telephony, videoconferencing allows real-time exchange of a vast amount of information effectively, without the need of the physical presence of every person in one central room.
The history
Ivan Sutherland, the great computer graphics pioneer, proposed the concept of Ultimate Display in 1965. It described a graphics display that allowed the user to experience a completely computer-rendered environment. In late 1960s, the ARPANET, conceived by the US Department of Defence, gradually evolved into the Internet. In 1998, Abilene, a backbone-research project was launched for the Internet2 research. The Internet2 needed an application that would challenge and stretch its networks capabilities. The head of Advanced Network and Services Inc. proposed tele-immersion as the application to drive forward the Internet2 research. Thus, the National Tele-immersion Initiative was formed.
The new broadband version of the Internet is capable of transmitting high volumes of data. Tele-immersion is the only application that can push it to its boundaries. In May 2000, researchers at University of North Carolina (UNC), University of Pennsylvania and Advanced Networks and Services Inc. reached a milestone in developing this technology. A user sitting in an office at UNC in Chapel Hill, NC, was able to see life-like, 3D images of colleagues hundreds of kilometres away.
Elements of a holographic environment
Initial tele-immersive displays required users to put on unique goggles and a head device that tracked viewpoints of users looking at the screen. On the other end, people who appeared as 3D images were being tracked with an array of eight ordinary video cameras, while three other video cameras captured real light patterns projected in each room to calculate distances. This enabled proper depth to be recreated on the screen. Hence, if an observer moved his head to the left, he or she could see corresponding images that would be seen if he or she was actually in the room with the person on the screen.
Images on the screen were split and polarised to create a different image for each eye. The goggles then combined these images so that the brain recognised only one 3D image. This process was similar to how 3D movie glasses worked. Early experiments experienced some problems, often encountered during normal videoconferencing. Scenes being projected were refreshed three times per second, which created an unsteady image. If that rate could be improved to ten frames per second, it would create a smooth projected image that would almost be like looking at another person through a window. Scientists have been working on new technologies to support this type of communication. Here are some components and aspects of the new technologies:
Tele-cubicle. By means of which users communicate, a tele-cubicle consists of a stereo-immersive desk surface and two stereo-immersive wall surfaces (Fig. 2). These three display surfaces join to form a corner desk unit. Walls appear as windows to other users’ environment, while desks join together to form a virtual conference table in the centre. This allows realistic inclusion of tele-immersion into the work environment as it consumes the average amount of desk space.
The Internet2. This will replace the current Internet infrastructure. It is a consortium made up of the USA government, industry and academia (over 180 universities) that was formed with the purpose of creating the Internet of the future. Good quality tele-immersion needs about 1.2Gbps. This new network will have a higher bandwidth and speeds 1000 times faster than today’s Internet. This high bandwidth and speed network is necessary to transfer large amounts of data that tele-immersion will produce.
Display technologies. Stereo-immersive displays would have to present a clear view of the scenes being transmitted. Haptic sensors would allow people to touch projections as if these were real. And, desktop supercomputers would complete trillions of calculations needed to create a holographic environment. Another possibility to support these environments would be a network of computers that split the total power.
Application areas
Tele-immersion can benefit a host of areas from being able to meet relatives you are unable to visit to treating very remotely-located critical patients. Here are the main application areas:
In medicine. Tele-immersion can prove to be tremendously helpful and even life-saving in the area of medicine. Traditionally, medicine has been taught and practiced face to face. However, with tele-immersion, 3D surgical learning for virtual operations is now possible. In times to come, we will be able to tele-immerse a surgeon into an operation theatre to carry out a surgery (Fig. 3).
This could imply that many lives could be saved, especially if there are patients who are in dire need of a certain process or equipment that is not available at their location. Some loose ends that need to be tied are the legal implications in case of any system or tele-immersed surgeon failure. Also, since the doctor carrying out the operation is actually not physically present, how effective will he or she be? Will the doctor be able to gauge what is transpiring in the back-end of medical services? Will the patient feel comfortable going through a critical procedure in this manner?
High-resolution 3D patient images have now become quite common, and it is vital to make this patient-specific data quickly available and usable to many specialists at various geographical locations. While web based systems can provide volume or surface rendering of medical data over networks with low fidelity, these are not able to manage stereoscopic visualisation or huge datasets effectively. This is where state-of-the-art tele-immersion techniques and high-speed networks are required to produce an environment for geographically-distributed clinicians to share these huge datasets in real-time basis.
In education. Tele-immersion allows geographically-distributed students to come together in a single environment. This is particularly important given the lack of specialist faculty at all required sites, and hence the need to resort to distance education. This technology will result in a much higher level of collaboration among educational institutes. Moreover, it will also help in mitigating the risk, for example, budding surgeons will get a much better experience and training w.r.t. complex situations before actually operating on their first patients. In the similar manner, a pilot will experience an almost life-like flight simulation before his or her maiden flight. School students will be able to access data or govern a telescope from a remote location.
Research activities at the Tele-immerson Lab at University of California, Berkeley, the USA (Fig. 4), include human-robot co-operation, human activity recognition from multi-modal data, development of individualised musculo-skeletal models, quantification of human performance, remote monitoring in health care and its privacy and security considerations, and modelling of driver interaction in semi-autonomous vehicles. The lab supports research in 3D stereo reconstruction, tele-immersion, motion capturing, remote monitoring and human-movement analysis.
The tele-immersion apparatus with 48 cameras has been developed to capture 3D data of users with accuracy of about 1cm to 2cm. Cameras are linked to 12 servers that process the stereo reconstruction algorithm, allowing full-body 3D reconstruction of user(s). Some applications that the lab is presently concentrating on include collaborative work on 3D data sets, such as medical (MRI, 3D X-ray), geo-scientific collaboration, collaborative design in CAD modelling, collaborative archaeology and remote teaching of physical activities/skills (such as in rehabilitation and dance performance).
In collaborative work. Tele-immersion will enable groups of engineers to work closely together across great distances on computerised designs, for say, developing innovative equipment (Fig. 5).
While different teams of engineers might be thousands of kilometres away, they would be able to work together on the same project as if they were sitting together on the same workbench.
Archaeologists from around the globe will have the same sort of experience during a critical excavation. Rarefied experts in building inspection or engine repair will be able to visit various sites without incurring time and money expenditure on air travel. Similarly, collaboration will be possible while working on mechanical CAD. A team of designers will be able to work in partnership from remote sites in an interactive design process. They will be able to influence a virtual model, right from the conceptual to the production stage.
What the future holds
One of the current research areas is how to make tele-immersion more natural, perhaps by doing away with the headgear and glasses. Efforts are on for making the tele-immersion experience possible by simply entering a tele-cubicle. One solution is to use a screen that transmits diverse information to each eye, using swivelling pixels that track either left or right eye.
Another probability is to turn the entire tele-immerion room onto a screen. Walls, chairs, roof, etc could be coated with special light-sensitive material. A camera would photograph surfaces, computers would calculate their shapes in 3D and projectors would screen pre-warped images, making it appear as if these filled the room.
In years to come, perhaps one of your office walls will sparkle and your teammate will be sitting right in front of you, as if in person. On the other side, your teammate will enjoy the same experience. This possibility will make air travel quite rare. This is definitely going to have a favourable impact on our environment too with less greenhouse gas emission, which will make this planet a better place to live on.
