A novel computer vision system transforms shiny objects into functional cameras, allowing observers to perceive their surroundings beyond obstacles or around corners.
When driving through narrow city streets, the reflections from glossy paint or side mirrors of parked vehicles can assist drivers in spotting previously obscured objects, such as a child playing behind the parked cars on the sidewalk.
MIT and Rice University researchers have developed ORCa, an AI-powered system that transforms shiny objects into functional cameras by capturing and mapping reflections. This technique improves autonomous vehicles’ visibility by utilizing reflections for observing obstacles and offering potential drone imaging applications. The technique uses object images from various angles to turn the object’s surface into a virtual sensor, capturing reflections. The AI system maps these reflections to estimate the depth and capture unique perspectives hidden from the observer’s view, like seeing around corners or beyond obstructions.
Reflecting on reflections
The researchers overcome the challenges of exploiting reflections with ORCa, a technique that works in three steps. They photograph the object from multiple angles, capturing various reflections. Using machine learning, ORCa converts the object’s surface into a virtual sensor, capturing light and reflections for each virtual pixel. The system then models the 3D environment from the object’s perspective using virtual pixels on its surface.
Catching rays
ORCa uses multiview reflections to estimate the depth and shape of the glossy object. It models the scene as a 5D radiance field, capturing the information of intensity, direction, and light rays. This 5D radiance field aids in accurate depth estimation and reveals hidden features blocked by corners or obstructions. Once ORCa captures the 5D radiance field, users can position a virtual camera anywhere in the scene and visualize its perspective. They can also add virtual objects or alter the appearance of existing objects, like changing from ceramic to metallic.
Accurate estimations
Researchers compared their technique with other reflection modeling methods, which were slightly different from ORCa’s task. Their method distinguished true object color from reflections and outperformed baselines in extracting accurate object geometry and textures. They also assessed ORCa’s depth estimations against simulated ground truth data, confirming its reliability in predicting object distances in the scene. Based on this proof-of-concept, researchers aim to use this technique in drone imaging. ORCa uses faint reflections to reconstruct scenes and plans to incorporate shadows and combine reflections for imaging hidden or additional parts of a scene.
Reference : “ORCa: Glossy Objects as Radiance Field Cameras” by Kushagra Tiwary, Akshat Dave, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraraghavan and Ramesh Raskar, 12 December 2022, Computer Science > Computer Vision and Pattern Recognition.
arXiv:2212.04531
Thanks for the informative and useful article.
It will be easy to read the article if the author mentions the acronym on the first appearance of the word.
ORCa: Objects as Radiance field Cameras