The dataset, capturing extensive real-world driving data from multiple vehicles over repeated trips, offers unique insights and is set to transform the development of AV systems.
A dataset promises to accelerate the development of autonomous vehicle (AV) technology by providing researchers with extensive real-world driving data captured from multiple vehicles over repeated trips. The MARS (MultiAgent, multitraveRSal, and multimodal) dataset, introduced by researchers from NYU Tandon School of Engineering in collaboration with May Mobility, offers unique features that distinguish it from previous efforts in the field. The NYU Tandon team presented a paper on MARS at the IEEE/CVF Computer Vision and Pattern Recognition (CVPR 2024) Conference.
Curated by Feng’s Automation and Intelligence for Civil Engineering (AI4CE) Lab and May Mobility engineers, the dataset was collected using a fleet of four autonomous Toyota Sienna Autono-MaaS vehicles operating within a 20-kilometer zone encompassing residential, commercial, and university areas in a U.S. city. The subscription service provided access to real-time and historical data from these vehicles, including sensor data (LiDAR, Camera), GPS/IMU, vehicle state, and more.The dataset’s “multitraversal” nature is particularly significant, involving 67 specific locations along the route and data from thousands of passes through these areas at different times and in varying weather conditions. This repeated observation is crucial for developing more robust perception and mapping algorithms.
Potential for Advanced AV Perception and Collaboration
Initial experiments by the team in visual place recognition and 3D scene reconstruction demonstrated the dataset’s potential. These tasks are fundamental to an AV’s ability to locate and understand its surroundings. The dataset allows to study both how multiple vehicles can collaboratively perceive their surroundings more accurately, and how vehicles can build up a detailed understanding of their environment over time. The team mentioned that they could not have assembled it without the unprecedented access May Mobility provided us to its large-scale real-world data. They emphasized the importance of transparency and data-sharing in fostering innovation within the AV industry. As they continue to build bridges with academia, their research will pave the way to more innovation at May Mobility and throughout the AV industry.
The researchers claimed that they are just scratching the surface of what’s possible with this dataset. They look forward to the possibilities that develop as we work with researchers to solve current and future challenges in autonomous driving. The release of the dataset comes at a critical time for the autonomous vehicle industry, aiming to move beyond controlled testing environments and tackle the complexities of real-world driving. This dataset, collected from multiple commercial vehicles in actual use, is poised to play a vital role in training and validating the AI systems that power AVs.
“Datasets for autonomous vehicle research have typically come from a single vehicle’s one-time pass of a certain location. MARS offers many more opportunities because it captures real interactions between multiple AVs traveling over fixed routes hundreds of times, including repeated passes through the same locations under varying conditions,” said Chen Feng, the lead researcher on the project and an NYU Tandon assistant professor specializing in computer vision for autonomous vehicles and mobile robots.
