The increasing interest in autonomous vehicles (AVs) stems from the promise of enhanced convenience, safety, and environmental benefits. Despite predictions of widespread AV deployment by 2020, their use remains limited to small-scale trials. A major obstacle is the challenge of achieving precise localization under various environmental and weather conditions. Sensor imperfections, especially under adverse weather such as rain, fog, and snow, significantly hinder accurate localization, a critical prerequisite for safe and reliable AV operation. Existing localization methods often rely heavily on GPS, which can be unreliable in urban environments due to signal degradation caused by obstacles and reflections. Furthermore, GPS typically provides only meter-level accuracy, lacking crucial orientation information. Imprecise localization can lead to dangerous situations, such as an AV misjudging lane position before a turn or failing to stop appropriately at an intersection. Ego-motion estimation (odometry), using onboard sensors, offers a complementary solution, predicting accurate relative self-position. A robust system must address sensor vulnerabilities arising from environmental conditions and sensor imperfections. AVs utilize various sensors, including cameras, lidars, radars, ultrasound, and GPS, to enhance situational awareness. Artificial intelligence (AI), machine learning, deep learning, and large datasets play crucial roles in processing this multisensory data for perception, localization, prediction, and motion planning. Accurate localization is fundamental to the success of AVs, and this research focuses on improving ego-motion estimation specifically to address the challenges of adverse weather conditions and sensor limitations.

Artificial intelligence in AV development heavily relies on public datasets, but the availability of accurate ground-truth data for supervision is limited due to manual labeling requirements and sensor deficiencies. Cameras and lidars are primary perception sensors, but their performance degrades significantly in inclement weather. Millimeter-wave radars offer a key advantage due to their immunity to adverse weather conditions, scene illumination, and airborne obscurants. Their longer wavelength allows them to penetrate or diffract around particles in fog, rain, and snow, and their radiofrequency nature makes them resilient to water and dust. Existing ego-motion estimation techniques often focus on lidar data, but these are not suitable for radar data, which is much coarser and noisier. While additional information from other sensors (wheel encoders, inertial measurement units) and intermediate predictions from other software modules can supplement ego-motion estimation, perception sensors such as cameras, lidars, and radars remain pivotal. Deep learning models provide state-of-the-art solutions for ego-motion estimation, but their performance suffers in adverse weather due to reduced sensing capabilities and domain shifts between training and deployment data. This study addresses these limitations by proposing a novel framework that leverages the strengths of multiple sensors while mitigating their individual weaknesses.

The proposed method, Geometry-Aware Multimodal Ego-Motion Estimation (GRAMME), is a self-supervised deep learning framework that uses cameras, lidars, and radars to estimate AV ego-motion by reconstructing 3D scene geometry under various conditions (day, night, rain, fog, snow). GRAMME is sensor-agnostic and supports sensors with varying configurations. A novel differentiable view-reconstruction algorithm incorporates range sensor measurements (lidars and radars), compensating for camera limitations in challenging conditions. The supervisory signal for training the neural networks comes from this view-reconstruction algorithm: given a multimodal input view, it reconstructs a new view from a different position. Visual reconstruction uses predicted per-pixel depth and ego-motion, while range reconstruction uses predicted ego-motion and range measurements, both using multimodal masks. A spatial transformer module implements view reconstruction in a differentiable manner. GRAMME features a modular design for independent operation of each modality (camera, lidar, radar) during training and inference, enhancing robustness. Although modules for depth, pose, and mask predictions are trained jointly, they can operate independently at test time, leading to uncorrelated failure modes. A reciprocal multimodal training technique enhances individual modality predictions through information flow across submodules. Late multimodal deep fusion uses unaligned ego-motion predictions from multiple modalities to predict final motion. This fusion involves two stages: reconstructing camera and range views using individual predictions, then interchangeably using each modality's predictions in the counterpart view-reconstruction algorithms. The resulting model is designed to be robust to sensor failures and capable of generalizing to various sensor configurations and weather conditions.

The paper evaluates GRAMME's performance in depth prediction and ego-motion estimation across five diverse settings (day, night, rain, fog, snow) using fivefold cross-validation on publicly available datasets (Oxford Robotcar and RADIATE). Experiments demonstrate that GRAMME's multimodal approach significantly improves robustness to adverse weather and outperforms state-of-the-art methods. Cross-condition evaluation, training models on day sequences and testing on challenging conditions, reveals GRAMME's strong generalization capability. The modular design enhances the performance of individual modalities even when others are unavailable, showing robustness to sensor failures. The method is shown to be sensor-agnostic, performing well with sensors of various resolutions and beamwidths. The analysis of depth prediction performance shows that multimodal fusion significantly improves generalization to diverse conditions. Camera-only models perform poorly in challenging conditions, highlighting the importance of range sensors. The predicted masks effectively remove unreliable regions in sensor measurements, improving performance. Multimodal fusion significantly improves both translational and rotational motion prediction accuracy compared to camera-only models. A game-theoretic approach (SHAP values) visualizes the model's interpretability. Camera-only models focus on scattered regions, while multimodal models focus on semantically consistent regions (objects, road boundaries), demonstrating improved interpretability. Data efficiency analysis shows that multimodal models achieve satisfactory performance even with limited training data (50%), although increased complexity makes them slightly more data-dependent than single-modality models. Comparison with baseline methods shows that GRAMME achieves state-of-the-art results in both depth reconstruction and ego-motion estimation, demonstrating its superiority particularly under adverse weather conditions.

GRAMME addresses five key challenges in autonomous driving: multimodal self-supervision, modularity, generalizability, interpretability, and data efficiency. The multimodal self-supervised approach improves robustness and generalizability, eliminating the need for laborious ground-truth collection. Modularity ensures robustness to sensor failures, while generalizability enables performance in unseen weather conditions. Interpretability is enhanced by focusing on semantically consistent regions in the scene, and data efficiency allows for satisfactory performance even with limited data. The results highlight the importance of multimodal sensor fusion for robust localization in autonomous vehicles. The superior performance of GRAMME, particularly under adverse weather, signifies a substantial step toward achieving reliable all-weather autonomous driving. Future work could integrate signal-to-noise ratios into the masking component, utilize Doppler measurements for object distinction, and extend GRAMME to lifelong and continual learning paradigms to enable continuous improvement of the system.

GRAMME presents a significant advance in robust ego-motion estimation for autonomous vehicles by effectively addressing the challenges of adverse weather conditions and sensor limitations. The use of a multimodal self-supervised deep learning approach, coupled with modular design and a focus on interpretability and data efficiency, leads to state-of-the-art performance across various challenging scenarios. Future research should explore the integration of additional sensor information and advanced learning techniques to further enhance the system's capabilities and robustness.

While GRAMME demonstrates strong performance across diverse weather conditions, the study's generalizability might be limited by the specific datasets used. The datasets, although comprehensive, may not fully represent the extreme range of weather conditions and scene complexities encountered in real-world driving. Furthermore, the complexity of the multimodal architecture increases the model's data dependency, meaning that performance might degrade with severely limited training data. Finally, the interpretability analysis, while insightful, relies on a specific game-theoretic approach, and alternative methods might yield different insights.