Autonomous vehicles (AVs) hold immense potential to revolutionize transportation safety and mobility. However, a crucial challenge is the lack of standardized procedures for testing and evaluating their driving intelligence – the ability to operate safely and efficiently without human intervention. Existing methods for human-driven vehicles are inadequate for AVs. The state-of-the-art approach employs an agent-environment framework, combining software simulation, closed-track testing, and on-road testing to evaluate AV agents within realistic driving environments. The primary challenge lies in three key aspects: First, AI-based AV agents often function as black boxes, hindering the use of traditional logic-based verification techniques. Second, the high dimensionality and stochastic nature of the driving environment complicate modeling and analysis. Third, the rareness of safety-critical events in naturalistic driving leads to extremely inefficient testing, potentially requiring hundreds of millions, or even billions, of miles of simulation to achieve adequate safety demonstrations. This paper addresses these challenges by proposing a novel approach to construct an intelligent testing environment that balances accuracy and efficiency while considering the high dimensionality and low frequency of critical events in the driving environment. Existing methods primarily rely on NDEs, such as those in sophisticated simulators like CARLA, AirSim, and Drive Constellation, but suffer from the inefficiency problem due to the high dimensionality of the environment and the rare occurrence of critical events. Scenario-based approaches using importance sampling (IS) theory have been proposed to improve efficiency, but their effectiveness is limited to scenarios involving simple maneuvers and short durations. These methods fall short of representing the complexity and variability of real-world driving environments involving numerous vehicles and extended time durations. Therefore, the problem of creating an effective AV testing environment that accounts for high-dimensionality and low-probability events is the focus of this research.

The authors extensively review existing approaches to AV testing, highlighting the limitations of naturalistic driving environment (NDE) simulations due to high dimensionality and the rarity of critical events. They discuss the limitations of current scenario-based approaches that leverage importance sampling, citing their inability to scale to the complexity of real-world driving scenarios involving many vehicles and long durations. The authors specifically mention existing simulators (CARLA, AirSim, Drive Constellation, etc.) and their inefficiencies. They also mention scenario-based methods that utilize importance sampling theory to address the rarity of events, but point out that their applicability is limited to scenarios with simple maneuvers and short durations, unsuitable for complex real-world driving scenarios.

The core of this research is the development and application of a Naturalistic and Adversarial Driving Environment (NADE) for AV testing. NADE builds upon the NDE but introduces sparse but intelligent adjustments to accelerate the evaluation process without sacrificing accuracy. The method uses a three-pronged approach: 1. **Naturalistic Driving Environment (NDE) Generation:** The authors propose a data-driven approach to generate a realistic simulation environment. They model the NDE using a Markov decision process, derive naturalistic distributions of vehicle maneuvers from naturalistic driving data (NDD), and sample vehicle maneuvers from these distributions. NDD is collected from the Safety Pilot Model Deployment (SPMD) program and the Integrated Vehicle-Based Safety System (IVBSS) at the University of Michigan. The data includes positions, speeds, accelerations, and distances between vehicles and lane markings, allowing for the categorization of maneuvers into six groups (free driving, car-following, cut-in, lane changes with varying numbers of adjacent vehicles). Empirical distributions are calculated for each maneuver category, given specific vehicle states. NDE simulation is achieved by sampling initial conditions and maneuvers from these distributions. 2. **Naturalistic and Adversarial Driving Environment (NADE) Generation:** To improve efficiency without sacrificing accuracy, the authors introduce the NADE. They combine the CMC and IS methods, leveraging the CMC's advantage in high-dimensional situations while using the IS method to focus on critical variables. The key is identifying critical moments and principal other vehicles (POVs) whose maneuvers significantly impact AV safety. Maneuver criticality is defined as the product of exposure frequency (from NDE) and maneuver challenge (the probability of an accident given the maneuver). Surrogate models (SMs), like the IDM and MOBIL models, are used to estimate the maneuver challenge, given the inherent uncertainty in real-world AV behaviors. Reinforcement learning is used to refine the maneuver challenge estimations, particularly for car-following scenarios. The importance function for POVs at critical moments is a weighted average of the exposure frequency and normalized criticality, inspired by defensive importance sampling. Manuevers of POVs are sampled from these importance functions, while other vehicles continue to follow naturalistic distributions from the NDE. 3. **AV Evaluation:** The authors construct two AV agents: one based on IDM and MOBIL, and another trained with deep reinforcement learning. They then evaluate these AVs in both NDE and NADE, comparing accident rates and required test miles. The relative half-width is used to quantify evaluation precision. A sensitivity analysis of the importance function weight (λ) is also conducted.

The key findings of the study demonstrate the superior efficiency and accuracy of the proposed NADE method compared to the traditional NDE approach for evaluating AV driving intelligence: 1. **NDE Generation:** The data-driven method for generating NDE produced naturalistic distributions of vehicle speeds and inter-vehicle distances, closely matching the ground truth from NDD. This indicates the successful generation of a realistic driving simulation environment. 2. **NADE Generation and Effectiveness:** NADE effectively generates a more adversarial yet still naturalistic environment by selectively adjusting maneuvers of POVs at critical moments. The adjustment frequency remains sparse, only modifying about 1.5-1.7% of the background vehicle maneuvers per mile. 3. **Accuracy and Efficiency of Driving Intelligence Testing in NADE:** The results showed that NADE achieves the same accuracy in estimating accident rates as NDE, but with significantly fewer test runs. For the tested AV models, NADE reduced the required number of tests by orders of magnitude (500x and 6000x, respectively), translating to millions of miles of driving distance saved. This efficiency is maintained for various importance function weight parameters (λ). 4. **Unbiasedness of Accident Type Estimation:** NADE produces similar weighted accident rates for different accident types to those in NDE, ensuring unbiasedness and demonstrating the integrity of the methodology. 5. **Generation of Adversarial Examples:** The authors further demonstrate that NADE is capable of generating more valuable adversarial examples, which are important for AV development. The simulation weight and diversity of the events involved are used as criteria to identify such examples. These findings collectively show that the NADE method significantly improves the efficiency and effectiveness of AV safety testing without compromising the accuracy of the results.

The results confirm the hypothesis that sparse but intelligent adjustments to the naturalistic driving environment can significantly improve the efficiency of AV testing. The NADE method addresses the critical issues of high dimensionality and rare events in the evaluation of autonomous driving systems. The findings demonstrate that the proposed methodology provides a statistically unbiased estimate of accident rates while drastically reducing the number of tests needed, hence achieving significant efficiency gains. The ability of NADE to generate more adversarial examples is valuable for identifying vulnerabilities in AV systems and driving safety improvements. The methodology's scalability suggests its applicability to larger and more complex driving scenarios, promising future applications in city-scale simulations. However, some limitations remain, such as the need for substantial NDD and potential approximation errors in maneuver challenge estimation. Future work could focus on addressing these issues and applying the method to scenarios beyond highway driving and to further refine the surrogate model and the reinforcement learning for maneuver challenge estimation.

This paper introduces NADE, a novel method for testing AV driving intelligence. NADE leverages a data-driven approach to generate a naturalistic driving environment (NDE) and then intelligently introduces adversarial scenarios to significantly accelerate the testing process without compromising accuracy. The results demonstrate NADE's superior efficiency compared to traditional NDE methods, requiring significantly fewer tests to achieve the same level of accuracy in accident rate estimations. This method represents a major advancement in AV testing, offering a pathway to more efficient and comprehensive evaluations that enhance the safety and reliability of autonomous driving systems. Future work should focus on extending NADE to more complex driving environments and investigating the use of NADE for accelerated training of AV systems.

The authors acknowledge several limitations. First, the method relies on the availability of a large amount of naturalistic driving data for accurate modeling of background vehicle behaviors. Second, the accuracy of the NADE method depends on the accuracy of the surrogate models used to estimate maneuver challenges. Approximation errors could arise from discrepancies between the surrogate model and the actual AV under test, and from uncertainties in predicting AV maneuvers in future time steps. Third, the study focused on highway driving scenarios with simplified maneuvers and vehicle-only interactions. Extending the methodology to more complex scenarios, including diverse road users and weather conditions, requires further investigation. The study also doesn't explicitly address perception-related tests, requiring further study to integrate these aspects within the NADE framework.