Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment

The study addresses how to accurately and efficiently evaluate the driving intelligence (safety capability) of autonomous vehicles (AVs) given three core challenges: AI-based black-box driving agents limit traditional verification methods; the driving environment is complex, stochastic, and high-dimensional; and safety-critical events (e.g., crashes) are rare, leading to extreme inefficiency of current naturalistic testing. Existing practices rely on on-road and high-fidelity naturalistic simulations, implying hundreds of millions to billions of miles to statistically demonstrate safety at human-driver levels. The research question is how to construct an intelligent testing environment that preserves statistical unbiasedness with respect to naturalistic driving while drastically improving efficiency. The authors propose NADE (Naturalistic and Adversarial Driving Environment), which sparsely and intelligently perturbs a naturalistic environment to provoke informative, safety-critical interactions, thereby overcoming the rareness-of-events and high-dimensionality issues without sacrificing unbiasedness.

Prevailing AV testing leverages the agent–environment framework via software simulation (e.g., CARLA, AirSim, Drive Constellation, CarCraft, AADS), closed-track, and on-road testing in naturalistic driving environments (NDE). Despite realism, these suffer from the curse of dimensionality and rarity of critical events, making evaluation inefficient. Scenario-based and importance sampling (IS) methods have been proposed to accelerate evaluation by focusing on designed critical scenarios (e.g., lane-change cut-in, car following). However, existing IS-based scenario generation typically covers short-duration, low-dimensional interactions among very few vehicles, failing to represent the full spatiotemporal complexity of real-world driving with many interacting agents over long horizons. The authors note theoretical limitations: crude Monte Carlo (CMC) is inefficient for rare events; IS efficiency degrades exponentially with dimensionality. They identify an underexplored advantage of CMC in high dimensions and propose combining IS on a small set of critical variables with CMC on the rest to overcome both rarity and dimensionality limitations.

Overview: The approach constructs NADE by sparsely adjusting behaviors of principal other vehicles (POVs) at critical moments within an otherwise naturalistic environment, guided by importance sampling. It ensures unbiasedness while improving rare-event sampling efficiency by twisting only a small set of critical variables. Naturalistic Driving Environment (NDE) generation: NDE is modeled as a Markov Decision Process (MDP) and probabilistic graphical model. Using naturalistic driving data (NDD) from the SPMD and IVBSS programs, the authors compute empirical exposure frequency distributions of vehicle maneuvers conditioned on states across six categories: free driving, car following, cut-in, and lane changes with zero/one/two adjacent vehicles. Maneuvers are discretized into 33 actions (left lane change, 31 longitudinal accelerations in [-4, 2] m s^-2 at 0.2 steps, right lane change), with zero longitudinal acceleration during a lane change. Initialization samples initial positions and speeds to match naturalistic speed and gap distributions; subsequent maneuvers for each vehicle are sampled independently from empirical distributions at 1 s time steps (vehicle dynamics integrated at 15 Hz). Validation compares simulated distributions (speeds, bumper-to-bumper ranges) to NDD using Hellinger distance and MAE. Identifying critical variables and NADE construction: At each time step, each background vehicle (BV) action is scored by criticality V(u_i|s) = P(u_i|s) × P(A_i|s, u_i), where P(u_i|s) is exposure frequency (naturalistic likelihood) and P(A_i|s, u_i) is the maneuver challenge (probability that the maneuver leads to an AV-involved crash). The BV with maximal criticality above a threshold is the POV; time steps with a POV are critical moments. Only the POV’s action distribution is adjusted via an importance function; all other vehicles follow naturalistic distributions. Defensive importance sampling is used by forming a weighted mixture q(u|s) = λ P(u|s) + (1−λ) normalized-criticality, with λ = ε ∈ (0,1], to balance exploitation and robustness against maneuver-challenge approximation errors. Maneuver challenge estimation: Since AV internals are unknown, surrogate models (SMs) approximate AV behavior: IDM for longitudinal control and a stochastic MOBIL for lane changes. For car-following, the challenge is learned via reinforcement learning (RL) by treating state-action values Q(s,a) as accident probabilities (reward 1 for AV accident, 0 otherwise); state includes BV speed, gap, and relative speed. For general scenarios, the BV’s challenge is computed as an expectation over the AV’s predicted maneuvers (from SMs) and the RL-estimated challenges in resulting car-following configurations. Evaluation with NADE: Accident rate P(A) is estimated using importance sampling with likelihood ratios recorded at critical moments. Tests run for a fixed distance (e.g., 400 m) or until an accident, enabling estimation of accident rate per test (convertible to per mile). Unbiasedness is guaranteed by ensuring q(u|s) > 0 whenever P(u|s) > 0 via λ > 0. Efficiency is analyzed theoretically: direct IS suffers variance growth with dimensionality (KL divergence scaling with O(NT)); the proposed method’s variance depends on divergences only over critical variables (POV maneuvers at critical moments), plus a term reflecting how well the chosen critical variables capture accident likelihood, thereby mitigating the curse of dimensionality. Experimental setup: Simulation uses CARLA (rendering) and a highway traffic simulator (highway-env). NDD is from SPMD (34.9M miles, 98 DAS sedans) and IVBSS (16 instrumented vehicles, 108 drivers). Data filtered for highway, 20–40 m s^-1, dry, daylight. About 1.86×10^8 points and 1.4×10^4 lane changes were identified. AV under test: two models—AV-I (IDM + MOBIL) and AV-II (deep RL). Metrics include distributions of gaps and TTC, event counts (accident, BV cut-in, BV hard brake, lane conflict, AV evasive lane change), accident rates per test, relative half-width (RHW) for precision, and computational cost. Sensitivity to λ (ε = 0.1, 0.3, 0.5) is evaluated. HPC experiments used 500 cores.

- NDE fidelity: Simulated NDE reproduces naturalistic distributions of speeds and bumper-to-bumper spaces closely to NDD, with reported Hellinger distances of 0.053 and 0.102 and MAEs of 0.006 and 0.001 for the compared distributions. Classical models (IDM, Wiedemann99) calibrated on other datasets produced unnatural distributions in this context. - NADE sparsity: For each driving mile, only about 6.51 (AV-I) and 5.43 (AV-II) POV action adjustments were made versus roughly 381.27 and 351.01 neighboring BV maneuvers, i.e., about 1.7% and 1.5% of maneuvers adjusted, maintaining naturalism. - Event enrichment: Over 2000 km simulations, NADE generated many more valuable events (accidents, BV cut-ins, lane conflicts) for both AVs, while such events were absent in NDE runs due to rarity; BV hard-brake counts were similar between environments. NADE also elicited more AV evasive lane changes at small gaps and relative speeds. - Unbiasedness across accident types: In NADE, unweighted accident rate for AV-II was 0.046 accidents per test (adversarially higher). After weighting by likelihood ratios, accident rates for five FARS-like crash types matched NDE within RHW < 0.3, preserving unbiasedness of both total and per-type rates. - Safety performance: Average miles per accident estimated from NDE were 5.13×10^5 (AV-I) and 1.54×10^6 (AV-II). Compared to human drivers on highways (≈4.79×10^5 miles per accident), AV-I is similar, AV-II better. - Efficiency gains: To reach RHW = 0.3 precision: • AV-I: NADE required 8.74×10^4 tests vs NDE 4.39×10^7 (≈500× faster), reducing ≈10 million miles. • AV-II: NADE required 2.32×10^4 tests vs NDE 1.41×10^8 (≈6000× faster), eliminating ≈35 million miles. - Computational savings (500-core HPC): For AV-I, NADE wall-clock times were ~197–325 s (ε=0.5–0.1) versus ~6.89×10^4 s for NDE; for AV-II, NADE ~4–17 s (ε=0.3–0.1) versus ~2.33×10^5 s for NDE. - Theoretical support: A variance bound shows dependence on divergences over low-dimensional critical variables rather than full dimensionality, addressing the curse of dimensionality while ensuring unbiasedness.

The findings demonstrate that sparsely perturbing a naturalistic environment at critical moments—guided by maneuver criticality and reinforced by defensive importance sampling—can drastically accelerate rare-event evaluation without compromising statistical fidelity to real-world accident rates and types. By adjusting only about 1.5–1.7% of surrounding maneuvers, NADE preserves overall naturalism yet provokes informative, safety-relevant interactions, leading to orders-of-magnitude improvements in sample and time efficiency. The unbiasedness with respect to both total and type-specific accident rates confirms that NADE can serve as a drop-in replacement for life-like NDE simulations for evaluation. The approach is scalable to long horizon, multi-vehicle, spatiotemporally continuous testing and can generate diverse adversarial examples useful for AV development and potentially for accelerated training. Theoretically, combining IS on a small set of critical variables (POV actions at critical times) with CMC on the remainder mitigates the dimensionality-induced variance growth that hampers conventional IS, thereby addressing both rarity and high dimensionality.

The paper introduces NADE, a naturalistic-and-adversarial testing environment for AVs that learns when and how background vehicles should execute adversarial maneuvers. Contributions include: (1) a continuous, data-driven NDE generator reproducing naturalistic behavioral distributions; (2) a principled NADE framework that identifies POVs via maneuver criticality and applies defensive importance sampling; (3) surrogate-model- and RL-based estimation of maneuver challenges; and (4) theoretical guarantees of unbiasedness and improved efficiency by focusing IS on low-dimensional critical variables. Experiments on highway driving show multiple orders-of-magnitude acceleration with preserved accuracy of accident rates and types, and only sparse deviations from naturalism. Future directions include extending NADE to urban settings and mixed traffic participants, enlarging action spaces, integrating perception-related perturbations (e.g., weather, sensor effects) via adversarial perception models, improving surrogate models with deep RL to reduce maneuver-challenge approximation errors, and leveraging NADE for accelerated AV training.

- Case study scope: Highway-only scenarios with simplified action spaces (discrete accelerations, lane changes) and vehicle-only interactions; at most one POV considered per time step. - Perception not targeted: NADE as implemented does not include perception-challenge variables (e.g., adverse weather, sensor artifacts); extension requires measuring perception challenge and identifying critical perception variables. - Data requirements: Accurate NDE/NADE modeling needs large-scale naturalistic driving data with relative positions and speeds for ego and surrounding vehicles; complex environments may require millions of data points. - Approximation error: Maneuver-challenge estimates rely on surrogate AV models and AV maneuver prediction; mismatches and prediction errors can reduce efficiency (though defensive IS mitigates bias).