Learning naturalistic driving environment with statistical realism

The study addresses a central challenge in autonomous vehicle (AV) development: building simulators that reproduce real-world, safety-critical traffic scenarios with distribution-level accuracy. Despite advances in vehicle dynamics, rendering, and sensor simulation, existing simulators often lack realistic background road user behavior and fail to capture the statistics of rare but safety-critical events. The naturalistic driving environment (NDE) is highly interactive, spatiotemporally complex, and high dimensional, and safety-critical events are extremely rare (“curse of rarity”), compounding the modeling difficulty alongside long-horizon distribution shift in learning-based simulators. The research question is how to learn and simulate multi-agent human driving behaviors such that both normal driving and safety-critical events (crashes and near-misses) match real-world frequency and pattern distributions. The paper proposes NeuralNDE to achieve statistical realism, aiming to close the sim-to-real gap for AV training and testing by accurately modeling human interactions and rare-event statistics.

Prior simulators emphasize vehicle and sensor fidelity but underrepresent realistic human driver behavior (e.g., CARLA, CarCraft/SimulationCity, Tesla’s simulator, AirSim, NVIDIA DRIVE Sim, Baidu AADS, Cruise). Traditional microscopic traffic simulators (SUMO, VISSIM, AIMSUN) rely on physics-driven, rule-based models (car-following, lane changing, gap acceptance), which limit fidelity and generalization in complex, interactive urban scenarios. Learning-based attempts (deep neural networks, Markov models, Bayesian networks, game theory) have improved specific behaviors or scenarios but struggle to scale to complex environments. Imitation-learning-based simulators (including GAIL-like methods) typically neglect distribution-level statistical realism, often yielding unrealistic crash rates and short simulation horizons that preclude full-trip evaluation. Reconstruction-based safety-critical simulation from police reports can reproduce fatal crashes but struggles with near-miss events due to missing information. This work complements prior approaches by explicitly targeting statistical realism, especially for rare events, in a scalable learning framework.

NeuralNDE is a deep learning-based framework designed to reproduce real-world driving statistics for both normal and safety-critical conditions by learning multi-agent interaction behaviors from trajectory data and refining safety-critical event generation. - Behavior modeling network: Formulated as imitation learning with large-scale real-world demonstrations. Each vehicle is modeled as an agent (token) and the network jointly predicts stepwise future action/state distributions for all N agents conditioned on their historical trajectories. A Transformer backbone (BERT-style layers) models inter-agent interactions via self-attention and intra-agent dynamics via feed-forward layers, offering scalability and permutation invariance. Inputs include recent historical states (positions and headings) across a temporal window; frequency encoding (sin/cos basis functions) projects inputs to a higher-dimensional space to better capture high-frequency variations. The model outputs stochastic predictions (e.g., Gaussian means and variances) and samples actions/states to simulate forward with a differentiable state transition. Training uses maximum likelihood via negative log-likelihood, predicting means and implicit variances (diagonal covariance) to capture uncertainty. - Generative adversarial training: To mitigate distribution shift and improve realism, a discriminator (MLP) is trained to distinguish real trajectory rollouts from simulated ones. The behavior modeling network is trained adversarially to fool the discriminator, combining NLL with adversarial loss under a minimax objective. - Conflict critic module: To achieve accurate safety-critical statistics despite rarity in training data, a conflict critic calibrates the acceptance probability of predicted conflicts/crashes during inference. For each predicted conflict/crash type, an acceptance probability pa(j) controls whether to accept the dangerous behavior or route it to the safety mapping network for rectification. Calibration proceeds in two steps: (1) fit the overall crash rate by finding a uniform acceptance probability pua; (2) fit the crash type distribution by setting pa(j) = pua * c_gt(j) / p(j), where c_gt(j) is the ground-truth crash type probability and p(j) is the observed crash type probability under pua. This controls both overall crash rate and crash type composition. - Safety mapping network: A neural mapper, pretrained on physics- and rule-based safety guards, rectifies unsafe actions to their nearest safe counterparts when an imminent crash is detected. Using a Transformer backbone, it takes current states and predicted actions and outputs rectified actions that respect safety constraints (trained via L1 loss to imitate a physics-based repulsive-force safety guard). It is kept fixed during joint training, providing a differentiable safety layer to reduce unrealistic crashes and decouple safety from behavior learning. - Simulation setup: Episodes initialize with a 2-second logged trajectory clip; then all agents are controlled by NeuralNDE. New vehicles arrive per-lane via a Poisson process calibrated to data; vehicles exit upon reaching exit areas. Each episode runs 3600 s with 0.4 s resolution; early termination occurs upon crash. Validation used approximately 15,000 simulation hours (all for crash-related metrics; 100 hours for other metrics). Experiments ran on a high-performance cluster (1000 CPU cores, 2000 GB RAM), with a simulation speed ratio of about 0.4 (simulation time/real time). - Datasets and evaluation: Primary validation uses a two-lane roundabout in Ann Arbor, Michigan (AA dataset) with trajectories (2.5 Hz), crash videos, and police reports for ground-truth safety-critical statistics. Additional normal-driving validation uses the German rounD dataset. Metrics compare simulated vs real distributions of instantaneous speed, inter-vehicle distance, yielding distance/speed, crash rate, crash type, crash severity (Delta-V and injury level), and near-miss PET and distance distributions, using Hellinger distance and KL-divergence. - Network architecture: Behavior model uses frequency encoding (order L=4), input embedding, 4 BERT layers (hidden 256, 4 heads, FFN 512), and prediction heads for stochastic positions and heading over a 5-step horizon (0.4 s step). The discriminator is a 4-layer MLP (1024-512-256-1) with LeakyReLU. The safety mapper shares a similar Transformer architecture and operates frame-by-frame to output rectified states.

- Statistical realism for safety-critical events: NeuralNDE reproduces the real-world crash rate for the studied roundabout: ground truth 1.21 × 10^-4 crash/km vs NeuralNDE 1.25 × 10^-4 crash/km. It also matches crash type and crash severity (Delta-V/injury) distributions derived from police reports (2016–2020). - Near-miss fidelity: Simulated near-miss statistics (closest distance under 10 m and PET distributions) align closely with real-world distributions, indicating accurate modeling of dangerous interactions short of crashes. - Normal driving realism: Instantaneous speed, inter-vehicle distance, and yielding behavior (yielding distance and speed) distributions in the roundabout closely match real data and outperform a SUMO baseline. - Qualitative crash realism: Generated crash scenarios (angle/failure-to-yield, sideswipe/improper lane usage, and rear-end/failure to maintain clear distance) resemble real-world crashes captured by roadside cameras and reports. - Long-horizon stability and scalability: Supports hour-level continuous simulation with interacting agents; a proof-of-concept network (intersection + roundabout) shows the approach scales by controlling critical nodes with NeuralNDE and connecting links with rule-based models, while maintaining statistical realism in both normal and safety-critical metrics. - Practical performance: Simulation speed ratio approximately 0.4 on an HPC cluster, enabling large-scale validation (~15,000 hours).

NeuralNDE directly addresses the need for statistically realistic traffic simulation by modeling multi-agent human driving interactions and explicitly controlling rare-event generation. By combining a Transformer-based behavior model, adversarial training to counter distribution shift, a calibrated conflict critic to match real-world crash frequencies and patterns, and a pretrained safety mapping network to rectify imminent unsafe actions, the framework reproduces both normal and safety-critical statistics with distribution-level accuracy. This statistical realism is crucial for AV development: it reduces sim-to-real gaps, supports comprehensive testing across crash types and severities, and provides realistic exposure to near-miss scenarios that influence AV decision-making and safety validation. The demonstrated fidelity at a complex roundabout and within a small road network suggests generalizability to larger networks by deploying NeuralNDE at critical interaction nodes and using rule-based models elsewhere. Overall, the results indicate that high-fidelity, statistically representative environments can provide a robust foundation for AV training, evaluation, and safety assessment.

This work introduces NeuralNDE, a learning-based simulation framework that achieves statistical realism in naturalistic driving environments, accurately reproducing both normal driving and long-tail safety-critical events. Key contributions include a multi-agent Transformer behavior model, adversarial training to mitigate distribution shift, a conflict critic to calibrate rare-event frequencies and types, and a safety mapping network to rectify unsafe behaviors. Experiments on a real-world roundabout show close matches to crash rates, crash type/severity distributions, near-miss metrics (distance and PET), and normal driving distributions, with qualitative crash examples resembling real incidents. A proof-of-concept network demonstrates scalability by combining NeuralNDE-controlled critical areas with rule-based links. Future work includes scaling to broader traffic networks with heterogeneous scenarios, incorporating interactions specific to AV-human driver dynamics, integrating accelerated testing methodologies with NeuralNDE, extending beyond one-step conflict handling, and leveraging richer data sources for broader generalization.

- Rarity and data dependence: Although the conflict critic addresses rare-event calibration, fidelity still depends on the availability and representativeness of real-world safety-critical data (crash and near-miss). - Modeling scope: Demonstrations focus on roundabouts and a small network; generalization to diverse road types and large-scale networks requires further validation. - One-step safety handling: The conflict critic and safety mapper are described primarily for one-step predictions, which may limit handling of longer-horizon interactions in complex scenarios. - Simplifying assumptions: Diagonal covariance for action/state uncertainty and independence approximations may limit capturing full joint uncertainties among agents. - AV-specific interactions: Human drivers may behave differently when interacting with AVs; current training from human-human interactions may not fully capture AV-induced behavior changes. - Computational resources: High-fidelity simulation and large-scale validation required substantial HPC resources, which may limit accessibility in some settings.