The development and deployment of autonomous vehicles (AVs) are rapidly transforming transportation systems, but safety remains a paramount concern. Simulation offers a valuable tool for developing and testing AVs, providing a controlled, efficient, and cost-effective environment. However, for simulation to be truly effective, it must achieve statistical realism, accurately reflecting the distribution of real-world driving scenarios, particularly safety-critical events. The complexity and high dimensionality of real-world naturalistic driving environments (NDEs), coupled with the rarity of safety-critical incidents, pose significant challenges to creating statistically realistic simulators. Existing simulators often focus on vehicle fidelity but fall short in accurately modeling background road user behavior. Background agents' behaviors are either replayed from logged data or simulated using oversimplified rules, creating a gap between simulation and reality. Accurately modeling human driving behavior is crucial for high-fidelity NDE simulation. While microscopic traffic simulators using car-following, lane-changing, and gap-acceptance models have been developed, their fidelity is limited by underlying parametric models and manually encoded rules. Neural network, Markovian, Bayesian, and game-theoretic approaches have shown promise in modeling specific behaviors or scenarios, but generalization and scalability to complex urban environments remain challenging. This study focuses on building a high-fidelity simulator that statistically represents real-world driving environments, particularly for long-tail safety-critical events like crashes and near-misses. This distinguishes our proposed NeuralNDE model from existing imitation learning-based simulators, which often lack statistical realism and may produce unrealistic crash rates or short simulation durations. Methods based on real-world event reconstruction also have limitations, particularly in reconstructing near-miss events due to data scarcity. The lack of statistical realism can mislead AV development by providing optimistic safety performance estimates. The high dimensionality, rarity of safety-critical events, and potential for distribution shift in long-time simulations further complicate NDE modeling.

Existing simulators for autonomous driving, such as CARLA, CarCraft, SimulationCity, Tesla's simulator, AirSim, DRIVE Sim, AADS, and Cruise's simulator, primarily focus on vehicle fidelity rather than the statistical realism of the driving environment, particularly the behavior of background agents. Microscopic traffic simulators like SUMO, VISSIM, and AIMSUN, while incorporating car-following, lane-changing, and gap-acceptance models, have limited fidelity due to their parametric nature and reliance on hand-crafted rules. Attempts to improve these models using neural networks, Markovian models, Bayesian networks, and game theory have shown some success in specific scenarios, but struggle with generalization and scalability in complex urban environments. Imitation learning techniques, including generative adversarial imitation learning, have been applied, but often fail to achieve statistical realism, leading to unrealistic crash rates (e.g., SimNet) and short simulation times (e.g., D2Sim). Real-world event reconstruction methods, while valuable, face challenges reconstructing near-miss events due to data limitations. The lack of statistical realism in these existing simulators hinders the accurate training and testing of autonomous vehicles.

NeuralNDE, a deep learning-based framework, addresses the challenge of simulating naturalistic driving environments (NDEs) with statistical realism. The framework uses imitation learning, leveraging large-scale real-world vehicle trajectory data. A behavior modeling network, employing a Transformer architecture, takes in historical states of all road users and predicts their future actions as a joint probability distribution (multivariate Gaussian). This network accurately models normal driving conditions but may not accurately represent safety-critical situations due to data scarcity. To address this, a conflict critic module monitors generated trajectories and probabilistically accepts or rejects potentially dangerous behaviors, thus controlling the frequency and patterns of safety-critical events. The acceptance probability is calibrated to match real-world safety statistics. A safety mapping network, pre-trained using physics-based rules, rectifies unsafe actions to ensure feasibility. Generative adversarial training (GAN) further enhances realism by employing a discriminator network that distinguishes between real and simulated trajectories, improving the distribution's accuracy. The simulation process iteratively updates road user states using the behavior modeling network, conflict critic module, and safety mapping network. The Transformer architecture enables efficient modeling of multi-agent interactions and long-term dependencies. The frequency encoding layer improves the model's ability to capture high-frequency variations in state spaces. The training process involves minimizing a negative log-likelihood loss for the behavior modeling network, combined with an adversarial loss to improve the realism of the generated trajectories. The safety mapping network is trained separately using a physics-based safety guard as the target. The conflict critic module's acceptance probabilities are calibrated in two steps: first, to match the overall crash rate, and second, to match the distribution of crash types. The entire pipeline is trained end-to-end. Two multi-lane roundabout environments (one in the US and one in Germany) were used for validation. The simulation includes vehicle generation and departure, with each episode lasting 3600 seconds and a 0.4-second resolution. Evaluation metrics include vehicle speed and distance distributions, yielding behavior, crash rate, crash type and severity distributions, and post-encroachment time (PET) distribution. Hellinger distance and KL-divergence measure the similarity between simulated and real-world distributions.

NeuralNDE demonstrates accurate reproduction of both normal and safety-critical driving statistics. In the Ann Arbor roundabout environment, NeuralNDE accurately replicates vehicle speed, distance, and yielding behavior distributions, significantly outperforming SUMO. Notably, NeuralNDE achieves a crash rate of 1.25 × 10^-4 crash/km, closely matching the real-world ground truth of 1.21 × 10^-4 crash/km. Furthermore, NeuralNDE accurately reproduces the distribution of crash types and severity, validated against police crash reports. The fidelity of NeuralNDE-generated crash events is further supported by comparison with real-world crash videos, showing similarities in crash patterns (angle crashes due to failure to yield, sideswipe crashes due to improper lane usage, and rear-end crashes due to unsafe following distances). NeuralNDE also accurately represents near-miss statistics, including vehicle distance and post-encroachment time (PET) distributions. Ablation studies confirm the importance of the Transformer-based behavior modeling network, generative adversarial training, conflict critic module, and safety mapping network in achieving these results. The scalability of NeuralNDE is demonstrated by its application to a larger road network comprising a four-way intersection and a roundabout. Even in this extended network, NeuralNDE maintains statistical realism, accurately representing normal driving statistics (speed, distance) and safety-critical statistics (near-miss distances and PET) in both the intersection and roundabout areas. The simulation speed ratio (simulation time/real-world time) is approximately 0.4, indicating efficient simulation.

NeuralNDE represents a significant advancement in simulating naturalistic driving environments, achieving statistical realism for both normal and, crucially, safety-critical driving conditions. This capability is unprecedented, allowing for more accurate evaluation of AV safety performance. The ability to accurately model long-tail rare events, such as specific crash types and severities, is particularly important for comprehensive AV testing. The use of a Transformer architecture allows for efficient handling of multi-agent interactions and long-term dependencies, contributing to the model's accuracy and scalability. Future work could incorporate the influence of AVs on the behavior of surrounding human-driven vehicles, and explore the integration of NeuralNDE with other high-fidelity AV simulators to create a comprehensive and realistic testing environment. The model's scalability is promising for simulating larger road networks, paving the way for more extensive and realistic AV testing. The framework is adaptable, allowing for different safety guards to be used to refine the safety mapping network. The success of NeuralNDE opens up exciting possibilities for developing safer and more reliable autonomous vehicles.

NeuralNDE offers a novel approach to naturalistic driving environment simulation, achieving unprecedented statistical realism, particularly for rare safety-critical events. This high-fidelity simulation environment is crucial for the rigorous testing and validation of autonomous driving systems. Future research could explore further enhancements to capture the nuanced interactions between AVs and human drivers, integrate NeuralNDE with other high-fidelity simulators, and apply it to a wider range of traffic scenarios and geographical contexts.

While NeuralNDE demonstrates significant progress, some limitations exist. The accuracy of the simulation depends on the quality and representativeness of the training data. The model's ability to generalize to unseen scenarios and traffic conditions needs further investigation. The computational cost of training and running the simulation, although improved by the simulation speed ratio of ~0.4, might be a limiting factor for extremely large-scale simulations. Further research is needed to fully account for the complex interaction between autonomous vehicles and human drivers, which could affect human behavior and thus the accuracy of the simulation.