Understanding catalytic reaction mechanisms is crucial for optimizing chemical processes and catalyst design. Traditional methods face challenges due to the complexity of multi-step reactions, short-lived intermediates, and the dynamic nature of catalysis. While Artificial Intelligence (AI), particularly machine learning, offers potential solutions, many existing methods are mechanism-agnostic, requiring human interpretation. Reinforcement learning (RL) shows promise for automated exploration of reaction networks, but faces challenges like non-stationary data, correlated sequences, finite-horizon rollout bias, and the high dimensionality and non-convexity of potential energy landscapes (PELs) in chemical systems. Previous RL applications in chemistry often rely on semi-empirical representations and reaction-specific encodings, limiting generalizability. This research aims to address these limitations by developing a more generalizable and efficient RL framework.

Existing research highlights the use of AI, especially machine learning, in chemistry. However, most approaches are mechanism-agnostic, requiring manual feature engineering. The combinatorial explosion of possible reaction pathways makes exhaustive enumeration impractical. Reinforcement learning (RL) offers a potential solution for automated pathway exploration, but faces challenges in applying it to complex chemical systems. Deep RL, while promising, requires careful state and action representation design, often relying on semi-empirical methods tied to specific reactions, limiting its generalizability. This work addresses these limitations by introducing a reaction-agnostic representation based on atomic positions, overcoming the need for human-defined features and reaction-specific encodings.

The HDRL-FP framework models catalytic reaction pathways as Markov decision processes (MDPs). States are defined by the normalized Cartesian coordinates of migrating atoms and their distances to target positions. Actions are stepwise atom movements in six directions. The reward function is derived from density functional theory (DFT) calculations, using the energy difference between states. HDRL-FP utilizes a high-throughput architecture, running thousands of concurrent RL simulations on a single GPU. This massive parallelism significantly improves training stability and reduces runtime. The actor-critic policy gradient algorithm with proximal policy optimization and the Adam optimizer is used for training. Rollout sampling uses the Monte Carlo method, collecting experience steps efficiently within the GPU's global memory. DFT simulations were performed using VASP software, employing the PBE exchange-correlation functional with D3 corrections for Van der Waals interactions. Nudged elastic band (NEB) calculations were used to refine transition states identified by HDRL-FP. Free energy corrections were calculated using the harmonic approximation and transition state theory was used to estimate reaction rates.

HDRL-FP was applied to the key hydrogenation step (2N∙NH₂ + 2H → 2N∙NH₃ + H) in Haber-Bosch ammonia synthesis on the Fe(111) surface, considering both Langmuir-Hinshelwood (LH) and Eley-Rideal (ER) mechanisms. The results show that the LH and ER mechanisms share the same transition state, with an energy barrier lower than that obtained from NEB calculations alone. High-throughput simulations (with 500 environment replicas) achieved an end-to-end training throughput of 0.23 million experience steps per second on a single GPU, significantly faster than previous methods. The framework's generalizability was further demonstrated by studying nitrogen atom and N₂ molecule diffusion on the Fe(111) surface. For N atom diffusion, both relaxed and unrelaxed DFT calculations, guided by HDRL-FP, revealed similar diffusion pathways. For N₂ diffusion, HDRL-FP predicted a pathway and energy barrier (0.52 eV) that was consistent with NEB calculations. In the Haber-Bosch process, a new, previously unobserved configuration of 2N∙NH₂ + 2H significantly reduced the energy barrier of the hydrogenation step compared to previous findings.

The HDRL-FP framework successfully addresses the challenges of applying RL to complex catalytic reaction mechanisms. The reaction-agnostic representation of states and actions, combined with the high-throughput computational architecture, enables efficient and stable exploration of the complex potential energy landscape. The lower energy barriers found for the key hydrogenation step in Haber-Bosch ammonia synthesis, compared to traditional NEB calculations, highlight the method's potential for discovering more efficient reaction pathways. The successful application to both single-atom (H migration) and two-atom (N₂ diffusion) systems demonstrates HDRL-FP's versatility. The improved understanding of the Haber-Bosch mechanism could lead to the development of more efficient catalysts and process optimization.

This study presents HDRL-FP, a novel high-throughput deep reinforcement learning framework for investigating catalytic reaction mechanisms. HDRL-FP offers significant advantages over existing methods through its reaction-agnostic representation and high-throughput capabilities, allowing efficient exploration of complex chemical systems. The successful application to ammonia synthesis and other diffusion processes showcases its potential to accelerate catalyst discovery and process optimization. Future work should explore the framework's application to a wider range of catalytic reactions and incorporating more advanced DFT techniques to account for anharmonic effects.

The study primarily uses the harmonic approximation for free energy corrections. Anharmonic effects, potentially significant at high temperatures, could affect the accuracy of free energy barriers. While the framework handles both single- and two-atom systems, extending it to reactions involving many more atoms might require further optimization. The current implementation relies on well-defined constraints for relaxed calculations, and extending this approach to less constrained environments requires further investigation.