Electron and scanning probe microscopies have become crucial tools for studying materials at atomic to mesoscopic scales. Advances in (S)TEM and scanning tunneling microscopy (STM) produce high-quality structural and spectral data. Aberration-corrected STEM, in particular, enables the study of single impurity atoms, grain boundaries, orbital and magnetic phenomena, plasmons, phonons, and anti-Stokes excitations. STM, while offering detailed fundamental studies, has lower throughput than STEM. Both techniques can induce structural changes in materials, with STEM offering advantages for exploring metastable configurations and beam-induced processes. Microscopy data can be analyzed using deep learning (DL) models to identify features, predict functional quantities, and create chemical or structural maps. The rise of computational power has also enabled advanced physical simulations, including DFT, MD, and Monte Carlo (MC) methods, offering insights into structural, thermodynamic, and electronic properties. However, integrating experimental data with simulations is challenging due to the significant disparity between the timescales accessible to simulations and microscopy, and the latencies of both methods. This paper aims to bridge this gap using a machine learning workflow.

Numerous studies have utilized datasets from electron microscopy combined with modern DL techniques such as convolutional networks and variational autoencoders to advance materials science understanding. Existing frameworks like Ingrained, EXSCLAIM, and BEAM, along with abTEM, demonstrate the use of literature data to create labeled datasets, optimize structures via forward modeling, and perform scalable data analyses and simulations. Several studies have used STEM to model electron beam effects on various materials, including atom assembly and manipulation. There are also reports on 2D materials exploring defect formation, dynamics, and stability using combined STEM observations and simulations. However, a systematic framework directly mapping experimental observations to computational studies using DL approaches has been lacking. The inherent time-scale differences between STEM observations and MD/DFT simulations (and their respective computational latencies) pose a significant challenge to this integration.

The workflow consists of three stages. Stage 1 involves using deep convolutional neural networks (DCNNs) to identify atomic features (type and position) from (S)TEM images. A U-Net type neural network is trained on labeled images to perform semantic segmentation, identifying atoms and defects pixel by pixel. An ensemble approach is used to improve robustness and provide uncertainty estimates. Stage 2 involves creating simulation objects (supercells) using the AtomAI framework, incorporating the predicted atomic coordinates. This step accounts for uncertainties in predictions and selects regions of interest. Stage 3 involves performing DFT and AIMD simulations to find optimized geometries and study temperature-dependent dynamics. DFT calculations, using VASP within the GGA framework, provide optimized structures that are then used as input for AIMD simulations. Selective dynamics configurations were also employed, fixing certain atoms to explore the impact of different constraints on the optimization process. Temperature-dependent simulations explored the behavior of the system with and without ad-atoms. The adsorption energy is calculated for various configurations. DFTB calculations were also performed to assess the possibility of simulating larger systems using faster, approximate methods. The workflow leverages Python-based codes for computational efficiency and open-access implementation.

The DCNN accurately identifies atomic features in graphene images, providing coordinates for building simulation objects. DFT simulations successfully reconstruct graphene geometries from the DL-predicted coordinates, with an average error of <5% in x and y directions. AIMD simulations reveal temperature-dependent dynamical evolutions of the graphene structure. At higher temperatures (4000K), the system reconstructs to form 5-7-7-5 defects. Studies with Cr ad-atoms show that the hollow site is the most stable adsorption site. Simulations with CHx (x=1,2,3) ad-atoms demonstrate graphene healing mechanisms, with complete or partial healing depending on the initial configuration and energy landscapes. Adsorption energies for CHx vary considerably depending on configuration and temperature. The workflow successfully bridges the gap between microscopy data and atomistic simulations, enabling the exploration of graphene physics at different length and time scales. The use of ensemble learning and uncertainty quantification improves the robustness and reliability of predictions. DFTB simulations offer faster computation times for larger systems.

The developed workflow successfully addresses the challenge of integrating (S)TEM data with DFT and MD simulations. The use of DL significantly speeds up the process of feature extraction and region selection from microscopy images, overcoming the timescale mismatch between experiments and simulations. The combination of DFT and AIMD simulations provides a comprehensive understanding of the graphene system's behavior at different temperatures and with various ad-atoms, illustrating the workflow's capabilities. The quantification of uncertainties associated with DL predictions helps to refine the simulations and to guide experimental design. The study demonstrates the feasibility of combining experimental and computational techniques in a synergistic manner. Future work could expand the workflow to other material systems and include more advanced theoretical methods for higher accuracy.

This work establishes a novel end-to-end workflow integrating (S)TEM data with DFT and MD simulations using deep learning. The workflow successfully addresses timescale disparities and observational biases, enabling the study of complex material systems like graphene with ad-atoms. This approach accelerates materials discovery and enables active causal feedback between experiments and simulations, improving the efficiency of materials research. Future research could explore integrating edge computing for on-the-fly analysis and expand the workflow to a broader range of materials and simulation techniques.

The current workflow is demonstrated on a specific graphene system. While the approach is generalizable, the transferability to other material systems needs further investigation. The accuracy of the simulations is limited by the chosen DFT functional and the size of the simulated system. Exploring more complex scenarios, such as high-concentration defects or multiple ad-atoms, could require larger simulation cells and more extensive computational resources. Finally, while DFTB methods are used for larger-scale calculations, they are approximations and might not capture all the physical details as accurately as DFT.