Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline

The study addresses the challenge of directly linking high-throughput microscopy data with atomistic simulations to study experimentally observed structures. Electron and scanning probe microscopy provide atomically resolved, dynamic views of materials, yet atomistic simulations (DFT, MD) operate on different length and timescales and with higher computational latencies. The purpose is to create an end-to-end, ML-driven pipeline that transforms STEM images into simulation-ready atomic coordinates, enabling selection of regions of interest and subsequent first-principles and molecular dynamics studies. This integration aims to overcome time/latency disparities, quantify uncertainties in feature extraction, and establish a feedback loop to guide experiments.

The paper surveys advances in STEM/STM enabling atomically resolved imaging of impurities, defects, and excitations, and highlights the promise and challenges of deep learning for feature extraction and property prediction from microscopy. It reviews multi-scale simulation methods (DFT, MD, MC, AIMD) and the growth of databases accelerating materials discovery. Prior frameworks bridging experiments and theory include Ingrained, EXSCLAIM, BEAM, and abTEM, as well as studies modeling electron beam effects and atomic manipulation. Extensive literature on 2D materials reports defect formation and dynamics from combined STEM and simulations. However, comprehensive DL-based workflows that map directly from experimental images to atomistic simulations with uncertainty quantification and feedback remain limited, motivating the present work.

Workflow: A three-stage pipeline links microscopy to simulations. Stage 1: Deep learning segmentation of STEM images to detect atom types and positions; Stage 2: conversion of high-confidence predictions into simulation objects (supercells with periodic boundary conditions) via AtomAI utilities; Stage 3: DFT geometry optimization followed by ab initio MD (AIMD) and, for scalability, DFTB/MD. Feature extraction (DL): Graphene STEM frames (896×896 pixels; 100 frames) with pixel size 0.104 Å are labeled for semantic segmentation (C, Si, background). A U-Net architecture (example block depths 2-3-3-4-3-3-2) is trained; ensemble learning (typically 20 models) yields robust predictions and pixel-wise uncertainty (standard deviation across ensemble). High-uncertainty regions indicate possible unknown species/defects for targeted simulations. From predicted blob masks, (x,y) centers are extracted to obtain atomic coordinates. Simulation object construction: Predicted atoms are placed in user-defined cells (cubic or other Bravais lattices) ensuring realistic bond lengths and PBCs. For cropped patches, twofold-coordinated edge C atoms are H-terminated to maintain charge neutrality/stoichiometry. Example: full frame detection found 2021 C and 22 Si atoms; a representative 91-C-atom patch (H-terminated) was selected for simulations. DFT geometry optimization: Performed with VASP using GGA-PBE and PAW potentials. Initial cell: a=b=c=25.786 Å, α=β=γ=90°, 91 C atoms plus H termination of twofold C. Relaxation until forces < 1e-3 eV/Å; plane-wave cutoff 400 eV; Monkhorst-Pack k-mesh 2×2×2 for geometry optimization. Configurations explored: (a) all perimeter atoms fixed; (b) x,y fixed with z-only relaxation; (c) all H atoms fixed; and full relaxation of all atoms and cell. AIMD: Starting from fully optimized H-terminated structure. VASP PAW-PBE, 400 eV cutoff, appropriate Monkhorst-Pack meshes. NVT via Nose–Hoover thermostat, 2000 steps with 1 fs timestep. Temperatures: 300, 500, 700, 900, 1200, 2000, 3000, 4000, 5000 K. Ad-atom studies: Cr ad-atom placed at bridge, hollow, and top sites; magnetism was neglected (non-spin-polarized). Hydrocarbon ad-atoms CH, CH2, CH3 placed in several initial configurations near hole/edge regions; AIMD at 300 K for 2 ps. Adsorption energy computed as E_adsorbate = E_system + E_adsorbate − E_(system+adsorbate). DFTB: DFTB+ (v21.1) integrated for larger-scale calculations, enabling optimization and MD for the full ~2043-atom graphene/Si system in hours, to capture nonlocal effects. Samples and imaging: Graphene grown by AP-CVD; STEM imaging on Nion UltraSTEM 200 (further details in cited refs). Computational performance: Training a typical U-Net on ~4500 sub-images (256×256) on an NVIDIA Tesla K80 takes ~0.5 h; ensemble of 20 models can be trained sequentially or in parallel. Feature prediction and simulation-object creation take seconds. DFT geometry optimization for 91-atom patch ~10 CPU hours; similar for 2-ps AIMD. Geometry optimization of full 2043-atom structure via DFTB takes a few hours.

- DL-based detection: From a graphene STEM frame, DL identified 2021 C and 22 Si atoms; minor misclassifications were mitigated by selecting high-certainty regions for simulation. - Geometry reconstruction (DFT): A 91-C-atom H-terminated patch relaxed with GGA-PBE reproduced realistic bonding. Fully relaxed average bond lengths: C–C ≈ 1.42 Å; C–H ≈ 1.09 Å. With selective constraints, C–C and C–H bonds varied in ranges 1.43–1.50 Å and 0.94–1.04 Å, respectively, consistent with charge-density variations near defects. Comparison of DL-predicted and DFT-optimized C atom positions showed <5% average error in x and y. - Temperature-dependent dynamics (AIMD): Up to ~1200 K, ring angles varied modestly (1–5 degrees) and structures showed limited reconstruction. At 2000 K, hexagonal rings remained with average bond length ~1.422 Å. At 4000 K after 2 ps, defects rearranged to a 5-7-7-5 Stone–Wales configuration; average bond lengths: seven-member rings ~1.430 Å, five-member rings ~1.606 Å. At 300 K, average bond lengths were pentagon 1.713 Å, hexagon 1.443 Å, heptagon 1.447 Å (from marked rings). Tables report detailed C–C distances for 300/1200 K rings and 4000 K 5-7-7-5 defects. - Cr ad-atom at 300 K (non-spin-polarized): Bridge site Cr rapidly diffused into the lattice causing bond breaking within 10 fs. Hollow site formed stable bonds to neighboring C atoms with average Cr–C ≈ 2.165 Å and persisted over the trajectory. Top site showed largest out-of-plane displacement; closest vertical separation ~2.237 Å. Total energies indicated hollow site most stable: bridge −910.514 eV, top −920.305 eV, hollow −923.351 eV; adsorption energies in this nonmagnetic picture exceeded ~−8 eV in magnitude. - Hydrocarbon ad-atoms at 300 K: CH3 frequently promoted partial or full healing of holes within 2 ps; adsorption energies for three CH3 configurations: −3.709, −7.220, −4.306 eV. CH and CH2 tended to bind to edge/neighboring C rather than fully heal at 300 K; adsorption energies across tested configurations ranged from −7.018 to 14.549 eV, reflecting strong dependence on initial geometry and local environment. - Scalability via DFTB: Enabled optimization and MD on the full ~2043-atom graphene/Si cell in hours, facilitating assessment of nonlocal effects (e.g., in self-healing). - Throughput and latency: DL inference plus simulation-box construction occur within seconds; representative DFT and AIMD runs complete within hours, making the pipeline amenable to near-real-time feedback during STEM.

The results demonstrate that a DL-to-simulation pipeline can transform raw STEM imagery into accurate atomistic models suitable for first-principles calculations. Ensemble DL segmentation provides robust atomic coordinates with uncertainty maps; selecting high-confidence regions yields reliable initial states for DFT, bridging experimental observations and theoretical modeling despite timescale and latency disparities. DFT reconstructions closely match DL-derived structures, validating image-based coordinate extraction. AIMD reveals temperature-dependent defect dynamics, including formation of Stone–Wales 5-7-7-5 defects at elevated temperatures, connecting observed defect motifs to thermal conditions. Ad-atom simulations show site-dependent stability and diffusion (Cr) and plausible healing mechanisms with hydrocarbons, offering insights to guide experimental exploration of doping and repair under electron-beam conditions. Integrating DFTB expands system sizes to thousands of atoms, capturing nonlocal effects while maintaining turnaround times compatible with experimental feedback. The framework supports a human-in-the-loop paradigm and adheres to FAIR principles, enabling active causal feedback from simulations to refine DL models, focus on regions of interest, and inform subsequent microscopy conditions and targets.

The study establishes an end-to-end workflow that directly couples STEM imaging with deep learning, DFT, AIMD, and DFTB to investigate real, experimentally observed graphene structures. It demonstrates automated feature extraction with uncertainty quantification, construction of simulation-ready cells, accurate geometry reconstruction, and temperature- and chemistry-dependent dynamics, including transition-metal adsorption and self-healing by hydrocarbons. The pipeline reduces latency between observation and simulation, enabling on-the-fly or near-real-time feedback to experiments, and is generalizable beyond graphene. Future work includes expanding training data to improve out-of-distribution robustness, integrating spin polarization and van der Waals interactions for more accurate adsorption/interaction energetics, extending timescales and sampling for defect dynamics, leveraging edge computing for real-time deployment, and closing the active feedback loop to guide experimental acquisition adaptively.

- Temporal and computational constraints: AIMD trajectories were short (≈2 ps) and may miss rare events; DFT runs are still resource-intensive. - Methodological approximations: GGA-PBE without explicit dispersion may underrepresent van der Waals interactions; Cr ad-atom simulations neglected spin polarization; more accurate quantum methods were beyond scope. - DL generalization and uncertainty: Misclassifications can occur under varying experimental conditions; reliance on high-confidence regions mitigates but does not eliminate out-of-distribution effects. - Limited configurational sampling: Only a few initial geometries/sites were explored for ad-atoms and hydrocarbons; adsorption energies showed strong dependence on initial configurations. - System focus: Demonstrations centered on graphene; broader material generalization requires retraining and validation. - Incomplete affiliations for some co-authors in the provided text restrict precise attribution here.