Machine learning (ML) has revolutionized electron microscopy, finding applications in atom localization, defect identification, image denoising, crystal structure classification, and microscope alignment. A major hurdle in applying ML to materials research is the need for large amounts of high-quality training data paired with ground truth labels. While simulated data offers a convenient source of both images and labels, simulated scanning transmission electron microscopy (STEM) images differ significantly from experimental images due to factors like detector noise, sample drift, alignment errors, radiation damage, and contamination. These discrepancies hinder the accuracy and generalizability of ML models trained on simulated data, requiring retraining for each change in experimental conditions. The researchers propose using a CycleGAN to bridge the gap between simulated and experimental STEM data, generating realistic training data while preserving ground truth labels for ML applications.

The introduction extensively reviews the existing literature on the application of machine learning to electron microscopy, highlighting various applications and the challenges associated with using simulated data for training. It cites numerous publications showcasing the use of ML in different aspects of electron microscopy, ranging from atom localization and defect identification to image processing and microscope automation. The review emphasizes the limitations of relying solely on simulated data due to the discrepancies between simulated and experimental images.

The core methodology revolves around using a CycleGAN to translate simulated STEM images into realistic images that closely resemble experimental data. The CycleGAN architecture incorporates both real-space and Fourier-space discriminators, ensuring that the generated images are realistic in both real and reciprocal space. The generators (G and F) learn mappings between the experimental and simulation domains, aiming to produce images that fool the discriminators. The training process minimizes a combination of adversarial losses, cycle-consistency loss, and identity loss, ensuring that the generated images are realistic and that the mappings are reversible and preserve important features like atomic defects. A fully convolutional network (FCN) is then trained on the CycleGAN-processed images and their corresponding defect labels to identify atomic defects. The experimental data consisted of large-scale, automatically acquired STEM images of various materials (graphene, WSe₂, SrTiO₃). Simulated images were generated using the incoSTEM package in Computem with minimal parameter optimization. The CycleGANs were trained separately for each material system due to the limitations of CycleGANs in handling large shape changes. Quantitative evaluation of the generated images was performed using the Fréchet Inception Distance (FID) and Kullback-Leibler (KL) divergence, comparing the generated images to experimental images. The FCN performance was evaluated using precision, recall, and F1 scores on manually labeled experimental test sets.

The CycleGAN successfully generated realistic STEM images that were quantitatively similar to experimental data, as demonstrated by low FID and KL divergence scores. The CycleGAN-processed images showed a significantly better match to experimental images than simulated images with or without manually added noise. The FCN trained on CycleGAN-processed images achieved high precision and recall in identifying atomic defects in experimental images, comparable to FCNs trained on manually optimized simulated data but with far less human intervention. The performance of the FCN was best when the CycleGAN was trained on the same dataset as the test data, highlighting the CycleGAN's ability to adapt to specific microscopy conditions. Using a small number of experimental images (as few as 6) for CycleGAN training proved sufficient to achieve comparable FCN performance, suggesting the potential for dynamic model adaptation.

The results demonstrate the effectiveness of using CycleGANs to generate realistic training data for ML applications in electron microscopy. The CycleGAN approach significantly reduces the need for manual parameter optimization and enables the development of more adaptable and robust ML models. The high performance of the FCN trained on CycleGAN-generated data suggests that this approach can significantly improve the efficiency and accuracy of automated data processing in large-scale microscopy experiments. The ability of the CycleGAN to adapt to changing experimental conditions by retraining with a small number of new images opens the door to real-time, dynamic ML models in microscopy. The authors suggest that their approach could be generalized to other imaging techniques.

This study presents a novel CycleGAN-based approach for generating realistic STEM images, significantly improving the performance of ML models for defect identification. The method drastically reduces the human intervention needed for training ML models, making automated and high-throughput data analysis feasible for large-scale experimental datasets. Future research could explore the application of this approach to other imaging modalities and more complex ML architectures.

The study focuses on specific materials and defect types. The generalizability of the CycleGAN approach to other materials, defects, or microscopy techniques needs further investigation. While the authors demonstrate the potential for dynamic model adaptation with limited retraining data, further exploration is needed to optimize the retraining process and determine the optimal number of images required for accurate adaptation. The reliance on manual labeling for evaluating FCN performance on experimental data remains a potential limitation.