The advancements in deep learning have yielded high-performance image interpretation algorithms, including object detection, semantic segmentation, instance segmentation, and image generation. These algorithms leverage the ability of neural networks to learn high-dimensional hierarchical features from extensive training data, achieving high generalization capabilities. While deep learning shows promise in automating tasks like medical diagnosis and biological image analysis, implementation remains challenging due to the need for joint development of hardware, datasets, and software. In the field of 2D materials, autonomous robotic assembly systems have accelerated the search for exfoliated materials and their assembly into van der Waals heterostructures. However, existing image recognition algorithms often rely on rule-based image processing using handcrafted features (color contrast, edges, entropy), requiring expert parameter tuning, which is time-consuming and potentially damaging to sensitive materials. This research aims to develop and implement a deep-learning-based solution to overcome these limitations, creating a robust and adaptable 2D material detector.

The paper reviews existing deep-learning approaches to image segmentation, categorizing them into fully convolutional approaches (SegNet, U-Net, SharpMask) and region-based approaches (Mask-RCNN, PSP Net, DeepLab). It highlights that region-based approaches often outperform fully convolutional methods, especially with sufficiently large annotated datasets. The authors also acknowledge previous work on autonomous robotic systems for 2D material searching and the limitations of conventional rule-based image processing in this context.

The researchers developed a deep-learning-assisted optical microscopy system comprising three components: a motorized optical microscope, a customized software pipeline for image capture, deep-learning algorithm execution, result display, and database recording; and a set of trained deep-learning algorithms for detecting 2D materials (graphene, hBN, MoS₂, and WTe₂). The core of the system is a Mask-RCNN model, built on the Keras/TensorFlow framework using the ResNet101 convolutional neural network as its backbone. The model predicts object location, bounding boxes, and segmentation masks. To train the model, a dataset of approximately 2100 annotated optical microscope images of the four target 2D materials was created. The images were manually annotated using a web-based labeling tool. Training was performed using a multitask loss function incorporating classification, localization, and segmentation mask losses. Transfer learning from a model pre-trained on the Microsoft Common Objects in Context (COCO) dataset was employed to improve performance, especially given the relatively small size of the 2D material dataset. The training process involved data augmentation techniques (color channel multiplication, rotation, flips, shifts) to increase the dataset size and improve model generalization. The training was structured in two steps: initial training on a mixed dataset of all four materials followed by transfer learning on individual material subsets to optimize for layer thickness classification. The software pipeline uses a server-client architecture for integrating deep learning inference with the optical microscope, allowing for remote operation. Performance was evaluated using precision and recall metrics based on manual checks of over 2300 images.

The developed Mask-RCNN model demonstrated effective detection of various 2D materials with accurate segmentation masks. The inference process was robust against changes in illumination conditions, unlike conventional rule-based methods. The system achieved a frame rate of approximately 1 fps, including image capture and result display overheads. Using WTe₂ as a test case, the automated search identified about 25 flakes on a 1 cm² substrate in one hour using a 50x objective lens. The quantitative analysis of the Mask-RCNN performance yielded a precision of ~0.53 and recall of ~0.93 for WTe₂, indicating that while some false positives occurred, the system reliably detected the usable 2D flakes. Graphene detection showed significantly higher precision and recall (~0.95 and ~0.97, respectively). The model's generalization ability was confirmed by successfully detecting graphene in images obtained from different optical microscope setups with varying conditions (white balance, magnification, resolution, illumination). Transfer learning significantly improved model performance, demonstrating the effectiveness of leveraging features learned across different materials. The training process is outlined, and training curves show the effect of data augmentation on model convergence and generalization.

The successful implementation and integration of a deep-learning-based image segmentation algorithm with a motorized optical microscope significantly improves the efficiency and robustness of searching for 2D materials. The algorithm's robustness to changes in microscopy conditions eliminates the need for constant parameter retuning, a major advantage over conventional methods. The high recall of the system ensures minimal loss of usable 2D flakes, which is crucial for practical applications. The server-client architecture makes the system accessible to researchers with limited deep learning infrastructure. The publication of the source code, model weights, and training dataset facilitates wider adoption and further development of the technique.

This study demonstrates a significant advancement in the automated search for 2D materials. The deep-learning-based system outperforms existing rule-based methods by offering robustness, speed, and ease of use. The availability of the code and data allows the community to build upon this work, expanding its applications to other 2D materials and advancing the development of fully automated systems for van der Waals heterostructure fabrication.

While the system shows high performance, the manual annotation of the training dataset remains a bottleneck. The current model's ability to classify layer thickness is limited to a coarse categorization (monolayer, few layers, thick layers), requiring further refinement for precise characterization. The generalization to entirely new 2D materials might require additional training data although some generalization is shown.