Deep learning, particularly convolutional neural networks (CNNs), has revolutionized computer vision, finding applications in image classification, object detection, and semantic/instance segmentation. In agriculture, CNNs are increasingly used for image-based phenotyping, including weed detection, disease diagnosis, and fruit detection. However, a major challenge is the need for large, labeled datasets, which are often difficult and expensive to create manually. While some specialized agricultural tasks require fewer annotations, the optimal amount remains unknown, and manual annotation remains a significant bottleneck. To mitigate this, researchers are turning to synthetic data and sim2real transfer learning. One promising approach is domain randomization, which involves training networks on synthetic images with wide variations in randomly sampled physical parameters, bridging the gap between synthetic and real-world data. Previous studies have shown the use of synthetic data in plant image analysis, for example, in estimating branching patterns or segmenting leaves. Seed morphology, encompassing seed size and shape, is a crucial agricultural phenotype affecting yield and germination. Existing methods for seed shape analysis are often labor-intensive, involving manual annotation or qualitative assessments. This study aims to leverage synthetic data with domain randomization to train an instance segmentation network for high-throughput, accurate crop seed phenotyping, thereby overcoming the limitations of manual annotation and enabling large-scale analysis.

Deep learning's success in computer vision has led to its application in various agricultural tasks, such as weed and disease detection and fruit identification. However, the need for large labeled datasets is a major hurdle. While some studies have successfully trained networks with relatively small datasets for specific tasks (e.g., sorghum head detection, crop stem detection), the generalizability of these findings remains limited. To address the annotation burden, synthetic data generation is becoming increasingly popular. Several studies have successfully used synthetic images, often generated with 3D models or Generative Adversarial Networks (GANs), to train plant image analysis networks. Domain randomization, a technique that leverages variations in synthetic images to improve the model's generalization to real-world data, has also shown promising results in robotics and object recognition. Previous work in plant phenotyping has explored synthetic data for leaf segmentation and counting. However, a comprehensive application of domain randomization to crop seed phenotyping, particularly handling densely packed seeds, remains unexplored.

This research utilizes a synthetic dataset generated through domain randomization to train a Mask R-CNN instance segmentation network for barley seed phenotyping. The process began with scanning images of barley seeds from 20 cultivars. Single-seed images were extracted, cleaned (background removed), and compiled into a "seed image pool." A "background image pool" was also created from the scanner background. Synthetic images were generated by randomly placing rotated seed images from the pool onto virtual canvases with background images. Simultaneously, ground-truth masks were created, assigning unique colors to each seed region. The Mask R-CNN was trained on these synthetic image-mask pairs. The model's performance was evaluated using a real-world test dataset of barley seed images from the same cultivars (images not used in synthetic data generation). The evaluation metrics included recall (at 50% Intersection over Union (IoU) threshold for bounding boxes) and average precision (AP) (at varying IoU thresholds for instance masks). To enhance the accuracy, a post-processing step was implemented to remove occluded or incompletely segmented seeds, non-seed objects, and seeds partially outside the image boundaries. The processed output was compared with manually annotated data to validate accuracy. Finally, for comprehensive morphological characterization, various analyses were performed on the processed output, including analysis of variance (ANOVA), principal component analysis (PCA) using simple morphological features and elliptic Fourier descriptors (EFDs), and latent space visualization using a variational autoencoder (VAE). The methodology was later extended to evaluate its effectiveness on other crops (rice, oat, lettuce, and wheat).

The Mask R-CNN model trained exclusively on the synthetic dataset achieved high accuracy in segmenting barley seeds in real-world images, with 96% recall and 95% AP50. The synthetic test dataset showed even higher AP values (73% AP@[.5:.95] and 96% AP50) than the real-world test dataset (59% AP@[.5:.95] and 95% AP50), which could be attributed to data leakage. The post-processing step effectively improved data quality, resulting in a strong correlation (Pearson correlation 0.97) between the processed seed area estimates and manual measurements. Multivariate analyses (ANOVA, PCA with simple features and EFDs, and VAE) revealed significant morphological differences among barley cultivars, allowing for classification and identification of key features. The method was successfully extended to other crops, demonstrating its broad applicability and efficacy in diverse seed morphologies. The VAE, in particular, showed promise in capturing complex features not easily captured by traditional methods.

This study successfully demonstrated the feasibility and effectiveness of using synthetic data generated through domain randomization to train instance segmentation networks for high-throughput seed phenotyping. The high accuracy achieved on real-world images, despite training solely on synthetic data, validates the efficacy of domain randomization in bridging the sim2real gap. The results highlight the potential of this method to significantly reduce the time and cost associated with manual data annotation in plant phenotyping. The ability to analyze densely packed seeds is a considerable advantage over traditional image analysis methods. The findings of the multivariate analyses provide valuable insights into the morphological diversity of barley cultivars and other crops, paving the way for enhanced genetic studies.

This research introduces a novel and highly efficient approach to crop seed phenotyping using a synthetic dataset and domain randomization to train an instance segmentation network. The method significantly reduces the reliance on manual annotation, enabling high-throughput, accurate analysis of seed morphology across multiple cultivars and species. The results demonstrate the broad applicability and scalability of this approach, promising to accelerate research in plant genetics and breeding. Future work should focus on optimizing the synthetic data generation parameters, exploring additional image augmentation techniques, and refining the post-processing steps to further enhance the robustness and accuracy of the pipeline.

The study's reliance on a specific scanner and image acquisition method could limit the generalizability of the findings. The effectiveness of the post-processing method is dependent on the assumption of a homogeneous seed population and may require adjustment for heterogeneous populations. The 'black box' nature of the neural network means a complete understanding of the learned features and their relation to real-world seed characteristics remains challenging. The impact of resolution and variance of the seed images used to generate the synthetic dataset requires further investigation.