Machine vision-based detections of transparent chemical vessels toward the safe automation of material synthesis

The study addresses safety risks in surveillance-free automated chemistry laboratories, where robotic systems handle corrosive or flammable chemicals. Frequent movements and placements of transparent vessels (flasks, beakers, vials) can fail due to robot malfunctions, programming errors, or environmental changes, potentially causing dangerous incidents. While computer vision-based object detection could mitigate these risks, popular detectors such as YOLO and SSD show insufficient accuracy in complex, noisy scenes typical of laboratory environments. Recent transformer-based detectors like DETR improve some aspects but still struggle to aggregate cross-layer information and focus on complex, transparent objects. The authors aim to develop a high-precision, robust detector tailored to detect the positions and failure states of transparent chemical vessels to enhance safety in automated material synthesis labs.

The paper reviews mainstream object detectors: YOLOv3 (single-stage, global image features, limited with irregular shapes and small object groups), SSD (introduces pyramidal feature layers to capture multi-scale global/local features), and DETR (transformer-based, end-to-end detection). While YOLOv6 further leverages pyramidal networks and improves multi-scale feature capture, these models insufficiently aggregate information across layers and have difficulty with complex, transparent objects in noisy backgrounds. For transparent objects, prior work includes ClearGrasp (RGB-D based 3D shape estimation and grasping; slower inference due to depth processing) and segmentation-focused models (Trans2Seg, TransMatting, Trans4Trans), which are not directly comparable for detection tasks. The authors therefore consider transformer-based detectors (DETR, TTG-Net references) as benchmarks but identify a need for improved cross-layer feature aggregation for robust detection of transparent vessels.

System workflow: A robot arm relocates vials from a storage box to holders on a stirrer while a camera provides a bird’s-eye view for machine vision. DenseSSD detects vial positions after each action; upon detecting a failure, the system halts and a safety alert is sent to researchers via a networked messenger module. Model architecture (DenseSSD): DenseSSD integrates a densely connected mechanism within both a mainstream network (dense blocks with direct connections from each layer to all subsequent layers to improve information flow) and a pyramidal feature cascading structure. The pyramidal component consists of feature blocks (FBs) that aggregate multi-scale features and reduction layers (avg pooling + 1×1 conv) to maintain depth and reduce complexity. This design enables richer global and local feature representations while reducing redundancy. Compared with YOLOv3, SSD, YOLOv6, and DETR, DenseSSD reuses feature maps across layers to improve learning and stability. Loss and training: The total loss L_total is a weighted sum of classification confidence loss (softmax cross-entropy) and localization loss (Smooth L1) with α=0.5. Optimization uses SGD with learning rate 1e-4, weight decay 1e-4, momentum 0.9, batch size 64, for 100 epochs on an NVIDIA Tesla V100 GPU. For baselines (YOLOv3, SSD, DETR, YOLOv6), authors fine-tuned official implementations under the same training protocol. Datasets and labeling: Standard 20 ml vials in real automated lab scenes. Initial dataset: 10 videos, 789 extracted images at 45° view; each image labeled vial-wise as success (correctly placed) or failure (fall-out, lie-down, lean-in, stand-on). Data augmentations (random flipping, brightness, saturation, hue, Gaussian filtering) applied to training/validation only. Split 60:40 (learning vs. test) with no overlap; 8,764 vial cases for learning and 1,502 for testing. For combined empty+filled vials: added 359 images with varied solution colors; total 17,174 vial cases for training and 2,282 for testing after augmentation. For camera angle robustness: images at 30°, 45°, 60°, 90°; 2,377 original images; transfer learning initialized from 45° model; 32,715 vial cases for training and 3,648 for testing. Evaluation: Metrics include AP per class and mAP across classes at IoU=0.5; stability assessed via precision-recall AUC. Computational efficiency measured via total parameters and FLOPs. Comparative models: YOLOv3, SSD, DETR, YOLOv6. Generalizability and environments: Tested across different stirring machine layouts (sparse, semi-sparse, dense) and illumination conditions (bright, dark). Also demonstrated adaptation via transfer learning to other vessel types (e.g., cuvettes). Safety alert module: Upon detection of failure cases, system halts hardware and sends images and metadata (time, vial info) to user messengers (e.g., Facebook Messenger, Telegram) via TCP/IP.

- DenseSSD achieved mAP of 95.2% on complex datasets involving both empty and solution-filled vials, surpassing DETR, YOLOv3, YOLOv6, and SSD by 11.3%, 53.4%, 10.5%, and 18.9%, respectively. - Class-wise AP (IoU=0.5) reached 99.9% (success) and 90.5% (failure) in one evaluation; in experiments contrasting training sets, failure AP improved to 90.8% when trained on both empty and filled vials (vs. 67.5% when trained on empty-only). - Stability: PR AUC was 0.97 for DenseSSD vs. 0.75 (YOLOv6), 0.74 (SSD), 0.71 (DETR), 0.49 (YOLOv3), indicating lower false alarms and better robustness. - Efficiency: DenseSSD used 7.9M parameters and 19.2M FLOPs, substantially lower than YOLOv6 (17.9M params, 44.2M FLOPs), SSD (23.9M, 22.5M), DETR (41.0M, 86.0M), and YOLOv3 (320.6M, 24.3M), enabling fast inference. - Angle robustness: mAP across test angles was 88.5% (30°), 94.8% (45°), 93.8% (60°), 84.9% (90°). DenseSSD maintained >93% between 45–60°, outperforming baselines whose mAPs dropped more at 30° and 90°; DETR was relatively consistent but lower (≈82–84%). - Unconstrained environments: DenseSSD outperformed DETR and YOLOv6 across stirring machine densities and lighting. Example values: for “dense” machine type, mAP 62.8% (bright) and 79.9% (dark) with DenseSSD; DETR and YOLOv6 were <49% in dense setups and lower across sparse/semi-sparse as well. Performance was consistently higher in dark backgrounds due to clearer contrast of transparent objects. - Qualitative analyses showed DenseSSD avoided common errors seen in baselines (missed detections, misclassifying failures as successes, duplicate/conflicting boxes), particularly for transparent vials between holders and overlapped objects. - Feature map visualizations indicated DenseSSD formed clearer, localized, and multi-scale representations that distinguished failure modes more effectively than YOLOv3, SSD, YOLOv6, and DETR.

The findings demonstrate that DenseSSD’s densely connected pyramidal architecture effectively aggregates global and local features across layers, enabling precise detection and classification of transparent vials and their failure states in complex, noisy laboratory scenes. High mAP and PR-AUC translate to fewer missed failures and fewer false alarms, directly supporting safety in surveillance-free automated synthesis by enabling immediate halting of operations upon failure detection and notifying responsible personnel via the alert module. DenseSSD’s robustness to solution color, camera angles (especially 45–60°), lighting conditions (notably strong performance in dark scenes), and hardware configurations (sparse to dense arrangements) suggests practical deployability across diverse lab setups. Comparative gains over YOLOv3, SSD, YOLOv6, and DETR in both accuracy and efficiency indicate that DenseSSD can deliver reliable, real-time detection under resource constraints. The successful transfer to other vessel types (e.g., cuvettes) and unconstrained environments underscores the model’s generalizability and potential for broader adoption.

This work introduces DenseSSD, a densely connected, single-shot object detector tailored for detecting positions and failure states of transparent chemical vessels. Across multiple challenging datasets—including empty and solution-filled vials, varied camera angles, and diverse lab configurations—DenseSSD achieved >95% mAP, superior PR-AUC, and lower computational cost compared to leading detectors (YOLOv3, SSD, YOLOv6, DETR). A safety alert module operationalizes the detector for real-time interventions in surveillance-free labs. The approach is robust and generalizable, showing promise for extension to other vessels and complex lab environments. Future work includes advancing adaptability to fully flexible automation with dynamic backgrounds and hardware motion, expanding datasets to more vessel types and configurations, integrating additional sensor modalities if needed, and translating the framework to other domains (autonomous driving, medical imaging, remote sensing) where high-precision detection is critical.

- The robotic manipulation paths and background scenes in this study were largely hard-coded and consistent; performance in highly dynamic, flexible automation systems with frequently changing backgrounds and hardware positions may require additional training data and adaptation strategies. - The primary datasets focus on standard 20 ml vials; broader validation across diverse vessel geometries and materials is partially explored (e.g., cuvettes) but not exhaustive. - Some performance degradation occurs at extreme camera angles (e.g., 30° with overlaps, 90° top-down), suggesting angle configuration remains an important deployment consideration. - While DenseSSD performs well in dark environments due to contrast, performance in highly reflective or glare-prone conditions was not extensively quantified.