Digital microfluidics (DMF), particularly electrowetting-on-dielectric (EWOD) systems, offers versatile and programmable control of individual droplets, finding applications in point-of-care testing, cell research, and environmental monitoring. Existing DMF systems rely on basic instructions and often incorporate imaging or impedance sensors for real-time monitoring and feedback control. Image processing methods, including background subtraction, edge detection, and object detection, are used for droplet tracking. However, these methods often struggle with the variable appearances of droplets during manipulation (changes in shape and color). This research addresses the limitation of existing DMF systems by leveraging advancements in convolutional neural networks and semantic segmentation to achieve automated control based on droplet appearance changes. The authors note the advantages of semantic segmentation over traditional image processing techniques in complex scenarios due to its precise object segmentation capabilities, support for multiple object categories, and environmental awareness. The challenge lies in establishing a suitable dataset for training the deep learning model, dynamically converting image data into droplet states, and translating state-related control processes into electrode voltage states. The µDropAI system aims to overcome these challenges.

The paper reviews the existing literature on digital microfluidics (DMF) systems and their control mechanisms. It highlights the use of sensors and image processing techniques for real-time monitoring and feedback control. The authors emphasize the limitations of existing methods in handling the variability in droplet appearance during manipulation. A key contribution is the adoption of semantic segmentation, a deep learning technique, which is presented as superior to traditional image processing methods for handling the complexity of droplet states and interactions. The review notes the success of semantic segmentation in other fields, such as autonomous driving and robotics, and proposes its application within the DMF context. The paper points out the lack of studies using semantic segmentation for DMF control and the challenges in creating training datasets and integrating the model into the DMF control process.

The µDropAI system consists of four main parts: (1) a hardware system for droplet actuation and video capture using an ITO-based DMF chip, a high-voltage source, a microcontroller, and a CCD camera; (2) an encoder-decoder semantic segmentation model based on U-Net architecture, optimized for processing DMF droplet images. The model uses stacked convolutional blocks for feature extraction, with varying numbers of layers for shallow and deep feature extraction. Direct twofold upsampling is used in the decoding stage to reduce computational cost. The final layer outputs a 5-channel image representing the four droplet states and background. (3) a region growing algorithm to further refine the segmentation results by connecting similar pixels and defining more precise droplet boundaries; (4) an automated control process implemented using a state machine. This state machine translates user commands (move, split, merge) into electrode-switching sequences. The camera captures images, the semantic segmentation model analyzes the images to identify droplet states and positions, and a feedback control method dynamically adjusts the electrode switching sequence based on the analysis. The DMF chip fabrication involved UV laser processing of ITO electrodes, Parylene C dielectric layer coating, and CYTOP hydrophobic layer coating. A multistate droplet dataset was created by capturing videos of droplet manipulations, converting them into images, and manually labeling them into four states: "unsplit", "splitting", "split", or "merging". The dataset was used to train and validate the semantic segmentation model. The model was trained using the Adam optimizer with cosine annealing learning rate scheduling, and Dice loss was used as the loss function. The mIoU metric was used to evaluate the segmentation performance.

The proposed semantic segmentation model achieved high accuracy in recognizing multistate droplets. The mIoU on the validation set was 89.73%, significantly higher than DeeplabV3+ (82.53%) and traditional U-Net (84.69%). The loss value was 0.11, substantially lower than DeeplabV3+ (0.22) and U-Net (0.14). The model successfully distinguished between the "splitting" state (hourglass shape) and other states, which is crucial for accurate control during droplet splitting. The system demonstrated autonomous control of droplet splitting, movement, and dispensing. The coefficient of variation (CV) of the volumes of split droplets was 2.74%, which is lower than the CV of traditionally dispensed droplets, indicating improved precision. The open-source dataset and the µDropAI framework are publicly available.

The results demonstrate the successful integration of AI, specifically semantic segmentation, into DMF systems for improved control. The high accuracy and precision achieved by the µDropAI system showcase the advantages of using deep learning for recognizing complex droplet states. The improvement in droplet splitting precision, as evidenced by the lower CV, highlights the potential of the system for applications requiring precise volume control. The open-source nature of the dataset and the framework promotes further research and development in AI-assisted DMF systems. The successful differentiation of the "splitting" state is a crucial contribution as this is a challenging aspect of droplet manipulation. The system's compatibility with existing DMF devices broadens its applicability and potential for integration with other AI techniques like reinforcement learning.

This study successfully developed and validated µDropAI, an AI-assisted DMF system for multistate droplet control. The system demonstrated high accuracy in droplet state recognition and improved precision in droplet manipulation. The open-source nature of the dataset and the system promotes future research on advanced AI techniques for fully automated and intelligent DMF systems. Future work could focus on integrating µDropAI with more advanced AI models, such as large language models, for more complex and flexible control.

The current study focused on a limited set of droplet manipulations (splitting, merging, moving). The generalizability of the model to other types of droplet manipulations or different DMF chip designs needs further investigation. The dataset, while substantial, might not fully capture the diverse range of conditions encountered in real-world DMF applications. Further improvements could involve incorporating more sophisticated algorithms for handling noisy or occluded images.