An artificial intelligence-assisted digital microfluidic system for multistate droplet control

The study addresses the need for intelligent, self-adaptive control in digital microfluidics (DMF), where droplets undergo complex, variable operations that change appearance (shape, color) and require precise real-time manipulation. Traditional automated DMF pipelines use imaging or impedance sensors with control algorithms focused mainly on localization and tracking, while appearance-state recognition has been underexplored. Semantic segmentation, successful in complex, multi-class vision tasks, is well-suited to DMF challenges (transparency, uncertain boundaries, high degrees of freedom). The research proposes µDropAI, a semantic segmentation-driven control framework that recognizes multistate droplet morphology and closes the loop via a state machine for robust, automated operations (move, split, merge, dispense). The work aims to create an open dataset, integrate a tailored encoder–decoder model, and translate recognized states into electrode actuation for reliable, fully automated DMF control, with broad compatibility with existing platforms.

The paper situates its contribution within DMF control literature involving EWOD-based actuation and standard image-processing methods (background subtraction, edge detection, object detection, classical segmentation) for noncontact droplet monitoring and tracking, including transparent droplets. It highlights the limitations of these approaches in robustly recognizing dynamic droplet states. Advances in semantic segmentation (e.g., U-Net, DeeplabV3+) offer precise, multi-class segmentation and environmental awareness proven in autonomous driving, robotics, and healthcare, suggesting suitability for DMF. The authors identify gaps: lack of datasets for DMF state recognition, methods to convert image-derived states to dynamic control actions, and translation of state-driven logic into electrode-level voltage sequences—gaps that µDropAI addresses.

System overview: µDropAI integrates (1) a hardware system for droplet actuation and video capture; (2) a deep learning semantic segmentation model tailored for DMF images; (3) a region growing algorithm to extract position and morphology; and (4) automated state-machine control to execute user-programmed droplet operations. Semantic segmentation model: An encoder–decoder architecture optimized from U-Net is used. The encoder stacks convolutional blocks to extract features: shallow blocks with 2 Conv2d (3×3, stride 1) layers to capture edges/shapes rapidly; deeper blocks with 3 Conv2d layers to learn complex semantic features (state, position). Each block uses ReLU activations followed by 2×2 max pooling to reduce spatial size and enhance feature expression. Compared with original U-Net encoders, consecutive conv and pooling reduce parameters while deep stacks improve representation. The decoder employs direct twofold upsampling to quickly restore resolution, reducing computation and memory versus stepwise upsampling. Eight convolutional layers in the decoder match encoder resolutions, with skip connections to retain spatial detail for accurate masks. The final 1×1 convolution outputs 5 channels matching label categories. Input frames are high-resolution (captured at 1920×1080), and outputs are full-resolution labeled images. Region growing for morphology: A region growing algorithm provides pixel-level division of droplet states in each frame and connects segments based on pixel similarity to refine droplets’ boundaries and compute state, position, and size robustly. State-machine control and feedback: User commands (move, split, merge, resend) are translated into specific electrode ON/OFF sequences and delays. After actuation, the camera acquires images; the model infers semantic masks and state/position. Feedback control adjusts electrode sequences dynamically to correct deviations. Control logic formalizes transitions among states including Initial, Unsplit, Splitting, Split, and Merging, with success/failed branches (e.g., resplitting if split fails), enabling autonomous move/split/merge workflows. DMF chip fabrication: ITO-based DMF chip assembled from a bottom plate (52×50 mm, 1.1 mm thick, ~8 Ω/sq) patterned by UV laser (355 nm, 5 W, 20 µs pulses) and an upper plate (20×40 mm, 0.7 mm thick, ~8 Ω/sq). Post-patterning cleaning with IPA and ultrapure water (15 min each), N2 dry. Bottom ITO receives air plasma (2 min) before ~3 µm Parylene C dielectric deposition; pad areas masked. Both plates are spin-coated with CYTOP hydrophobic layer. Plates assembled with ~0.5 mm double-sided tape spacer. Hardware setup: High-voltage source (ATG-2081, Aigtek) is switched via an optocoupler array controlled by MCU (STM32L432), receiving instructions from Python (with AI control). Imaging via microscope and CCD camera (IUA20000KPA, ToupTek) at 2560×1440 resolution. Processing runs on a PC (R7 5700X CPU, 16 GB RAM, RTX 3070 GPU). Dataset: Videos captured at 30 fps; frames extracted to 1920×1080 images using OpenCV. Manual labels (LabelMe) into four states: unsplit, splitting, split, merging, converted to PASCAL VOC format. Train/val split 0.8/0.2 with 3503 train and 876 val images. Dataset released publicly: https://github.com/Eric1105/-DropAI. Training setup and metrics: Experiments in PyTorch (Python 3.8, Windows) using dual RTX 3090 12 GB GPUs. Optimizer: Adam. Initial LR 1e-4 with min 1e-6; cosine annealing LR schedule reducing every 5 epochs for smooth convergence. Loss: Dice loss to handle strong background/object imbalance. Evaluation metric: mean Intersection over Union (mIoU) across classes.

- Semantic segmentation enables accurate recognition of four droplet states (unsplit, splitting, split, merging) and their interactions; the system achieves autonomous control without user intervention. - Recognition flexibility: droplets of different colors and shapes are recognized with an error rate < 0.63%. - Precision in droplet splitting: coefficient of variation (CV) of volumes of split droplets limited to 2.74%, lower than that of traditionally dispensed droplets, improving volume control precision. - Benchmark performance: On the validation set, proposed model mIoU = 89.73%, outperforming DeeplabV3+ (82.53%) and U-Net (84.69%). Validation loss stabilizes at 0.11 vs 0.22 (DeeplabV3+) and 0.14 (U-Net). - Qualitative robustness: Correctly segments hourglass “splitting” neck and distinguishes merging vs split states where DeeplabV3+ and U-Net misclassify, attributed to deeper semantic extraction and efficient twofold upsampling in the decoder. - Practical demonstrations: Successful automated droplet splitting, movement, dispensing, and merging through state-machine feedback control on a custom EWOD DMF platform.

The results demonstrate that semantic segmentation-driven feedback provides the missing capability to recognize and act upon transient droplet states, directly addressing variability in DMF operations. By reliably identifying critical intermediate morphologies (e.g., hourglass splitting, initial contact in merging), the controller can adapt electrode sequences in real time, improving operation success and precision (e.g., lower CV in split volumes). Superior mIoU and lower loss versus DeeplabV3+ and U-Net confirm that the tailored architecture and decoding strategy capture both fine boundaries and high-level semantics needed in DMF. The open dataset and compatibility with existing camera-equipped DMF controllers (DropBot, openDrop, DropLab) position the approach for broad adoption and integration with advanced AI (e.g., reinforcement learning, MLLMs) to achieve higher autonomy and task complexity in microfluidic workflows.

µDropAI integrates a purpose-built semantic segmentation model, morphology extraction, and a state-machine controller to achieve multistate, closed-loop droplet manipulation on EWOD DMF platforms. The system recognizes droplets across colors and shapes with <0.63% error, improves split-volume precision (CV 2.74%), and outperforms DeeplabV3+ and U-Net in mIoU and loss. An open, labeled multistate dataset and open-source framework are provided, and the approach is compatible with common DMF systems, enabling scalable deployment. The work paves the way for semantic-driven, fully automated DMF integrating reinforcement learning and multimodal LLMs for complex, high-throughput microfluidic tasks. Future directions include expanding the state taxonomy, scaling datasets across devices/fluids, optimizing real-time performance, and coupling with higher-level planners for task-level autonomy.