AlphaFlow: autonomous discovery and optimization of multi-step chemistry using a self-driven fluidic lab guided by reinforcement learning

The study addresses how to autonomously discover and optimize complex multi-step chemistries in high-dimensional, data-scarce environments. Self-driving labs (SDLs) have shown promise but are typically constrained to well-studied, limited parameter spaces. Multi-step syntheses such as colloidal atomic layer deposition (cALD) present exponential growth in decision space (“curse of dimensionality”), compounded by hidden states, nondeterministic outcomes, and costly experimentation. Conventional retrosynthetic planning often relies on physics-based models and extensive literature data, which is less applicable to under-studied, heterogeneous nanoparticle systems with poor reproducibility across labs. The authors propose AlphaFlow, integrating reinforcement learning (RL) with an automated microdroplet platform, to navigate multi-step decision processes by modeling state-action-reward trajectories rather than static input-output mappings. The research aim is to autonomously discover and optimize cALD-based shell growth routes for CdSe/CdS core–shell quantum dots without prior knowledge of conventional reagent sequences, demonstrating that RL-guided closed-loop experimentation can effectively handle 40+ dimensional reaction spaces.

The paper situates AlphaFlow within SDL advances using robotics and microfluidics for automated experimentation and closed-loop optimization. Prior SDLs have optimized nanoparticle syntheses and perovskite systems using supervised learning or Bayesian optimization, but typically for single-step or low-dimensional problems with significant prior data. Retrosynthetic planning systems in batch and flow have synthesized small molecules via literature-informed models but are less suitable for under-characterized, multi-step nanoparticle systems. cALD and related shell-growth techniques offer monolayer precision at room temperature but involve complex, sequence-dependent steps and hidden surface states. RL has excelled in large decision spaces (e.g., AlphaGo) and has been demonstrated in silico for process synthesis and route discovery; however, integration with real-time, closed-loop experimental platforms remained limited. The authors leverage these insights to apply RL to experimental multi-step chemistry, addressing the need for algorithms capable of handling sequence-dependence, hidden states, and sparse, in-house data.

AlphaFlow comprises a modular single-microdroplet (≈10 µL) flow platform and an RL agent guiding multi-step syntheses. Hardware modules: (i) formulation (precise reagent injection via one-hot selectable channels and optical positioning), (ii) synthesis (oscillatory mixing with in situ optical sampling per oscillation), (iii) characterization (UV-Vis absorption and photoluminescence to extract first absorption peak wavelength λ_AP, peak intensity I_AP, peak-to-valley ratio R_PV, photoluminescence intensity I_PL), and (iv) in-line phase separation (timed inert gas segmenting to remove immiscible phase based on droplet-length calibration). The system includes automatic reactor washing and syringe refilling for multi-day, low-failure (<1% injection) operation. A reusable in-line separator avoids membranes to minimize fouling and droplet loss; droplet lengths and phase volumes are estimated from transit times and spectral signatures to adapt separation timing. Software/RL: The reactor is the environment; the agent uses a short-term memory (STM) state (default last four injections; extended with cycle/injection indices, volumes, times in optimization) and a reward based on trajectory: the slope of improvements in a weighted local reward (combining Δλ_AP, R_PV, I_PL) vs. |Δλ_AP| over an eight-point moving window, counting only positive local reward gains to favor routes that increase shell growth while preserving quality. The agent’s belief model is an ensemble neural network regressor (predicting reward) and a gradient-boosted decision tree classifier (predicting terminal/unviable states). The rollout policy forward-simulates action sequences four steps ahead, applying a discount via cumulative viability probabilities; an Upper Confidence Bound (UCB) policy balances exploration and exploitation during exploration phases; exploitation maximizes mean predicted reward. Campaigns: (1) Sequence selection with four reagents—oleylamine (OAm), sodium sulfide (Na2S·9H2O), cadmium acetate (Cd(Ac)2·2H2O), formamide (FAm)—at fixed volumes and times to autonomously discover a viable, high-performing 20-step sequence. (2) Volume and time optimization using the discovered sequence, optimizing per-step injection volume (1–10 µL) and reaction time (1–10 oscillations; 40–400 s) across multiple cALD cycles and for three starting CdSe core sizes, within ~700 injection steps per core size (>9000 condition evaluations via in situ sampling). Data processing includes robust spectral phase isolation, feature extraction (Savitzky–Golay smoothing for feature detection only; trimmed means), terminal condition definitions, and calibration of droplet metrics. Digital twin studies compare RL to Bayesian optimization (BO) and basin-hopping (L-BFGS) under an evaluation-limited, high-dimensional setting.

- RL autonomously discovered a repeating cALD-like sequence: OAm–Na2S–FAm–Cd(Ac)2–OAm (with phase separations where needed), differing from the conventional OAm–Na2S–FAm–FAm–Cd(Ac)2–FAm–FAm by replacing certain wash steps and inserting OAm after Cd(Ac)2. - Performance vs. conventional sequence (six cycles, same starting CdSe QDs): • First absorption peak wavelength shift (λ_AP) was 26 nm higher for the RL-discovered route. • Photoluminescence intensity increased by 450% relative to the conventional route. • Prior to the conventional route’s plateau (after ~3 cycles), the RL route achieved on average 9 nm larger λ_AP shift per cycle, despite fewer injections per cycle. • The RL route is two injections shorter per cycle, reducing experimental cost. - Exploration challenges quantified: 45% of all possible combinations of the first three injections led to terminal conditions; RL with UCB rapidly avoided unviable regions. - Volume/time optimization (for a 480 nm starting λ_AP core): achieved λ_AP shifts comparable to the sequence-discovery exploitation while improving R_PV by ~40% by the fourth cycle; similar strong results across two additional core sizes, demonstrating transferability. - RL identified non-myopic optima: selected interim steps that transiently reduced R_PV to enable higher downstream rewards; forward prediction beyond immediate steps was crucial to avoid delayed terminal states and locate true optima in volume/time space. - Digital twin benchmarking (evaluation-limited, 40D): • BO (ENN surrogate, UCB) failed to find any viable 20-step condition set after 100 experiments (~2000 injections). • RL found a viable 20-step condition set after 4 experiments and reached 94% of the digital twin’s known optimum after 100 experiments; 12 viable sets identified in 100 attempts. • Global optimum via basin-hopping (L-BFGS) required >50,000 simulated experiments (~1,000,000 injections), impractical for real-world; RL attained near-optimal with 100 experiments. - Real-world validation of digital optima: basin-hopping predicted conditions overestimated λ_AP by ~15 nm; real-world outcomes underperformed predicted local reward by ~10% and remained below real-world RL-optimal conditions, highlighting the value of real-time adaptation. - Platform reliability: <1% injection failure; sustained autonomous operation for days; sequence reproducibility across replicates; automated refilling reliable over ≥14 cycles. - Practical insights: Reduced washing and added OAm after Cd(Ac)2 likely enhance passivation and suppress homonuclei formation, enabling continued room-temperature shell growth (akin to sub-monolayer SILAR behavior).

AlphaFlow demonstrates that RL-guided closed-loop experimentation can effectively navigate and optimize high-dimensional, multi-step chemistries with hidden states and nondeterministic outcomes, directly addressing the curse of dimensionality in cALD-like processes. By modeling trajectories and forward-predicting multi-step outcomes, the agent balances exploration and exploitation, discovers non-intuitive sequence modifications (reduced washes, OAm insertion), and tunes stepwise conditions to jointly improve shell growth (λ_AP) and quality (R_PV, I_PL). Compared to BO and offline global optimization, RL’s real-time belief updates and viability-aware rollouts lead to rapid identification of viable, high-reward sequences under strict experimental budgets. The discovered routes outperform conventional cALD in both extent of shell growth and photophysical quality while reducing the number of injections. The approach is transferable across core sizes and replicable, suggesting robustness. Beyond nanoparticles, the framework is applicable to other multi-step processes (e.g., ALD/MLD, telescoped reactions) where sequence, exposures, and purging times interact with unstable intermediates and hidden surface states. The findings advance SDL capabilities from low-dimensional optimizations to autonomous discovery of complex, multi-stage reaction strategies using only in-house, reproducible data.

The work introduces AlphaFlow, the first integration of reinforcement learning with an automated, modular microdroplet platform for autonomous discovery and optimization of complex multi-step chemistries. AlphaFlow autonomously discovered a superior cALD-inspired sequence and optimized per-step volumes and times, achieving larger λ_AP shifts, substantially higher photoluminescence, and improved nanoparticle quality with fewer injections compared to conventional protocols. RL outperformed Bayesian optimization under evaluation constraints and approached digital global optima with orders of magnitude fewer experiments, underscoring the value of forward-looking, viability-aware, real-time decision-making. Future directions include applying the framework to diverse multi-step chemistries (e.g., different chalcogenide sources, ligand systems, temperatures; ALD/MLD parameters), scaling strategies for broader adoption, and community standards for SDL data and metadata to facilitate reproducible data mining and knowledge discovery.

- Microdroplet reactor scalability and handling of solids pose challenges; translating optimized conditions to batch may be synthesis-dependent (e.g., mass-transfer-limited systems may not transfer directly). - RL state definition uses a limited short-term memory (e.g., last four injections), a heuristic that may miss longer-range dependencies and could require more data if extended. - Reagent variability (notably sodium sulfide aging and byproducts) introduces hidden-state drift; performance depends on careful reagent preparation and age windows. - Global optimality cannot be guaranteed due to the enormous sequence space (>10^12 possibilities); discovered routes are empirically superior but may be local optima. - Reward relies on in situ spectral metrics (λ_AP, R_PV, I_PL); properties not captured by these metrics are not directly optimized. - Viability classification and hardware operations can yield occasional false terminals or operational errors, though the failure rate was low (<1% of injections). - BO comparison used a surrogate trained on RL-generated data; alternative BO settings or more sampling might improve BO but at higher experimental cost.