Self-driving labs (SDLs) integrate automated physical experimentation with digital data processing and algorithm-guided experiment selection to explore chemical and materials science problems with unprecedented speed and efficiency. However, current SDLs struggle with complex multi-stage chemistries due to the curse of dimensionality (exponentially increasing parameter space with increasing dimensionality) and data scarcity. This is particularly challenging in multi-step syntheses, common in materials science, where even a few steps dramatically increase complexity. Colloidal atomic layer deposition (cALD) is a prime example of a high-dimensional, multi-stage chemistry. It involves sequential injection, removal, and washing of reactants and ligands to grow heteronanostructures layer-by-layer. The self-limiting, monolayer precision of cALD offers precise control over luminescent and electronic properties, while preserving the size dispersity of starting quantum dots (QDs). However, the exponential growth of the parameter space with each step makes traditional exploration methods impractical. Existing SDLs employing retrosynthetic planning algorithms or supervised learning (SL) struggle with cALD's complexity and data scarcity. Retrosynthetic approaches rely on extensive literature data and physics-based models, which are often unavailable for novel or poorly understood reactions. SL methods, while applicable to autonomous reactors, require substantial pre-existing datasets and do not readily handle the sequence-dependent nature of multi-step processes. Reinforcement learning (RL), a powerful subset of machine learning, offers a potential solution. Unlike SL, RL monitors the system's current state, maps actions to responses, and learns through trial-and-error, making it ideal for multi-step processes. By breaking down decisions into isolated steps and predicting future effects, RL can efficiently navigate high-dimensional, dynamic systems, as demonstrated by AlphaGo's success in the game of Go. This paper introduces AlphaFlow, an RL-guided SDL designed to autonomously discover and optimize complex multi-step chemistries.

Several studies have leveraged SDLs with retrosynthetic planning algorithms for on-demand production of small molecules through elaborate multi-step synthesis routes using both batch and flow reactors. These studies, however, rely heavily on existing literature data and physics-based models for each reaction step, limiting their applicability to under-studied or immeasurable reaction routes. Many SDL studies involving nanoparticles rely solely on data generated by a single reactor, hindering generalizability. This highlights the need for ML techniques capable of handling sequence-dependent processes with limited data, particularly in the context of nanoscience where reproducibility and accessible literature data are often scarce. Previous work has explored RL in silico for process synthesis and synthetic route discovery, but real-time integration with closed-loop experimentation has been lacking. Miniaturized automated experimentation platforms offer the potential to bridge this gap by facilitating trial-and-error exploration with minimal material and time loss.

AlphaFlow integrates a modular fluidic microdroplet reactor with reinforcement learning algorithms. The microreactor, shown schematically in Figure 2b, comprises four integrated modules: (i) formulation, (ii) synthesis, (iii) in-situ characterization, and (iv) in-line phase separation. The formulation module prepares the reactive droplet, the synthesis module oscillates the droplet to ensure mixing and repeated spectral analysis, the in-situ characterization module non-invasively monitors the reaction using optical spectroscopy (measuring first absorption peak wavelength (λAP), first absorption peak intensity (IAP), absorption peak-to-valley ratio (RPV), and photoluminescence peak intensity (IPL)), and the in-line phase separation module separates immiscible phases using timed nitrogen injection. The system also includes automated washing and reagent refilling mechanisms for continuous operation. The AlphaFlow software uses a reinforcement learning agent that interacts with the microreactor environment. The agent's state is represented by a short-term memory (STM) containing the four prior injection conditions, considering that recent injections have the most impact on current decisions. The response is a reward based on the in-situ spectral characteristics. The agent’s belief model includes an ensemble neural network regressor (ENN) that predicts rewards for given states and actions, and a gradient-boosted decision tree that classifies state-action pairs as viable or unviable. The belief model is constantly retrained with new data. The reward function is crucial and is designed to prioritize improvements in multiple parameters (λAP, RPV, and IPL), while avoiding conditions that lead to high λAP but poor overall quality. The reward is calculated as the slope of local reward improvement (weighted sum of parameters) as a function of λAP changes, favoring consistent increases while maintaining high RPV and IPL. The viability classifier assigns probabilities to state-action pairs resulting in terminal conditions (e.g., undetectable droplet volume, no measurable spectral features, insufficient nanoparticle concentration). The agent uses a model-based rollout policy to predict outcomes of future action sequences using an upper confidence bound (UCB) decision policy during exploration and a mean reward maximization policy during exploitation. Two campaigns were conducted: (1) autonomous discovery of a viable 20-step reagent addition sequence for cALD using four reagent options (oleylamine, sodium sulfide, cadmium acetate, and formamide); and (2) optimization of reagent injection volumes and reaction times for each step of the discovered sequence using three different starting CdSe QD sizes. The experimental hardware and software were designed with modularity and flexibility to accommodate system modifications and exploration of different reaction pathways.

In the first campaign, AlphaFlow autonomously discovered a novel cALD reagent addition sequence that significantly outperformed conventional methods. Without prior knowledge, AlphaFlow identified a sequence that resulted in a 26 nm higher first absorption peak wavelength and a 450% higher photoluminescence intensity than the conventional sequence after six cycles. The AlphaFlow-selected sequence also included fewer steps, reducing experimental costs. Analysis suggests the improved performance stems from a modified role of oleylamine, replacing some washing steps and potentially aiding in surface passivation and preventing homonuclei formation. The algorithm's sequence selection was guided by its ability to predict rewards multiple steps ahead, effectively avoiding detrimental conditions. The second campaign focused on optimizing reagent injection volumes and reaction times using the RL-identified sequence. AlphaFlow successfully optimized these parameters for three different starting CdSe QD sizes, leading to significant improvements in spectral properties. For example, with 480 nm QDs, the optimized conditions yielded a similar λAP shift as the sequence selection exploitation results, but with a 40% higher RPV by the fourth cALD cycle. AlphaFlow demonstrated the ability to select conditions that temporarily lower RPV to achieve a greater RPV later, showcasing its capacity for long-term strategic decision-making. Comparison with alternative methods using a digital twin model revealed AlphaFlow's superiority in high-dimensional optimization. Bayesian Optimization (BO) failed to find a viable 20-step sequence even after 100 attempts, whereas AlphaFlow achieved a viable sequence quickly and then continuously improved performance. A manual model-driven study using a global optimization method in the digital twin identified an optimized set of conditions, but this approach required significantly more computational effort and failed to account for real-world experimental deviations. The AlphaFlow system, with its real-time adaptation and closed-loop feedback, overcame this limitation and achieved superior results compared to both the model-driven study and Bayesian Optimization.

AlphaFlow overcomes significant limitations in algorithm-guided multi-step chemistry, enabling efficient exploration and optimization in complex, high-dimensional parameter spaces. Its high throughput and real-time decision-making capabilities surpass the capabilities of manual experimentation. The work provides a significant advance in the field of autonomous experimentation, demonstrating the potential for RL to accelerate the discovery and optimization of complex chemical processes. Future applications of AlphaFlow could include the exploration of other cALD-based chemistries, atomic layer deposition, molecular layer deposition, and telescoped reactions involving unstable intermediates.

AlphaFlow represents a significant advancement in autonomous experimentation. It successfully navigated a 40-dimensional parameter space, outperforming conventional methods and alternative optimization algorithms in discovering and optimizing a complex multi-step chemical synthesis. Future work will explore the application of AlphaFlow to a wider range of chemical systems and further refine its capabilities for even more complex and challenging syntheses. The high-throughput data generation capacity of AlphaFlow will also be valuable for data mining and the generation of fundamental insights into multi-stage chemistries.

The current AlphaFlow system relies on a short-term memory (STM) of four previous injections for state representation, which is a simplifying assumption that may limit its performance in scenarios with longer-range dependencies. The reward function, while effective, was tailored to the specific cALD reaction studied. Generalization to other systems might require modifications to the reward function and possibly the state representation. While AlphaFlow exhibited high reproducibility, some level of experimental error is inherent, particularly due to the aging of the sodium sulfide reagent. The system's ability to handle unexpected events or hardware failures could be further improved.