Large language models (LLMs), especially transformer-based models, have shown remarkable progress in various fields, including natural language processing, biology, chemistry, and programming. The recent release of GPT-4 represents a significant advancement in LLM capabilities, exhibiting strong problem-solving skills across diverse domains, including chemistry-related problems. Simultaneously, advancements in laboratory automation have enabled the development of autonomous systems for chemical research, such as automated reaction discovery and optimization. This research investigates the potential of combining powerful LLMs with laboratory automation to create a system capable of autonomously designing and executing scientific experiments. The study aims to explore the capabilities of LLMs in the scientific process, determine the achievable level of autonomy, and understand the decision-making processes of such autonomous agents. This work builds upon existing research in autonomous agents and aims to contribute to the field by introducing Coscientist, a novel AI system designed for autonomous chemical research.

The paper reviews existing literature on the applications of LLMs in various scientific domains, highlighting the success of transformer-based models in natural language processing, biology, and chemistry. It also examines previous work on the automation of chemical research, including autonomous reaction discovery and optimization, automated flow systems, and mobile platforms. The literature review establishes the context for the development of Coscientist and its potential contribution to the field. Specific examples of prior work cited include the use of LLMs for computational chemistry, generative models for molecular structure prediction, and robotic platforms for automated organic synthesis. The review underscores the need for a system that integrates the capabilities of LLMs and laboratory automation for autonomous scientific experimentation.

Coscientist, the AI agent developed in this study, is built upon a multi-LLM architecture. It interacts with multiple modules, including web and documentation search, code execution, and experimental automation. The core module, 'Planner' (a GPT-4 chat completion instance), plans experiments based on user input by using four commands: 'GOOGLE', 'PYTHON', 'DOCUMENTATION', and 'EXPERIMENT'. The 'GOOGLE' command uses a web searcher module (another LLM) to retrieve information from the internet. The 'PYTHON' command executes code within an isolated Docker container. The 'DOCUMENTATION' command utilizes a 'Docs searcher' to retrieve and summarize information from relevant documentation, such as robotic API documentation. The 'EXPERIMENT' command interfaces with laboratory automation APIs, such as the Opentrons Python API and the Emerald Cloud Lab (ECL) Symbolic Lab Language (SLL), to perform experiments. The system's architecture is designed to handle both high-level and low-level instructions, enabling the control of liquid handling instruments and other laboratory hardware. The study evaluates Coscientist's performance across six tasks: (1) planning chemical syntheses, (2) navigating hardware documentation, (3) executing high-level commands in a cloud laboratory, (4) precisely controlling liquid handling, (5) integrating multiple hardware modules, and (6) optimizing reaction conditions based on experimental data. Different LLMs (GPT-3.5, GPT-4, Claude, Falcon) were compared for web searching, and different methods were explored for information retrieval from documentation, including inverted search index and vector database approaches. The experimental setup includes an Opentrons OT-2 liquid handler, a UV-Vis plate reader, and the Emerald Cloud Lab platform for cloud-based experiments. The study involved testing the system's ability to plan syntheses of various compounds, execute experiments on the OT-2 robot, and analyze data to improve subsequent experiments. Evaluation metrics include scoring of synthesis plans and analysis of reaction optimization results using normalized advantage and normalized maximum advantage metrics.

Coscientist demonstrated significant capabilities in autonomous chemical research. In chemical synthesis planning, the GPT-4-powered web searcher substantially outperformed other models, achieving high accuracy in generating detailed and chemically accurate synthetic procedures. The system's ability to effectively use technical documentation to control laboratory hardware was also demonstrated, with Coscientist successfully executing complex protocols on the Opentrons OT-2 liquid handler. Coscientist could successfully integrate multiple hardware modules, such as the liquid handler and UV-Vis plate reader, to solve complex tasks. In reaction optimization experiments, Coscientist demonstrated the ability to learn from previous experimental data to improve the yield of palladium-catalyzed Suzuki and Sonogashira coupling reactions. Compared to Bayesian optimization, Coscientist showed a higher normalized maximum advantage, indicating superior performance in maximizing reaction yield. The system's ability to reason about chemical reactivity was evident in its ability to select appropriate reagents and conditions for different reactions, while avoiding chemical errors. Importantly, the model often provides justifications for its choices, demonstrating understanding of concepts like reactivity and selectivity. The successful execution of both Suzuki and Sonogashira coupling reactions, with product confirmation via GC-MS, further validated Coscientist's experimental capabilities. The study also explores the use of SMILES strings as input for reaction prediction, demonstrating the system's adaptability and versatility.

The findings demonstrate the considerable potential of LLMs in accelerating scientific discovery. Coscientist's ability to autonomously design, plan, and execute complex chemical experiments significantly reduces the time and effort required for research, potentially leading to faster breakthroughs. The system's capacity for advanced reasoning and chemical intuition, as demonstrated in the reaction optimization experiments, showcases the power of integrating LLMs with laboratory automation. The successful integration of multiple hardware modules further highlights the system's versatility and scalability. While Coscientist's current capabilities are limited to specific reaction types and readily available resources, future improvements could expand its applicability to a wider range of chemical transformations and experimental designs. The development of more comprehensive and integrated scientific tools for LLMs is crucial for unlocking their full potential in scientific research.

This research presents a proof-of-concept for an AI agent capable of semi-autonomous scientific experimentation. Coscientist demonstrates significant potential for accelerating chemical research through its ability to integrate various tools and reasoning capabilities. While limitations exist, notably regarding the scope of reactions and current automation level, the successful execution of complex experiments and reaction optimization underscores the substantial promise of this approach. Future work should focus on expanding Coscientist's capabilities to encompass a broader range of chemical transformations, improving data integration from diverse sources, and enhancing the level of autonomy in experimental design and execution. Addressing safety and ethical considerations associated with autonomous agents in chemical research is also crucial for responsible development and deployment of such systems.

The current version of Coscientist relies on existing datasets and readily available resources. Its ability to explore truly novel chemistry is limited by the constraints of the available data. While the system demonstrated successful integration of several hardware modules, the complete automation of the experimental process remains an area for future improvement. The study's focus on specific reaction types limits the generalizability of the findings to other chemical domains. Moreover, the interpretation of the LLM’s reasoning remains reliant on the analysis of generated text, which can be subject to limitations in explainability.