Advanced scientific user facilities, such as next-generation x-ray light sources, nanoscience centers, and neutron facilities, are undergoing significant upgrades, revolutionizing our understanding of materials across various scientific disciplines. However, these improvements also lead to increased instrument and experimental complexity. The intricate nature of modern experiments makes it difficult for scientists, particularly new users, to effectively design and conduct research that fully leverages the capabilities of these advanced instruments. This challenge necessitates innovative approaches to streamline the experimental process and enhance accessibility for a wider range of researchers. Large language models (LLMs), a rapidly advancing area of artificial intelligence, offer a promising avenue for addressing this complexity. LLMs excel at complex information retrieval, knowledge-intensive tasks, and providing guidance on tool usage. Their ability to process and synthesize large amounts of information makes them ideally suited for assisting scientists in navigating the intricacies of advanced scientific instruments and experiments. The potential applications within scientific user facilities are vast, including aiding in experimental design, providing data summaries, and even assisting with publication writing. This research explores the application of LLMs in this context. Using x-ray light sources, nanoscience computing facilities, and nanoscience centers as case studies, the researchers conducted preliminary experiments with a Context-Aware Language Model for Science (CALMS). The goal was to evaluate the efficacy of LLMs in assisting scientists with various aspects of experiment design and execution, ranging from answering basic operational questions to guiding instrument operation in a conversational manner. The ultimate aim is to determine whether LLMs can significantly improve the efficiency and accessibility of these crucial scientific facilities.

The transformative potential of large language models (LLMs) is increasingly recognized across diverse fields. Conversational agents built upon LLMs, exemplified by ChatGPT, are impacting various industries, including education, talent acquisition, and many others. McKinsey & Company estimates that generative AI, including LLMs, could add trillions of dollars annually to global productivity. In science, LLMs have the potential to revolutionize all aspects of the scientific process. This includes accelerating literature searches, assisting in experimental design, summarizing complex datasets, and aiding in the writing and editing of scientific publications. Furthermore, LLMs have demonstrated rapid learning capabilities, acquiring domain expertise with limited training examples, as seen in applications like materials property prediction and inverse design. While the potential is substantial, it's crucial to acknowledge the limitations of LLMs, including the tendency to produce inaccurate or hallucinated information, requiring careful validation of their outputs. Prior research has investigated using LLMs to improve different aspects of scientific workflows, however, this study looks to bridge the gap between LLMs and the operation of physical instruments.

This research focuses on developing and evaluating a Context-Aware Language Model for Science (CALMS) specifically tailored for scientific user facilities. CALMS comprises four key components: 1. **Large Language Model (LLM):** The core of CALMS is an LLM that powers the conversational agent. The researchers used two state-of-the-art LLMs for comparison: OpenAI's GPT-3.5 Turbo (a closed-source model) and an open-source model, Vicuna. The choice of LLMs allows for a comparison between closed-source and open-source models. 2. **Memory Component:** This component maintains the agent's memory, crucial for tracking conversation history and context within interactions that exceed the LLM's token limit. 3. **Document Store:** A repository of relevant documents (facility manuals, instrument guides, etc.) that CALMS can access to answer user queries and enhance its context awareness. Context retrieval was done using semantic search, and two methods were explored: fine-tuning and providing additional context. 4. **Experiment Planning Assistant:** This component interacts with the model, providing user-specific information when requested. The methodology involved querying CALMS with questions related to experimental planning assistance, instrument operation, and the ability to successfully drive an instrument. The researchers compared the performance of GPT-3.5 and Vicuna across these tasks, both with and without context retrieval. The evaluation metrics included the relevance, trustworthiness (absence of hallucinations), and completeness of the LLM responses. The experiments were conducted using representative examples from different user facilities (APS, CNMS, ALCF). Furthermore, the researchers integrated CALMS with software tools and APIs (Materials Project API, SPEC instrument control software), enabling the LLM to directly interact with and control scientific instruments. This integration was facilitated by using the Chain-of-Thought prompting and ReAct framework, allowing the LLM to parse and execute commands to control instruments.

The experiments revealed several key findings: 1. **Context is Crucial:** The results strongly highlight the critical role of context in improving the accuracy and reliability of LLM responses. Without proper context (e.g., relevant facility documentation), both LLMs exhibited a significant tendency to hallucinate, providing incorrect or misleading information. With appropriate context, however, both models provided consistently relevant and accurate answers. This emphasizes the importance of retrieval-augmented generation (RAG) techniques in enhancing LLM performance for domain-specific applications. 2. **Performance Differences between LLMs:** The closed-source model, GPT-3.5 Turbo, consistently outperformed the open-source model, Vicuna, particularly in terms of completeness and the ability to handle complex queries. The ability to successfully drive a real-world diffractometer experiment via API calls was only possible with GPT-3.5. This reflects the maturity gap between open-source and commercially available LLMs. 3. **Successful Tool Integration:** The successful integration of CALMS with software tools and APIs demonstrated the feasibility of using LLMs to directly control and operate scientific instruments. The experiment with controlling a diffractometer highlights the potential of fully automating routine experimental procedures. This successful control was executed using GPT-3.5. 4. **Importance of Prompt Engineering:** The study also showcased the importance of prompt engineering. Careful selection of prompt structures and parameters within the CALMS framework was crucial for eliciting accurate and helpful responses from the LLMs. The researchers also observed that, in instances where context was absent, even when providing a truthful answer, it may not be directly helpful in the specific context of the scientific query. For example, when asked about a tomographic scan, an LLM without context provided an accurate yet irrelevant description of a medical CT scan.

The findings from this study demonstrate the significant potential of context-aware LLMs for enhancing the efficiency and accessibility of advanced scientific user facilities. By providing scientists with immediate access to relevant information and guidance on complex experimental procedures, LLMs can accelerate research workflows and reduce the learning curve for new users. The successful integration of LLMs with software tools and APIs opens up exciting opportunities for automating various aspects of experimental design and execution, leading to substantial time savings and increased productivity. The differences observed between the performance of closed-source and open-source models highlight the ongoing advancements in the field of LLMs. As open-source models continue to mature, we expect the performance gap to narrow. The integration of LLMs with e-logs and the development of fully autonomous experimental workflows represent promising future research directions.

This research successfully demonstrated the potential of context-aware LLMs, such as CALMS, to significantly enhance the efficiency and accessibility of advanced scientific user facilities. The integration of LLMs with scientific instruments through tool augmentation provides a powerful mechanism for automating complex experimental tasks and empowering scientists to fully leverage the capabilities of these facilities. Future work will focus on improving the robustness and reliability of open-source LLMs, expanding tool integration to a wider range of instruments, and developing fully autonomous experimental workflows.

This study represents preliminary work, and further investigation is needed to fully explore the potential and limitations of LLMs in this context. The reliance on context retrieval highlights the importance of maintaining accurate and up-to-date documentation for scientific user facilities. The performance differences between closed-source and open-source models may reflect the current state of the field, but this gap is expected to reduce with continued advancements. The current implementation is limited to certain instruments and interfaces, and expanding to diverse instruments presents a future challenge. Finally, thorough ethical considerations, such as bias and fairness in LLMs, are crucial for responsible implementation and deployment.