Manual data extraction from research papers is a time-consuming and laborious process. Automated methods based on natural language processing (NLP), language models, and large language models (LLMs) offer a significant improvement in efficiency. However, existing automated methods often require substantial upfront effort, expertise, and coding, involving the preparation of parsing rules, model fine-tuning, or retraining. This work addresses this limitation by proposing ChatExtract, a novel method that leverages the capabilities of conversational LLMs to achieve highly accurate data extraction with minimal initial effort. Conversational LLMs, like ChatGPT, possess pre-trained general language abilities and zero-shot capabilities, making them well-suited for this task. ChatExtract harnesses these abilities through a carefully engineered set of prompts that guide the LLM to identify relevant sentences, extract data, and verify its correctness through a series of follow-up questions. This approach effectively mitigates common LLM issues like factual inaccuracies and hallucinations. The simplicity and transferability of ChatExtract, combined with its high accuracy, make it a promising tool for researchers across various fields.

Numerous studies have explored automated data extraction using NLP and LLMs, leading to the creation of various materials science databases. The emergence of large language models has significantly improved the accuracy of extracting complex data. However, previous automated methods frequently demand substantial effort for setup, such as designing parsing rules, fine-tuning models, or preparing training data. The authors cite several examples of existing database creation efforts using NLP and LLMs, highlighting the resources and expertise typically needed. The use of prompt engineering in image generation to ensure high-quality results is also noted, suggesting a parallel opportunity for improving data extraction accuracy using LLMs.

The ChatExtract method uses a two-stage process. Stage A involves a simple relevancy prompt applied to all sentences to identify those containing relevant data. Stage B extracts the data from sentences classified as relevant in Stage A. To enhance performance, several engineered prompts are used in Stage B. The workflow addresses single- and multi-valued sentences differently. For single-valued sentences, direct questions about the material, value, and unit are posed, explicitly allowing for negative answers to prevent hallucinations. Multi-valued sentences are handled by requesting the model to present the data in a tabular format, followed by follow-up questions to verify each field's accuracy. These follow-up questions introduce uncertainty and redundancy, prompting the model to re-analyze the text and reduce the chance of errors. The prompts are embedded within a single conversation, leveraging the LLM's information retention capabilities while repeatedly presenting the relevant text. Responses are constrained to a Yes/No format for easier automation. The text analyzed includes the sentence's title, the preceding sentence, and the target sentence, often containing a complete 'Material, Value, Unit' triplet. The authors emphasize that while the prompts are optimized for GPT models, the general method is adaptable to other LLMs. A Python code implementation of ChatExtract is provided.

The ChatExtract method was evaluated using several datasets. A test dataset for bulk modulus demonstrated the ability of the method to extract data from sentences containing multiple potentially confusing values and similar units. The best performing LLM (ChatGPT-4) achieved 90.8% precision and 87.7% recall. Single-valued sentences showed higher recall than multi-valued sentences. Comparative analyses revealed the importance of follow-up questions and conversational information retention in achieving high precision. Removing follow-up questions drastically reduced precision, emphasizing their role in preventing hallucinations. The conversational aspect also proved crucial, as starting a new conversation for each prompt significantly reduced recall. The LLaMA2-chat model was also tested, demonstrating that ChatExtract can be used with other LLMs, although performance was slightly lower than ChatGPT-3.5. Comparison with ChemDataExtractor2 (CDE2) showed ChatExtract's superior performance in terms of precision and recall on both bulk modulus and critical cooling rate datasets. ChatExtract was used to build two real-world databases: one for critical cooling rates of metallic glasses and another for yield strengths of high-entropy alloys. For critical cooling rates, ChatExtract achieved 76.9% precision and 63.1% recall on the raw database, improving to 91.9% precision and 84.2% recall on the standardized database. The standardized metallic glass database contained 217 unique material compositions, exceeding the size of previously published hand-curated databases. For high-entropy alloy yield strengths, the standardized database contained 636 unique material compositions, also significantly larger than existing databases. The datasets generated using ChatExtract are made available via Figshare.

The results demonstrate the effectiveness of ChatExtract in extracting accurate materials data from scientific literature. The high precision and recall achieved across different properties and LLMs highlight the method's robustness and transferability. The successful generation of large, high-quality databases for critical cooling rates and yield strengths showcases the practical applications of ChatExtract. The study's findings underscore the potential of conversational LLMs and prompt engineering to automate scientific data extraction, overcoming the limitations of traditional methods. The ability to adapt ChatExtract to various data types and LLMs suggests its broad applicability across different scientific disciplines.

ChatExtract offers a simple yet powerful approach to accurate materials data extraction from research papers. Its high accuracy, combined with its minimal requirements for setup and expertise, positions it as a valuable tool for researchers. The method’s ability to adapt to various LLMs ensures its continued relevance as LLMs evolve. Future work could explore extending ChatExtract to handle more complex data types and integrate additional constraints or conditions into the data extraction process.

While ChatExtract demonstrates high accuracy, the performance may be affected by the quality and structure of the source texts. The standardization of databases may require manual effort, particularly for complex materials or ambiguous descriptions. The prompts used were optimized for GPT models, and further optimization might be needed for other LLMs. The current implementation has limited error handling, and improvements in this area could enhance robustness.