Testing the reliability of an AI-based large language model to extract ecological information from the scientific literature

The emergence of AI-based large language models (LLMs) offers the potential to revolutionize scientific research by automating time-consuming tasks. However, concerns exist regarding the reliability of LLMs due to their propensity for inaccuracies and biases in training data. This study addresses this concern by directly comparing the performance of an LLM against a human expert in extracting ecological data from scientific reports focusing on emerging infectious diseases (EIDs) in plants. EIDs provide an ideal case study due to the vast and rapidly growing volume of related publications. The goal is to evaluate the LLM's capabilities and limitations, thereby informing its potential application as a valuable research tool in ecology and related fields. The high volume of EID reports published annually makes manual data extraction a significant challenge, highlighting the potential efficiency gains offered by LLMs.

Existing literature explores the potential applications of LLMs in scientific research, acknowledging both their promising capabilities and inherent limitations. Some studies highlight the potential of LLMs to improve efficiency and productivity in science, while others caution against the risk of inaccuracies and biases introduced by these models. The "hallucination" phenomenon, where LLMs generate factually incorrect information, is a major concern. The lack of transparency in LLM training data further complicates the assessment of their reliability. The use of AI in ecological studies, particularly in generating hypotheses and predictions, necessitates a thorough understanding of their capabilities and limitations. This study builds upon existing research by focusing on a specific ecological application and directly comparing LLM performance against a human expert.

The study utilized 100 reports on emerging infectious tree diseases from a previous study by Gougherty (2023) as source material. These reports were characterized by their brevity and data density, making them suitable for testing the LLM's data extraction capabilities. The Google text-bison-001 model, a generative text model chosen for its ability to return only relevant text, was employed. A custom prompt was developed to guide the LLM to extract specific data points, including pathogen and host scientific names, incidence of infection, year and location of detection, and geographic coordinates. The prompt was iteratively refined to ensure accuracy and consistency in data extraction, addressing issues like varied coordinate formats and the need for clear column delimiters in the output. A human reviewer independently extracted the same data from the reports to serve as a benchmark for comparison. The accuracy and speed of both the LLM and human reviewer were compared and analyzed.

The LLM completed data extraction in approximately 5 minutes, while the human reviewer required 268 minutes, demonstrating a more than 50-fold increase in speed. The LLM exhibited high accuracy in identifying pathogens (98.1% match with the reviewer), hosts (91.7% match), and countries (100% match). However, accuracy was lower for year (72.1% match) and incidence of infection (23.8% match). Discrepancies in year data often occurred when the EIDs were observed across multiple years. For incidence, the LLM frequently assigned 100% incidence when no data were provided in the report. The LLM showed a notable ability to geocode locations even when coordinates weren't explicitly provided, with high accuracy (98.6%) but uncertain precision. The LLM occasionally struggled with converting geographic coordinates to decimal degrees and occasionally returned coordinates that were not present in the original report or that represented locations outside of the expected range. The analysis of the key findings identified errors of commission, where the LLM added data not present in the source material, and errors of omission, where data were missing from the LLM's output. Notably, the LLM demonstrated the ability to distinguish between pathogen and host names based on the context of the text.

The results highlight the remarkable speed and relatively high accuracy of the LLM for extracting specific types of ecological data, particularly categorical and discrete information. This suggests significant potential for accelerating large-scale data collection in ecological research. However, the lower accuracy in quantitative data extraction, especially concerning incidence, underscores the need for robust quality assurance procedures. The LLM's tendency to "fill in" missing incidence data with 100% values raises significant concerns. Fine-tuning the LLM or utilizing different models might address some of these limitations. The unexpected ability of the LLM to geocode locations adds substantial value but requires further investigation into its methods. The study's limitations include the use of relatively short, data-dense reports, which might not fully represent the LLM's performance on longer, more complex texts. The simplicity of the data extracted might also have influenced the accuracy rates. Despite these limitations, the study’s findings have important implications for future ecological research.

LLMs offer substantial potential for accelerating ecological data extraction from scientific literature, drastically reducing the time required for this task. However, the limitations revealed, particularly concerning quantitative data and the need for quality assurance, must be carefully addressed. Future research could focus on fine-tuning LLMs for ecological applications, comparing the performance of different LLMs, and developing automated quality control mechanisms. The ability to reliably extract even discrete/categorical data offers valuable opportunities for identifying novel ecological relationships and accelerating scientific discovery.

The study's results may not be generalizable to all types of ecological data or all lengths of scientific texts. The specific LLM used and the nature of the source reports might influence the results. The relatively small sample size of 100 reports limits the statistical power. Further research is needed to evaluate the generalizability of the LLM's performance across diverse data types, text lengths, and LLMs.