The integration of artificial intelligence (AI) with various disciplines is profoundly impacting numerous sectors, including academic research. Psychology, situated at the intersection of humanities and natural sciences, continuously strives to understand the complexities of human behavior and cognition. While AI models have been developed based on brain structures and attention systems, and have in turn offered insights into human cognition, psychology has relied largely on theory-driven methodologies. This reliance contrasts with the data-centric approaches prevalent in AI research. Hypothesis generation, a cornerstone of psychological research, is often based on existing frameworks, observations, data anomalies, or interdisciplinary discoveries. Causal graphs provide a systematic framework for modeling complex systems in psychology, enabling a holistic view of bio-psycho-social interactions. However, their use is labor-intensive and requires multidisciplinary expertise. Large Language Models (LLMs) like GPT-3, GPT-4, and Claude-2 offer promising avenues for hypothesis generation due to their ability to comprehend and infer causality from text. This study proposes a framework that combines the strengths of LLMs and causal graphs to generate psychological hypotheses. The researchers aim to construct a comprehensive causal network of semantic concepts within psychology by analyzing a large corpus of psychological articles, extracting causal pairs, and utilizing link prediction algorithms to identify novel causal relationships and generate hypotheses. This methodology allows for the exploration of potential causal relationships that might not be apparent through traditional research methods.

The introduction extensively discusses the intersection of AI and psychology, emphasizing the limitations of traditional theory-driven approaches in psychology compared to the data-centric methods used in AI. It highlights the importance of hypothesis generation in psychological research and the potential of causal graphs as a systematic framework for modeling complex psychological phenomena. The limitations of traditional methods in processing large amounts of data are also pointed out. The authors then introduce LLMs as a powerful tool for extracting causal knowledge from text, emphasizing the complementary strengths of LLMs and causal graphs and the potential of their synergy. Existing literature on LLMs and their applications in various fields is also referenced, particularly focusing on their ability to understand and infer causality.

The proposed LLM-based causal graph (LLMCG) framework comprises three main steps: literature retrieval, causal pair extraction, and hypothesis generation. Step 1 involved retrieving approximately 140,000 psychology-related articles from the PMC Open Access Subset, applying keyword filters and using metadata to refine the selection process. A subset of 43,312 articles was ultimately used. Step 2, causal pair extraction, involved multiple sub-steps: (1) cost analysis to determine feasibility, (2) text extraction and cleaning using PyPDF2 to remove irrelevant sections, (3) causal knowledge extraction using GPT-4 to identify causal relationships and format them in JSON, and (4) graph database storage using Neo4j to represent the causal pairs as nodes and relationships. The extraction process included prompt engineering, filtering, and expert validation. An exploratory study was conducted to assess GPT-4's accuracy in distinguishing between causality and correlation. Step 3, hypothesis generation, involved using node2vec for vector embedding and the Jaccard similarity index for link prediction to identify potential causal relationships between unconnected concepts in the Neo4j database. GPT-4 was then used to generate hypotheses based on these predicted relationships. The methodology also included a rigorous quality control process, incorporating expert review and iterative refinement.

The researchers generated 130 hypotheses focusing on "well-being" using their LLMCG algorithm. These hypotheses were compared to hypotheses generated by 13 PhD students (Control-Human group, 41 hypotheses, 30 selected), and the Claude-2 LLM (Control-Claude group, 50 hypotheses, 30 selected). Three expert psychology professors rated the hypotheses (120 total) for novelty and usefulness. Statistical analysis (ANOVA, Bonferroni post-hoc tests) revealed significant differences: The LLMCG hypotheses, both randomly selected (Random-selected LLMCG) and expert-selected (Expert-selected LLMCG), showed significantly higher novelty scores compared to the Control-Claude hypotheses. The Control-Human group also exhibited significantly higher novelty scores than the Control-Claude group. There was no significant difference in usefulness scores between the groups. Deep semantic analysis using BERT and t-SNE visualizations showed that the Control-Human group had a larger semantic distance compared to other groups, while the LLMCG groups showed broader topic dispersion. An ablation study comparing GPT-4-only generated hypotheses with LLMCG hypotheses demonstrated significantly higher novelty scores for LLMCG, without impacting usefulness scores. This suggests that integrating a causal graph with an LLM significantly improves the novelty of generated hypotheses.

The findings demonstrate the potential of integrating LLMs with causal graphs for generating novel and useful hypotheses in psychology. The superior performance of the LLMCG algorithm in generating novel hypotheses, comparable to those of human experts, highlights the effectiveness of combining the strengths of LLMs and causal graph techniques. The deep semantic analysis further supports these findings by showing a broader semantic spectrum in the LLMCG hypotheses. The study's results challenge the conventional belief that hypothesis generation in psychology heavily relies on human expertise, offering a promising path towards automating the discovery process. The findings also underscore the importance of integrating structured knowledge, such as causal graphs, with the generative capabilities of LLMs to enhance both novelty and interpretability.

This research presents a novel framework for automating hypothesis generation in psychology by combining large language models and causal graphs. The results indicate that this approach yields hypotheses of comparable novelty to those generated by human experts, significantly outperforming LLM-only methods. This suggests a powerful new tool for accelerating discovery within the field. Future research could explore the application of this framework to other areas of psychology and investigate ways to further improve the accuracy and efficiency of the hypothesis generation process. Refining the causal graph construction and expanding the validation process to a broader range of hypotheses are key directions.

The study acknowledges limitations such as potential inaccuracies in causal relationship graph construction (approximately 13% of relationship pairs did not align with human expert estimations). The validation process was limited to 130 hypotheses, and the evaluation of hypotheses by expert panels showed some inconsistencies. Future research should focus on improving the accuracy of causal graph construction and expanding the validation process to encompass a more extensive range of hypotheses. The study's focus on "well-being" might limit the generalizability of findings to other areas of psychological research.