Classroom dialogue is crucial for learning, particularly from socio-cultural perspectives, where it mediates between collective and individual thinking. Dialogic pedagogy, emphasizing participation and collaborative understanding, has shown academic benefits across subjects. However, traditional qualitative analysis methods (content, discourse, and thematic analysis) for classroom dialogue are inefficient and subjective. The coding process, involving categorizing data based on themes, is particularly time-consuming and prone to researcher bias. This research explores the potential of Large Language Models (LLMs), specifically GPT-4, to streamline and enhance the analysis of classroom dialogue, leveraging their natural language processing capabilities to process and interpret verbal exchanges in educational settings. The study focuses on classroom dialogue to explore LLMs' role in understanding and enhancing communication within educational contexts, offering a novel perspective on analyzing educational discourse and potentially revolutionizing educational research and practice.

Existing literature highlights the importance of classroom dialogue in learning and the challenges of traditional qualitative analysis methods. Studies have shown the benefits of dialogic teaching approaches on student reasoning, problem-solving, and academic performance. Researchers have proposed various coding schemes for analyzing classroom dialogue, focusing on features like authentic questions, extended contributions, critical engagement, and consensus-building. The work of Alexander, Mercer, and Littleton on productive dialogue forms, and Hennessy et al.'s framework for analyzing classroom dialogue across various contexts, significantly influenced the coding scheme used in this study. Recent research also explores the application of AI in classroom discourse analysis, including work on annotated datasets and the use of deep learning to model teacher discourse. The study draws on these existing studies to provide a foundation for evaluating the application of GPT-4 in classroom dialogue analysis.

The study used classroom dialogue transcripts from middle school mathematics and Chinese classes. Audio and video recordings were transcribed using Lark software. Data included approximately 150,000 characters from six lessons in each subject. A 15-category coding scheme (Table 1), revised from the Cambridge Dialogue Analysis Scheme, was used. This scheme categorized dialogue turns based on features like elaboration invitations, reasoning, coordination, agreement, querying, and references (Table 1). Two coding methods were used: manual coding by experienced researchers and automated coding using a customized GPT-4 model with a retrieval-augmented generation (RAG) system. The manual coding served as a benchmark against which the GPT-4's performance was measured. Inter-human-rater reliability was assessed using Cohen's Kappa (κ). The GPT-4 model was fine-tuned using prompt engineering to align its coding with the established scheme. The evaluation focused on three aspects: time efficiency (comparing manual and automated coding time), inter-coder agreement (percentage of agreement between human and GPT-4 coding), and inter-coder reliability (using Cohen's Kappa to account for chance agreement). SPSS 21.0 was used for statistical analysis.

The study found significant time savings using GPT-4 for coding classroom dialogues. For a single 41-minute math lesson, manual coding took approximately 2.5 hours, while GPT-4 took 5 minutes, a 30-fold time saving (Fig. 1). Inter-coder agreement between human coders and GPT-4 was high, exceeding 90% for both math and Chinese lessons (Fig. 2). However, Cohen's Kappa (κ) values varied across coding categories (Table 3). High Kappa values (near-perfect agreement) were observed for categories like Elaboration Invitation (ELI) and Elaboration (EL). Lower Kappa values (indicating less agreement) were found for categories such as Coordination Invitation (CI), Simple Coordination (SC), and Reasoned Coordination (RC). An example illustrating inconsistencies between human and GPT-4 coding is provided in Table 4. The study also notes that the performance of GPT-4 varied slightly between Chinese and math lessons for some categories, potentially due to differences in dialogue complexity and contextual nuances between the two subjects. The study achieved over 90% inter-coder agreement, surpassing previous studies using specialized intelligence which reported ~80% agreement using a seven-category framework. This showcases the advancement in AI capabilities and cost-effectiveness compared to the limitations of specialized intelligence.

The findings demonstrate the significant potential of LLMs like GPT-4 for automating the qualitative analysis of classroom dialogues, offering substantial time savings and high coding consistency in most categories. The high agreement rates suggest that AI can effectively mirror human coding practices, facilitating large-scale automated assessment in educational contexts. However, the lower consistency in certain categories (CI, SC, RC) highlights the challenges of automating the interpretation of complex and nuanced interactions. These discrepancies may arise from GPT-4's reliance on explicit textual cues compared to human coders' use of broader context and implicit meanings. Future research should explore ways to enhance AI's ability to capture subtle contextual nuances. The study's results are promising but need further validation across a broader range of disciplines, age groups, and learning environments to ensure generalizability.

This study demonstrates the transformative potential of LLMs, particularly GPT-4, in analyzing classroom dialogues. The significant time savings and high inter-coder agreement highlight the practicality and efficiency of AI in educational research. While limitations exist, particularly concerning the interpretation of nuanced interaction categories, the findings suggest a promising direction for scalable qualitative educational diagnosis and assessment. Future research should address these limitations by expanding the scope to include diverse subjects, age groups, and learning environments, thus enhancing the generalizability of findings and further refining the application of AI in educational research.

The study's sample size and scope are limited. The findings may not generalize to all subjects, age groups, and learning environments. Variations in interaction frequency across different subjects and age groups were not fully represented. The complexity of certain coding categories and the reliance on textual cues by GPT-4 may contribute to lower inter-coder reliability in those areas. Future research needs to address these limitations for broader applicability.