Investigating students’ programming behaviors, interaction qualities and perceptions through prompt-based learning in ChatGPT

In today's tech-driven culture, programming education is becoming increasingly vital due to its role in enabling efficient handling of repetitive tasks and fostering problem-solving skills essential for modern, data-driven workplaces. Despite its benefits, students face obstacles in programming education. Rapid advances in AI have aligned with programming learning, with resources like ChatGPT aiding coding ability and task understanding by providing explanations, responding to queries, and offering broad support. ChatGPT’s natural language capabilities and conversational interface can enhance accessibility and efficiency, yet limitations exist: it may present single, unevidenced answers, potentially leading students to accept outputs uncritically. Many students use ChatGPT haphazardly, assuming accurate outputs regardless of input quality, which underscores the need for scaffolding. Prompt-based learning (PbL) offers a no-code approach to instruct LLMs through structured prompts, potentially improving outcomes. The study investigates whether including prompts in ChatGPT interactions changes its usefulness for programming education and identifies effective approaches. Acceptance of technology, modeled by the Technology Acceptance Model (TAM), influences learning performance; TAM posits perceived usefulness and ease of use shape attitudes and behavioral intention. While TAM has been applied to AI-based learning and programming contexts, little research examines students’ perceptions of using prompts with ChatGPT in programming. This study explores the impact of prompts on college students’ programming behaviors, interaction qualities, and perceptions when using ChatGPT, comparing prompt-based learning (PbL) and unprompted learning (UL). Research questions: - RQ1: How does PbL affect learners’ programming behaviors versus UL? - RQ2: How does PbL affect learners’ interaction quality with ChatGPT versus UL? - RQ3: How does PbL affect learners’ perceptions of ChatGPT versus UL? Insights aim to inform policymakers, curriculum designers, and educators on integrating prompts with ChatGPT in programming instruction, potentially improving programming education practices and contributing to global discourse on AI-supported learning.

Advancing programming education with ChatBots: Programming is a practical and essential skill for digital innovation. AI tools—including chatbots, interactive coding platforms, and virtual instructors—can enhance engagement and accessibility in programming education, support clearer code and quicker debugging, and promote self-directed learning at individualized paces. ChatGPT, notably, provides personalized, natural language explanations and demonstrates strong performance across programming tasks (e.g., program repair, hint generation, grading feedback, peer programming, task development, contextualized explanations). It supports learners at varying levels, assists with numerical methods, and can generate, debug, enhance, complete, and modify code across languages. While effective for many tasks, limitations appear on complex tasks requiring integrated explanations or multiple class files. Proper understanding of ChatGPT’s constraints helps design assessments that reduce cheating while remaining effective. ChatGPT can also make complex topics accessible via real-world examples and analogies, reduce programming time, expand solution options, and guide users through explanations, examples, and code optimization. Prompt utilization in ChatBots study: Explicit queries and instructions (prompts) steer LLM outputs. Prompt-based techniques enable zero-shot performance without additional training and have broad applicability, supporting diverse instructional goals. Carefully crafted prompts direct discourse, elicit targeted information, and improve response accuracy. Prompt-based learning involves constructing templates describing tasks and inserting inputs to guide predictions; in this study, learners use prompt templates to communicate with LLMs. Benefits include directing chatbot behavior, obtaining precise information, and improving learning outcomes by aligning dialogue with student interests. Frameworks exist for domain-relevant prompt creation and benchmarking student queries, and tools like Promptly help teach prompt construction in programming courses. Industry resources (e.g., DeepLearning.AI’s ChatGPT Prompt Engineering for Developers) provide principles and tactics—such as clear instructions, giving the model time to think, using delimiters, requesting structured output, checking conditions, and few-shot prompting—which can be applied to programming education. Prompts mediate between users and LLMs, shaping analysis and generation to meet varied user goals.

Research context and participants: The study took place in a mandatory Python Programming course for Educational Technology students at Hangzhou Normal University in Spring 2023. A quasi-experimental design compared two conditions: control (Unprompted Learning, UL) and experimental (Prompt-based Learning, PbL), each with 15 learners. UL group: 9 females, 6 males; PbL group: 7 females, 8 males. Both groups were taught by the same instructor with consistent materials and guidance. All students had prior C programming experience. Ethical clearance was obtained. Instructional procedures: The course spanned six 90-minute sessions across three phases. Phases I–II covered Python basics (intro, data and control structures, functions/methods like recursion and lambda). Phase III required students to develop a Python project aided by ChatGPT within 90 minutes. Both groups received lectures and demonstrations, then performed self-guided programming tasks. The ChatGPT Next platform (deployed by the first author) with gpt-3.5-turbo enabled question–answer interactions. UL students interacted based on their own approaches (e.g., natural/oral language); PbL students used additional prompts. Prompt design: Prompts were designed per DeepLearning.AI’s “ChatGPT Prompt Engineering for Developers” guidelines, emphasizing two principles: crafting clear, precise instructions and giving the model time to think. Tactics included using delimiters, requesting structured output, checking satisfaction of conditions, and few-shot prompting; and, for thinking time, specifying steps and instructing the model to reason before concluding. The prompt set was contextualized to a Word Cloud project with good/bad examples (e.g., generic vs. specific function requests) and stepwise task guidance. Data collection and analysis: Three data sources were used. 1) Programming behaviors: Screen-capture videos (final session; 90 min per learner; total 2,700 min) and platform logs were analyzed. Clickstream analysis identified behaviors; videos were coded iteratively using a pre-established framework (e.g., Understanding task, Coding in Python, Debugging, Checking output/console, Asking new questions, Pasting code/messages to/from ChatGPT, Reading feedback, Copy/paste, Referring to materials, Idle, and ChatGPT failures). Lag-sequential analysis (LSA) with Yule’s Q measured transitional associations and visualized behavior networks across conditions. 2) Interaction with ChatGPT: Platform logs captured learner queries and ChatGPT feedback. Using a coding framework for knowledge inquiry (irrelevant, superficial, medium, deep), feedback inquiry (superficial, medium, deep), and ChatGPT feedback (general, specific, correct, incorrect), interactions were segmented (PbL: 281 units; UL: 459 units) and coded by two researchers with interrater reliability of 0.81. Data were reformatted for cognitive network analysis and analyzed via webENA to compare epistemic networks. Mann–Whitney tests assessed group differences on network dimensions. 3) Perceptions: Surveys (7-point Likert) based on TAM (Venkatesh & Davis, 2000; Sánchez & Hueros, 2010) measured perceived usefulness, perceived ease of use, behavioral intention to use, and attitude toward using ChatGPT. Pre- and post-tests were analyzed with independent t-tests and descriptive statistics.

Programming behaviors (RQ1): Frequency analyses showed distinct emphases. In PbL, dominant behaviors included reading feedback (RF=538), asking new questions (ANQ=414), and understanding Python code (UPC=249). In UL, Python coding (CP=928) and reading feedback (RF=927) were most frequent. Lag-sequential analysis highlighted group-specific patterns: PbL learners commonly coded then debugged (CP→DP; Yule’s Q=0.83) and copied code from ChatGPT then debugged (CPC→DP; Q=0.74). UL learners frequently pasted Python code into ChatGPT then asked new questions (PPC→ANQ; Q=0.94) and copied code from ChatGPT then continued coding (CPC→CP; Q=0.90). Interaction quality (RQ2): Epistemic network analysis (ENA) revealed that UL students had stronger links involving superficial-level knowledge inquiry with correct/specific feedback (e.g., SKI–CFC, SKI–SFC) and medium-level feedback inquiry with correct/specific feedback (MFI–CFC, MFI–SFC). PbL students showed stronger connections involving medium-level questioning and correct feedback (e.g., MFI–CFC). Subtracted network comparisons showed a significant difference along the primary ENA dimension (MR1): PbL vs UL (Median PbL=0.14; Median UL=-0.10; U=54.00; p=0.00; r=0.58). PbL learners displayed enhanced interconnections among SKI–MKI, MKI–CFC, MKI–SFC, and MKI–GFC, suggesting more mid-level, self-processed queries leading to accurate feedback; UL learners engaged more at a surface level yet still often received correct feedback. Perceptions (RQ3): Pre-test showed no significant differences in perceived usefulness, behavioral intention, or attitude; UL had higher perceived ease of use (t(28)=2.27, p=0.03). Post-test means favored PbL across all constructs. A significant difference emerged for attitude toward using ChatGPT favoring PbL (t(28)=-2.26, p=0.03). Descriptives (post): Perceived usefulness—UL M=4.68 (SD=1.22), PbL M=5.23 (SD=1.06); Perceived ease of use—UL M=4.73 (SD=1.46), PbL M=4.86 (SD=1.18); Behavioral intention—UL M=5.02 (SD=1.35), PbL M=5.53 (SD=0.86); Attitude—UL M=3.82 (SD=1.28), PbL M=4.75 (SD=0.94; p<0.05). Learners preferred prompt tactics such as requesting structured output and using delimiters, which improved understanding of problem-solving steps and efficiency.

Findings address the research questions by showing that prompt-based learning (PbL) shapes both behaviors and the cognitive quality of interactions with ChatGPT. PbL fostered iterative cycles of coding and debugging and encouraged students to formulate more processed, mid-level questions, aligning with constructivist and self-regulated learning principles. In contrast, UL students displayed more hands-on coding but relied more on copying and adjusting code from ChatGPT and posed more superficial questions, indicating surface-level engagement. ENA demonstrated that PbL strengthened connections between mid-level knowledge inquiry and accurate feedback, supporting deeper cognitive processing and integrated understanding. Perception results indicate that structured prompts scaffolded learners’ expectations and effective use of ChatGPT, improving attitudes and perceived ease of use post-intervention. The preference for structured output and delimiters underscores the value of clarity and organization in AI-supported learning. Overall, the pedagogical context—specifically structured prompting—amplifies ChatGPT’s educational benefits, improving interaction quality, promoting reflective inquiry, and enhancing learner attitudes.

This quasi-experimental study demonstrates that structured prompt-based learning with ChatGPT can positively influence college students’ programming behaviors, deepen the cognitive quality of their interactions, and improve perceptions toward using ChatGPT. PbL led to iterative coding–debugging cycles, more self-processed questioning, and stronger links to accurate feedback, while also improving learners’ attitudes toward ChatGPT. The results suggest that integrating structured prompts into programming instruction can enhance the effectiveness of AI tools like ChatGPT by aligning with constructivist pedagogy and scaffolding student learning. Future research should generalize these findings across varied prompt designs, investigate prompt integration at different instructional stages (including prompts that encourage reflection and critical thinking), and conduct longer-term studies to examine sustained effects on learning outcomes and perceptions.

The study’s generalizability is limited by the use of a single prompt version; future work should test diverse prompt variations and structured prompt designs. Instructional design plays a crucial role; studies should explore integrating prompt-based methods across instructional stages, especially prompts facilitating reflection, feedback uptake, and critical thinking development. The six-week duration limits insights into long-term effects; longitudinal research is needed to understand sustained impacts of prompt utilization on learning outcomes and perceptions.