Investigating students’ programming behaviors, interaction qualities and perceptions through prompt-based learning in ChatGPT

ChatGPT has proven to facilitate computer programming tasks through the strategic use of prompts, which effectively steer the interaction with the language model towards eliciting relevant information. However, the impact of specifically designed prompts on programming learning outcomes has not been rigorously examined through empirical research. This study adopted a quasi-experimental framework to investigate the differential effects of prompt-based learning (PbL) versus unprompted learning (UL) conditions on the programming behaviors, interaction qualities, and perceptions of college students. The study sample consisted of 30 college students who were randomly assigned to two groups. A mixed-methods approach was employed to gather multi-faceted data. Results revealed notable distinctions between the two learning conditions. First, the PbL group students frequently engaged in coding with Python and employed debugging strategies to verify their work, whereas their UL counterparts typically transferred Python code from PyCharm into ChatGPT and posed new questions within ChatGPT. Second, PbL participants were inclined to formulate more complex queries independently, prompted by the guiding questions, and consequently received more precise feedback from ChatGPT compared to the UL group. UL students tended to participate in more superficial-level interactions with ChatGPT, yet they also obtained accurate feedback. Third, there were noticeable differences in perception observed before and after the ChatGPT implementation, UL group reported a more favorable perception in the perceived ease of use in the pre-test, while the PbL group experienced an improvement in their mean scores for perceived usefulness, ease of use, behavioral intention to utilize, and a significant difference regarding the attitude towards utilizing ChatGPT. Specifically, the use of structured output and delimiters enhanced learners' understanding of problem-solving steps and made learning more efficient with ChatGPT. Drawing on these outcomes, the study offers recommendations for the incorporation of ChatGPT into future instructional designs, highlighting the structured prompting benefits in enhancing programming learning experience.