Applying the Rasch model to analyze the effectiveness of education reform in order to decrease computer science students' dropout

Higher education institutions face increasing pressure to reduce student dropout rates. This study focuses on computer science (CS), a field with particularly high attrition (60% at the studied university between 2010 and 2016, mirroring trends globally). The high dropout rate presents not only an educational challenge but also an economic one, given the significant demand for computer scientists. Introductory Mathematics is a frequent stumbling block, with around 30% of students leaving CS after the first semester and 60% by the end of the first year. The study aims to analyze the CS curriculum to identify factors contributing to dropout and to evaluate the effectiveness of an implemented educational reform designed to improve student retention. Several theories attempt to explain student persistence, focusing on individual characteristics, institutional integration (academic and social), and the interaction between students and the institution. Interactional theories suggest that strong student-student and student-teacher relationships are crucial for retention. However, there's a lack of conclusive evidence showing that these factors alone provide a reliable predictive model for student engagement. The current research delves into this gap by investigating specific academic factors and implementing an intervention program to address them. Prior research indicates that difficulties in core CS subjects, particularly mathematics and programming, contribute significantly to attrition. Success rates in introductory programming courses are often below 70%, and difficulties in foundational courses frequently discourage students from continuing their studies. Studies also highlight the role of prior math proficiency in high school, memory skills, and academic self-efficacy in predicting success in engineering and related fields. Several intervention strategies have been proposed, including targeted support for challenging courses, self-selected learning groups, orientation sessions, academic advising, and counseling services. These interventions, however, often show limited effectiveness or focus on students already at risk of dropping out.

The literature review extensively examines existing theories of student persistence, highlighting the interactional model proposed by Tinto (1975, 2012), which emphasizes the interplay between student characteristics and their integration into the academic and social environment. Pascarella and Terenzini (1983) support the importance of social and academic integration. Braxton and Hirschy (2004) stress the role of campus community in social integration, while Terenzini and Reason (2005) and Reason (2009) emphasize the interaction between pre-college experiences and institutional factors. The literature review also delves into studies on CS dropout, pointing towards difficulties in mathematics and programming courses as major contributors. Studies by Divjak et al. (2010), Bennedsen and Caspersen (2007), and Watson and Li (2014) confirm the significant failure rates in these areas. Further, the literature explores student characteristics, such as prior math performance (Pearson and Miller, 2012), memory skills (Bacon and Stewart, 2006; Rawson et al., 2013), and academic self-efficacy (Robbins et al., 2004), as predictors of academic success. Finally, the review discusses various intervention techniques used to reduce attrition, including targeted course support, peer learning groups, mentoring, and academic counseling (Blanc et al., 1983; Tinto, 2005; Gregerman et al., 1998; Pascarella et al., 1986; Bowman et al., 2019; Humphrey, 2005; Hwang et al., 2014; Kot, 2014; Wlazelek and Coulter, 1999; Mellor et al., 2015; Morisano et al., 2010; Sneyers and De Witte, 2018; Herpen et al., 2019). However, many of these interventions are voluntary, lacking the comprehensive approach adopted in the current study.

This study employed a quasi-experimental design to evaluate the impact of an educational reform on CS student attrition. The reform involved two key interventions: (1) making all theoretical lectures compulsory, and (2) introducing a mandatory "Preparation course for university studies and developing learning skills" for all first-year students. This course included training in study skills, time management, and motivation techniques, along with mentoring and peer support. The study analyzed data from 3673 undergraduate CS students from 2010 to 2018, comparing attrition rates and student performance before and after the reform implemented in 2016. The main analytical method was Item Response Theory (IRT) based on the Rasch model, applied to student grades in various CS subjects. The Rasch model allows for the simultaneous scaling of student ability and item difficulty, providing a more nuanced understanding of student performance than traditional grading methods. The model incorporates two key parameters for each subject: difficulty and slope. Difficulty indicates how challenging the subject is, while slope reflects the item's ability to differentiate between students with different levels of ability. This approach enables examination of how the reform influenced both the difficulty of subjects and their ability to discriminate between students with varying abilities. The analysis used STATA15 software to analyze final grades in key mathematics and programming courses, comparing results from the pre-reform (2010-2015) and post-reform (2016-2018) periods.

The study found a significant decrease in the CS student dropout rate after the 2016 educational reform. The dropout rate fell from 48% (1776 out of 3671 students) between 2010 and 2015 to 20% (168 out of 809 students) between 2016 and 2017, representing a 28% reduction in attrition. Rasch model analysis revealed changes in the difficulty and discriminatory power of various subjects. In mathematics-related courses, such as Discrete Mathematics, the difficulty of achieving lower grades decreased after the reform, suggesting that students with lower ability levels were more likely to attempt and pass exams. Conversely, in programming courses, the difficulty of achieving lower grades increased, indicating a higher bar for success. However, even in these programming subjects, the achievement of higher grades became easier for students once they had passed the initial hurdle. This suggests that the reform created a more centralized effect, making it harder to pass the initial level of some programming subjects, but easier to achieve higher marks once this level was reached. This suggests that the intervention had increased the overall rigor of the program, while also ensuring that it did not result in a disproportionate increase in dropouts. The analysis of specific courses revealed that the educational reform made achieving passing grades easier in some subjects, and particularly in discrete mathematics. This resulted in more students with lower-level abilities attempting the examinations.

The findings strongly suggest that the implemented education reform played a significant role in reducing CS student attrition. The decrease in the dropout rate from 48% to 20% directly addresses the research question and shows the effectiveness of the intervention. The compulsory attendance policy and the introduction of the learning methodology course likely improved student engagement and provided crucial support, thereby contributing to higher success rates. The Rasch model analysis offers a more granular understanding of the impact on specific subjects, demonstrating how the reform affected both the difficulty of courses and their ability to differentiate between students with varying abilities. The results align with interactional theories of student persistence, indicating that the reform improved both academic and social integration. The comprehensive nature of the intervention, encompassing all first-year students rather than just those at risk, appears to be a crucial factor in its success. The study's findings have significant implications for CS education, highlighting the potential of mandatory interventions in improving student retention. The observed changes in the difficulty and discrimination parameters of different subjects provide valuable insights for curriculum design and teaching strategies.

This study demonstrates the effectiveness of a comprehensive educational reform in significantly reducing CS student attrition. The mandatory attendance policy and the newly introduced learning methodology course, focusing on study skills and peer support, proved highly effective in improving student success and retention. The Rasch model analysis provided a detailed picture of the impact on individual courses, indicating that the reform made some subjects more accessible to lower-ability students, while maintaining the rigorous standards of other subjects. Further research could explore the long-term effects of the reform, investigate the specific elements of the intervention most responsible for success, and examine the generalizability of these findings to other academic contexts and disciplines.

The study's conclusions should be viewed with caution due to certain limitations. The data were collected from a single university in Hungary, limiting the generalizability of the findings to other contexts. The study's focus on academic performance and attrition may not fully capture other factors influencing student decisions to leave CS, such as career opportunities or personal reasons. Future research could incorporate additional variables, such as student demographics, pre-college preparation, and career aspirations, to provide a more comprehensive understanding of student attrition. Furthermore, the study did not include a control group; thus, any observed changes could be due to other factors rather than the intervention program. The analysis relies on data collected from a single institution and country, and might not be generalizable to other educational contexts. Future research should address these limitations for broader implications.