Applying the Rasch model to analyze the effectiveness of education reform in order to decrease computer science students' dropout

The study addresses the persistent problem of high dropout rates in computer science (CS) programs. At a large European public university, overall dropout is about 30%, but in Informatics it averaged 60% (2010–2016), mirroring trends across Europe and worldwide. Foundational first-year courses—especially mathematics and introductory programming—are key bottlenecks driving early attrition. Guided by interactional theories of persistence (e.g., Tinto) emphasizing academic and social integration, the authors aim to analyze academic success patterns and subject characteristics linked to retention, and to evaluate whether an education reform introduced in 2016 affected attrition and grade patterns. The study focuses on two research questions: (1) To what extent will an education reform affect student attrition? (2) Can evidence be found in subject-level grade patterns using IRT/Rasch analysis?

Prior work identifies first-year STEM gateway courses (mathematics and programming) as critical to attrition, with high failure rates in math (Divjak et al., 2010) and substantial difficulty in introductory programming; a systematic review reported a 67.7% pass rate in introductory programming (Watson & Li, 2014). Early difficulties can reduce persistence in CS (Barker et al., 2009). Pre-college preparation, particularly in mathematics (e.g., calculus), and related high school GPA correlate with engineering/CS success and degree completion (Pearson & Miller, 2012; Hopkins et al., 2016). Psychosocial factors such as academic goals, self-efficacy, and study skills show moderate to strong associations with GPA and retention (Robbins et al., 2004). Multi-factor models (e.g., Giannakos et al., 2017) explain substantial variance in retention via perceived usefulness, cognitive gains, and supportive environment. Interventions targeting difficult entry-level science courses have increased retention by up to ~10% (Blanc et al., 1983; Tinto, 2005), and mentoring, advising, counseling, and small-class interventions often improve GPA and reduce attrition (Wlazelek & Coulter, 1999; Kot, 2014; Mellor et al., 2015; Morisano et al., 2010; Bowman et al., 2019). Meta-analytic evidence shows student–faculty mentoring yields strong positive effects (Sneyers & De Witte, 2018). Despite extensive work, compulsory, universal interventions are rare; thus, further research on structural reforms integrated into curricula is needed.

Design and setting: The study analyzes academic achievement patterns of CS BSc students at a large public European university from 2010 to 2018 using Item Response Theory (IRT) with Rasch-model analysis. The institution grades subjects on a 1–5 scale (1=fail; 2–5=pass). The analysis uses final subject grades. Participants: N=3673 full-time first-year CS students (mean age 19.81). Of these, 2863 started before 2016 and 809 after 2016. Longitudinal counts show 3671 starters (2010–2015), with 1776 (48%) dropping out and 24% retained (N=894). In 2016–2017, 809 registered and 168 (20%) dropped out; others were in progress. Intervention (education reform): Building on earlier mentoring/tutoring efforts (since 2006) and a successful 2015 pilot, a compulsory education reform was introduced in 2016 for all first-year students: (a) all theoretical lectures became mandatory (previously only practice sessions were mandatory), and (b) a compulsory course, “Preparation course for university studies and developing learning skills,” combining an intensive pre-semester training (18 lessons) and an in-semester component (12 hours), delivered by psychologists and peer counselors focusing on motivation, organization, time management, concentration, study strategies (especially for mathematics and programming), and building peer/teacher relationships via fixed mentor groups of ~20 students. The program aimed to enhance competence, autonomy, relatedness, and intrinsic motivation. Analytical approach: Rasch modeling in STATA 15 placed student ability and subject difficulty on a common latent scale. Two parameters were examined for each subject and grade threshold: (1) difficulty (d) at ordered grade thresholds (1–5), and (2) slope/discrimination (s), reflecting how sharply the probability of attaining a grade changes with ability. The probability model used: P(grade at a given ability) = 1 / (1 + e^(−s(a−d))), where a is student ability. Positive slopes indicate non-random differentiation. Subjects analyzed: representative first-year mathematics and programming courses, including Discrete Mathematics 1 (lecture), Discrete Mathematics 2 (practical), Law and Management Theory, and Functional Programming. Pre-2016 and post-2016 periods were estimated separately to assess reform effects on slopes and difficulty thresholds.

Descriptive outcomes: • Between 2010–2015: 3671 starters; 1776 (48%) dropped out; 24% retained (N=894). • 2016–2017: 809 registered; 168 (20%) dropped out; remaining students’ degrees in progress. • Overall, attrition decreased by 28 percentage points following the reform (from 48% to 20%). Rasch-model results (Table 2): • Discrete Mathematics 1 (lecture): – Slope increased from 3.69 (pre) to 4.03 (post), indicating strong and stable discrimination. – Difficulty thresholds shifted downward at lower grades, especially Difficulty 1: −0.599 (pre) to −0.71 (post), suggesting passing became achievable at lower ability levels; more lower-ability students attempted exams. Other thresholds: D2: −0.205→−0.236; D3: 0.735→0.501; D4: 1.298→1.26; D5: 1.661→1.806. • Discrete Mathematics 2 (practical): – Slope: 3.32 (pre) → 3.52 (post), maintaining strong discrimination. – Difficulty thresholds decreased at lower grades (D1: −0.799→−0.947; D2: −0.544→−0.76), with mid–upper thresholds also lowering but to a lesser extent (D3: 0.469→0.054; D4: 1.084→0.737; D5: 1.643→1.353). • Law and Management Theory: – Slope increased from 1.405 to 2.052, indicating improved discrimination. – High-grade thresholds shifted markedly easier post-reform, especially at grades 3–5 (e.g., D3: 0.125→−0.246; D4: 0.802→0.082; D5: 1.353→0.525), implying students with lower abilities could achieve better grades. Lower thresholds changed little (D1: −0.663→−0.685; D2: −0.620→−0.545). • Functional Programming: – Slope slightly increased: 1.613→1.705 (moderate discrimination). – Lower thresholds shifted upward (harder to pass): D1: 0.792→1.111; D2: 0.834→1.144; D3: 1.022→1.199. – Upper thresholds became somewhat easier (D4: 1.414→1.461; D5: 1.912→1.766), indicating that once students surpassed the initial passing threshold, higher grades were more attainable—producing a centralizing effect. Overall patterns: • Mathematics-related subjects became achievable at lower ability levels post-reform, encouraging lower-ability students to attempt and pass exams. • Programming/professional subjects (e.g., Functional Programming) became harder at the passing threshold while retaining discrimination, but higher grades were more reachable once past the initial hurdle. • The observed changes in slopes and thresholds align with increased exam participation and success in mathematics and maintained differentiation in programming. • The education reform coincided with a substantial reduction in early attrition (48% to 20%).

Findings directly address the research questions. First, the education reform coincided with a substantial reduction in attrition (28 percentage-point decrease). Second, Rasch-based grading pattern analyses provide subject-level evidence consistent with increased accessibility of mathematics courses post-reform—lowered difficulty thresholds at passing levels and sustained high discrimination in Discrete Mathematics—indicating more lower-ability students attempted and succeeded in exams. Law and Management Theory showed improved discrimination and notably easier attainment of high grades post-reform, while Functional Programming became harder at the passing threshold but allowed easier attainment of higher grades once the threshold was crossed, maintaining a meaningful spread of performance. Collectively, these patterns suggest that the compulsory learning-skills course, mandatory lecture attendance, and structured mentoring fostered academic and social integration consistent with Tinto’s framework, supporting students to engage, attempt assessments, and progress, especially in mathematically intensive subjects that often drive early dropout. The universality and compulsory nature of the reform distinguishes it from many voluntary interventions reported in the literature, potentially explaining its broad impact across the cohort. Ongoing iterative evaluation each semester is being used to refine the program to further mitigate attrition.

The study demonstrates that a compulsory, cohort-wide education reform—mandating lecture attendance and integrating a structured learning-skills and mentoring course—was associated with markedly reduced early attrition in a CS BSc program and with favorable shifts in subject-level grading patterns. Mathematics-oriented subjects became achievable at lower ability levels, encouraging exam attempts and passes among students who might otherwise disengage, while programming subjects retained discrimination with a higher initial bar but improved access to higher grades once passed. These outcomes highlight the value of embedding non-voluntary, skills-focused interventions into curricula to reach all first-year students, including those unlikely to self-select into support. Future directions include extending analyses to additional variables (e.g., motivation, background preparation), rethinking curricular sequencing, and developing targeted supports for mathematics and programming to sustain motivation and success throughout the degree.

The study is limited to CS students from a single Hungarian university, restricting generalizability. Available data were constrained by ethical restrictions and did not include broader covariates (e.g., comprehensive demographic or psychosocial variables). Analytical scope focused on representative subjects rather than the entire curriculum in detail. Outcomes post-2016 include students whose degrees were still in progress, so longer-term attrition/completion impacts require follow-up. Future research should incorporate additional variables and methods to characterize profiles of dropouts and graduates, and to evaluate curricular restructuring effects.