The impact of the first wave of COVID-19 on students' attainment, analysed by IRT modelling method

During the first wave of COVID-19, universities rapidly shifted from on-campus to fully online instruction. This study investigates how that abrupt transition affected student dropout and academic performance in a Hungarian context, focusing on a large public university where online education began in March 2020. The paper addresses broader issues like inequality and accessibility but centers on academic outcomes. The research formulates three hypotheses: (H1) Grade point averages (GPAs) differ between cohorts experiencing on-campus versus online education in their second semester; (H2) In several core computer science subjects, passing became easier during online education; (H3) Online education increased dropout compared with on-campus education. Understanding these effects is important because GPA predicts dropout and determines scholarships in Hungary, and because high attrition in computer science threatens workforce needs.

Theoretical models of student persistence (Tinto, 1975; 2012) emphasize academic and social integration, personal characteristics, and institutional fit; social integration and campus community are crucial (Braxton & Hirschy, 2004). High dropout has economic and educational costs, drawing policy attention (Di Pietro, 2006; Belloc et al., 2011; Cabrera et al., 2006). Online education predates COVID-19 and offers flexibility and accessibility, with engagement-enhancing strategies (Molins-Ruano et al., 2014; Jovanovic et al., 2019). Humanizing online learning aims to reduce equity gaps (Pacansky-Brock & Vincent-Layton, 2020). Non-traditional students often face external constraints influencing persistence (Bean & Metzner, 1985). Historically, online courses report higher dropout than on-campus (Carr, 2000; Willging & Johnson, 2019; Levy, 2007; Patterson & McFadden, 2009). Persistence relates to learner autonomy and self-regulation (Wang et al., 2013; Serdyukov & Hill, 2013). COVID-19 introduced emergency remote teaching, raising concerns about learning loss, motivation, and wellbeing (Andrew et al., 2020; Bayrakdar & Guveli, 2020; Brown et al., 2020; Bol, 2020; Rahiem, 2021; Abilleira et al., 2021; Daumiller et al., 2021). Evidence is mixed: some studies report decreased perceived success and increased cheating (Daniels et al., 2021), higher anxiety and challenges in math (Mendoza et al., 2021), no performance difference (Said, 2021), or improved performance during lockdown (Iglesias-Pradas et al., 2021; Gonzalez et al., 2020; Yu et al., 2021). Teachers faced disruptions in assessment and adopted innovative digital strategies with unclear grading impacts (Burgess & Sievertsen, 2020; Adedoyin & Soykan, 2020; Gamage et al., 2020; Rapanta et al., 2020; Watermeyer et al., 2021). Given the mixed findings, the present study examines objective performance data using IRT to assess changes in subject difficulty and differentiation.

Design and context: Comparative cohort study at a large European public university’s Computer Science BSc programme. Cohorts: students who started in 2018 (studied on-campus throughout first year) versus those who started in 2019 (second semester moved online in March 2020). The programme has six semesters; each subject is graded 1 (fail) to 5 (excellent). Only final grades per subject were analyzed. Sample: 862 students total: 438 (2018 cohort; on-campus) and 447 (2019 cohort; second semester online). Dropout counts: 50 (2018) and 19 (2019). GPA of second semester: 3.3 (2018) vs 2.5 (2019). Measures and subjects: Final grades from first-year core CS subjects whose content and assessment remained stable across years were included: Mathematical Foundations; Programming; Computer Systems (lecture+practice); Imperative Programming; Functional Programming; Object-oriented Programming (lecture+practice); Algorithms and Data Structures I (lecture and practice); Discrete Mathematics I (lecture and practice); Analysis I (lecture and practice). Analytic approach: Item Response Theory (IRT) using a 2-parameter Graded Response Model (GRM) for ordered categories (grades 1–5), implemented in STATA 15. For each subject, item parameters estimated were: (a) slope (discrimination) and (b) difficulty thresholds for each grade category. The modelling aligns student ability and subject difficulty on a common latent scale (Rasch, 1960; Forero & Maydeu-Olivares, 2009). Parameters were estimated separately for 2018/2019 (on-campus) and 2019/2020 (online in spring) to compare changes in difficulty and discrimination. The study previously applied related IRT approaches to curriculum reform evaluation (Takács et al., 2021). Hypotheses: H1 compared second-semester GPAs across cohorts; H2 tested whether passing thresholds decreased (i.e., subjects became easier to pass) during online education; H3 compared dropout between cohorts. Descriptive comparisons were reported in line with the administrative decision context.

- GPA difference (H1): Second-semester GPA was lower for the cohort experiencing online education (2019/2020: 2.5) than for the on-campus cohort (2018/2019: 3.3). - Dropout (H3): Dropout counts were 50 (on-campus, n=438) vs 19 (online-spring, n=447). The study concludes no significant difference in dropout between cohorts after the first year; online education did not increase dropout. - Subject difficulty and passability (H2): IRT GRM analyses showed that, in most subjects—Basic/Mathematical Foundations practice, Programming, Imperative Programming (lecture+practice), Functional Programming, Object-oriented Programming (lecture+practice), Algorithms and Data Structures I (lecture+practice), Discrete Mathematics I practice, and Analysis I practice—difficulty thresholds shifted downward during online education, indicating it was easier to obtain passing grades (grades 2–3). However, obtaining high grades (4–5) often became more difficult, suggesting elevated thresholds at the top end. - Discrimination: Some subjects (e.g., Discrete Mathematics I practice) maintained high slopes (>3–4) before and after 2019, indicating strong differentiation across ability levels in both modes. - Behavioral/assessment shifts: More students attempted exams during online education (lower threshold even for grade 1), and potential shifts in teacher grading strategies and student exam-taking strategies likely contributed to the observed patterns. - Scholarship implications: Lower GPAs among online cohort could disadvantage them in GPA-based scholarship systems, despite easier passability in several subjects.

The study addresses whether emergency online education altered performance and dropout. Results show substantial GPA decrease for the online cohort, while dropout did not increase, suggesting that although subjects became more passable at lower grade levels, high achievement (grades 4–5) was harder to attain online. This pattern may stem from combined effects of student strategies (greater exam registration even when underprepared; altered study habits in isolation) and teacher strategies (compensatory grading or adjusted assessments to ensure passability), alongside modality-specific assessment constraints (e.g., theoretical lecture courses relying on oral exams without materials). Relative to mixed prior literature, these findings align with studies reporting performance changes during COVID-19 but diverge from work finding either no differences or overall improvements. By using IRT, the study disentangles changes in pass thresholds and discrimination, showing that online education can reduce barriers to passing while not facilitating high achievement, which in turn lowers average GPA. Practically, unchanged dropout suggests that online instruction, at least in the short term, did not harm retention. However, GPA-based scholarship systems may become inequitable across modalities due to differential grading thresholds and assessment conditions. Institutions should consider modality-sensitive scholarship policies and targeted academic support to help students achieve higher-grade thresholds in online settings.

The study contributes evidence on the impact of the first COVID-19 wave and the rapid shift to online teaching on CS undergraduates’ outcomes. Key contributions: (1) Confirmed H1: on-campus students attained higher second-semester GPAs than those studying online; (2) Confirmed H2: in many core subjects, it became easier to earn passing grades online, while high grades were more difficult; (3) Did not confirm H3: dropout did not significantly differ between cohorts, indicating online education did not increase attrition in the first year. These results suggest that while online delivery supported retention and passability, it hindered top-end performance, with implications for GPA-based scholarships and equity. Institutions should provide organizational support, flexible structures, and modality-aware assessment and scholarship policies. Future research should broaden variables, include mixed methods, and extend analyses beyond the first two semesters to capture longer-term academic trajectories.

- Scope limited to the first two semesters of a CS BSc; findings may not generalize to later years or other disciplines. - Potential confounders not measured (e.g., prior in-person relationships, baseline knowledge, adaptability to online learning) could affect GPA and dropout. - Administrative data focus may not fully reflect student engagement; qualitative insights were not included. - The study period captures the initial emergency transition, with unprepared institutions and evolving teacher practices; results may differ under mature online implementations. - Future work should include additional variables, employ diverse data analysis techniques, and use larger mixed-methods designs to validate and deepen understanding.