DeepLMS: a deep learning predictive model for supporting online learning in the Covid-19 era

The study addresses the urgent need to support and evaluate online learning quality during and beyond the Covid-19-driven shift from face-to-face to digital instruction. While LMSs provide platforms for interaction, educators lack predictive tools to gauge and respond to learners' engagement quality. The authors propose forecasting the Quality of interaction (Qol) within LMS environments to inform timely, personalized feedback and instructional decisions. The research explores whether deep learning—specifically LSTM networks—can model and predict users' LMS-based Qol across diverse contexts, time periods (pre- and during Covid-19), and populations (professors and students), thereby enhancing metacognitive support for learners and evaluation capabilities for educators.

Prior work in online and blended learning highlights the importance of interaction quality in OLEs as a predictor of success. LMS logs have been used for descriptive analytics and predictive tasks such as early detection of at-risk students, learning dispositions, and performance prediction, but typically do not evaluate Qol itself. Earlier Qol approaches include: (i) FuzzyQoL, translating expert knowledge and LMS metrics into a normalized Qol index for users; (ii) semiotic frameworks identifying human information interaction issues via questionnaires; and (iii) Fuzzy Cognitive Maps (FCM-QoI) modeling influential concepts and dynamics of Qol. Other studies clustered LMS behaviors (effort, time, procrastination) and linked them to achievement, but lacked generalization and focused on grades rather than interaction quality. The gap remains for predictive modeling of LMS-based Qol that can drive proactive, personalized feedback. DeepLMS aims to provide such predictive capability using LSTM networks, building upon the FuzzyQoL-derived Qol as ground truth while overcoming FCM-QoI limitations (e.g., reliance on mean behaviors).

Datasets: Three LMS Moodle datasets were used. DB1 (Portugal, 2009/2010; pre-Covid): 75 professors and 1,037 students across five undergraduate courses (358 days), totaling 610,775 interactions. DB2 (UAE, Spring 2020; during-Covid): 3 professors and 180 students in an engineering design course (76 days), 9,646 interactions. DB3 (Greece, Spring–Fall 2020; during-Covid): 1 professor and 52 students in an advanced signal processing discipline (181 days), 27,056 interactions. All users had de-identified data and ethics approvals as applicable. Qol estimation: From 110 LMS interaction metrics (M1–M110), 14 categories (C1–C14) were formed and input to the FuzzyQoL model to produce daily per-user Qol in [0,1]. For uniformity, Qol sequences were represented over 358 days; DB2 and DB3 were linearly interpolated to this length while figures showed original durations. Predictive model (DeepLMS): An LSTM-based time series forecaster predicts Qol(k+1) from historical Qol up to time k. Architecture: sequence input layer; LSTM layer with 1200 hidden units; fully connected layer; regression output layer. Training used Adam optimizer, 300 epochs, initial learning rate 0.005 with drop after 150 epochs by factor 0.2, gradient threshold 1, mini-batch size 128, and L2 regularization (lambda=0.0005). Training was performed on an HPC cluster (KUST) using 24 Ivy Bridge nodes in parallel. Train/test protocol: For each user’s Qol sequence, first 90% used for training and last 10% for testing. The model learns one-step-ahead prediction at each time step. Evaluation metrics: (a) RMSE between FuzzyQoL Qol and DeepLMS-predicted Qol on test segments; (b) Pearson correlation coefficient r between test and predicted Qol (p≤0.05); (c) r_a between derivatives of test and predicted Qol to assess trend dynamics. Distributions across user groups are presented with boxplots. Feedback path: The difference dQol(k)=QolDeepLMS(k+1)−QolFuzzyQoL(k) in [−1,1] informs personalized feedback: negative values signal warnings, positive values signal rewards. Segmentation of [−1,1] into bins allows granularity of feedback content and tone.

- Overall predictive performance: Across datasets and user groups, DeepLMS achieved average testing RMSE < 0.009 and average r ≥ 0.97 (p<0.05), as highlighted in the abstract. - DB1 (Pre-Covid; 358 days; Professors n=75, Students n=1037): Median (±95% CI) metrics: Professors RMSE 0.0065 ± 0.0022; r 0.98 ± 0.06; r_a 0.87 ± 0.08. Students RMSE 0.0086 ± 0.0012; r 0.99 ± 0.01; r_a 0.86 ± 0.02. Time series examples showed accurate tracking of varied Qol patterns over two semesters. - DB2 (During-Covid; 76 days; Professors n=3, Students n=180): Professors RMSE 0.0043 ± 0.0095; r 0.96 ± 0.03; r_a 0.66 ± 0.34. Students RMSE 0.0038 ± 0.0046; r 0.94 ± 0.01; r_a 0.74 ± 0.04. The model captured shifts to near-constant Qol ≈ 0.11 during project demo days with reduced LMS interaction, accurately reflecting trend changes. - DB3 (During-Covid; 181 days; Professor n=1, Students n=52): Professor RMSE 0.0172; r 0.99; r_a 0.90. Students RMSE 0.0039 ± 0.0098; r 0.99 ± 0.08; r_a 0.90 ± 0.09. The model predicted alternating Qol patterns similar across train/test segments. - Group and context robustness: No significant performance differences between professors and students (DB1 Wilcoxon rank sum p=0.070). Cross-country/time/scale comparisons showed no significant differences (RMSE p-values: DB1 vs DB2=0.207; DB1 vs DB3=0.219; DB2 vs DB3=0.387). Sex and age had no significant effect (various p>0.36 for professors; p>0.16 for students across DB1–DB3). - Bias assessment: The authors report balanced male/female distributions, broad age coverage, uniform sampling, equal LMS access, and consistent evaluation metrics, arguing absence of historical, representation, measurement, evaluation, population, sampling, interaction, or self-selection bias. - Baseline comparison: Compared to the related FCM-QoI model (applied on DB1), DeepLMS achieved lower test RMSE and higher r, providing personalized per-user forecasts rather than relying on mean behaviors.

DeepLMS demonstrates that LSTM-based sequence modeling of LMS-derived Qol can reliably forecast short-term interaction quality for both professors and students across diverse institutions, countries, course scales, and pandemic periods. By accurately tracking both levels (r) and dynamics (r_a) of Qol, the model enables a feedback mechanism that can nudge learners toward improved online engagement and inform instructors of emerging disengagement or unstructured interaction, complementing traditional content-based assessment. The robust performance and lack of significant demographic effects suggest strong generalizability. DeepLMS outperforms an FCM-based baseline and can be aggregated to support broader stakeholder decision-making (departmental to institutional). The approach aligns with and can integrate into broader learning analytics efforts (e.g., at-risk detection, engagement communication, skill modeling), and sets a foundation for fair and extensible predictive support in online education.

This work introduces DeepLMS, an LSTM-driven predictive framework that forecasts users’ LMS-based Quality of interaction (Qol) and operationalizes a personalized feedback path. Validated on three real-world datasets spanning pre- and during-Covid-19 periods and multiple educational contexts, DeepLMS achieves low error (median RMSE typically ≤ ~0.009) and high correlation (r up to ~0.99), outperforming an FCM-based baseline while providing individualized predictions. The contributions include: (i) a novel predictive use of LMS-derived Qol; (ii) a portable modeling pipeline agnostic to course content; and (iii) an actionable feedback mechanism to scaffold engagement. Future research directions include correlating Qol predictions with academic outcomes (quizzes/exams), longitudinal validation across multiple academic years, expansion to additional domains and secondary education, fusion with other quality dimensions (Quality of Collaboration and Affective Engagement) toward a holistic a/b/c/d-TEACH framework, and analyses spanning pre/during/post Covid-19 periods to assess robustness to time-related disruptions.

- No correlation analysis was performed between predicted Qol and content evaluation outcomes (e.g., quizzes, midterms, finals), limiting conclusions about links to academic performance. - Temporal scope was limited to one (DB1) or partial (DB2/DB3) academic years; longer-term, multi-year follow-up is needed to assess stability and consistency. - Small numbers of professors in DB2 (n=3) and DB3 (n=1) limit generalizability for instructor-level conclusions in those settings. - Qol ground truth relies on the FuzzyQoL model; while established, it encapsulates expert-defined constructs and may introduce modeling dependencies. - Interpolation to a common 358-day length for DB2/DB3, while practical for modeling, may smooth temporal idiosyncrasies in shorter courses.