The COVID-19 pandemic drastically shifted education online, leading to a surge in the use of video conferencing platforms and Learning Management Systems (LMSs) such as Moodle, Blackboard, and Google Classroom. While these platforms provide the infrastructure for e-learning, effective methods to predict learner behavior and provide support are lacking. This research addresses this gap by exploring the application of deep learning to predict the quality of interaction (QoI) within an LMS environment. QoI is a crucial factor in the success of online learning, influencing both learning development and cost efficiency. The ability to predict QoI offers valuable support to educators by providing early warnings of potential issues, allowing for timely interventions. It also empowers learners through personalized feedback, promoting self-reflection and adjustment of their learning strategies. The rapid growth of the online learning industry, expected to reach $325 billion by 2025, underscores the need for such predictive tools to optimize the learning experience and ensure effective resource allocation. Furthermore, the environmental benefits of online learning, with 90% less energy and 85% fewer CO2 emissions compared to face-to-face instruction, further justify the investment in improving its effectiveness.

Existing research on QoI in online learning often relies on descriptive statistics of user interactions or utilizes methods like fuzzy logic (FuzzyQoL) to quantify QoI based on expert knowledge. Other approaches employ semiotic frameworks to analyze interaction issues or group student behaviors based on factors like time spent and procrastination. However, these methods lack the predictive power needed for proactive interventions. This study uniquely explores the potential of deep learning to predict QoI, filling a significant gap in the existing literature. Previous studies using LMS data have focused on early detection of at-risk students, identifying learning dispositions, predicting learning success and performance, and predicting learner behavior in MOOCs. However, none have focused on directly predicting QoI as a key indicator of online learning effectiveness. The current work aims to address this limitation by leveraging the predictive capabilities of deep learning to improve the overall quality of online learning.

DeepLMS utilizes a Long Short-Term Memory (LSTM) recurrent neural network architecture to predict QoI. The model is trained and tested on three datasets (DB1, DB2, DB3) collected from different countries (Portugal, UAE, Greece), educational settings (Higher Education Institution, course level, focused discipline level), and time periods (pre- and during the COVID-19 pandemic). DB1, comprising data from 75 professors and 1037 students over two semesters (358 days), was obtained from previous work by the authors using the FuzzyQoL model. DB2 and DB3 consist of data from engineering design and advanced signal processing courses, respectively, collected during the COVID-19 pandemic. The FuzzyQoL model, which considers user interactions based on LMS usage, provides the ground truth QoI values. These values are then used to train and evaluate the LSTM model. The input features for DeepLMS are 14 categories derived from 110 LMS Moodle metrics, capturing various aspects of user interaction. The LSTM network architecture consists of four layers: a sequence input layer, an LSTM layer with 1200 hidden units, a fully connected layer, and a regression output layer. The model was trained using the Adam optimizer, with hyperparameters selected through early testing. Techniques like dropout and L2 regularization were used to prevent overfitting. The training involved 300 epochs and a mini-batch size of 128. Model performance was evaluated using RMSE, correlation coefficient (r) between the predicted and actual QoI values, and correlation coefficient (ra) between their derivatives. The analysis was conducted separately for professors and students in each dataset to assess the model’s performance across different user groups and settings. Ethical approvals were obtained where necessary, and data were de-identified before analysis.

DeepLMS demonstrated high predictive accuracy across all three datasets. The average testing RMSE was consistently below 0.009, and the correlation coefficient (r) was consistently above 0.97 (p<0.05) for both professors and students across DB1, DB2, and DB3. Figures 2, 3, 5, and 7 visually demonstrate the model's ability to accurately track QoI trends for individual users. Figure 4 and Figure 6 show the distribution of RMSE, r, and ra values across all users within DB1 and DB2, highlighting the consistent high performance of DeepLMS. Table 1 summarizes the median and 95% confidence intervals for these metrics, further supporting the model's robustness. Statistical tests showed no significant differences in DeepLMS performance based on user type (professor/student), country, time period (pre/during COVID-19), or user demographics (sex, age). A comparison with the FuzzyQoL model revealed that DeepLMS achieves significantly better predictive performance (lower RMSE and higher r values). The model's ability to handle sparse and variable interaction patterns underscores its practical applicability in real-world online learning environments, demonstrating the robustness of the DeepLMS model. Importantly, no significant data biases were identified in the datasets, ensuring the fairness and generalizability of the DeepLMS model's findings. Figure 8 demonstrates the DeepLMS performance with the DB3 data. Figure 10 shows the convergence of training RMSE during model training.

DeepLMS's high predictive accuracy and generalizability across diverse datasets demonstrate its potential as a valuable tool for both educators and learners. The ability to predict QoI enables proactive interventions to improve online learning engagement. For educators, DeepLMS offers an additional layer of assessment, providing insights into learner motivation and participation beyond traditional content-based evaluations. For learners, the personalized feedback generated by DeepLMS acts as a metacognitive trigger, encouraging self-reflection and adjustment of learning strategies. The model’s insensitivity to factors like user type, country, time period, and demographics highlights its broad applicability. The superior performance compared to existing models like FuzzyQoL further solidifies its potential to enhance the effectiveness of online learning. The model's ability to adapt to diverse learning contexts and handle sparse data makes it a versatile and robust tool. Further, the successful application across varied subjects (human kinetics, engineering design, advanced signal processing) shows its adaptability to multiple disciplines. Future research could explore the integration of DeepLMS with other measures of online learning quality, such as collaboration and affective engagement.

DeepLMS offers a significant advancement in supporting online learning by providing accurate and timely predictions of QoI. Its high predictive performance, generalizability, and adaptability make it a promising tool for enhancing both the learner experience and the educator's capacity for effective teaching and evaluation. Future work will focus on expanding the model to incorporate other factors affecting learning quality, validating its performance over longer time periods, and exploring its application in various educational settings.

While DeepLMS shows promising results, certain limitations exist. The study lacked correlation analysis between QoI and content evaluation outcomes (quizzes, exams). Further, the data used represent only one or half an academic year, limiting the assessment of its long-term predictive power. Future studies should address these limitations by incorporating assessments and longitudinal data to better understand the model's sustained performance.