Interpretable early warning recommendations in interactive learning environments: a deep-neural network approach based on learning behavior knowledge graph

The rise of online interactive learning environments has transformed learning by enabling collaboration, data-driven feedback, and temporal monitoring of learner behavior. Yet such environments also produce large volumes of inefficient behaviors, negative affect, and assessment failure, driven by unclear content sequencing, lack of interpretable guidance, and cognitive overload. This study targets the research problem of designing an interpretable early warning recommendation mechanism that connects learning content, concept classes, features, and temporal sequences to guide timely interventions. Specifically, the authors propose mining large-scale learning behavior instances, constructing an interpretable knowledge graph (KG) linking content and concepts, and developing a deep-neural network approach that operates on temporal sequences of behavior. The purpose is to improve prediction, interpretability, and actionable recommendations to increase learner success and engagement. The importance lies in enabling transparent, temporally-aware guidance and early warnings that align with learners’ knowledge structures and course concept classes.

The related work examines three interpretability dimensions: (1) Interpretable methods—Model-Agnostic (generalizable explanations across models) versus Model-Specific (tailored to a particular model) and public (global) versus private (instance-level) interpretations. Model-Agnostic approaches offer broader applicability across heterogeneous environments. (2) Interpretable tools—e.g., LIME, SHAP, PDP, InterpretML, Alibi, H2O—differ in data type compatibility, stability, and usability; lack of standardization and complexity limit direct application at massive scales. (3) Interpretable recommendation mechanisms—deep learning–based recommenders can personalize services and reduce overload, but need interpretable processes to enhance trust and support early warnings. The authors argue that an early warning mechanism should be grounded in a learning behavior knowledge graph, integrating interpretable recommendations to capture semantics, flexibility, and adaptive tracking.

The authors design an interpretable early warning mechanism by coupling a knowledge graph (KG) of learning behavior with a deep-neural network that processes temporal sequences of learner actions. Key components: 1) Definitions: - Knowledge graph: KG = {(h, r, t)} where entities (learners, learning contents, concepts) are nodes and relationships are directed edges. Relationship types include contains (LC→Concept), isUpperTo (concept hierarchy), and isPrerequisiteTo (concept prerequisites). Learner–content interactions IR(l, interaction, c) are integrated into KG. - Vector decomposition: For CNN output h(a)=W·a+b, weight vectors are decomposed onto an orthogonal basis of concept-feature vectors q, with least-squares solution s=C+·w (C+ pseudo-inverse), and residual r to capture features outside the basis, improving interpretability. - Concept features: Augment basis with residuals to ensure CNN-mined features are represented; scores decompose into contributions from concept labels and residuals. 2) DNNA algorithm (core steps): - Step 1: Recognize temporal sequences of learning behavior via CNN to obtain weight vectors and confidence. - Step 2: Decompose CNN weight vectors into interpretable feature vectors (from training set). - Step 3: Visualize outputs by activating the last CNN layer and backpropagating to validate decomposition. - Step 4: Rank feature importance by average contribution; dynamically track top-5 features over temporal sequences. - Step 5: Re-score sequences using interpretable feature–sequence inputs; compute accuracy, recall, precision, F1. - Step 6: Evaluate reliability by comparing pre/post scores and confidences. - Step 7: Construct a KG based on relations among decomposed feature vectors. - Step 8: Compute temporal sequence similarity via Jaccard coefficient. - Step 9: Mine discriminant features by extending interpretable results. - Step 10: Test learner credibility using MSE over TF-IDF of discriminant features. 3) Data processing and KG construction: - Large-scale AI-enabled platform dataset (≈1.3 PB) with two challenges: sparsity and behavioral uncertainty. To address sparsity, enrich features beyond assessment results; to address uncertainty, model behavior as temporal sequences capturing dynamics. - Entities: learners (28,707), learning contents (97), concepts (1,204), total entities (1,542), relationship types (3), triples (2,620), topology paths (5,133). - Relationship rules: LC contains Concept; Concept m order Concept n; Concept m level Concept n. - Attributes: Learner ID; LC attributes (ID, name, utility, class/series, difficulty) and observed behavior (watch duration/progress/date). - Preprocessing: Cluster features into categories (resource, questionnaire, upload, download, quiz, data sampling, experiment, interaction, cooperation). Classify concepts into six categories: Related Content, Theoretical Basis, Difficulties of LC, Key Points of LC, Application Background, Context of LC. - Temporal structure: A complete learning period spans 20 weeks; build KG over Learning Content → Concept Class → Feature Class → Temporal Sequence. - Relationship types for interpretability: Interpretation (one-way, feature → temporal sequence) and Juxtaposition (two-way among features within the same sequence). 4) Experimental design: - Baselines: LR, FM, DNNFM, DNNCross, AutoInt, AFN. - Metrics: AUC; Relative Improvement (RI) with α=0.5 (random classifier); F1; MTL-Gain = MMTL − Msingle. - Sampling strategies for similarity-based experiments: R-method (random); FR-method (ranked by feature similarity, N=10); L-method (by learner description similarity, N=8); FS-method (feature–feature similarity threshold 0.569); DNNA adaptive interpretable feature feedback (automatic similarity and sampling). - Knowledge graph significance testing: Structural equation models test LC→Concept and Concept→Feature effects across key temporal intervals for pass/fail groups. - Additional analyses: Temporal sequence similarity (Jaccard), discriminant feature mining, and learner credibility (MSE with TF-IDF).

- Overall recommendation performance: DNNA achieved higher AUC than baselines (Table 4): DNNA 0.9035; LR 0.8279; FM 0.8501; DNNFM 0.8577; DNNCross 0.8446; AutoInt 0.8702; AFN 0.8811. - Interpretable feature recommendation (Table 5): DNNA outperformed baselines on F1 and AUC with multi-feature learning gains: DNNA F1=0.8101, AUC=0.9124, MTL-Gain=+6.89%. Baselines showed lower F1/AUC and smaller MTL gains. - Similarity-based negative sampling (Table 6): On negative samples, DNNA attained best AUC (0.8615) versus R-method 0.8218, FR-method 0.7549, L-method 0.8485, FS-method 0.8477. - KG-driven temporal insights: • Concept classes: strong correlations centered on “Difficulties of LC” and “Key Points of LC” with correlation interval [0.63, 0.92]; other concept classes correlate indirectly. • Feature classes: strong inter-feature correlations with interval [0.59, 0.89], forming stable learning behavior paths that vary across the 20-week period. - Critical learning paths by LC clusters: • Cluster I: interaction (resource → search → download, wiki), questionnaire & Q&A → quiz, resource (Q&A → interaction, forum), download. • Cluster II: interaction (forum → quiz, upload → download), Q&A → (resource → download), interaction, forum. • Cluster III: interaction → (data sampling → experience, experience, cooperation) → (cooperation, Q&A); experience → (Q&A → quiz, quiz). - Key early warning temporal intervals for failers (from passers’ critical paths as references): • Cluster I: weeks 3–7 and 15–19. • Cluster II: weeks 7–15 and 17–20. • Cluster III: weeks 2–9 and 17–20. - Significance testing: • Passers (Table 7): Significant effects of LC→Concept and Concept→Feature across multiple temporal intervals (e.g., Cluster I: [3rd–7th], [7th–10th], [15th–19th]; Cluster II: [3rd–15th], [17th–20th]; Cluster III: [2nd–9th], [9th–11th], [11th–20th]) with p<0.05, p<0.01, p<0.001. • Failers (Table 8): No significant KG effects in corresponding intervals; some individual features show effects but feature-class effects are absent or reversed. - Model applicability: DNNA training showed high clustering with a 94.85% success rate; five learning contents were outliers associated with low participation, unclear purposes/key points, weak context, and sparse/discrete behaviors. - Early warning logic: For passers, early warning operates with “OR” over key sequences (selective intervention); for failers, “AND” logic is recommended (continuous tracking across all relevant sequences). - Practical implication: Aligning content classes, concept classes (especially Difficulties and Key Points), feature classes, and temporal intervals enables interpretable, actionable early warnings and personalized interventions.

The findings demonstrate that coupling an interpretable KG with a deep-neural network over temporal sequences addresses the core research problem: providing accurate, explainable early warning recommendations in interactive learning environments. DNNA’s superior predictive metrics (AUC, F1, MTL-Gain) confirm that interpretability grounded in concept/feature classes and sequence-aware modeling improves recommendation quality. The KG significance for passers indicates that successful learners’ behaviors align with coherent LC→Concept→Feature relationships during specific temporal intervals. Conversely, failers lack such structured patterns, reinforcing the need for targeted, sequence-sensitive interventions. The identified critical paths and intervals provide actionable points for guidance (e.g., emphasizing interactions and assessments in weeks 3–7, 15–19 for certain content clusters). The OR logic for passers facilitates selective support, while the AND logic for failers emphasizes comprehensive, continuous monitoring across key sequences. Overall, the study underscores that interpretable organization of content (via concept classes), feature classes, and temporal alignment enables credible, scalable early warnings and enhances learner engagement and success.

This study introduces an interpretable early warning recommendation mechanism built on a learning behavior knowledge graph and a deep-neural network with interpretable vector decomposition (DNNA). Contributions include: (1) a scalable KG linking learning contents, concept classes, feature classes, and temporal sequences; (2) an interpretable DNN that decomposes feature contributions and visualizes temporal importance; (3) empirical evidence that DNNA outperforms strong baselines (AUC, F1, MTL-Gain) and yields significant, actionable patterns for passers while revealing structural deficits for failers; and (4) practical early warning strategies using OR logic for passers and AND logic for failers across identified key intervals. Future research will refine temporal tracking, enhance the KG’s logical/topological design, handle dynamic, large-scale behavior data more flexibly, and further validate models across diverse content types and concept topologies to improve early warning accuracy and feedback reliability.

- Behavioral complexity and dynamics: Learning behaviors evolve over time and are influenced by group dynamics and individual traits, complicating stable modeling and generalization. - Data sparsity and uncertainty: Some courses have low participation, unclear goals/context, and sparse/discrete behaviors, limiting reliable inference and early identification. - Semantic granularity: Current platforms lack precise semantic management for all concepts, making exhaustive concept-level interpretability impractical; analysis relies on concept classes rather than all individual concepts. - External validity: Results are derived from a specific large-scale platform and content clusters; transferability to other contexts and domains requires further validation. - Metric/reporting inconsistencies: Some reported RI figures and content counts show inconsistencies; additional replication and reporting standardization would strengthen conclusions.