Comparing models of learning and relearning in large-scale cognitive training data sets

Real-world skill acquisition differs significantly from controlled laboratory settings due to irregular practice schedules and durations. While spaced repetition's benefits for long-term retention are well-established, laboratory studies often lack the complexity of real-world learning schedules. This research utilizes a large dataset from the Lumosity cognitive training platform to investigate learning and retention over extended periods with variable practice intervals. The study aims to characterize naturalistic learning features absent in laboratory studies and to evaluate a hierarchy of learning models to identify the most accurate representation of long-term learning incorporating skill, task-set preparedness, and forgetting. The highly variable practice schedules introduced by user-controlled learning pose unique challenges to traditional theoretical approaches. Learning curves from the Lumosity data show significant gains within short periods (single sessions) and substantial gaps due to breaks in practice. Analyzing gameplays instead of chronological time reveals traditional learning curves but obscures the patterns of learning and forgetting. The distribution of time between consecutive sessions shows regular intervals (days, weeks) indicating systematic practice scheduling by users. A comprehensive theory of learning must accommodate these varied time scales.

Existing research extensively documents the spacing effect, where spaced practice enhances long-term retention. However, limitations in controlled laboratory settings hamper our understanding of real-world learning complexities, particularly those involving irregular schedules and extended breaks. Studies on metacognitive control demonstrate the benefits of self-regulated learning across various domains. Long-term learning functions reveal two timescales: rapid within-session gains and slower, steady across-session gains, with evident losses and rapid recovery between sessions. The warm-up decrement, a drop in performance following a rest period with subsequent rapid recovery, highlights the interplay between skill and task-set preparedness. Existing models, such as those incorporating power-law forgetting, or those with latent variables mediating learning and forgetting, attempt to capture these aspects, but may not fully account for the wide range of time scales found in naturalistic learning. Models lacking forgetting mechanisms produce inaccurate predictions of long-term learning. Two-timescale models incorporating skill and task-set preparedness provide improved accuracy, but still fail to completely capture the observed rapid recovery from extended practice breaks.

The study uses data from the Lumosity cognitive training platform, specifically focusing on two games: a flanker task ('Lost in Migration') and a task-switching game ('Ebb and Flow'). The dataset, comprising gameplay event history for 194,695 users over five years, includes 389,389 learning curves and 41,006,715 single gameplay events. The analysis uses a subsample of 19,463 users for the flanker task and 19,694 for the task-switching game, selected based on having a ratio of total gameplays to total sessions >1.5 and more than 50 game sessions. Each gameplay event is of a fixed duration, and performance is assessed by the number of correct trials completed, a metric closely related to the in-game score. The data were processed to cluster gameplays into sessions based on time elapsed between gameplays. Sessions with breaks of >1 hour were considered separate. The dataset displays a wide range of intervals between sessions, from a few days to over two years. Four learning models were formulated: M1 (baseline, no within-session learning or forgetting), M2 (two-timescale learning, no explicit forgetting), M3 (two-timescale learning with forgetting that varies with delay), and M4 (two-timescale learning with an interactive component between skill and task-set preparedness). Model parameters were estimated using maximum-likelihood procedures, employing automatic differentiation and the Adam algorithm for adaptive gradient descent. Proportional performance changes were analyzed to account for individual performance variations. The Predictive Performance Equation (PPE), a benchmark model for spacing and forgetting effects, was also applied and compared to the four models. Data were split into training and testing sets using a conditional sampling approach, ensuring the inclusion of trials with large delays in both sets.

Analysis of learning curves reveals a saw-toothed pattern, with performance dropping between sessions and rapidly recovering at the start of each new session. This effect is observed even after delays of almost a year. Analyzing performance across sessions for different gameplays within each session, revealed that learning across sessions mirrors traditional learning curves but that the magnitude of the learning effect diminishes with each subsequent gameplay within the session. Analyzing the first and second gameplay performance as a function of the retention interval shows substantial performance drops even after very long breaks (over two years), but also rapid recovery within the first few gameplays of each session. Model comparison based on root mean squared error (RMSE) on out-of-sample data showed that for the 'Lost in Migration' game, the interactive model (M4) yielded the most accurate predictions, while for 'Ebb and Flow', models M3 and M4 were equivalently accurate. M4 consistently provided a better general account of performance across different delay intervals. The interactive model (M4) uniquely and accurately captured the steeper growth for earlier gameplays within a session (seen in Figure 5) and the shallower forgetting function for subsequent gameplays within a session (seen in Figure 6), indicating the interactive nature of skill and task-set learning. The other models failed to capture this pattern. The table 1 summarizes predictive performance of models assessed by RMSE on out-of-sample data, showing how M4 or M3 perform best depending on the game and delay period.

The findings support the existence of multiple timescales in learning, with two key components: long-term skill and short-term task-set preparedness. Skill represents fundamental task knowledge, while task-set preparedness encompasses factors affecting performance (attention, task knowledge, familiarity). Task-set preparedness is susceptible to forgetting between sessions, while skill is relatively permanent. The study's results extend prior research on two-timescale models by demonstrating the importance of incorporating forgetting that varies with the delay between practice sessions. Crucially, the interactive nature of skill and task-set preparedness, where relearning is faster with greater skill, is essential to accurately model naturalistic learning. This interactivity, as shown in model M4, provides a much more accurate representation of long-term learning than models that treat the processes as independent. These findings highlight the limitations of extrapolating from controlled laboratory experiments to real-world learning conditions.

This study reveals the importance of considering multiple timescales of learning, incorporating forgetting, and acknowledging the interactive nature of skill and task-set preparedness in modeling real-world learning. The interactive model (M4) provided the best fit for the data, suggesting that future models of learning should incorporate this crucial feature. Further research should focus on exploring individual differences in learning parameters and investigating the influence of various factors on the speed of relearning.

The study's reliance on a specific online platform (Lumosity) with a potentially biased user base (predominantly female, Western nations) limits the generalizability of the findings. The reliance on self-reported data also introduces potential biases. Future research could benefit from using more diverse datasets and validating the findings with experimental studies. Also note that the models make the simplifying assumption that the forgetting parameter γ is shared across participants. Further research could explore participant-specific parameters.