The research focuses on bridging the gap between the efficacy of social-psychological interventions demonstrated in controlled trials and their real-world effectiveness when self-administered by students. Many interventions, such as the Strategic Resource Use intervention, have shown significant improvements in student learning in controlled settings. However, there's a lack of research on their translational effectiveness when students have the autonomy to choose whether or not to utilize them. This study addresses this gap by evaluating the "Exam Playbook," a digital adaptation of the Strategic Resource Use intervention, designed to guide students' self-reflection on resource use during exam preparation. The Exam Playbook was implemented using ECoach technology, enabling customization and scalability across various classes. The study aimed to determine if self-administration of the Exam Playbook predicted academic achievement, identify conditions under which it's more or less effective, and analyze heterogeneity across student demographics. The study hypothesizes a significant positive correlation between Exam Playbook use and exam performance, although smaller than the effect sizes observed in previous randomized controlled trials (RCTs). The study's significance lies in providing crucial insights into the practical effectiveness of self-administered interventions, helping to refine intervention design, improve prediction, and enhance scalability for broader application in educational settings.

The introduction cites numerous studies demonstrating the efficacy of social-psychological interventions in improving student learning and well-being within controlled environments, such as randomized controlled trials (RCTs). These interventions target various learning mechanisms, but their effectiveness in real-world settings where students self-administer them is less understood. The authors highlight the need for “effectiveness” studies that complement “efficacy” studies. Efficacy studies demonstrate the intervention's effects under controlled conditions, while effectiveness studies assess its impact in real-world settings. The authors reference several successful interventions that have shown strong efficacy, including the Strategic Resource Use intervention, values affirmation, social-belonging, and growth mindset interventions. These interventions have demonstrated effectiveness in controlled settings, but their effectiveness when self-administered needs further investigation. The authors aim to address this gap by examining the real-world impact of a self-administered version of the Strategic Resource Use intervention.

The study used a large-scale naturalistic design. The researchers adapted a previously validated Strategic Resource Use intervention into a user-friendly online app called the "Exam Playbook." This app guided students through self-reflection on their resource use before exams. It involved anticipating exam format, reviewing available resources, selecting useful resources, explaining their rationale, and planning resource use. The Exam Playbook was delivered through the ECoach platform, which allowed for class-specific customization and scalability. The intervention was implemented in 14 large introductory STEM and Economics classes across two semesters at a large US public university. 12,065 students were included in the study. ECoach tracked student engagement with the Exam Playbook and their exam performance. Data analysis involved multiple approaches, including a meta-analysis treating each class as a separate experiment to estimate the overall effect of Exam Playbook use on exam scores. Robustness checks were performed by controlling for college entrance exam scores and analyzing exam-level data. Additional analyses examined heterogeneity by class and student demographics, including stratified matching to assess the effects of adopting and dropping the Exam Playbook, and analyses of dosage and timing effects. Mixed-effects models were used to examine the moderating effects of gender, race, and first-generation status on Exam Playbook effectiveness.

The meta-analysis revealed that students using the Exam Playbook scored 2.17 percentage points higher on average than non-users (d = 0.18), a substantial effect size for a low-cost, scalable intervention. This effect remained significant even after controlling for college entrance exam scores (1.65 percentage points). Examining the data at the exam level yielded a similar significant effect (2.91 percentage points, d = 0.22). Heterogeneity in effect sizes across classes was observed. Introductory Statistics had the largest effect size, likely due to the intervention's initial design and testing within this course. Other courses showed smaller but still significant effects. Class climate (peer usage and teacher support) was positively associated with effect size. Stratified matching analyses revealed that adopting the Exam Playbook was associated with a 1.75 percentage point improvement in subsequent exam scores, while dropping it was associated with a 1.88 percentage point decrease. Dosage and timing effects were also found: higher usage frequency and earlier usage predicted better performance. Females and first-generation students were more likely to use the Exam Playbook and benefited more from its use, potentially reducing gender and first-generation achievement gaps. Black and Hispanic students were less likely to use the intervention.

The findings demonstrate the real-world effectiveness of a self-administered social-psychological intervention, extending previous RCT findings. The significant positive relationship between Exam Playbook usage and exam performance, robust across various analytical approaches, supports the intervention's efficacy in a naturalistic setting. The identification of class climate, timing, dosage, and student characteristics as moderators of effectiveness highlights the importance of contextual factors in intervention design and implementation. The observed benefits were not attributable to the use of other learning resources available on the ECoach platform. The results suggest that fostering a supportive class environment and promoting early and consistent intervention usage can enhance its effectiveness. The finding that females and first-generation students benefited more underscores the potential for interventions to address equity gaps. The lower uptake among Black and Hispanic students indicates a need to address potential barriers, such as identity threat, to improve intervention reach and impact.

This study provides valuable insights into the real-world effectiveness of a self-administered social-psychological intervention, highlighting the influence of contextual factors and student characteristics. The findings suggest the importance of considering class climate, timing, dosage, and potential equity-related issues in the design and implementation of such interventions. Future research should explore ways to optimize intervention delivery and address barriers to access and engagement, particularly for underrepresented groups. Further investigation into the interplay between individual and classroom-level effects is warranted.

The study's correlational nature limits the ability to definitively establish causality. The reliance on self-reported data on resource usage and the potential influence of unmeasured confounding variables should be acknowledged. The specific context of the study (large introductory STEM and Economics courses at a single university) may limit generalizability to other educational settings and disciplines. Future studies utilizing more rigorous designs, such as RCTs, and including measures of additional contextual and individual factors are necessary to strengthen the understanding of these interventions' broader impact.