Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalies

The paper addresses how to design online learning experiences that promote active belief revision without instructor feedback. In large-scale settings like MOOCs and Wikipedia, learners often fail to revise misconceptions when faced with conflicting facts and may simply memorize or ignore anomalies. The research asks: which reflective prompts most effectively promote belief revision; what characteristics of anomalies (facts contradicting misconceptions) best facilitate revision; and how many anomalies should be presented? The approach explores whether prompting learners to engage in explanation can elicit cognitive processes that reveal gaps in understanding and guide them toward correct principles, providing scalable, domain-general strategies for online instruction.

Prior work shows that prompting learners to generate self-explanations improves understanding beyond receiving explanations passively. Explanation tends to push learners to interpret instances as part of broader patterns and principles, supporting generalization. Although anomalies—facts that contradict prior beliefs—have potential to trigger conceptual change, simply presenting anomalies often fails unless learners process them appropriately. Prompts that elicit explanation may help learners reconcile anomalies by abandoning incorrect rules in favor of rules that make anomalies intelligible. Related lines of research include intelligent tutoring systems that provide hints and adaptive feedback, peer discussion and assessment in online classes, and findings that confusion can benefit learning when properly supported. Together, these literatures suggest that carefully designed reflective prompts, especially explanation prompts directed at anomalies, could promote belief revision at scale without real-time feedback.

Two randomized controlled experiments examined reflective prompts and anomaly design in an online statistics learning task about comparing student grades across courses.

• Content and misconceptions: Participants learned the correct ranking rule: more deviations above the average (akin to higher z-score). Three prevalent misconceptions were targeted: Higher Score (choose higher raw score), Greater Distance from Average (choose larger absolute difference from class mean), and Closer to Maximum (choose score nearer the class maximum). A ranked pair is an anomaly with respect to a misconception if the observed ranking contradicts that misconception’s prediction.
• Materials and interface: Each study trial displayed a pair of students with their scores, and the class average, deviation, minimum, and maximum, plus the university’s stated ranking for that pair. A Reflection Prompt appeared below with a text box. Prompts (between-subjects): Explain (e.g., Explain why Tom was ranked higher by the university) versus Write Thoughts (Write out any thoughts you have about this information). Each ranked pair screen was shown for exactly two minutes.
• Design factors: • Experiment 1 (N=659): 2x2 between-subjects: Reflection Prompt (Explain vs Write Thoughts) × Number of anomalies (One vs Four anomalies contradicting each misconception across five ranked pairs). Anomalies overlapped: each anomalous pair contradicted all three misconceptions simultaneously. • Experiment 2 (N=261): 2x2x2 between-subjects: Reflection Prompt (Explain vs Write Thoughts) × Number of anomalies (Two vs Four per misconception across six ranked pairs) × Distribution of anomalies (Overlapping vs Distributed). Overlapping: a pair was either anomalous or consistent with respect to all three misconceptions (as in Exp. 1). Distributed: anomalies were spread to maximize the number of pairs that contradicted at least one misconception, allowing some pairs to challenge only a subset of misconceptions.
• Procedure: Participants completed a Pre-Test of four unranked pairs where the correct rule predicted a different ranking than all three misconceptions. After the study phase with prompts, a Post-Test presented four isomorphic unranked pairs (different names/numbers). Belief revision was operationalized as Accuracy Increase (Post minus Pre).
• Participants and compensation: Recruited from Amazon Mechanical Turk; N=659 (Exp. 1) and N=261 (Exp. 2); study length 20–40 minutes; compensation approximately $3–$6 per hour.
• Dependent measure and analysis: Primary outcome was Accuracy Increase. ANOVAs tested main effects and interactions among Prompt type, Number of anomalies, and Distribution (Exp. 2).

Experiment 1 (N=659):

• Main effects: Explanation prompts produced greater belief revision than Write Thoughts (F(1,659)=13.23, p<0.01). More anomalies yielded greater belief revision (F(1,659)=24.53, p<0.01).
• Interaction: Prompt × Number of anomalies (F(1,659)=8.20, p<0.01). Explaining improved learning when 4 anomalies were presented (r(260)=4.07, p<0.01), but not when only 1 anomaly was presented (t(367)=0.62, p=0.54).

Experiment 2 (N=261):

• Interaction: Prompt × Distribution of anomalies (F(1,259)=6.11, p<0.05). With Overlapping anomalies, Explain > Write Thoughts (t(127)=2.20, p<0.05). With Distributed anomalies, no significant benefit of Explain over Write Thoughts (t(129)=1.32, p>0.19).
• Additional effect: Main effect of Number of anomalies with more anomalies improving learning (F(1,259)=8.20, p<0.01). No Prompt × Number interaction (likely due to 2 vs 4 rather than 1 vs 4 as in Exp. 1).

Overall: Explanation prompts drive belief revision toward the correct z-score rule, but primarily when multiple anomalies are presented and when anomalies overlap to simultaneously rule out multiple misconceptions. Generic reflective writing without targeted explanation does not significantly improve learning.

The findings show that not all reflective prompts are equally effective for online learning without feedback. Prompts to explain why anomalous rankings are true lead learners to revise misconceptions toward the correct statistical principle. However, the success of explanation depends on the anomaly set: multiple anomalies are needed, and anomalies should overlap so that each observation simultaneously contradicts several misconceptions. When anomalies are distributed so that each observation challenges only one misconception, learners can rationalize using alternative misconceptions, weakening the guidance toward the correct rule. These results address the research questions by demonstrating that explanation is the preferred reflective prompt, that more anomalies produce stronger belief revision, and that anomaly overlap is critical. This provides actionable guidance for scalable online lesson design to elicit cognitive processes that support conceptual change without instructor feedback.

The paper contributes experimentally validated design principles for scalable reflective prompts in online learning: (1) prompt learners to explain anomalies rather than merely write thoughts, (2) present multiple anomalies, and (3) design anomalies to overlap so they simultaneously rule out multiple misconceptions. Together, these principles enable belief revision toward correct concepts without instructor feedback. Future work should explore automated or crowdsourced methods to identify misconceptions and generate overlapping anomalies, test generalization to other domains and learner populations, and apply these prompts in informal platforms (e.g., Wikipedia) and varied instructional contexts.

Identifying prevalent misconceptions and crafting corresponding overlapping anomalies can be difficult and time-consuming, and this work leveraged prior domain research. Generalizability beyond the studied statistics task and MTurk population is untested; learners in other settings may have different motivations and backgrounds. Additional studies are needed to evaluate applicability across domains, materials, and informal learning environments.