A whole learning process-oriented formative assessment framework to cultivate complex skills

The paper addresses the challenge of assessing and cultivating complex 21st-century skills (e.g., creativity, complex problem solving, collaboration) in education, with a particular focus on vocational contexts. Formative assessment integrated into teaching is positioned as critical for monitoring progress and guiding learning in complex, ill-structured problem-solving. However, formative practices for complex skills remain scarce and under-specified regarding what to assess, when, and how. The authors propose to construct a whole learning process-oriented formative assessment framework grounded in the 4C/ID model and systems perspectives (I-E-O), and to empirically examine its implementation and effects in an industrial robot programming course. The study is guided by three research questions: (1) How to establish a model to assess learning progress dynamically, longitudinally, and comprehensively? (2) How to design and administer formative assessment during complex skill learning processes? (3) How to evaluate and understand the effect of formative assessment on complex skill learning?

The paper synthesizes research on complex skill learning and assessment. The 4C/ID model (Van Merriënboer & Kester) is highlighted as a foundation for complex learning by integrating tasks, supportive information, procedural information, and part-task practice to manage cognitive load and foster schema construction, automation, and transfer. Prior applications of 4C/ID demonstrate improved complex skill acquisition and transfer, including in Chinese vocational contexts. Formative assessment literature underscores its role in diagnosis, motivation, and metacognitive support, yet practical frameworks for complex skills are limited. Existing approaches often embed feedback in ICT environments (e.g., corrective and cognitive feedback), but challenges remain: aligning instruction and assessment criteria for learner understanding, providing teachers with actionable process feedback, and evidencing effects beyond summative outcomes. Calls exist for process-focused measurement and intervention during learning, moving beyond end-of-course assessments.

Design: Controlled experimental study implementing a whole learning process-oriented formative assessment framework for the complex skill 'Industrial Robot Trajectory Programming.' Participants: 35 second-year Mechatronics students at the Shanghai Technical Institute of Electronics & Information (STIEI), with no prior industrial robot programming experience. Two intact classes were pre-tested on course-related knowledge and showed similar low baseline scores; classes were randomly assigned to experimental (n=16) and control (n=19) groups. Ethics: Approved by East China Normal University Research Ethics Panel; informed consent obtained; data handled confidentially. Framework and instruments: The authors constructed the Spiral Complex Skill Learning Framework (SCSLF) integrating I-E-O systems thinking and 4C/ID, depicting an upward spiral across schema construction, automation, and transfer. A formative assessment indicator system was derived from a skill hierarchy for industrial robot programming, distinguishing recurrent and non-recurrent constituent skills and yielding 18 key indicators (KI1–KI18) covering knowledge, process, rules, and comprehensive implementation/transfer. An academic performance scale encompassed 58 items mapped to these indicators across three dimensions: knowledge mastery, schema construction (processes), and rule automation. Assessment tasks: Four formative tasks monitored different phases—Task 1 (knowledge and process, including flowcharting), Tasks 2–4 (rules and procedure execution for framework creation, program establishment, debugging/optimization), and one summative Task 5 (comprehensive implementation and transfer in a similar, realistic problem setting, covering all 58 items). Environment: A human-computer collaborative system, the Complex Skills Automatic Assessment System (CSAAS), collected submissions (answers, flowcharts, code packages), automated some scoring (objective items), supported manual scoring (flowcharts, rules), and provided individualized student feedback and aggregated class analytics to teachers for targeted intervention. Procedure: Eight lessons over four weeks (two per week). Both groups followed identical task sequences and scenarios. The experimental group received process-oriented formative assessment via CSAAS after Tasks 1–4 with immediate teacher feedback and interventions (whole-class concept clarification, process breakdown, one-on-one operational coaching). The control group completed tasks without formative evaluation or feedback; only data were collected. One week after course completion, all students completed the summative Task 5. Additional measures: Post-course questionnaires assessed cognitive load (intrinsic, extraneous, germane; 14 items, 5-point Likert; validated with Cronbach’s alpha >0.97 across subscales; KMO 0.893) and self-efficacy for industrial robot programming (4 items; alpha 0.931; KMO 0.844). Data analysis: Descriptive statistics, within-group comparisons (formative vs corresponding summative points), and between-group independent-samples t-tests for academic performance dimensions and questionnaire scales; effect sizes (Cohen’s d) reported for performance.

• Formative vs summative within experimental group: Process-integrated formative assessment with immediate feedback improved knowledge and concept mastery for most items; exceptions (e.g., items q7–q8 on turning motion parameter relationships) likely due to limited review time before summative testing. Schema construction: formative scores for KI6–KI10 were initially <60%, but post-intervention summative performance increased markedly. Schema automation: while many rules were already reasonably mastered, formative assessment further enhanced weaker points (e.g., KI13, KI15). Overall transfer: mastery rates per indicator in the summative task generally reached about 80% in the experimental group. - Between-group academic outcomes (Summative Task 5): • Complex skill transfer/implementation: experimental > control (t=2.56, p=0.015, Cohen’s d=0.86). • Schema construction: experimental > control (t=2.38, p=0.024, d=0.87). • Schema proficiency/automation: experimental > control (t=2.083, p=0.045, d=0.72). • Knowledge concepts: no significant difference (t=0.038, p=0.970). - Cognitive load and self-efficacy: • Germane cognitive load higher in experimental group (t=2.19, p=0.035), indicating optimized allocation of effort toward learning. • No significant differences for intrinsic (t=0.849, p=0.402), extraneous (t=0.949, p=0.352), or total cognitive load (t=1.656, p=0.107). • Self-efficacy: no significant difference (t=0.727, p=0.472). - Equity effect: Formative assessment reduced academic gaps and improved overall performance on complex processes and rule application, though gains in declarative knowledge accuracy were limited.

The study demonstrates that a whole-process, spiral formative assessment framework aligned with 4C/ID can dynamically and comprehensively monitor and guide complex skill learning. Embedding indicator-aligned assessments into schema construction, automation, and transfer tasks, and delivering targeted teacher feedback via CSAAS, led to significant improvements in process mastery, rule automation, and transfer without increasing overall cognitive load. The selective increase in germane load suggests more productive cognitive effort. Limited impact on declarative knowledge indicates that feedback and assessment modalities for objective concepts may require refinement to better support conceptual understanding. The framework operationalizes alignment between instruction and assessment, offers teachers actionable analytics for timely intervention, and supports students’ self-monitoring. These findings address the research questions by providing a concrete model, implementation approach, and empirical evidence of effects on complex skill learning.

This paper proposes and validates a whole learning process-oriented formative assessment framework for complex skills grounded in the 4C/ID model and systems/spiral learning perspectives. Implemented in an industrial robot programming course, the framework—comprising a skill hierarchy-derived indicator system, staged formative tasks, and a human-computer collaborative assessment environment—significantly enhanced schema construction, rule automation, and transfer performance, while increasing germane cognitive load without adding overall burden. Contributions include a practical, scalable design for aligning instruction and formative assessment across the full learning trajectory and evidence that process-focused formative interventions can elevate complex performance outcomes. Future research should generalize the framework to broader domains and learner populations, integrate richer supports for conceptual knowledge acquisition (e.g., targeted practice, reflective discussion, VR/simulation), allow sufficient consolidation time post-feedback, and incorporate a wider set of non-cognitive measures (motivation, affect) to holistically optimize complex skill learning.

• Sample and context: Small sample (n=35) confined to a single vocational institute and an industrial robotics course, limiting generalizability. - Scope: Primary focus on cognitive outcomes; non-cognitive factors (emotion, motivation) were not comprehensively assessed. - Timing: Compressed schedule provided limited time for review after feedback, potentially dampening knowledge gains and some feedback effects. - Domain breadth: Effects in other complex problem scenarios and subject areas remain to be tested.