A whole learning process-oriented formative assessment framework to cultivate complex skills

The 21st century demands learners acquire complex skills, encompassing creativity, problem-solving, and collaboration. These skills integrate knowledge, technique, and attitude, enabling effective solutions to real-world problems. However, assessing these complex 21st-century skills (21CS) poses a significant challenge. While formative assessment, integrating instruction and evaluation, is crucial for promoting skill mastery, its application to complex skill learning remains scarce. Existing formative evaluation practices are inadequate due to the intricate interconnectedness of knowledge and competencies involved in 21CS. Key issues requiring exploration include how and when formative assessment should be implemented to effectively monitor students' learning progress within complex contexts. This study aims to address these gaps by establishing a whole learning process-oriented formative assessment framework for complex skills and testing its feasibility and impact on student learning.

Research on complex problem-solving and skill acquisition highlights the significance of transferable skills for diverse work scenarios. Mayer (1992) emphasizes the role of experience in shaping cognitive skills, while Merrill (2002a) proposes the Pebble-In-The-Pond model for complex skill training. Van-Merriënboer and Kester's (2014) Four Components Instructional Design (4C/ID) model offers a detailed theoretical explanation of complex learning, emphasizing the integration of knowledge, skills, and attitudes. This model, used extensively in complex learning, underscores the importance of schema construction and automation to avoid excessive cognitive load. While existing research often focuses on summative assessment within the 4C/ID framework, limitations have emerged, including minimal impact of learning tasks and teacher-led interventions on student performance. This necessitates exploring formative assessment design for enhanced complex skill learning. Prior studies have examined the use of corrective and cognitive feedback in ICT-integrated learning environments, but classroom application and teacher feedback mechanisms remain underdeveloped. The current research addresses these shortcomings by focusing on the holistic formative assessment process, integrating the 4C/ID model with the I-E-O model and spiral curriculum theory, offering a comprehensive framework to investigate the efficacy of formative assessment.

This study employed a controlled experiment to investigate the effectiveness of a whole learning process-oriented formative assessment framework in cultivating complex skills. Thirty-five mechatronics students, with no prior industrial robot programming experience, were divided into experimental and control groups. The experimental group received instruction incorporating the proposed formative assessment framework, while the control group received traditional instruction. The course focused on "Industrial Robot Trajectory Programming," requiring students to configure a robot workstation in a simulated environment. The formative assessment framework was developed based on the 4C/ID model, incorporating several assessment points related to knowledge acquisition, schema construction and automation, and skill transfer. The assessment points were categorized into four types of tests: knowledge tests (objective questions), process tests (flowcharts), rule tests (practical problem-solving), and implementation/transfer tests (comprehensive tasks). The Complex Skills Automatic Assessment System (CSAAS) was employed to facilitate data collection, processing, and feedback delivery. Both formative and summative assessments were conducted. Summative assessment included comprehensive transfer tasks, cognitive load questionnaires, and self-efficacy questionnaires to comprehensively evaluate students' learning outcomes, cognitive load, and self-efficacy. Data analysis involved descriptive statistics, t-tests, and line/area charts to compare the performance of both groups across different assessment measures. The cognitive load questionnaire measured intrinsic, extraneous, and germane cognitive loads. The self-efficacy questionnaire assessed students' confidence in using their learned skills. Ethical approval was obtained from the Research Ethics Panel of East China Normal University. Informed consent was secured from all participants and teachers involved.

The results demonstrate the effectiveness of the proposed formative assessment framework in enhancing complex skill learning. Specifically, the experimental group showed significantly better performance in schema construction (t=2.38, p=0.024), schema automation (t=2.083, p=0.045), and complex skill transfer (t=2.56, p=0.015) compared to the control group. While the difference in knowledge mastery was not statistically significant, the formative assessment framework successfully reduced the academic gap between students. Analysis of formative and summative assessment scores indicated that formative assessment and feedback effectively promoted knowledge mastery, particularly in areas where students initially demonstrated weaker understanding. The formative assessment improved schema construction and automation significantly, even improving performance on already well-mastered aspects. Further analysis revealed that the experimental group showed significantly higher germane cognitive load (t=2.19, p=0.035) compared to the control group, suggesting that the formative assessment framework effectively engaged students in deeper cognitive processing without increasing their overall cognitive load significantly. The self-efficacy levels of both groups did not differ significantly, potentially due to the challenging nature of the tasks that revealed areas for improvement.

The findings support the integration of formative assessment into the complex skill learning process. The proposed framework, integrating elements from 4C/ID, I-E-O, spiral curriculum, and HPI, successfully guided students' learning journey. The deconstruction of learning objectives into measurable assessment points and the design of varied formative assessment tasks facilitated effective monitoring and feedback. The results align with previous research emphasizing the benefits of continuous assessment and intervention in consolidating learning. The absence of a significant increase in overall cognitive load highlights the framework's efficiency. However, the limited improvement in objective knowledge mastery suggests potential areas for enhancement, such as providing more targeted feedback and exercises to address specific knowledge gaps. This study demonstrates the feasibility and efficacy of applying a whole learning process-oriented formative assessment framework in real classroom settings.

This study presents a novel formative assessment framework that effectively cultivates complex skills in vocational education without significantly increasing cognitive load. The framework's success lies in its integration of established learning theories and its focus on a holistic, dynamic assessment process. Future research should explore the framework's applicability to different skill domains and incorporate non-cognitive factors such as motivation and emotion. Further investigation into more effective feedback methods to address specific knowledge gaps is also warranted.

This study's limitations include the relatively small sample size and focus on a specific vocational skill. The generalizability of the findings to other contexts and student populations needs further investigation. The study's limited timeframe may have influenced the assessment results. Future studies should address these limitations by employing larger, more diverse samples and longer intervention periods.