Crafting a framework: a Delphi method approach to formulating a maker literacy assessment model for primary school students in China

The Maker Movement has drawn global attention in K-12 education, supported by advances in open-source technologies and the proliferation of makerspaces. Maker education, grounded in constructivism and Dewey’s learning-by-doing, emphasizes hands-on, project-based, learner-centered, and iterative learning aimed at cultivating 21st-century competencies (e.g., critical thinking, communication, problem-solving, collaboration) and innovative thinking. Traditional assessment approaches that focus on summative, standards-based outcomes are often ill-suited to capture the higher-order, multi-dimensional skills developed in maker activities, creating an assessment gap that constrains the growth of maker education. In China, integrating Maker education with classroom instruction and STEM has shown promise for elementary learners. Prior assessment efforts (e.g., process/understanding/work; affect/cognition/participation; scales on critical thinking, innovation, collaboration, work habits) remain too general to capture the multifaceted performance involved in making artifacts. This study addresses the “Maker contribution research gap” by proposing a holistic, context-aligned framework for assessing Maker literacy in Chinese primary schools, encompassing design thinking, technological and practical abilities, collaboration, and attitudes/values. The study’s purposes are to: (1) develop a Maker literacy assessment model for Chinese primary students that specifies essential characteristics, abilities, and values required for Maker activities; and (2) determine the weights of the framework’s indicators. The model aims to guide teaching objectives, content selection, and evaluation of instructional effects. Establishing a literacy-oriented evaluation system aligns with international trends and China’s innovation-focused educational reforms, recognizing “Maker literacy” as the social practice of making/remaking artifacts using diverse materials and technologies.

A content-analytic literature review retrieved ~200 publications on Maker activities, design thinking, and innovation education from teaching theory, psychology, and philosophy of technology (Web of Science, CNKI with back-to-back translation). Through coding, the authors identified potential dimensions and indicators of Maker literacy and built an initial conceptual framework that informed the first-round Delphi questionnaire. Two theoretical lenses informed framework construction: - Performance Evaluation Theory: Emphasizes assessing students in authentic or simulated contexts to evaluate application of knowledge/skills, creativity, problem-solving, teamwork, and reasoning beyond paper-and-pencil tests. In Maker education, performance evaluation considers both process (strategies, iterations, problem-solving) and product (creativity, functionality, alignment with goals). - Evidence-Centered Design (ECD): Guides evaluation design around learner, evidence, and task models. For Maker literacy, the learner model specifies knowledge (technical), skills (hands-on practice), and attitudes (innovation consciousness). The evidence model identifies observable behaviors during problem-solving, making, and collaboration, supporting valid inferences about literacy. ECD strengthens explanatory and inferential validity by aligning tasks with targeted evidence across knowledge, skills, and attitudes.

Design and phases: The study developed a Maker literacy assessment model via two phases. Phase 1 conducted a literature review and content analysis to construct a preliminary framework and indicators. Phase 2 employed a two-round Delphi process to refine domains/indicators and establish consensus, followed by weighting via Analytic Hierarchy Process (AHP) using YAAHP software. Statistical analyses (e.g., medians, interquartile ranges, percentages) were conducted in SPSS 24.0. Participants: Eighteen experts were invited; sixteen volunteered and completed both Delphi rounds. Inclusion criteria: active employment in Chinese elementary/secondary education or related institutions; residence and work in China; expertise in STEAM, IT education, labor education, science education, etc.; >5 years experience. Demographics: 44% male, 56% female; ages 31–40 (25%), 41–50 (38%), >50 (38%); education: bachelor (19%), postgraduate (6%), doctorate (75%); teaching experience: <5 years (6%), 6–10 (13%), 11–15 (13%), >15 (69%); positions: professors (56%), associate professors (44%). Affiliations included universities (n=12), MOE research institute (n=1), and primary/secondary schools (n=3). Delphi procedures: Data were collected via Wenjuanxing (an online survey platform). Round 1 (May–June 2022) solicited expert judgments (5-point Likert) on the inclusion, importance, modification, and categorization of domains/indicators; follow-up reminders were issued every four days over a two-week window. Expert feedback informed revisions. Round 2 (July–August 2022) asked panelists to re-evaluate relevance and to conduct pairwise comparisons for AHP-based weighting. Consensus and analysis: Consensus criteria included interquartile range (IQR) ≤ 1 and coefficient of variation (CV) ≤ 18%, with an additional median-based threshold. Items not meeting acceptance criteria were reconsidered for modification or deletion by the panel. YAAHP 10 computed weights; IBM SPSS 24.0 supported descriptive statistics. Refinement from Round 1 to Round 2: Based on expert input, domains and indicators were adjusted. Notable changes: “Creation practice” was renamed to “Materialized practice” and integrated with “Technology application”; “Scheme selection” became “Scheme formulation”; “Material technology” and “Tool understanding” merged into “Technical understanding”; “Visual representations” was absorbed into “Prototype construction”; “Technical inquiry” merged with “Craftsman spirit” and renamed “Maker spirit”; “Ethical responsibility” became “Maker responsibility”; “Self-monitoring” was subsumed under “Technology application and materialized practice.” A revised model with three domains and twelve indicators was then validated in Round 2 and weighted using AHP.

Final framework: Three domains with twelve indicators were established for Chinese primary students’ Maker literacy: - Design thinking (weight 0.365): Requirement definition (0.193), Creative idea (0.295), Scheme formulation (0.222), Prototype construction (0.179), Iterative optimization (0.111). - Technology application and materialized practice (weight 0.282): Technical understanding (0.248), Knowledge integration (0.290), Making (0.171), Information gathering (0.124), Cooperation and communication (0.167). - Maker spirit and responsibility (weight 0.348): Maker spirit (0.571), Maker responsibility (0.429). Round 2 ratings and consensus: Domain means were high (Design thinking 4.938, Technology application and materialized practice 4.750, Maker spirit and responsibility 4.625) with full-score rates for domains up to 93.75% (Design thinking). Indicators all averaged >4.0. Full-score rates exceeded 80% for Creative idea, Knowledge integration, Maker spirit, and Maker responsibility. IQR values were reported as less than 1.8 for indicators, with several showing very high concentration (e.g., Design thinking, Creative thinking, Knowledge integration, Maker spirit, Maker responsibility). Although Iterative optimization, Technical understanding, and Information gathering had lower full-score rates, their means remained >4, and they were retained. Model coherence and applicability: Experts agreed the model reflects key abilities and values for primary school Maker activities in China and required only minor refinements to indicator descriptions. The AHP results positioned Design thinking as the central domain, with strong emphasis also on affective/attitudinal dimensions (Maker spirit and responsibility).

The study addresses the assessment gap in Maker education by operationalizing Maker literacy for Chinese primary schools into a validated, weighted framework spanning cognition (design thinking), skills/practice (technology application and materialized practice), and attitudes/values (maker spirit and responsibility). The prominence of Design thinking (highest domain weight) reflects its role as the foundation for organizing and implementing Maker activities and driving learning outcomes. Within this domain, Creative idea has the largest indicator weight, aligning with Maker education’s core of cultivating innovation. For Technology application and materialized practice, Knowledge integration receives the highest indicator weight, underscoring higher-order cognitive processes that synthesize interdisciplinary knowledge for problem-solving. Instrumental abilities (e.g., Information gathering) are comparatively less emphasized. In Maker spirit and responsibility, Maker spirit outweighs Maker responsibility, indicating the centrality of curiosity, perseverance, and enthusiasm in sustaining iterative, hands-on creation, while ethical and environmental responsibilities remain necessary but less distinctive to Maker education. Collectively, the findings clarify the hierarchical structure of Maker literacy, provide a culturally and contextually aligned assessment model for China, and offer a foundation for instructional design, curriculum development, and evaluation of Maker activities. The framework’s weighted indicators enable prioritization and tailoring of instruction to foster balanced development across thinking, doing, and being.

This study proposes and validates a Maker Literacy Assessment Model tailored to Chinese primary school students, delineating three domains and twelve indicators with empirically derived weights. The model captures essential competencies and values for Maker activities, aligns with China’s educational policies and practices, and offers practical guidance for teachers to set goals, select content, and assess outcomes. It also supplies a scientific basis for developing measurement tools to monitor students’ Maker literacy and evaluate instructional effectiveness. Future work should focus on constructing and validating specific assessment instruments based on this framework, extending data collection across additional regions and contexts, and iteratively refining the model as Maker education evolves. By emphasizing experiential learning, ingenuity, collaboration, and responsibility, the model supports preparation of students for future challenges and contributes to cultivating a creative, innovative generation.

Several limitations apply: (1) Delphi consensus reflects expert agreement rather than definitive truth; further testing and validation are needed. (2) The framework is grounded in the current Chinese Maker education context, so international comparisons require caution. (3) The AHP weighting approach is subjective and may introduce bias. (4) The study’s geographic scope limits generalizability; future research should collect broader data, including from additional Chinese provinces and developing countries (e.g., India, Pakistan, Nepal, Bangladesh, Thailand, Vietnam).