Design, content validity, and inter-observer reliability of the 'Digitization of Cultural Heritage, Identities, and Education' (DICHIE) instrument

The paper addresses the growing importance of digitizing cultural heritage for protection, conservation, research, dissemination, and public engagement, in line with EU policy initiatives (e.g., Horizon 2020, Europeana, and the 2019 pan-European declaration). It situates digital technologies—AR/VR, multimodal and interactive systems, crowdsourcing, and metadata co-creation—as catalysts for access, participation, and identity construction. Despite these opportunities, evaluations of Spanish heritage education programs involving advanced technologies reveal low educational quality and design specificity, highlighting a need for rigorous, validated instruments. The study’s purpose is to design and validate DICHIE, a focus-group instrument to collect qualitative data on needs, challenges, and solution pathways for digitizing cultural heritage in Castile and Leon, and to inform education and cultural policy.

The review outlines European policy frameworks promoting digitization and access (European Commission; 2019 declaration), and research emphasizing user engagement, co-creation, and immersive experiences (Ridge; Roussou & Katifori; Portalés et al.). It covers participatory approaches such as crowdsourcing for metadata enrichment and archival semantics (Alemu; Goy et al.; Rettig et al.), noting heterogeneity across domains (Davies) and limits in museum indexing practices (Golub et al.). Educational research in Spain connects digital heritage with citizenship education and learning (DIDPATRI, DESYM, DiCSO, GIPYPAC, ARGOS), yet evaluations show lower quality when advanced technologies are included (Ibáñez-Etxeberria et al., 2020). Digital heritage supports identity, historical understanding, reconstruction, and community memory (Martínez et al.; Saygi et al.; Johanson; Tasker & Liew; Beel & Wallace; Purkis). Technological innovations (3D, modeling, laser printing) affect public contextualization and management (Tamborrino & Wendrich; Masciotta et al.; Ocón). However, few recent validated instruments exist to assess professional needs and challenges in digital heritage across conservation, management, and education, justifying the creation of DICHIE.

Design: Quantitative-descriptive, instrumental psychometric study focused on reliability and content validity. The process included two evaluation rounds. Instrument: DICHIE comprises 15 key questions across 5 theoretical dimensions: (1) Access and use; (2) Potentialities; (3) Participative cultural behavior; (4) Memory and collective identities; (5) Obstacles and limitations. Questions were derived ad hoc from prior studies (see Table 1 in the paper for sources per question). Participants: 20 expert judges selected via intentional sampling, representing three professional groups: heritage conservation (n=6), heritage education in Castile and Leon (n=8), and cultural management (n=6). Experts rated each question on five evaluation criteria: clarity (EC1), internal coherence (EC2), bias/induced response (EC3), linguistic adaptation (EC4), and precision (EC5). Procedure: Phase 1—independent evaluation by each professional group of question consistency and coherence. Phase 2—rewording items not meeting criteria, followed by a second evaluation round with the same experts to confirm improvements (notably Question 5). Measures and scales: 4-point ordinal scale (1=low, 2=insufficient, 3=moderate, 4=high) for each EC at both global instrument and item levels. Content validity assessed with Aiken’s V. Statistical analysis: Inter-judge agreement assessed with Kendall’s W; inter-observer reliability with Fleiss’s Kappa and Krippendorff’s alpha. Intra-group agreement assessed with Bangdiwala’s original (BN) and weighted coefficients (BWN), suitable for ordinal data. Inter-group differences tested with Kruskal–Wallis H, effect size via epsilon-squared (ε²_R), and Games–Howell post hoc tests. Software: SPSS v25 and RStudio 1.4.1717 (vcd, vcdExtra). Ethics: informed consent obtained; anonymity and data protection ensured; study approved by University of Burgos Bioethics Commission (July 14, 2021).

- High inter-judge agreement across both evaluation rounds: Kendall’s W global concordance W=0.693 (p=0.000) first round; W=0.706 (p=0.000) second round, with dimensional significance across ECs. - Item-level improvement: Question 5 initially showed lower clarity and coherence; after rewording, second-round ratings reached optimal values. Aiken’s V for Q5 improved to Vb=0.83 (CI 0.72–0.91) and up to 1.00 (CI 0.94–1.00). - Inter-observer reliability: Global Fleiss’s Kappa indicated optimal strength of agreement—κ=0.712 (p=0.000) first round; κ=0.725 (p=0.000) second round. Krippendorff’s alpha corroborated reliability—α=0.859 (first), α=0.878 (second), with criterion-specific alphas ranging approximately 0.846–0.974. - Intra-group agreement: Bangdiwala weighted coefficients showed excellent intra-professional agreement—BWN≥0.913 across groups; global BWN up to 0.984 (with BN up to 1.000 in some dimensions). - Inter-professional differences: Statistically significant differences appeared in EC2 for Q7, EC3 for Q9, EC3 for Q12, and EC5 for Q13 (Kruskal–Wallis p≤0.05), with moderate effect sizes (ε²_R≥0.33). Post hoc comparisons indicated differences between conservation and cultural management for EC2–Q7 and EC3–Q9 (p=0.027), and between conservation and heritage education for EC3–Q12 and EC5–Q13 (p=0.025). Generally, educators and managers assigned lower scores than conservationists on several criteria. - Overall, DICHIE demonstrated optimal internal reliability and comprehensive content validity for use with professionals in heritage conservation, education, and cultural management.

The validated DICHIE instrument directly addresses the need for rigorous, reliable tools to assess professional needs, challenges, and solution pathways in digital cultural heritage. Strong agreement and reliability metrics (Kendall’s W, Fleiss’s κ, Krippendorff’s α, Bangdiwala’s BWN) demonstrate that the questions effectively capture clarity, coherence, bias control, linguistic adequacy, and precision, fulfilling the study’s aims. The identified inter-professional differences (notably on items about public-centered designs, metadata co-creation, and identity-related narratives) help anticipate focal points of debate in focus groups, facilitating targeted dialogue among conservationists, educators, and managers. The findings reinforce the importance of inter-professional collaboration and public participation (e.g., co-creation, crowdsourcing, gamification) in designing and evaluating digital heritage initiatives that support qualitative learning, identity construction, and community memory. Thus, DICHIE provides a robust framework to guide discussions and inform educational and cultural policy decisions in the regional context.

The study offers a validated, reliable focus-group instrument (DICHIE) comprising 15 questions across five dimensions to evaluate needs, challenges, and solutions in digitizing cultural heritage within Castile and Leon. Evidence from multiple concordance and reliability indices confirms strong content validity and inter- and intra-observer reliability, making DICHIE suitable for professional contexts spanning conservation, education, and cultural management. The instrument complements established evaluation tools in Spanish heritage education by focusing on digital heritage processes and inter-professional/public collaboration. Future research could apply DICHIE in focus-group implementations across diverse regions and institutions, integrate findings into policy and program design, and expand the instrument with additional items or dimensions based on empirical use cases and longitudinal evaluations.

The paper does not include a dedicated limitations section or explicit discussion of limitations.