Assessing scale reliability in citizen science motivational research: lessons learned from two case studies in Uganda

Citizen science (CS) is an approach where members of the public contribute to scientific research. It offers benefits such as boosting research capacity, diversifying knowledge, and enhancing social capacity. However, participant retention is vital for long-term success. Motivation is the driving force behind participation, varying across individuals and contexts. Understanding motivations is crucial for designing effective recruitment and retention strategies. Existing literature on CS motivation is skewed towards the Global North, with limited research in the Global South. This study aimed to address this gap by investigating the motivations of CS participants in southwest Uganda, focusing on three projects: Digital citizen science for community-based resilient environmental management (D-SIRe), Natural HAzards, RISks and Society in Africa (HARISSA), and Action Towards Reducing Aquatic snail-borne Parasitic diseases (ATRAP). These projects involved citizen scientists collecting data on natural hazards and snail-borne diseases. The research utilized established theoretical frameworks to understand motivations: the Volunteer Functions Inventory (VFI) and the Theory of Planned Behaviour (TPB). The study specifically focuses on the reliability of these frameworks when applied to the unique context of CS in the Global South, particularly addressing the limitations of solely relying on Cronbach's alpha for reliability assessment.

CS motivational research often draws from volunteerism, psychology, and sociology. The functional approach to volunteering, describing six psychological functions (values, understanding, social, career, protective, enhancement), is frequently used. The Volunteer Functions Inventory (VFI) assesses these functions. The Theory of Planned Behaviour (TPB) predicts behavioral intentions based on attitude, subjective norms, and perceived behavioral control, sometimes including self-identity and moral obligation. These frameworks have shown strong psychometric properties in various settings. However, their application in CS research, especially concerning scale evaluation (validity and reliability), is limited, particularly in the Global South. This study addresses this gap by examining the reliability of VFI and TPB in the Ugandan context.

This study employed a quantitative research design using the VFI and TPB frameworks. The study population consisted of two groups: active citizen scientists from two CS networks (ATRAP and D-SIRe/HARISSA) and a control group of candidate citizen scientists. The CS networks involved participants collecting data on natural hazards (GO) and snail-borne diseases (ATRAP). Participant recruitment followed specific criteria, including Ugandan nationality, age, English proficiency, and community involvement. Active participants received resources such as financial compensation, equipment, and training. A semi-structured questionnaire, incorporating VFI and TPB items adapted for the CS context, was administered via face-to-face interviews in both individual and group settings. Data analysis involved three stages: (1) a priori reliability analysis using Cronbach's alpha; (2) examination of alpha's underlying assumptions (normality, unidimensionality, uncorrelated errors, tau-equivalence) through item analysis, Shapiro-Wilk test, confirmatory factor analysis (CFA) using Lavaan package, and residual correlation matrix analysis; and (3) reliability analysis using alternative indices (McDonald's omega and Greatest Lower Bound (GLB)). Data transformations (log and inverse) were explored to address skewed distributions. The analysis was conducted using R version 4.2.2, and statistical significance was set at p ≤ 0.05. Metadata from field notes were also utilized to provide qualitative context.

The study involved 161 participants (74% male), aged 20-59 (mean 34, SD 8). A priori Cronbach's alpha values varied significantly across factors, ranging from 0.91 to -0.03. Most VFI factors showed relatively adequate alpha values, while only two TPB factors did. The PBC, intention, moral obligation, and self-identity factors consistently showed low, unacceptable alpha values. Examination of alpha's assumptions revealed violations in most factors. Most items were negatively skewed, and the Shapiro-Wilk test rejected normality for all items. Data transformations did not significantly improve alpha values. CFA model fit indices indicated violations of unidimensionality for more than half of the factors. The assumption of uncorrelated errors was also violated for many factors. Standardized factor loadings revealed non-tau-equivalence in all factors. Alternative reliability indices (omega and GLB) generally yielded higher values than alpha, but the pattern of low reliability for certain TPB factors persisted. Metadata analysis revealed that many problematic items, with weak item-total correlations, caused participant confusion. Excluding these items improved omega values. The TPB factors (PBC, intention, moral obligation, self-identity) consistently exhibited low reliability across all estimates.

The study demonstrates the limitations of solely relying on Cronbach's alpha when assessing the reliability of adapted psychometric tests in novel contexts. The frequent violations of alpha's assumptions highlighted the need for less restrictive measures like omega and GLB. The consistent pattern of low reliability for some TPB factors suggests potential issues with either the conceptual fit of the TPB to the CS context or the specific item wording. The weak item-total correlations and participant confusion identified through metadata analysis underscore the importance of careful item construction and context-specific validation. The homogenous nature of the sample, possibly due to participant selection bias, might have contributed to the clustered scores and low alpha values. The study highlights the challenges of directly transferring frameworks developed in WEIRD societies to diverse cultural settings. This emphasizes the need for considering cultural nuances in questionnaire design and the development of context-specific instruments for CS research in the Global South.

This study reveals the limitations of applying established frameworks like VFI and TPB in citizen science contexts without careful consideration of their reliability and cultural applicability. Cronbach’s alpha is shown to be insufficient and less restrictive measures like omega and GLB are recommended for future research. The findings highlight the need for rigorous scale validation, including careful item wording, and consideration of cultural context when conducting citizen science motivational research in the Global South. Future research should focus on developing context-specific instruments and exploring different methodologies to assess participant motivation more accurately.

The study's primary focus was not on psychometric evaluation, limiting the assessment of other reliability types (e.g., test-retest). The questionnaire's length and use of English might have influenced responses. The control group was quite similar to the CS group, and socially desirable responses might have occurred due to the interviewer-participant dynamic. The small sample size might also limit the generalizability of the results. Future research should address these limitations by including other reliability methods, using local languages, and employing a more diverse control group.