Evaluations of training programs to improve capacity in K*: a systematic scoping review of methods applied and outcomes assessed

The paper investigates how K* training programs—activities that build capacity to connect research, policy, and practice—are evaluated. K* encompasses functions such as knowledge translation, exchange, mobilization, and brokering, intended to make research useful, usable, and used outside academia. The context is growing interest and investment in evidence use across sectors (e.g., education and health) and recognition that individual capacity building is a key lever to promote research use. Despite increased funding and programming, there is no consensus on which outcomes or methods should be used to evaluate K* training. The study aims to map and synthesize evaluation methods and outcome indicators used in evaluations of K* training programs and to identify gaps and future directions to inform both the literature and the authors’ own program evaluation.

Prior work indicates that multi-pronged strategies are needed to promote research use, with capacity building for individuals as an important component. Various related concepts populate the K* landscape (knowledge brokering, mobilization, translation, exchange, boundary spanning, dissemination and implementation). Reviews have described diverse system-level strategies (e.g., funding, policies, networks) to support knowledge use. However, there is limited consensus on evaluation indicators and methods for K* training. Existing reviews and commentaries highlight growing investment in training but note variability in terminology and approaches, and a dominance of health fields. This backdrop motivates a scoping review to map current evaluation practices and outcomes assessed for K* training.

Design: Systematic scoping review guided by Arksey and O’Malley’s five-stage framework: (1) identifying the research question, (2) identifying relevant studies, (3) study selection, (4) charting the data, and (5) collating, summarizing, reporting results. Scope: Population included K* professionals (knowledge brokers, researchers and graduate students, policymakers, practitioners, community members). Intervention included any K*-related training or capacity-building activity. Study designs and outcomes were intentionally broad to capture the range of evaluation approaches. Search strategy: Searched eight multidisciplinary databases (ProQuest, ScienceDirect, JSTOR, EBSCO, PubMed, Web of Science, Academic OneFile, Scopus) and Google Scholar for articles published before August 2022. Also screened reference lists of included studies and relevant reviews. Inclusion criteria required studies to describe evaluations of K* training programs and report training descriptions, evaluation methods, and outcomes; English only. Exclusions included literature reviews without primary data, studies describing training without evaluation, or inaccessible full texts. Screening and selection: 1,297 citations identified; after deduplication, 824 unique records screened by three independent reviewers based on titles/abstracts, with discrepancies resolved by discussion; 127 full texts assessed; 47 documents included. Data extraction and analysis: Used a standardized template; two researchers extracted, a third checked; disagreements resolved by consensus. Employed scientometric analysis (VOSviewer v1.6.19) to map publications, co-authorships, bibliographic coupling, and keyword co-occurrence. Conducted qualitative content analysis to synthesize methodological characteristics and outcomes assessed. Outcomes categorized using the Kirkpatrick four-level model (reaction, learning, behavior, results). Reporting followed scoping review principles (mapping evidence without critical appraisal of individual study quality).

Scope and trends: Of 824 unique records screened, 47 studies met inclusion. Publications grew markedly after 2012 (43/47; 91%), with increases noted after 2014 as well. Most studies were from the United States and Canada. Journals: The top five journals accounted for 25/47 (53%); Implementation Science alone published ~36% of included articles. Highly cited works included Meissner et al. (2013) with 147 citations, Straus et al. (2011) with 124, Brownson et al. (2017) with 56, Stamatakis et al. (2013) with 52, and Padek et al. (2018) with 51. Collaboration and networks: Most collaborative authors included Ross Brownson (n=7), Enola Proctor (n=5), Karen Emmons (n=4), and Sharon Straus (n=4). Institutions collaborating most included Washington University (n=8), National Cancer Institute (n=4), McMaster University (n=4), and St. Michael’s Hospital (n=4). By country, the USA (n=18) and Canada (n=13) led. Bibliographic coupling showed strongest link strengths for Moore et al. (2018; 99), Padek et al. (2018; 90), and Brownson et al. (2017; 89). Keyword co-occurrences emphasized knowledge translation, implementation, dissemination, capacity building, training, evaluation, education, mixed methods, and implementation science. Methodological characteristics (n=47): - Types of evaluation (non-exclusive): process 68.1%, outcome 61.7%, impact 12.8%, formative 4.3%, cost-benefit 0%. - Methods: mixed methods 89.4%; qualitative 8.5%; quantitative 2.1%. - Design: one-group 91.5%; two-group 2.1%; not applicable 6.4%. - Sample sizes: not reported 19.1%; ≤29 participants 36.2%; 30–84 participants 29.8%; 85–781 participants 14.9%; ≥782 participants 0%. - Data collection techniques: surveys 76.6% (of these, reaction 86.1%, pre/post competency 52.8%, post-only 30.6%, network 8.3%); interviews 29.8%; document analysis 14.9%; participant feedback 10.6%; critical reflection 10.6%; focus groups 10.6%; student data 8.5%; bibliometrics 4.3%; observation 4.3%; program data 2.1%. - Timing of data collection: before intervention 27.7%; during 17.0%; immediately after (≤1 month) 42.6%; 1–5 months 4.3%; 6–11 months 21.3%; ≥12 months 21.3%; unclear 36.2%. Outcomes assessed via Kirkpatrick model: - Reaction: assessed in 37/47 (78.7%), commonly via self-report Likert and open-ended surveys, sometimes interviews/focus groups; focused on satisfaction, relevance, organization and clarity of curriculum, delivery format, trainer competence, cohort learning, engagement, and areas for improvement. - Learning: assessed in 38/47 (80.9%); measured knowledge/skills, confidence, and commitment via pre/post self-assessments, interviews, focus groups, observation, and student data; often used mean ratings and significance tests; some relied on post-only assessments; multiple timepoints (pre, during, 6- and 12-months) were occasionally used. - Behavior: assessed in 32/47 (68.1%); used self-report surveys, interviews, student data, and social network surveys; examined engagement in K* activities, use of resources, influence on colleagues, and development of collaborations. - Results (downstream impacts): least assessed, in 20/47 (42.6%); often via self-report; also bibliometrics (e.g., publications and grants vs control), participant feedback on employment, social network analysis of sustained collaborations, and qualitative reports of organizational changes. Common limitations of included studies: reliance on simple, largely single-group designs; small samples; short-term follow-up; infrequent curriculum evaluation; few long-term and Level 4 (results) assessments; limited evaluation outside health fields.

The review answers its research question by mapping how K* training programs have been evaluated, what methods are used, and which outcomes are assessed. Findings show a young and growing field concentrated in health and implementation science, with increased activity post-2012. Evaluations predominantly use mixed-methods, single-group designs, and surveys, focusing on immediate reactions and learning, with fewer studies measuring behavior change and especially downstream results. Collaboration networks exist but are limited, suggesting opportunities to share resources and avoid duplication. The significance lies in highlighting methodological gaps—limited rigor, small samples, and lack of long-term follow-up—that constrain causal inference and understanding of real-world impacts of K* training. The synthesis underscores the need to strengthen evaluation designs, standardize measures, extend timelines to capture behavior and results, and broaden the disciplinary scope beyond health to fields like education and the social sciences.

This scoping review combines scientometric mapping with qualitative content analysis to provide a comprehensive overview of K* training evaluations, identifying influential publications, collaboration patterns, commonly used methods, and outcome assessment practices. The review documents steady growth in the literature and prevalent use of surveys and mixed-methods designs, but also widespread weaknesses, including simple designs, small samples, and limited long-term and results-level assessments. The authors recommend increasing rigor via theory-informed logic models, inclusion of both process and impact components, appropriate pre/post measures (self-report and/or competency assessments), triangulation (e.g., 360-degree feedback), adequate sample sizes, and, where feasible, two-group designs. For impact assessment, contribution analysis and longitudinal follow-up at multiple intervals (e.g., 6–24 months) are encouraged. Future research should evaluate curriculum components, consider contextual and spillover effects, use standardized measures, and expand studies beyond health to other sectors.

The search, though broad across eight databases, Google Scholar, and reference lists, may have missed unpublished or inaccessible evaluations and studies using alternative terminology for K*. Only English-language reports were included. The included corpus is relatively small (n=47), limiting generalizability. The review did not analyze the structural details of the training programs themselves or how evaluation findings were used to improve programs. As a scoping review, it maps evidence without critically appraising individual study quality.