Indicators for tracking programmes to strengthen health research capacity in lower- and middle-income countries: a qualitative synthesis

The study addresses how health research capacity strengthening (RCS) initiatives in low- and middle-income countries are evaluated and tracked using indicators, and whether current indicators adequately capture progress and outcomes across levels (individual, institutional, and national/regional/network). Contextually, despite growing recognition of the need for LMICs to generate and use health research, systematic assessments of health RCS effectiveness have been hindered by heterogeneity and complexity of initiatives. Funders and coordinating bodies (e.g., ESSENCE) have emphasized theories of change and indicator frameworks, yet gaps persist in measurement quality and linkage of activities to outcomes. The objective of this study was to describe the design quality of health RCS evaluations, catalogue the indicators used, and assess linkages among activities, outputs, and outcomes to inform more rigorous tracking and accountability.

Background literature highlights: (1) increasing global emphasis on LMIC capacity for equity-oriented health research and various frameworks/resources for RCS; (2) the SMART criteria for development evaluation indicators and calls from research impact evaluators to include indicators beyond knowledge production to policy and practice change; (3) the importance of explicit theories of change and linking activities through outputs to outcomes within evaluation frameworks (e.g., ESSENCE); and (4) persistent challenges in systematically assessing RCS effectiveness due to initiative heterogeneity and limited evaluation strategies. The literature also notes variable terminology at the macro level (societal, environment, network) and the need to clarify scale for consistent comparisons.

Design: Qualitative synthesis incorporating report identification, quality appraisal, indicator extraction, and synthesis against a framework. Ethics: University of Toronto Health Sciences Research Ethics Board approval (#26837). Report identification: Consulted ESSENCE member funders and other LMIC research funders using a snowball approach. Inclusion: publicly available, English-language evaluations since 2000. From 54 reports identified, 18 reports covering 12 evaluations were purposively selected using maximum variety sampling to ensure diversity of RCS types, funders, countries, and evaluation approaches. Quality appraisal: Based on OECD DAC standards. Key questions: clarity of evaluation purpose; description of methodology/analysis; explicitness and justification of indicators. Particular attention to evaluation design, indicator measurement/collection, and bias. Two reviewers independently appraised each evaluation. Indicator extraction: Systematic framework analysis. Extracted explicit and implied indicators and their context, coding them to ESSENCE Planning, Monitoring and Evaluation matrix categories (individual, institutional, national/regional/network), creating new categories as needed. Each report double-coded by two researchers; discrepancies resolved through discussion/third reviewer. Stopped at analytic saturation. Synthesis: Searched for links between aims and indicators and along pathways from activities to outputs to outcomes (theory-of-change consistent). Because single evaluations rarely provided full pathways, synthesized cross-evaluation indicator examples to illustrate potential linkages. Interim findings were iteratively validated with ESSENCE stakeholders.

- Evaluation designs: All 12 evaluations stated clear purposes; most used mixed methods drawing on document reviews, surveys, interviews, and site visits. Complexity varied by initiative scope and evaluation timing. Constraints included lack of monitoring/evaluation frameworks, short review timeframes, lack of baseline data in most evaluations, and only one considering a potential control comparison. These limited attribution and effectiveness estimation. - Indicator quality and use: Indicators commonly addressed activities, outputs, or outcomes, but rarely their inter-relationships. Few indicators met SMART criteria fully; measurement properties were seldom addressed. Some evaluations used bibliometrics (e.g., publication counts, citations, impact factors, norm-referencing). Stakeholder-designed frameworks with intervention logic were occasionally applied. - Bias and validity: About half addressed potential biases (e.g., response bias, recall), with triangulation via multiple sources/site visits; only one described a formal pilot of a survey instrument. Low response rates were noted in several evaluations. - Individual-level indicators: More quantitative/comparable and sometimes equity-attentive (e.g., disaggregation by gender, nationality, country income level, discipline, award level). Indicators covered research skills training (PhDs, MScs, fellowships), mentoring, conference/workshop participation, curricula development, job outcomes, return to LMIC research, and new funding. Equity dimensions beyond sex/nationality were rare. - Institutional-level indicators: Highly diverse; linked individual support to institutional strengthening. Indicators covered mentoring capacity, research infrastructure and management (hard and soft), governance, quality assurance, accreditation, research support centers, strategic planning, and collaboration (North–South/South–South), with attention to ownership and visibility. - National/regional/network-level indicators: Included engagement and communication for research uptake (plans, media outputs, website metrics), development of national health research systems (commitment of ministries, councils, priority-setting, legal frameworks), networking outputs/outcomes (multidisciplinary platforms, harmonized regional activities, policy/practice impacts), and financial sustainability. Indicators of research users’ capacities were rare. - Pathways: Within single evaluations, linkage from activities to outputs to outcomes was limited; across evaluations, the authors constructed potential pathways with corresponding indicators (illustrated via Tables 2–4). - Equity: Disaggregation by equity categories was uncommon despite global equity priorities; some promising practices noted (e.g., targeted recruitment of disadvantaged groups).

The study’s synthesis shows that while many evaluations track activities, outputs, and some outcomes of health RCS, they often lack rigorous design features (baselines, comparisons), validated measures, and explicit theories of change linking indicators along impact pathways. This undermines attribution and learning about what works. Individual-level indicators are more standardized and sometimes equity-informed, whereas institutional and national/network indicators are heterogeneous, reflecting system complexity. Findings underscore the need to improve indicator quality (measurement properties, validity, reliability), expand indicator scope to encompass relationships with research users and uptake, and clarify levels of analysis and scale (e.g., separating provincial-national environments, international-global contexts, and networks). Early stakeholder engagement in indicator selection and prospective monitoring, aligned with clear frameworks, can enable stronger contribution assessments and accountability across funders, institutions, researchers, and users.

This qualitative synthesis catalogs indicator types used in LMIC health RCS evaluations and highlights wide variability in evaluation design and indicator quality, with limited linkage along pathways of change. Advancing the field requires: rigorous, prospective evaluation designs; systematic attention to indicator framing, measurement, and equity disaggregation; incorporation of relationships with knowledge users; and explicit theories of change guiding indicator linkage from activities to outcomes. Implementing comprehensive, long-term monitoring and evaluation systems within RCS programs can generate robust evidence on effectiveness and impact to justify and optimize investments.

Study limitations include incomplete access to all funder evaluations; labor-intensive extraction limiting the number of evaluations analyzed; occasional need for discussion to allocate narrative extracts/indicators to framework categories; most evaluations provided only single time points with few longitudinal views; and limited integration across broader health development and research efforts. Nonetheless, the selected evaluations spanned diverse funders, contexts, and RCS types, and analysis reached thematic saturation.