The global need for health research to inform policy and practice is widely recognized, yet significant gaps persist, particularly in LMICs. While resources are allocated to health RCS, its effectiveness remains inconsistently assessed. This study addresses the lack of systematic assessment of health RCS initiatives by examining existing evaluations to describe their design, indicators used, and linkages among activities, outputs, and outcomes. The goal is to provide evidence for designing rigorous evaluations and selecting appropriate indicators to demonstrate value to all stakeholders (funders, organizations, researchers, trainees, and users). The complexity and heterogeneity of health RCS initiatives have hampered systematic assessments of their effectiveness. A clear need exists for improved strategies and frameworks for RCS tracking, particularly given the increasing scale and complexity of RCS initiatives. The use of indicators to monitor performance, measure achievement and demonstrate accountability is widespread in health programmes, and well established criteria for development evaluation exist. However, research impact evaluations require indicators of not just knowledge production and capacity development, but also changes in health systems and practices. This study investigated existing health RCS evaluations to examine their design, indicators used, and linkages between activities, outputs and outcomes, aiming to inform the design of more rigorous evaluations and effective indicator selection for tracking progress and impacts.

The paper refers to several existing frameworks and guidelines for evaluating health research capacity strengthening, including the ESSENCE on Health Research Initiative's Planning, Monitoring and Evaluation framework, and the Development Assistance Committee standards. It cites previous research highlighting the challenges in evaluating the effectiveness of health RCS initiatives due to their heterogeneity and complexity, and the need for improved monitoring and evaluation strategies. The authors note the use of SMART (Specific, Measurable, Attainable, Realistic, and Timely) indicators in development evaluations and the suggestion to include indicators of changes in health system policies, programs, and practices alongside knowledge production and capacity development.

This study employed a qualitative approach involving report identification, evaluation quality appraisal, indicator extraction, and synthesis. Stakeholder consultation with LMIC health research funding agencies was conducted. Ethical approval was obtained from the University of Toronto Health Sciences Research Ethics Board. Report identification involved contacting funding agencies and using a snowballing process to gather reports. The researchers analyzed 54 reports from 31 funding agencies to select 18 reports from 12 evaluations using maximum variety sampling. Quality appraisal of the evaluations used questions derived from the Development Assistance Committee standards, focusing on the clarity of the evaluation purpose, the description of the methodology, and the explicit justification of indicators. Indicator extraction involved a systematic framework analysis, extracting text related to indicators and their context. The ESSENCE Planning, Monitoring and Evaluation matrix was used as a framework for categorizing the indicators. The team employed a systematic coding process to ensure reliability, and two researchers independently extracted text from each report, resolving discrepancies through discussion. Synthesis involved reviewing extracted material, creating additional categories as needed, and attempting to identify and document links between aims, indicators, and the pathway from activities to outputs and outcomes. Interim findings were shared and discussed with the ESSENCE on Health Research initiative steering committee.

The twelve evaluations examined varied widely in duration, evaluators, evaluation stages, and approaches. While all evaluations stated their purpose clearly, the quality of their designs varied significantly. Few evaluations utilized baseline data or control comparisons, hindering the assessment of change and attribution to the health RCS program. Most evaluations used mixed-methods designs and drew on existing data. The quality of indicators varied widely, and although many were specific, attainable, realistic, and timely (SMART), they struggled to be consistently measurable. Only one evaluation formally piloted its questionnaires. Half the evaluations addressed potential biases, while some used triangulation and iterative feedback to enhance their validity. Indicators were categorized into individual, institutional, and national/international levels. Individual-level indicators tended to be quantitative, comparable, and considerate of equity. Institutional and national/international level indicators showed substantial diversity. Although individual evaluations did not always show a clear pathway from activities to outputs to outcomes, the analysis was able to construct such pathways across the evaluations, linking common indicators. Individual level indicators commonly included training in research skills, professional skills, quality of training, and trainee satisfaction. Equity-related disaggregations of data by gender, nationality, country income, discipline and award level were present, but equity indicators by socio-economic status or minority status were less common. Institutional level indicators included support for individual grantees, retrospective funding analyses, mentoring capacity, research infrastructure and management, and institutional collaborations. National and international level indicators focused on activities with policymakers, research uptake (including stakeholder engagement), national research capacity (eg. research councils), and the promotion of research sustainability.

The study revealed a broad set of indicators across evaluations, but also highlighted important gaps. The rarity of equity-related disaggregations in the data is concerning, given the global focus on health equity. The lack of indicators addressing ongoing relationships among stakeholders for facilitating research conduct and use was also noted. Nomenclature for national/international level indicators was inconsistent. The study highlights the need for clarification of scale and separation of components (provincial-national, international-global, and research networks). The quality of the indicators, including measurability, was often insufficient to meet SMART criteria. This may result from divided responsibility for data collection and limited investment in evaluation. Most evaluations were retrospective, hindering assessment of long-term impacts and causal relationships. The limited use of theories of change further hampered the assessment of program contributions to impact.

This research synthesizes new knowledge about evaluation designs and indicators for tracking health RCS initiatives. Using more rigorous designs and improved measurement within clear evaluation frameworks will produce robust evidence of effectiveness and impact to better justify investments in health RCS. Recommendations are provided for funding agencies, decision-makers, producers, users, and evaluators to improve the quality and effectiveness of evaluations, including adequate resource allocation, systematic indicator development, attention to equity, use of theories of change and prospective monitoring systems.

Not all contacted funders provided reports, and the analysis was limited by the number of evaluations examined. The labor-intensive nature of data extraction and analysis constrained the scope of the analysis. However, the diversity of the included evaluations and the emergence of common themes suggest that additional evaluations would not significantly change the findings. The allocation of extracts to framework analysis categories sometimes required discussion to reach consensus. Most evaluations captured only a single point in time, limiting longitudinal insights. Few evaluations incorporated the contributions of various health development efforts, research programs and RCS initiatives to the development of a broader health research system.