Expect the unexpected? Challenges of prospectively exploring stakeholder engagement in research

The study investigates how stakeholder engagement (SE) in health research can be prospectively examined and what methodological challenges arise when researching engagement in real time. Motivated by evidence that interaction between researchers and potential users is often associated with greater impact, the authors note that most prior analyses are retrospective, which can obscure process detail and introduce recall or selection biases. This paper reports on SEE-Impact, a longitudinal prospective study conducted alongside EQUIPT, a 3-year EU-funded project developing and disseminating a tobacco control return on investment (ROI) tool across five countries (Germany, Hungary, the Netherlands, Spain, UK). SEE-Impact set out to track SE as it occurred in EQUIPT to understand how co-production is experienced, the extent to which engagement contributes to impact, and the practical challenges of implementing and studying SE. The purpose is to inform researchers and funders about when and how to apply SE to enhance research uptake and impact, and to reflect on the trade-offs of prospective versus retrospective approaches in this context.

The paper situates SE within a broad literature highlighting its role in legitimizing findings, enhancing knowledge production, and promoting impact (e.g., Weiss 1977; Phillipson et al. 2012; Buxton & Hanney 1996). Reviews in health research associate interaction with policymakers/managers with impact (Innvær et al. 2002; Hanney et al. 2003; Lavis et al. 2005; Bullock et al. 2012), though often based on retrospective analyses where timing and nature of interactions are unclear. Methodological literature contrasts prospective and retrospective designs: prospective studies can reduce measurement error and capture change and process in real time but face challenges in recruitment, sampling, attrition, cost, and time (Euser et al. 2009; White et al. 1998; Van Ness et al. 2011; Plano Clark et al. 2015). Retrospective designs are efficient but constrained by existing data and susceptible to recall and cognitive biases (Bitektine 2008). The authors also reference debates on the costs and benefits of co-production, potential burdens and risks for participants and researchers, and the need for methods to evaluate the policy/practice impact of evidence (Oliver et al. 2019; Oliver & Boaz 2019). Prior protocol work suggests prospective studies of research processes face resource and access constraints (Greenhalgh et al. 2017).

Design: SEE-Impact is a longitudinal prospective mixed-methods study conducted in parallel with EQUIPT (October 2013–September 2016) to track SE in real time across five countries (Germany, Hungary, the Netherlands, Spain, UK). Stakeholder identification and categories:

• EQUIPT recruited stakeholders via country teams, forming a Research Advisory Group (RAG; n=9) with expertise in health policy/practice, health economics, and tobacco control. Wider stakeholders were categorized as: decision makers, purchasers of services/pharma products, professional service providers, evidence generators, and health promotion advocates.
• SEE-Impact categorized stakeholders as engaged (EQUIPT team and RAG; linked stakeholders recruited into EQUIPT within the five categories) and unengaged (unlinked stakeholders who declined EQUIPT participation; potential stakeholders not contacted by EQUIPT but identified by SEE-Impact). Planned EQUIPT SE methods (intended):
• Two stakeholder surveys (baseline and follow-up; target 75–100 stakeholders).
• Stakeholder interviews.
• Eighteen SE workshops/meetings: 4 with EQUIPT team/RAG and 14 with wider stakeholders to inform ROI tool design, validation, policy proposals, and dissemination. Planned SEE-Impact data collection (intended):
• Two stakeholder surveys (baseline and end) across the five countries, including open/closed items on expectations, understanding, and intended engagement intensity, enabling longitudinal comparison.
• Stakeholder interviews: at least 35 interviews, with 5–10 repeat interviews later in the project.
• Observations of all 18 EQUIPT stakeholder events to assess nature, timing, and level of SE; detailed field notes. Ethics: Approval from the Faculty Research Ethics Committee, Faculty of Health, Social Care and Education, St George’s University of London and Kingston University, granted 03/18/2014. Actual data collection (executed):
• Surveys: SEE-Impact items were incorporated within EQUIPT’s surveys to avoid overburdening stakeholders. Baseline: one SEE-Impact question embedded in EQUIPT’s survey; Follow-up: several SEE-Impact questions embedded in EQUIPT’s usability survey. Baseline n=93 (DE 17, HU 16, NL 28, ES 18, UK 14); Follow-up n=66 (DE 14, HU 16, NL 15, ES 14, UK 7).
• Interviews: 45 SEE-Impact stakeholder interviews conducted (country totals: DE 6, HU 8, NL 13, ES 9, UK 9), including 16 EQUIPT team members, 19 wider engaged stakeholders, and 10 unengaged stakeholders; two repeat interviews one year later. Additionally, 5 interviews with EQUIPT researchers (one per country) to understand recruitment, non-participation reasons, and general attitudes.
• Observations: Observed 6 EQUIPT stakeholder events (vs 18 planned): 4 EQUIPT team/RAG events and 2 wider stakeholder events (locations: Maastricht, Brussels [two], Budapest, London, Zagreb; attendance 22–60, spanning the five stakeholder categories). Also observed 6 EQUIPT team teleconference meetings; additional observations were limited by scheduling/notification and technical issues.
• Analytical focus: Level and nature of SE, timing, stakeholder types and motivations, interactions during events, and influence of SE on the project, documented through field notes and qualitative interview analysis.

• EQUIPT implemented two stakeholder surveys and interviews as intended, but reduced stakeholder events from 18 to 6 (4 team/RAG; 2 wider stakeholder events). This significantly constrained SEE-Impact’s planned observational data.
• SEE-Impact survey data were embedded within EQUIPT’s instruments to minimize burden: Baseline included one SEE-Impact question (“What would you like to get out of involvement in the EQUIPT project?”) with n=93 responses across five countries; Follow-up included several SEE-Impact questions with n=66 responses. High response rates were achieved but baseline depth was limited by a single question.
• Interviews exceeded targets for unique stakeholders (n=45) but follow-up interviews were limited to two due to attrition. Additional EQUIPT researcher interviews (n=5) provided context on recruitment, non-participation, and attitudes.
• Observations covered all 6 SE events that occurred and 6 internal team teleconferences; fewer events limited opportunities to observe high-level co-production dynamics.
• Reasons for reduced SE included: non-negotiable funder deadlines; early project delays; complexity in modelling inter-country decision support tools within a 36-month window; concerns about overburdening stakeholders.
• Dependency of SEE-Impact on EQUIPT’s SE plans created lack of control over data availability, volume, and timing. Despite this, cross-team collaboration (including a gatekeeper role) facilitated access and transparency.
• Actual SE level aligned more with “participation” (per INVOLVE framework) than with intended “co-production/involvement.” Stakeholders primarily provided feedback via surveys and interviews rather than being actively involved as co-producers.
• Data quality remained high: interviews and internal meeting observations yielded rich insights into SE expectations, challenges, internal tensions, and adaptations; however, total data quantity was lower than planned.
• Potential response bias noted because EQUIPT researchers administered the surveys containing SEE-Impact questions (social desirability/conflict of interest risk).

The study set out to prospectively examine how SE unfolds within a live research project and what methodological challenges arise. Findings show that when a prospective study depends on the target project’s SE activities, alterations in those activities—driven here by strict funder timelines, resource constraints, and burden considerations—directly constrain data collection. The observed shift from intended co-production to actual participation highlights a frequent gap between SE aspirations and implementation under real-world pressures. This addresses the research aim by elucidating the conditions under which SE is prioritized, scaled back, or reshaped. The power-dependence dynamic explains SEE-Impact’s limited control over data volume and timing; however, no overt power imbalance was experienced between SEE-Impact and EQUIPT due to strong collaborative relationships, gatekeeping support, and shared updates. A more pronounced asymmetry was observed between EQUIPT and its funder, where fixed deadlines and deliverables shaped SE intensity and scheduling. Methodologically, the study demonstrates the value of a prospective approach: real-time observation of evolving plans, team deliberations, and emergent tensions provided nuanced understanding unattainable through retrospective recall. SEE-Impact adopted a flexible, bricolaged design—integrating interviews with EQUIPT researchers and observing internal teleconferences—to compensate for reduced events. While this reduced standardization, it preserved analytical depth and enabled the study to meet its objectives. The work underscores trade-offs: embedding SEE-Impact items within EQUIPT surveys achieved high reach but limited independent control and may have introduced social desirability bias. Reduced events curtailed observation of higher-level co-production processes. Nonetheless, the mixed-method, adaptive strategy captured critical insights into how SE operates under constraints and how it may influence knowledge translation and potential impact.

This paper illuminates key challenges in prospectively studying stakeholder engagement within an ongoing research project. The SEE-Impact study, conducted alongside EQUIPT, revealed that strict timelines, resource constraints, and stakeholder burden concerns can lead to substantial reductions in planned engagement activities, thereby constraining prospective data collection. Despite lower-than-anticipated data quantity, the adaptive, mixed-method design produced rich, real-time insights into the nature, timing, and dynamics of SE, including a shift from intended co-production to participation. Lessons learned include: the inherent lack of control in prospective, embedded studies; the need for funder flexibility when projects involve SE; and the importance of iterative, flexible research designs that can accommodate changes without compromising rigor. While generalizability is limited due to study-specific features, the findings offer practical guidance for future prospective evaluations of SE: plan for contingencies, minimize participant burden, develop strong cross-team relationships and gatekeeping arrangements, and consider embedding measures judiciously to balance data quality with feasibility. Future research should further develop and test indicators for SE with potential for impact, explore strategies to safeguard against bias when host teams collect data, and examine funding models that support meaningful co-production timelines and resources.

• Generalizability is limited; many challenges were specific to the SEE-Impact/EQUIPT context.
• Data quantity was lower than planned due to EQUIPT’s reduction of stakeholder events (18 planned to 6), and embedding SEE-Impact items within EQUIPT surveys (baseline limited to one question).
• Follow-up interview attrition limited longitudinal depth (only two repeat interviews achieved), influenced by relocation, scheduling, and costs.
• Potential response bias: EQUIPT researchers administered surveys containing SEE-Impact questions, raising social desirability/conflict of interest concerns.
• Observational coverage of internal teleconferences was constrained by scheduling notifications and technical issues, potentially limiting insight.
• As a prospective, embedded study, SEE-Impact lacked control over timing, availability, and scope of SE activities, affecting sampling and data completeness.