Peer reviewers' dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

The paper investigates how external subject experts experience and enact the first stage of grant peer review: independent assessment of applications. Although peer review underpins legitimacy and merit-based allocation of research funding, concerns about bias, reliability, innovation-stifling effects, inefficiency, and limited predictive validity have prompted scrutiny of funding processes. Prior work—especially Lamont and colleagues’ ethnographies of research panels—highlights that grant evaluation is shaped by contextual, psychological, and social factors, and that fairness emerges through discipline-appropriate epistemic styles rather than universal criteria. The authors argue that different stages of peer review (independent external reviews, internal panel reviews, and panel deliberations) involve distinct decision processes; independent reviews, in particular, may be uniquely affected by time investment, reliance on personal expertise, and absence of group dynamics. The study aims to capture the phenomenological experience of individual external reviewers evaluating real applications in near real time, focusing on their “horizons” (anticipated possibilities beyond what is written) and “background assumptions” (implicit pre-judgements) that shape interpretations and choices.

The introduction synthesizes a broad literature critiquing grant peer review: evidence and claims of bias against women, interdisciplinary work, and less-known applicants; low inter-reviewer reliability, especially among inexperienced reviewers; tendencies to discourage innovation; inefficiencies and burdens on applicants; and weak prediction of downstream success. While some findings are mixed, much prior research correlates applicant/application attributes with outcomes, revealing systemic issues but offering limited insight into reviewers’ real-time judgement processes. Qualitative research on panels (Lamont; Mallard et al.) shows tensions between evaluation and self-interest, and between democratic consensus and expertise, with fairness grounded in discipline-appropriate evaluation styles. Research on decision-making contrasts individual versus group vulnerabilities (biases and cognitive limits versus groupthink/peer pressure). These backgrounds motivate direct observation of individual reviewers’ cognitive and moral deliberations during independent review, a comparatively under-studied stage.

Design: Qualitative study using a phenomenological lens and a “Big Q” qualitative paradigm to explore reviewers’ thoughts as they appear to consciousness. Methods combined simultaneous think-aloud verbal protocols with adaptations of the Critical Decision Method (CDM) to elicit fine-grained decision processes about real, non-routine tasks. Sampling and participants: Purposive recruitment of reviewers in medical humanities and social sciences who had evaluated applications to the Wellcome Trust Awards in Humanities and Social Science (July 2018 and January 2019 rounds). Of 44 invited reviewers, 16 participated (36% response) between July and December 2019. Participants spanned roles (lecturer, reader, professor), disciplines (predominantly history, plus sociology, ethics/law, philosophy, economics, anthropology), and geographies (UK regions and international). Procedure: After reviewers submitted their original evaluations to the funder, the funder (separately from funding decisions) shared consented participant details, the specific application reviewed, and the written review with the research team. Remote Zoom interviews (~60 min) were conducted. Interview flow: (1) unstructured account of the review process; (2) think-aloud re-review of the same application as if first encountering it, with the interviewer recording salient decision points; (3) general appraisal and clarification questions. A practice think-aloud on a generic abstract preceded the focal re-review. Screen and audio were recorded. Demographics and review experience were collected. Participants received a debrief and £45 voucher. Interviews were transcribed verbatim. Ethics: Approved by Kingston University Faculty of Business and Social Sciences ethics committee (ref: 181950); conducted per Declaration of Helsinki, BPS code, and institutional policies. Informed consent obtained; confidentiality and withdrawal rights assured. Analysis: Transcripts were managed in Dedoose and analyzed using Braun and Clarke’s six-phase thematic analysis. An inductive–deductive approach identified novel themes aligned with the research aim, focusing on motivations, experiences, perceptions, and practices. AW coded all transcripts; TV double-coded a subsample. Themes were developed iteratively with AW, TV, and GVT; discrepancies resolved by discussion. Quality criteria included grounding interpretations in data excerpts, credibility checks, and coherence across the study.

- Sample and process: 16 reviewers (36% of 44 invited) across medical humanities and social sciences completed ~60-minute think-aloud/critical decision interviews about actual Wellcome Trust grant applications they had reviewed. Participants included 4 women and 12 men; disciplines were predominantly history (10/16), with a range of ages and roles. - Core result: Reviewers encountered five interrelated, non-linear dilemmas during independent evaluation, reflecting moral conflict and social considerations rather than simple heuristics: 1) Whether to accept an invitation to review: balancing workload and time constraints against topic fit, interest, familiarity with applicants/mentors, perceived project value, potential benefit to one’s own research, and gratitude/reciprocity toward the funder. 2) Whether to rely exclusively on application content: tension between impartiality (judging only “what’s on the page”) and completeness (using prior knowledge or searching for applicant/institution information). Strategies varied: disclosure of prior knowledge; self-imposed sequencing (reading proposal before CVs); or Googling unknown applicants versus explicitly rejecting external searches as inappropriate. 3) Whether to attend to institutional prestige: ambivalence about signals from elite affiliations, funder networks, and high-profile recommenders. Some saw prestige as informative; others critiqued deference to elite institutions and favored redressing systemic imbalances by supporting strong proposals from less-resourced institutions (“Robin Hood” tendency). 4) Whether to comment outside one’s expertise: discomfort evaluating multidisciplinary content, budgets, or public engagement plans. Some limited comments to their domain and deferred other aspects to relevant experts; others expressed cynicism about the value or role of public engagement in academic grants but acknowledged its importance to the funder. 5) Whether to prioritize originality despite risks versus err on the side of caution: some reviewers overlooked shortcomings when they trusted known applicants/teams or deferred issues to panel interviews; others emphasized feasibility and value-for-money, particularly for larger budgets. Several perceived the system as dampening “excitement” and favoring safe, feasible proposals over bold, innovative ones. - Idiosyncrasy and trade-offs: Reviewers engaged in deliberations and trade-offs of varying complexity; interpretations of the “right” course were influenced by personal values, preferences, experience, and perceived norms/guidelines. Approaches to the same dilemma were sometimes diametrically opposed, indicating potential sources of variability in outcomes.

The study illuminates the lived, cognitive–moral work of independent reviewers and reframes peer review as a process marked by interpretive flexibility. Grant applications, review forms, and criteria documents operate as boundary objects with stable identities yet divergent meanings across reviewers, panels, and funders. The invisible work—seeking external information, navigating conflicts, and weighing risks—gets sanitized from final ratings and short comments, explaining how equally expert reviewers can reasonably diverge. Applying March’s logic of appropriateness highlights how decisions arise from situation recognition, identity, and (implicit/explicit) rules. For example, the dilemma of seeking information beyond the application is construed either as fairness (avoid bias, judge only the text) or as thoroughness (inform judgement with additional data), leading to different heuristics (do not Google vs disclose conflicts and/or search externally). Such heterogeneity suggests gaps in explicit guidance, leaving reviewers to rely on personal heuristics and values. The findings urge caution against attributing disagreement to bias or instinct alone and invite embracing cognitive diversity as a potential asset, especially in panel deliberations where rules can be negotiated. At the same time, certain practices (e.g., familiarity bias, prestige signals, obligatory comments outside expertise) may systematically shape evaluations and equity, indicating areas for clearer policies and redesigned forms.

Using a think-aloud/CDM approach to capture near real-time cognition, the study reveals five recurring dilemmas that structure independent reviewers’ judgements: whether to accept reviews, whether to use information beyond applications, how to treat institutional prestige, whether to evaluate beyond one’s expertise, and how to balance originality versus feasibility. These dilemmas are navigated through reviewers’ identities, situational recognition, and heuristic rules, producing reasonable yet divergent pathways and outcomes. Improving peer review requires shifting focus from individual flaws to system-level, evidence-based and consensus-driven guidance that leverages expertise and diversity while curbing inappropriate subjectivity. Future research should further make visible the hidden socio-psychological processes of review and examine how guidance, training, and form design influence decisions across funders and disciplines.

- Sample composition: Small, purposive sample (N=16) from medical humanities and social sciences; underrepresentation of women (25%) and overrepresentation of White, 50s age group, and historians may limit generalizability. - Potential biases: Re-reviewing the same application may elicit post hoc rationalizations; social desirability could have constrained disclosures on sensitive topics. - Method choices: While think-aloud enabled candid reflection, similar dilemmas might be elicited through standard interviews; comparative methodological work is needed. - Scope: Findings pertain to a single prestigious funder and specific schemes; replication across disciplines, funders, and review formats (e.g., blinded elements) is warranted.