Grant funding relies heavily on peer review, a system often criticized for inconsistencies, biases, and lack of reliability. While previous research has examined correlations between applicant characteristics and funding decisions, this study focuses on the lived experience of individual reviewers during the initial, independent evaluation stage. The research, part of the TORR project, aims to understand the phenomenological experience of peer review, exploring the 'horizons and background assumptions' that shape reviewers' decisions by observing their review processes in near real-time. This qualitative approach offers a deeper understanding of the decision-making process compared to studies focusing solely on outcomes.

The peer-review process, typically involving independent review by experts, internal review by panels, and panel discussions, is central to research funding. Critics have pointed to various flaws including bias against women, interdisciplinary research, and unknown or disagreeable applicants. Additionally, the reliability of reviews is questioned, particularly from inexperienced reviewers. Studies have also highlighted the inefficiency and burden of the system for applicants, and the questionable ability of peer review to predict research success. The existing literature is methodologically diverse and inconclusive. Lamont's work offers a valuable counterpoint, emphasizing the deeply contextual, psychological, and social influences on peer review, showing it to be affected by tensions between appraisal and self-interest, and between democratic values and expertise. This research acknowledges peer review as a human endeavor shaped by individual values and conventions.

Employing a qualitative approach with a Big Q emphasis on subjectivity, the study used a mixed method combining the think-aloud method and the Critical Decision Method (CDM). Sixteen peer reviewers from the medical humanities and social sciences, recruited from the Wellcome Trust's Awards in Humanities and Social Science, participated. The study combined simultaneous and retrospective verbal protocol approaches. Participants first provided an unstructured account of their review process. Then, they used the think-aloud method to review a grant application they had previously assessed for the Wellcome Trust. Finally, they gave a general appraisal. Interviews lasted approximately 60 minutes and were recorded. Data analysis followed Braun and Clarke's six-phase thematic analysis, combining inductive and deductive approaches. Two researchers independently coded the transcripts, resolving differences through discussion until consensus was reached.

The analysis revealed five key dilemmas faced by peer reviewers: 1. **Accepting an invitation to review:** Reviewers considered the costs and benefits, including expertise match, interest in the topic, familiarity with the applicant, and the project's potential benefit to their own research. Gratitude towards the funder also played a role. Time constraints were mentioned as a factor, though not a reason for rejecting invitations. 2. **Relying exclusively on application information:** Reviewers struggled with using prior knowledge of applicants versus remaining objective. Some felt prior knowledge could influence their evaluation, while others saw it as enabling more informed decisions. Accessing information via Google was a common practice, viewed differently by participants; some considered it part of the process, while others felt it unethical. 3. **Paying attention to institutional prestige:** Reviewers' attitudes were ambivalent; they recognized the importance of prestige but also voiced concerns about potential bias towards elite institutions. Some reviewers expressed support for applicants from less prestigious institutions. Application sections highlighting institutional affiliations or previous funding from the same organization inadvertently prompted a bias towards well-established applicants. 4. **Offering comments on aspects outside their expertise:** Reviewers grappled with multidisciplinary applications, feeling a responsibility to comment on aspects beyond their specialized knowledge. Budgeting and public engagement were particularly challenging areas. The willingness to comment on these areas often depended on how comfortable reviewers felt providing such feedback. 5. **Balancing risk-taking with caution:** Reviewers faced the challenge of evaluating applications with outstanding aspects alongside substandard ones. Some prioritized strengths and overlooked weaknesses, while others focused on feasibility and were less willing to overlook flaws. The decision to take risks was often influenced by the applicant's familiarity and the amount of funding requested.

The study reveals that peer review is a complex, morally charged process shaped by reviewers' individual values and experiences. The concept of interpretive flexibility, borrowed from science and technology studies, helps understand how the grant application, reviews, and funder's criteria are interpreted differently by various stakeholders. March's 'logic of appropriateness' framework is useful to explain dilemmatic decision-making, demonstrating how reviewers' identities, situational recognition, and implicit rules shape their choices. The study highlights the lack of explicit rules, leading reviewers to rely on heuristics, which can lead to inconsistent outcomes. The observed differences in decision-making styles underscore the importance of providing clearer guidance and addressing potential biases within the application process itself.

This study offers novel insights into the complex decision-making processes of peer reviewers, emphasizing the importance of moving beyond focusing on individual reviewer flaws and towards creating a more equitable and transparent system. Future research should focus on developing clearer guidelines and training to aid reviewers in navigating ethical dilemmas and reducing biases. This includes providing guidance on acceptable levels of risk, the appropriateness of external information gathering, and funder expectations regarding feedback. Revising application forms to minimize the unintentional reinforcement of biases is also crucial.

The study's relatively small sample size, while diverse in terms of experience and discipline, might limit the generalizability of findings. The overrepresentation of certain demographics and underrepresentation of others raises concerns about potential biases in the sample. The study also acknowledges the potential for post hoc rationalization during the second review of the application and the possibility that social desirability bias might have influenced responses. Future studies should address these limitations by employing larger, more representative samples and exploring alternative methodologies to minimize bias.