Demystifying the process of scholarly peer-review: an autoethnographic investigation of feedback literacy of two award-winning peer reviewers

Peer review is the central quality assurance mechanism in scholarly publishing, intended both to assess research quality and to improve manuscripts. Despite its importance, evidence on its effectiveness is mixed and the process can negatively affect researchers—particularly ECRs—when feedback is perceived as unfair or unprofessional. Formal training for peer reviewers is rare; most learn informally via journal guidelines or mentorship, leaving gaps in how reviewers develop effective feedback practices. This study responds to these issues by investigating the development and enactment of feedback literacy among peer reviewers through the authors’ own experiences. Research questions: (1) What are the exhibited features of peer reviewer feedback literacy? (2) What are the forces at work that affect the development of feedback literacy? The study positions feedback literacy within broader assessment literacy and examines it through an ecological systems lens to capture contextual influences.

The paper situates feedback literacy as a subset of assessment literacy, drawing on sociocultural perspectives from new literacy studies. Foundational work (Sutton, 2012; Carless & Boud, 2018) conceptualizes student feedback literacy in terms of abilities to understand the formative role of feedback, make evaluative judgments against standards, manage emotions, and take action. Subsequent research has extended feedback literacy to teachers and students in higher education, often focusing on classrooms and workplace learning, with limited attention to academics’ feedback literacy in scholarly peer review. The authors extend feedback literacy to peer reviewers, conceptualizing it as both abilities and orientations (beliefs, tendencies, habits) shaped by experiences, cultures, and politics. They synthesize prior models and embed them within an ecological systems framework (micro, meso, exo, macro, chrono) to account for the multilayered contextual forces influencing reviewers’ feedback practices. The paper details the five manifestations of peer reviewer feedback literacy—engineering feedback uptake, navigating responsibilities, maintaining relationships, reflecting on feedback experiences, and understanding standards—while arguing that feedback literacy is malleable, situated, and emergent rather than a binary state.

Design: Collaborative autoethnography (CAE) was used to systematically analyze the authors’ personal experiences as peer reviewers to illuminate sociocultural dimensions of feedback practices. CAE emphasizes dialogic sense-making, reflexivity, and credibility/validity through transparent, systematic narrative reconstruction.

Participants: The two authors (both ECRs, <5 years post-PhD; awardees of the 2019 Reviewer of the Year Award from Higher Education Research & Development and Taylor & Francis). Both are active authors and reviewers in education-related fields and hold academic positions (UK; Japan).

Data sources: (1) Individually written reflective essays produced via recalling critical moments, incorporating written reflections and excerpts from prior review feedback; (2) Two online discussion meetings (~90 minutes each), audio-recorded and transcribed, where authors annotated each other’s essays, posed questions, and dialogically analyzed experiences.

Analysis: Guided by the feedback literacy ecological framework. The team employed Miles & Huberman’s (1994) approach: data reduction, data display (tabular visualization), conclusion drawing, and verification. Coding and writing were done collaboratively in Google Docs. The analysis generated descriptive codes and overarching themes organized by manifestations of feedback literacy and ecological systems.

Ethics: Conducted under the School Research Ethics Committee, Queen’s University Belfast (Ref: 005_2021). Privacy safeguards were observed; pseudonyms or anonymization were used as appropriate. As CAE, the researchers were also the participants.

Contextual reviewer experience at data collection (Table 1): Author 1: first review Aug 2017; 44 reviews for 38 manuscripts across 14 journals (fields: educational assessment, educational technology, higher education, education, language education). Author 2: first review May 2017; 23 reviews for 15 manuscripts across 8 journals (fields: education, higher education, bibliometrics).

Manifestations of peer reviewer feedback literacy:

• Engineering feedback uptake: Reviews are clear, explicit, specific, and actionable to help authors improve, including in rejection cases. Emphasis on promoting a growth mindset and separating critique of the work from the person. Longer, more detailed feedback often accompanies rejections to support future improvement.
• Navigating responsibilities: Despite limited formal recognition and time pressure, reviewing is seen as service and identity-building in the academic community. Review processes are personalized: A1 conducts holistic read-throughs with annotated PDFs and composes structured reports mirroring paper sections; A2 builds reports iteratively in bullet points during short work windows. Both provide examples and questions rather than exhaustive rewrites, and use confidential editor comments for borderline cases to triage effort and uphold standards.
• Maintaining relationships: Even under double-blind conditions, reviewers frame feedback as a respectful, empathetic professional dialogue while maintaining directness and rigor. They aim not to ‘sugar coat’ but to avoid personalizing critique. Strategies include summarizing the paper to demonstrate engagement and offering to review revisions to signal commitment.
• Reflecting on feedback experience: Experiences as authors (both constructive and unprofessional reviews) shape commitment to tone and substance. Informal learning occurs by comparing one’s reviews with others’ and via editorial feedback on reviews. Editorial roles (A1) provide insight into reviewer practices and effective use of confidential comments.
• Understanding standards: Journal guidelines help, but familiarity through authorship and repeated engagement with a journal most strongly builds confidence and calibration to journal-specific expectations.

Ecological systems influences:

• Microsystem: Universities’ research agendas emphasize publications over reviewing; limited institutional training for peer review leads to self-directed learning. Journal communities and editorial experiences provide situated learning. Twitter serves as a reflective community for sharing norms and cautionary tales (e.g., ‘Reviewer 2’). Prior teaching experience (A2) shapes attention to clarity, developmental guidance, and emotional impact.
• Mesosystem: Interactions across microsystems (institutional gaps, journal/editorial engagement, online communities) foster strategic feedback practices (focusing on argumentation over copyediting) and self-learning through exposure to others’ reviews.
• Exosystem: Exposure to language advising approaches (learner-centered, partnership mindset) informs question-asking, resource-pointing, and offering multiple suggestions. Skepticism toward formal, asynchronous reviewer training courses; practical reviewing experience considered more formative for feedback literacy.
• Macrosystem: Sociocultural backgrounds shape formality, respect norms, and identity. A1’s Hong Kong upbringing and teacher identity promote formal, structured, respectful tone; evolving from third- to second-person address for interactivity. A2’s Anglophone status in Japan cultivates empathy for non-native English authors and avoidance of English-only judgments; first-generation scholar identity influences thoroughness and confidence.
• Chronosystem (embedded across systems): Over time, both authors became more strategic, efficient, and partnership-oriented in feedback, informed by cumulative experiences as authors, reviewers, and editors.

Illustrative data points: At data collection, A1 had completed 44 reviews for 38 manuscripts across 14 journals; A2 had completed 23 reviews for 15 manuscripts across 8 journals. Both began reviewing in 2017.

The findings address RQ1 by articulating five core manifestations of peer reviewer feedback literacy—engineering uptake, navigating responsibilities, maintaining relationships, reflecting on feedback, and understanding standards—and demonstrating how these are enacted in practice through tone, structure, and actionable guidance. For RQ2, the ecological framing reveals that feedback literacy develops through layered contextual influences, with limited institutional support offset by journal communities, editorial exposure, and online professional networks. Over time, reviewers’ practices become more strategic and developmental, balancing empathy with rigor and aligning with journal standards via familiarity and authorship experience. The study argues that feedback literacy is not a static trait but an emergent, situated orientation shaped by experiences, roles, and sociocultural contexts. Practically, hands-on, situated opportunities (e.g., co-reviewing, editorial mentoring) and community engagement appear more impactful than purely knowledge-based training modules. Framing review as both quality assurance and researcher development underscores the broader academic and human dimensions of peer review.

This study contributes an insider, collaborative autoethnographic account of how two award-winning ECR reviewers develop and enact feedback literacy, advancing a framework that integrates five feedback literacy manifestations with an ecological systems perspective. It highlights that reviewers’ effective feedback practices are highly personalized, contextually embedded, and largely developed through informal, experiential pathways spanning authorship, reviewing, editorial work, and professional communities. Implications suggest prioritizing hands-on, situated training (e.g., co-reviewing, editorial shadowing), transparent sharing of exemplar reviews, and cultivating online/offline communities of practice to foster feedback literacy. Future research should: (1) extend to larger, more diverse samples to map broader patterns; (2) analyze actual reviewer reports to triangulate narratives with textual evidence; (3) evaluate the efficacy of formal reviewer training on feedback literacy and practices; and (4) include authors’ perspectives on interpreting and acting on reviews to better understand the communication dynamics of peer review.

The study is intentionally idiosyncratic, focusing on two ECR reviewers within the education discipline, which limits generalizability. Data rely on self-authored narratives and discussions without including the actual reviewer reports, constraining triangulation. Institutional and cultural contexts (UK/Japan) may not transfer to other disciplines or systems. As a reflective CAE, recall and interpretive biases are possible despite efforts to ensure credibility and validity.