Introduction
Peer review is crucial for ensuring research quality and improving manuscript quality. However, there's limited formal training for peer reviewers, particularly early-career researchers (ECRs), leading to inconsistent practices and potentially negative experiences for authors. Negative experiences, especially unfair feedback, can significantly impact researchers' professional identity and well-being. This study aims to understand how experienced peer reviewers develop their feedback literacy skills and navigate the complexities of the peer review process by examining the experiences of two award-winning reviewers. The research questions are: 1. What are the exhibited features of peer reviewer feedback literacy? 2. What are the forces at work that affect the development of feedback literacy?
Literature Review
The study grounds itself in the concept of feedback literacy, drawing from new literacy studies and assessment literacy research. Feedback literacy is viewed as a multi-dimensional construct encompassing epistemological (knowledge of feedback), ontological (capability to understand feedback), and practical (engagement with feedback) dimensions. Existing models emphasize abilities such as understanding feedback's formative role, making evaluative judgments, managing emotions, and taking action. The study extends this concept to scholarly peer review, considering feedback literacy not only as abilities but also as orientations (beliefs and habits). It integrates ecological systems theory (Bronfenbrenner, 1986) to examine contextual influences on peer reviewers' feedback literacy, considering microsystem (direct interactions), mesosystem (interactions between microsystems), exosystem (indirect influences), macrosystem (broader societal influences), and chronosystem (changes over time).
Methodology
This study employs a collaborative autoethnography (CAE) approach, combining the authors' personal experiences with ethnographic analysis of their sociocultural context. Data collection involved separate reflective essays and two online discussion meetings. The reflective essays focused on significant moments impacting the authors' peer-review trajectories. The discussions provided opportunities for collaborative reflection, identifying similarities and differences in experiences. Data were coded using a four-stage process (data reduction, display, conclusion drawing, and verification) based on Miles and Huberman (1994), guided by the feedback literacy framework. The authors, both recipients of a Reviewer of the Year Award, have considerable experience in peer review, detailed in Table 1. Ethical considerations, including honesty and privacy, were carefully addressed.
Key Findings
The analysis revealed five key manifestations of feedback literacy:
1. **Engineering feedback uptake:** Reviewers prioritized clear, explicit, specific, and actionable feedback to promote author learning and manuscript improvement, emphasizing growth mindset. Longer reviews were given for rejections to offer comprehensive feedback.
2. **Navigating responsibilities:** Reviewers acknowledged time constraints and the lack of formal recognition for peer review. They developed personalized strategies for efficient review, such as reading the entire manuscript initially or reviewing in smaller increments. They focused comments on key areas and delegated tasks appropriately.
3. **Maintaining relationships:** While anonymity is maintained, reviewers strived to maintain a professional and supportive relationship with authors, offering feedback with respect and empathy but without 'sugar coating'. Directness was focused on the work, not the person. Explicit commitment to the review process, including a willingness to review revisions, fostered a supportive relationship.
4. **Reflecting on feedback experience:** Reviewers drew upon their experiences as authors (both positive and negative) to inform their reviewing practices. They learned from constructive criticism and observed diverse review styles, comparing their feedback to those of other reviewers. This process enhanced their understanding of good and bad practice in delivering constructive feedback.
5. **Understanding standards:** Reviewers relied on journal guidelines and their own experience of publishing with different journals to develop an understanding of journal-specific standards. Familiarity with a journal strongly influenced their confidence and efficiency in reviewing.
The study also identified the influence of five ecological systems:
* **Microsystem:** University environment, journal communities, and Twitter shaped feedback practices through expectations, learning opportunities, and community feedback.
* **Mesosystem:** The interaction between microsystems highlighted the importance of self-learning and strategic feedback approaches.
* **Exosystem:** Influences from language advising techniques and online peer-review training courses, although not directly engaged with, contributed to the development of the reviewers' feedback literacy.
* **Macrosystem:** The diverse sociocultural experiences of the reviewers shaped their approach to feedback.
* **Chronosystem:** The authors' feedback practices evolved over time due to accumulated influence from various ecosystems.
Discussion
The study provides an in-depth understanding of how two highly successful peer reviewers developed their feedback literacy. The findings highlight the importance of personalized, contextualized approaches to peer-review training, emphasizing hands-on experience and opportunities for reflection and collaboration. The lack of formal support within universities underscores the need for alternative learning pathways. The various layers of ecological systems revealed the complexity of factors affecting feedback practices. This study contributes to the literature by providing an insider perspective of the peer review process, highlighting individual experiences in relation to the multifaceted feedback literacy framework.
Conclusion
This study offers a valuable insider perspective on the development of feedback literacy in scholarly peer review. The personalized and contextualized nature of this development emphasizes the need for flexible, experiential training programs. Future research could explore larger cohorts and incorporate reviewer reports to identify broader trends. The integration of author perspectives and further investigation into the role of peer-review training courses would also enrich our understanding.
Limitations
The study's findings are limited by its small sample size (two reviewers) and focus on educational research. The idiosyncratic nature of the experiences limits generalizability. The study did not include the reviewer reports themselves, which could provide additional insights. Furthermore, the focus is on the reviewers' experience, while authors' perspectives are not included, representing a gap in understanding the full peer review communication process.
Related Publications
Explore these studies to deepen your understanding of the subject.