Academics face increasing pressure to demonstrate the societal impact of their research, driven by funders and governments seeking to justify public funding. The UK's Research Excellence Framework (REF) in 2014 and 2021, and similar systems internationally, assess research impact through case studies scored from 1 to 4*. The REF2014 evaluation has been criticized for its association with a 'market logic', concerns about distorting research priorities, and the subjectivity of the peer-review process. While some have raised concerns about the negative consequences of the impact agenda, others have defended its benefits, highlighting enhanced stakeholder engagement and the long-term societal benefits of incentivizing research impact. This paper empirically assesses the content and language of high-versus low-scoring REF2014 impact case studies to understand how these characteristics relate to the evaluation of impact. The research questions address how high-scoring and low-scoring case studies articulate and evidence impacts, and whether they exhibit differences in linguistic features and styles.

The literature review examines criticisms and defenses of the research impact agenda. Critics point to the ‘market logic’ driving the impact agenda, its potential to destabilize professional identities, and its negative unintended consequences. Concerns have been raised about the potential distortion of research priorities towards less discovery-led research. In contrast, proponents highlight the value of a focus on changing the culture outside academia, enhanced stakeholder engagement, and the long-term societal benefits of incentivizing research impact. The literature also discusses the implicit criteria and subjectivity in evaluating impact, with emphasis on the role of narrative, style, and structure in shaping the evaluation outcomes. Prior research indicates that the subjectivity in the peer-review process led to biased interpretations and potential ‘groupthink’ in evaluating impact case studies.

The study employed a mixed-methods approach combining quantitative linguistic and qualitative thematic analysis. The quantitative linguistic analysis used a sample of 124 high-scoring and 93 low-scoring impact case studies. The analysis focused on sections containing predominantly text (Summary of the impact, Underpinning research, and Details of the impact). Lexical bundles (frequent word combinations) were extracted and compared using Log Likelihood and Log Ratio to identify statistically significant differences. Readability was assessed using the Flesch Reading Ease and Coh-Metrix tools. A separate qualitative thematic analysis was conducted on a sub-sample of 85 high-scoring and 90 low-scoring case studies to analyze content and themes related to the articulation and evidence of impacts. The thematic analysis used a coding framework based on the REF2014 impact case study template, focusing on themes such as impact type, pathway to impact, evidence of significance and reach, and quality of corroborating evidence. Intercoder reliability was assessed at over 90%.

The study revealed several key findings. High-scoring case studies provided specific, high-magnitude, and well-evidenced articulations of significance and reach, focusing on the content of the impact itself rather than solely on the pathway to impact. They showcased examples of global reach and used context to create robust arguments for impressive reach, even in cases of geographically limited impact. Quantitative linguistic analysis confirmed that high-scoring case studies included more phrases specifying reach and significance. High-scoring case studies used distinct features to establish links between research (cause) and impact (effect), employing attributional phrases to explicitly establish causal links. Quantitative analysis showed that high-scoring case studies were significantly more likely to include attributional phrases establishing attribution to impact, whereas low-scoring case studies were more likely to attribute to research or pathways. Low-scoring case studies used more ambiguous or uncertain phrases, suggesting a lack of clarity in demonstrating impact. Qualitative analysis found that high-scoring case studies more clearly linked underpinning research to claimed impacts, used higher-quality corroborating evidence, and were more likely to provide evidence of policy implementation as well as policy design. High-scoring case studies achieved higher scores on the Flesch Reading Ease measure, reflecting clearer writing style. Coh-Metrix analysis revealed significant differences in 'deep cohesion' and 'connectivity,' with high-scoring case studies showing more explicit causal and logical connections. High-scoring case studies were more likely to clearly identify individual impacts using subheadings and paragraph headings. Qualitative analysis identified differences in the descriptions of underpinning research between high and low scoring case studies, with low scoring case studies focusing more on research processes and outputs rather than findings.

The findings address the research questions by demonstrating clear differences in the content, language, and style between high- and low-scoring impact case studies. High-scoring case studies effectively communicated impact by providing specific evidence, clear causal links, and strong narrative coherence, while low-scoring case studies often lacked clarity, specificity, and strong connections between research and impact. The study highlights the importance of focusing on the 'content' (the impact itself) rather than solely on the 'process' (the pathway to impact). The significant differences in readability scores suggest that clear and concise writing contributes to the successful communication of research impact. The study also suggests that while explicit criteria in the REF guidelines played a role, implicit rules related to writing style also influenced scores. The results indicate that high-scoring case studies adapted to a new genre of writing that is more direct, less uncertain, and easier to read than typical academic writing.

This study provides empirical evidence of the linguistic and stylistic factors associated with high-scoring impact case studies in REF2014. High-scoring case studies demonstrated clear articulation of significant impacts, strong causal links between research and impact, high-quality corroborating evidence, and clear and concise writing. The findings offer valuable insights for authors, institutions, and those designing impact assessment frameworks. Future research could explore the causal mechanisms linking linguistic features to evaluation outcomes and investigate the generalizability of these findings across different disciplines and evaluation systems.

This retrospective study cannot definitively establish a causal relationship between linguistic features and evaluation panel judgments. The sample, while the largest to date, represents a small percentage of all REF2014 case studies and may not be fully representative of all disciplines. The study's findings are specific to REF2014, and changes in expectations and evaluation practices in REF2021 and other impact assessment systems should be considered. The limited number of case studies focused on public engagement and pedagogy prevents definitive conclusions about these impact types.