Survey of open science practices and attitudes in the social sciences

Open science practices, such as data and code sharing and pre-registration of analyses, are increasingly advocated in the applied sciences but their actual prevalence and long-term adoption remain unclear. Existing research provides an incomplete picture, often focusing on point prevalence in published articles rather than lifetime adoption or researchers' opinions. This study aims to address this gap by surveying social scientists (published authors in top journals and PhD students in top departments) from economics, political science, psychology, and sociology. The study employs improved methodology, including high remuneration to boost response rates, broad definitions of open science practices, and verification of self-reported behavior through web scraping and manual audits. This comprehensive approach offers a more accurate and up-to-date understanding of open science adoption, attitudes, and perceived norms across disciplines and career stages.

Previous research on open science practices in the social sciences has yielded partial or unrepresentative results. Some studies used audit approaches to assess point prevalence, focusing solely on practices mentioned in published articles, neglecting lifetime adoption and researchers' views. Others employed narrow definitions of open science, limiting the scope of analysis. This study addresses these limitations by employing a broader approach, encompassing lifetime use and diverse opinions, and utilizing a larger, more representative sample of social scientists.

A survey was administered twice (2018 and 2020) to a sample of social scientists, including published authors from top-10 journals and PhD students from top-20 North American departments across four disciplines (economics, political science, psychology, sociology). High incentives were offered to maximize response rates. The survey measured self-reported lifetime use of and attitudes toward open science practices (data/code posting and pre-registration), and perceptions of field norms. Self-reported behavior was verified through web scraping and manual audits. Power calculations ensured sufficient sample size (N=3200) to detect meaningful differences. The primary analysis used participants' most recent survey responses; supplementary materials included results using average responses across both waves. Indices were created aggregating awareness, attitudes, and behavior across three open science practices (posting data/code, posting study instruments, pre-registration) to assess open science engagement across various factors. To validate survey responses, manual and automated audits were conducted on a subset of economists and psychologists, comparing self-reported behavior with actual online evidence. The audits assessed the accuracy of self-reported data, the effectiveness of web-based scraping in capturing open science trends, and the degree of selection bias in the sample. Propensity score matching was employed to address potential non-response bias. Analyses included correlations between stated support and usage of open science practices, and comparisons between actual and perceived norms of behavior and support across disciplines.

Lifetime usage of open science practices was high overall (approximately 87% by 2020, up from 49% in 2010). Data/code posting was more common than pre-registration. While there was variation across disciplines, usage rates exceeded 45% in all fields. Published authors exhibited higher rates of open science practice than PhD students, likely due to greater opportunities. Support for open science practices was high, exceeding 85% for data/code posting and 50% for pre-registration. There was a strong positive correlation between stated support and actual lifetime usage of open science practices, especially among published authors. However, social scientists underestimated the actual prevalence of both open science practices and favorable attitudes towards them, indicating a lag in perceived norms. This underestimation was particularly strong for data/code posting. The year of adoption of open science practices varied across disciplines, with economics and political science showing earlier and faster adoption of data/code posting than psychology and sociology. Experimental researchers exhibited the highest rates of open science practice adoption, while qualitative researchers had the lowest. Hand audits confirmed high accuracy (91% for economists, 75-80% for psychologists) of self-reported open science practices. Automated audits, however, underestimated open science behavior due to the diversity of data and code repositories. Analyses adjusting for selection bias revealed limited evidence for systematic differences in behavior between survey respondents and non-respondents.

The findings reveal a significant increase in the adoption of open science practices in the social sciences over the past decade. High levels of support for these practices exist across disciplines, even among those with less institutionalization of such methods (e.g., sociology). The strong positive correlation between stated support and actual behavior suggests that attitudes towards open science are meaningful predictors of practice. The underestimation of actual prevalence of open science practices, however, points to a gap between actual behavior and perceived norms. This suggests that raising awareness of the current levels of open science adoption could help shift norms towards greater acceptance and adoption. The differences in adoption rates across disciplines and methodological approaches highlight the need for targeted strategies to promote open science in specific contexts.

This study demonstrates widespread support for and usage of open science practices in the social sciences. The considerable increase in adoption over the last decade and the strong correlation between attitudes and behavior are encouraging. However, the underestimation of open science prevalence underscores the need for interventions to improve the visibility of these practices and align perceived norms with actual behavior. Future research should investigate the impact of specific open science practices on research quality and explore strategies to foster broader adoption, particularly in fields with lower current levels of engagement.

The study's sample focused on highly-cited journals and top-ranked institutions, potentially limiting the generalizability of findings to the broader social science community. Despite high incentives and two waves of data collection, the response rate was 53%, raising the possibility of non-response bias, though analyses were performed to mitigate this. The survey's measures of data and code sharing were combined, not separated; and the study did not address all open science practices. Finally, the discrepancy between self-reported practices and perceived norms might stem from the survey sample being more favorable to open science than non-respondents.