Survey of open science practices and attitudes in the social sciences

The paper investigates how widespread core open science practices—public posting of data/code and pre-registration of hypotheses or analyses—are within major social science disciplines, and how scholars privately view these practices and perceive their fields’ norms. Motivated by ongoing debates and uneven institutionalization of open science, the study aims to move beyond article-level audits to measure lifetime adoption, attitudes, and norm perceptions among active researchers. It addresses gaps in prior work that provided only snapshots of usage or used narrow definitions, thereby missing broader lifetime engagement and scholars’ opinions. By surveying authors from highly cited journals and PhD students at top programs in economics, political science, psychology, and sociology, and by validating self-reports with audits, the study seeks to provide a comprehensive, up-to-date picture of open science adoption, support, and perceived norms across disciplines and career stages.

Prior research has offered partial or potentially unrepresentative views of open science prevalence. Article-level audits (e.g., Hardwicke et al.) found generally low usage when examining transparency practices within randomly sampled papers, but such approaches do not capture researchers’ lifetime adoption nor their attitudes. Some studies have applied narrow definitions (e.g., counting only practices documented within articles or specific repositories), potentially underestimating usage. Broader debates in economics, psychology, and other fields have discussed benefits and drawbacks of practices like pre-registration and data sharing, with concerns about creativity, fit for non-experimental research, and quality variation in pre-analysis plans. There is evidence of institutional and technological changes promoting open science, yet field-based opposition persists. Overall, the literature indicates uncertainty regarding actual prevalence, support, and norms, motivating the present survey-based, lifetime-focused assessment with audit validation.

Design and registration: Two-wave survey of social scientists measuring lifetime usage, attitudes, and perceived norms regarding open science practices. Analyses were pre-registered on OSF (https://osf.io/zn8u2/registrations), and the study received IRB approval from Princeton University (Protocol #11972). Data collection used a custom interface on top of Qualtrics. Analyses were conducted in R (4.0.2). Sample and recruitment: The target population comprised two career stages—published authors and PhD students—across economics, political science, psychology, and sociology. Published authors were randomly drawn from authors with at least one publication during 2014–2016 in 10 of the most-cited journals per discipline, plus each discipline’s Annual Review (total 45 journals). PhD students were sampled from top-20 North American universities per Times Higher Education Social Science ranking (fall 2017). PhD students who were also published authors were sampled only as students. Over 22,000 authors/students formed the sampling frame; 6,231 individuals were invited in wave 1 and again in wave 2 (6,114 successfully contacted). Overall, 3,257 responded in at least one wave (53% response rate). Forty-two percent identified as women. Waves and incentives: Wave 1 ran April–August 2018; Wave 2 ran March–July 2020. Wave 1 randomized incentive levels: published authors received $75 or $100; PhD students received $25 or $40. Higher incentives modestly increased student response rates (+8.2 pp) and had minimal effect for authors (+0.8 pp). Wave 2 offered standard incentives to all. Measures: The survey focused on two core practices: (1) posting data or code online and (2) pre-registering hypotheses or analyses. It also measured posting study instruments (for awareness, attitudes, behavior indices only). Lifetime prevalence was coded as 1 if the respondent reported having ever engaged in a practice. Attitudes used Likert items (1=Not at all in favor to 5=Very much in favor) asking about the importance of each practice for progress in the discipline. Perceived descriptive norms asked respondents to estimate what percentage of researchers in their discipline have ever engaged in each practice. Perceived prescriptive norms asked respondents to allocate percentages across five opinion categories about each practice using a dynamic histogram. Indices: For each practice, awareness (1 item), behavior (3 items), and attitudes (2 items) were averaged; indices across the three practices were averaged to form composite Awareness, Behavior, and Attitudes indices. A broad Overall Personal Support index averaged the three sub-indices. Details appear in Supplementary Tables 7–8. Validation audits: To assess reliability and selection, the team conducted (a) manual hand audits and (b) automated scraping audits. Manual audits validated pre-registration and data/code posting for random samples of published authors (economics and some psychology) among respondents and non-respondents by searching a wide set of websites and authors’ recent publications (audit periods: March 2019 for wave 1; June–November 2020 for wave 2 updates). Automated audits (August 2020) queried major repositories/registries (AEA RCT Registry, EGAP Registry, Dataverse, AsPredicted, OSF) for items posted before March 1, 2020. Comparisons across self-reports, manual, and automated audits assessed accuracy and coverage. Statistical approach and power: Power analyses indicated N≈3,200 sufficient to detect small effects (d≈0.2; d≈0.14 when pooling groups) at 80% power, alpha=0.05, two-tailed. Analyses employed two-tailed t-tests assuming IID data with potentially heteroskedastic errors. When individuals completed both waves, the primary analysis used their most recent response (robustness using averages across waves provided in Supplementary). Selection analyses used propensity score adjustments and controls for observable covariates (e.g., subfield, publication record, job type, institution) to test representativeness. Ethics and openness: Informed consent obtained; data and code (de-identified) posted on OSF (https://osf.io/zn8u2/).

- Adoption and support are widespread. As of 2020, nearly 90% of scholars had used at least one open science practice at least once; lifetime prevalence rose from 49% (2010) to 87% (2020) among those with PhDs by 2009. - Support levels: Overall support for posting data or code was about 88% (very much or moderately in favor); support for pre-registration was lower but still majority at about 58%. - Usage levels: Self-reported lifetime posting of data/code was 52%, while lifetime pre-registration was 25% overall, with smaller gaps in psychology. - Discipline and method differences: Economists and political scientists showed near-universal posting of data/code; pre-registration was highest in psychology. Experimental researchers exhibited the highest support and usage, followed by quantitative non-experimental; qualitative/theoretical researchers showed the lowest but still substantial support and usage. - Attitudes-behavior link: Correlations between support and lifetime usage were r=0.32 (p<0.001; 95% CI [0.29, 0.35]) for posting data/code and r=0.32 (p<0.001; 95% CI [0.28, 0.35]) for pre-registration. Among published authors: r=0.44 (p<0.001; 95% CI [0.40, 0.49]) for posting data/code and r=0.36 (p<0.001; 95% CI [0.32, 0.41]) for pre-registration. - Norm misperceptions: Respondents consistently underestimated actual lifetime rates of both behaviors and levels of support in their disciplines, especially for posting data/code and for pre-registration in psychology and political science. - Validation: Manual audit broadly confirmed self-reports (overall verification around 80%; ≈91% agreement for economists; ≈75–80% for psychologists). Automated scraping missed about 70% of data/code postings relative to manual/survey, indicating repository-based scraping underestimates behavior. - Persistence: Those who adopted a practice were more likely to use it in their most recent project, indicating persistent usage. - Trend differences by field: Economics and political science showed rapid increases in posting data/code across the period; psychology accelerated most in recent years; sociology remained lowest but increased steadily.

The findings indicate that open science practices have become common and are widely supported across social science disciplines, with stronger endorsement and uptake for posting data/code than for pre-registration. Adoption is especially prevalent among experimental researchers and in fields with stronger experimental traditions (e.g., psychology, parts of economics), consistent with the greater ease of fitting pre-registration to experimental designs. The positive correlations between attitudes and behaviors suggest that stated support is meaningful and translates into practice, particularly among published authors who have more opportunities to implement these practices. Despite this, scholars systematically underestimate both the actual prevalence of open science behaviors and the level of support within their fields. This lag in perceived norms relative to actual behavior may slow diffusion, as perceived norms can influence adoption decisions. Making open science usage more visible—e.g., via journal badges or clearer documentation—could help align perceptions with reality and further shift norms. Audit results bolster confidence in the survey data: self-reports align closely with observed behavior in manual audits, while automated repository scraping substantially undercounts data/code posting due to the variety of platforms used. Selection analyses suggest limited bias after controlling for observables, supporting the representativeness of estimates for the sampled populations of top-publishing authors and top-program PhD students.

This study provides a comprehensive, validated picture of open science in four major social science disciplines, showing substantial growth in lifetime adoption (49% in 2010 to 87% in 2020) and broad support, especially for posting data/code. Pre-registration, while less universally supported than data/code sharing, still enjoys majority support and is most prevalent in psychology and among experimental researchers. Scholars underestimate both peers’ usage and support, indicating that perceived norms lag behind actual practice. Future work should identify which open science practices most effectively improve credibility and research quality relative to their costs, and examine how pre-registration translates to non-experimental contexts. Continued efforts to increase visibility and standardization (e.g., clearer repositories, journal badges, registered reports) may help correct norm misperceptions and promote beneficial adoption across methods and disciplines.

- Measurement scope: The survey combined data and code sharing into single items, potentially masking differences in attitudes/behaviors between the two. Pre-registration attitudes were not differentiated by type or level of detail (e.g., full pre-analysis plans vs. shorter registrations). Other practices (e.g., pre-publication verification, registered reports) were not measured. - Construct mismatch: Comparisons of own behavior (lifetime ever-use) with perceived norms (estimates of field behavior) are not perfectly aligned, which may bias differences. - Sample generalizability: The sample focuses on authors from highly cited journals and students at top programs; results may not generalize to entire disciplines. Response rate was 53%, leaving potential for nonresponse bias despite adjustments and audit-based assessments. - Audit coverage: Manual audits validated only economics and part of psychology due to resource constraints; automated audits undercount data/code postings (missed ~70%), limiting their standalone accuracy. - Selection concerns: Some evidence of selection on behavior (particularly pre-registration) among economists, largely attenuated after controlling for subfield and other observables; residual unobserved selection cannot be ruled out. - Recall and self-report: Lifetime adoption and year-of-first-use rely on self-reports and recall, which may introduce noise.