Measuring Science: Performance Metrics and the Allocation of Talent

The efficient allocation of talent is crucial for scientific advancement and economic growth. Both private firms and universities increasingly rely on performance metrics to identify and attract talented individuals. In academia, these metrics, particularly citations and publications, influence hiring, promotions, funding, and prestige. While the use of such metrics is widespread, concerns exist regarding overreliance and potential biases. This paper addresses this gap by providing the first systematic evidence on how performance metrics specifically affect the organization of science, focusing on the impact of citation metrics on talent allocation and scientific careers. The study leverages the introduction of the Science Citation Index (SCI) in 1963, which created quasi-random variation in the visibility of scientists' citations, allowing for a causal analysis of their effects.

The paper contributes to the economics of science literature, which highlights the increasing knowledge processing demands on scientists, the importance of superstar scientists, peer effects on productivity, and the role of editors. Recent research also emphasizes inefficiencies like the Matthew Effect, gatekeepers, and discrimination. While the existing literature utilizes publication and citation data, there's a lack of evidence on how the *observability* of citation metrics affects scientific careers. Previous studies have shown that citation metrics predict career outcomes, but this paper uniquely causally examines the impact of *observability*, disentangling differences in underlying academic quality from differences in observed quality based on citation metrics. It adds to the performance metrics in labor markets literature, where estimating the impact of such metrics is challenging due to the lack of valid counterfactuals. This study overcomes this by observing the exact information set available to decision-makers and what was missing.

The researchers utilize a unique dataset combining historical faculty rosters from the World of Academia Database with publication and citation data from Clarivate Web of Science. They meticulously reconstruct the SCI's coverage of citations, distinguishing between citations visible to contemporaries in the 1960s and those invisible at the time but observable today. This allows them to compare the predictive power of visible versus invisible citations on career outcomes. The main analysis employs regression models where the dependent variable is department rank in 1969, and independent variables include visible and invisible citation counts, controlling for publication records, subject fixed effects, and department-level clustering of standard errors. The study addresses potential biases by conducting robustness checks that focus on over-time variation in citation visibility (holding journal quality constant) and across-journal variation (holding citation timing constant). A placebo test uses 'pseudo-visible' citations (from journals covered in the first SCI but in years not covered) to ensure that the differential impact of visible citations arises only when the SCI was in operation. Furthermore, the study examines heterogeneity by analyzing the impact of citations across citation percentiles, for 'hidden stars' (highly cited scientists in lower-ranked departments in 1956), and for various minority groups (women, Hispanics, Asians, Jews). Finally, a separate regression analyzes the impact of citation metrics on promotion to full professor between 1956 and 1969.

The study's key findings demonstrate that the availability of citation metrics significantly impacted the organization of science. Visible citations are substantially more predictive of department rank in 1969 than invisible citations (four times larger effect in the fully controlled model), even after controlling for detailed publication records. This finding holds across various robustness checks addressing potential biases related to journal quality and citation timing, and is confirmed by a placebo test. The increased assortative matching is driven by two mechanisms: scientists with low visible citation counts leaving academia, and highly cited scientists moving to higher-ranked departments. Heterogeneity analysis shows that the benefits of citation metrics were disproportionately large for scientists in the top percentiles of citation distribution, especially those initially at lower-ranked departments ('hidden stars'). However, this benefit did not extend to minorities; no significant differences were found in the impact of citation metrics for women, Hispanics, Asians, or Jewish academics. Finally, visible citation ranks had a significant positive effect on promotion to full professor, while invisible citations did not. These results are robust to various robustness checks and placebo tests.

The findings highlight the substantial causal impact of measuring and revealing citation metrics on talent allocation and career progression in academia. The increased assortative matching suggests a more efficient allocation of talent based on observable performance, but the lack of benefit for minorities reveals potential persistence of discrimination despite the introduction of seemingly objective metrics. The disproportionate benefit for 'hidden stars' indicates that citation metrics can help to identify and reward talent that might otherwise be overlooked in the pre-SCI era. The impact on promotions reinforces the importance of citations in resource allocation within the academic system. These findings have important implications for ongoing debates surrounding the use and potential biases of performance metrics in academia.

This study provides compelling evidence of the significant and immediate impact of systematically measuring scientific citations on talent allocation and career outcomes. The introduction of the SCI led to increased assortative matching, disproportionately benefiting top-cited scientists, particularly those in lower-ranked institutions. However, the lack of similar benefits for minority groups indicates that citation metrics alone did not eliminate existing biases. Future research could explore the long-term consequences of citation metrics on scientific innovation and explore the specific mechanisms of discrimination that remain.

The study's analysis is limited to a specific historical period (1956-1969) and focuses primarily on scientists in the natural and biomedical sciences. The reliance on name-based identification of minority groups may introduce some error. The findings might not generalize to other fields or time periods. Further, the causal interpretation hinges on the assumption that visible and invisible citations would have had equal predictive power in the absence of the SCI.