What can mathematical modelling contribute to a sociology of quantification?

The sociology of quantification, a growing field encompassing various disciplines, examines the creation, usage, and impact of numbers across diverse aspects of life. Existing research extensively covers the production of numbers in data science, algorithms, quantified self-tracking, and indicators. However, it has overlooked crucial areas like the science reproducibility crisis and the role of statistics, which has been criticized for enabling misuse or abuse of statistical tests. The 'Statistics Wars,' highlighting these issues, have largely excluded mathematical modeling, partly due to its multifaceted nature and the lack of universally accepted norms for model adequacy and quality control. Defining a mathematical model is inherently complex; descriptions range from simple system explanations to sophisticated representations of phenomena acting as mediators across various tasks. Models act as tools for explaining things and calculating possibilities, as crafting instruments and representational tools, and as metaphors facilitating dialogue among stakeholders addressing shared problems. The widespread use of models across various domains from population dynamics to financial pricing underscores their versatility. This paper investigates how two mathematical modeling frameworks—sensitivity analysis (SA) and sensitivity auditing (SAUD)—can enhance the transparency, adequacy, and fairness of numbers in quantitative disciplines, contributing to the sociology of quantification.

The study of quantification draws from diverse fields including sociology, history, political science, law, and economics. Two prominent schools of thought are the Foucauldian studies of quantification and the Economics of Conventions, which have spurred movements like 'statactivism,' advocating for alternative quantifications. Researchers across various disciplines, including data scientists, jurists, and economists, have explored threats posed by quantification, encompassing metric misuse, legal system disruptions, and surveillance capitalism. Concerns about the seduction of numbers, their performative nature, and their increased presence in all aspects of life are leading to resistance and public discussions. This has generated interest in the concept of an 'ethics of quantification' overseen by societal actors.

This paper examines the potential of sensitivity analysis (SA) and sensitivity auditing (SAUD) from mathematical modeling to improve other quantification methods. SA assesses the influence of model inputs on outputs, identifying factors contributing most to output uncertainty. SAUD expands this to the entire modeling process, investigating potential biases, interests, and overlooked factors. Both approaches aim to address the opacity of algorithms and apportion uncertainty and ambiguity to underlying assumptions. Real-world models often use uncertain inputs rather than precise numbers, thus SA and SAUD assess the quality of numbers on technical and normative levels, aligning with Amartya Sen’s dual requirement of quantification. Both methodologies are rooted in post-normal science (PNS), an approach suitable for situations with uncertainties, conflicting values, and urgent decisions. The authors propose applying SA and SAUD’s principles to develop an epistemology of quantification, suggesting a 'hermeneutics of quantification' approach to decipher the complexities of model interpretation. The paper then defines uncertainty quantification, SA, and SAUD, exploring their extension to various quantification instances using responsible modeling guidelines. It illustrates relevant aspects of modeling, such as the candor of SA and the modeling of the modeling process, examining their implications for sociology of quantification studies. Finally, it addresses the policy implications of the proposed approach.

The paper's central argument is that sensitivity analysis (SA) and sensitivity auditing (SAUD), frameworks from mathematical modeling, can significantly improve the transparency, adequacy, and fairness of numbers used in various quantitative disciplines. The authors suggest that SA can address the technical aspects of quantification by identifying and quantifying uncertainties and their sources within models. This aligns with the call for a more transparent and accountable use of models in policy-relevant settings. SAUD, on the other hand, tackles the normative aspects of quantification by examining the underlying assumptions, biases, and value judgments embedded within the models and the entire modeling process. The paper highlights several key aspects related to improving the quality of quantitative analysis: 1. **Mind the Assumptions:** SA helps to assess uncertainty and sensitivity, revealing the fragility of evidence and encouraging analysts to anticipate criticism. The paper cautions against both the artificial compression and inflation of uncertainty to manipulate the relevance of assessments. 2. **Mind the Hubris:** Model complexity can hinder relevance, and the paper recommends balancing model complexity with parsimony using information criteria when validation data is available or using uncertainty quantification and SA when such data is not available. 3. **Mind the Framing:** Models reflect the designers' values and worldviews; therefore, transparency and participatory processes are essential. The paper explores the “statactivist” movement’s strategies for fighting against statistical abuse. 4. **Mind the Consequences:** Quantification can have unintended consequences; therefore, SA and SAUD can deconstruct indicators with significant social impact, helping to expose reductionist or technocratic tendencies. The paper gives examples, including the World Bank's Doing Business Index. 5. **Mind the Unknowns:** Acknowledge ignorance in quantification; this helps to prevent the transformation of political problems into purely technical ones and allows for more realistic and responsible decision-making. The paper provides examples of how SA and SAUD can be used in practice, such as using SA to ensure algorithms do not implicitly use protected attributes.

The paper's findings directly address the research question by demonstrating the potential of SA and SAUD to enhance the quality and trustworthiness of quantitative analyses. The significance of these results lies in their contribution to a more critical and reflexive approach to quantification within the social sciences. The frameworks presented offer practical tools for assessing both the technical and normative aspects of models and other forms of quantification, moving beyond a simplistic focus on numerical output. This is particularly relevant in policy contexts where numbers are often used to justify decisions with significant social consequences. By highlighting the potential for bias, uncertainty, and value judgments embedded within quantitative methods, the paper encourages a more nuanced and ethical approach to their use. This has broad implications for various fields, particularly those dealing with complex social issues where the impact of quantitative analyses is substantial.

This paper makes a significant contribution to the sociology of quantification by highlighting the potential of mathematical modeling techniques, specifically SA and SAUD, to improve the quality, transparency, and fairness of numerical evidence. These methods can address both the technical and normative dimensions of quantification, leading to more robust and ethically sound results. Future research should focus on developing and applying these methods in diverse contexts, exploring their practical implications across various quantitative disciplines. Further investigation into the integration of qualitative and quantitative methods would strengthen the tools provided in this paper.

While the paper provides a compelling argument for the use of SA and SAUD, it primarily focuses on methodological aspects. Further research is needed to explore the practical challenges of implementing these techniques in various real-world settings, considering the potential constraints related to data availability, computational resources, and the complexity of social systems. In addition, the paper primarily addresses the perspective of modelers and researchers. Future work should examine the perspective of those who use the data and models and consider the broader societal and political implications of quantification in more detail.