On responsible machine learning datasets emphasizing fairness, privacy and regulatory norms with examples in biometrics and healthcare

The rapid advancement of artificial intelligence (AI) and machine learning has led to significant improvements in various fields, but concerns regarding bias, privacy, and manipulation have emerged. While much attention has focused on developing trustworthy AI algorithms, the quality of training data is paramount. Deep learning algorithms are highly susceptible to biases present in their training data, potentially leading to unfair or discriminatory outcomes. This paper highlights the critical role of responsible data in AI development, emphasizing the need to address fairness, privacy, and regulatory compliance during dataset creation. The focus is on biometric and healthcare datasets due to the sensitive nature of the data and the increasing prevalence of data protection legislation. The research aims to provide a framework for evaluating dataset responsibility and offer recommendations for improving dataset curation practices within the scientific community. The global impact of AI necessitates the development of trustworthy systems, and data quality plays a pivotal role in achieving this goal. This study is timely and essential for the ongoing evolution of AI technologies.

Existing research addresses dataset quality through qualitative (interviews, discussions) and quantitative methods. Gebru et al. proposed datasheets for increased transparency and accountability, aligned with frameworks for responsible data development. Other works analyze data representation in accessibility datasets and the importance of documenting context within image datasets. There's also a growing recognition of the need to incorporate socio-cultural considerations, including consent and inclusivity, into data collection processes. While research has explored bias in biometric data at the model level and audited algorithms, few have jointly quantified dataset responsibility across fairness, privacy, and regulatory compliance. Previous studies have quantified bias and privacy in various contexts, but none comprehensively assess the three dimensions jointly. Various approaches exist for measuring privacy (l-diversity, k-anonymity, t-closeness, M-invariance), but these are not always integrated with fairness and regulatory considerations.

The study introduces a responsible rubric to assess machine learning datasets, focusing on fairness, privacy, and regulatory compliance. The rubric consists of three main components: **1. Quantifying Fairness (F):** This metric considers diversity, inclusivity, and the reliability of labels. Inclusivity measures the representation of different groups (gender, skin tone, ethnicity, age) in the dataset. Diversity uses Shannon's diversity index to quantify the distribution of these groups. The label score reflects the reliability of the labels (self-reported, classifier-generated, or apparent). **2. Quantifying Privacy (P):** This component identifies vulnerable annotations that could lead to privacy leakage, including name identification, sensitive attributes, accessories, critical objects, location inference, and medical condition information. Privacy Leakage (PL) is the sum of these attributes found in the annotations, and Privacy Preservation (P) is calculated as (1 - A) * PL, where A is the presence of the attributes in the annotations. **3. Quantifying Regulatory Compliance (R):** This score assesses institutional approval, individual consent, and the facility for data expungement/correction. Each factor is scored as 0 or 1, with a maximum score of 3 indicating full compliance. The authors surveyed over 100 datasets, focusing on 60 accessible datasets (52 face-based biometric and 8 chest X-ray healthcare datasets) for detailed analysis. They used equations (1) through (7) to calculate fairness, privacy, and regulatory scores, creating a three-dimensional representation of dataset responsibility. DBSCAN clustering was applied to analyze the dataset distribution.

The analysis revealed that many datasets inadequately address fairness, privacy, and regulatory compliance. The average fairness score (F) was 0.96 ± 0.64 (maximum 5), indicating a significant need for improvement. Healthcare datasets showed slightly higher fairness scores (1.34 ± 0.17) compared to biometric datasets (0.90 ± 0.67). The DroneSURF dataset was the most privacy-preserving (maximum 6), while the FB Fairness Dataset (Casual Conversations) achieved the maximum regulatory compliance score (maximum 3). A fairness-privacy paradox was observed, where including sensitive attributes to enhance fairness increased privacy risk. The study highlighted that most datasets performed poorly on both fairness and regulatory compliance, with 91% of datasets receiving a fairness score of 2 or less, and 89% having a compliance score of 0 or 1. Analysis by year of publication and data source showed a trend toward increasing fairness and regulatory compliance over time, and higher scores for non-web-collected datasets. A 3D scatter plot and DBSCAN clustering analysis revealed distinct clusters of datasets based on their fairness, privacy, and regulatory compliance scores, identifying the FB Fairness Dataset (Casual Conversations), IISCIFD, and UTKFace as the most responsible datasets based on a weighted average of the three scores.

The findings highlight a significant gap between the current state of dataset creation and the principles of responsible AI. The prevalence of fairness, privacy, and regulatory compliance issues underscores the need for substantial changes in dataset development practices. The fairness-privacy paradox is a significant challenge, demanding creative solutions that balance inclusivity with data protection. The use of synthetic datasets is suggested as a potential approach to mitigate these challenges; however, the generation of unbiased synthetic datasets currently requires the use of large datasets that have inherent privacy and fairness issues. The study's recommendations provide actionable steps for improving dataset responsibility. The results underscore the importance of considering ethical considerations throughout the entire data lifecycle, from design to deployment, and the need for greater transparency and accountability in the release and use of machine learning datasets.

This research emphasizes the crucial role of responsible dataset creation in building trustworthy AI systems. The comprehensive audit and proposed rubric offer a valuable tool for evaluating dataset responsibility across fairness, privacy, and regulatory compliance. Recommendations for improved dataset creation and management practices, combined with the development of comprehensive datasheets, are vital for the responsible advancement of AI. Future work should focus on addressing the fairness-privacy paradox, developing more sophisticated quantification methods, and extending the framework to other domains, such as datasets concerning objects rather than only human-centric applications. The development of methods for mitigating the risks of training data leakage in synthetic datasets is crucial.

The fairness quantification focuses on label distribution and does not fully capture image diversity. The study acknowledges that unequal distribution might be necessary for some applications, and the current formulation of scores is primarily tailored to human-centric datasets. The recommendations presented represent high standards that may be challenging to achieve immediately, requiring further research. The specific choice of metrics within the framework could influence the outcome; however, the authors note the flexibility of their framework.