Accessible data curation and analytics for international-scale citizen science datasets

The Covid Symptom Study (CSS) is a smartphone-based citizen science project that, by May 23, 2021, amassed over 360 million self-assessments from more than 5 million participants. Its scale poses major curation challenges: standard Python tools (e.g., Pandas) cannot readily process such data on commodity hardware, while alternative scalable technologies increase technical complexity. The authors present ExeTera, an open-source Python package that delivers Pandas-like analytics for datasets approaching terabyte scale. ExeTera’s design enables scalable, reproducible curation and analysis by emphasizing columnar storage, streaming operations, and sorted-key algorithms. They demonstrate ExeTera’s role as a core component of an end-to-end curation pipeline for CSS, enabling reproducible research across an international collaboration.