Accessible data curation and analytics for international-scale citizen science datasets

The Covid Symptom Study (CSS) is a smartphone-based citizen science project that, by May 23, 2021, amassed over 360 million self-assessments from more than 5 million participants. Its scale poses major curation challenges: standard Python tools (e.g., Pandas) cannot readily process such data on commodity hardware, while alternative scalable technologies increase technical complexity. The authors present ExeTera, an open-source Python package that delivers Pandas-like analytics for datasets approaching terabyte scale. ExeTera’s design enables scalable, reproducible curation and analysis by emphasizing columnar storage, streaming operations, and sorted-key algorithms. They demonstrate ExeTera’s role as a core component of an end-to-end curation pipeline for CSS, enabling reproducible research across an international collaboration.

Benjamin Murray, Eric Kerfoot, Liyuan Chen, Jie Deng, Mark S. Graham, Carole H. Sudre, Erika Molteni, Liane S. Canas, Michela Antonelli, Kerstin Klaser, Alessia Visconti, Alexander Hammers, Andrew T. Chan, Paul W. Franks, Richard Davies, Jonathan Wolf, Tim D. Spector, Claire J. Steves, Marc Modat, Sebastien Ourselin