Harmonizing government responses to the COVID-19 pandemic

From lockdowns to travel bans, government responses to SARS‑CoV‑2 affected nearly every societal dimension. Comprehensive, high-quality, timely PHSM data are critical for understanding policy scope, timing, and impacts on health, economy, environment, and society. Many trackers documented COVID‑19 PHSM but each provides an incomplete portrait, with several ceasing collection due to funding. This article demonstrates how harmonization—making compatible conceptually similar data—can yield a more comprehensive dataset than any single source. The authors introduce a rigorous methodology to harmonize PHSM data (Dec 31, 2019–Sep 21, 2021) from eight major efforts (ACAPS; COVID AMP; CIHI; CoronaNet; HIT‑COVID; OxCGRT; WHO EURO; WHO CDC) into the CoronaNet taxonomy. With 500+ research assistants, ~150,000 external observations are harmonized against CoronaNet (>180,000 observations), preserving sources from trackers that stopped and enabling richer analyses. The work illuminates challenges in harmonizing incomplete and dirty data, compares dataset strengths and weaknesses, and details methodological steps. The harmonized data significantly outperform the WHO PHSM aggregation in scale and quality. The results section outlines dataset selection criteria (geographic and temporal coverage, volume, taxonomy similarity to CoronaNet, partner collaboration capacity), challenges of harmonizing diverse taxonomies and dirty data, and an assessment of gains and remaining limitations. The paper concludes by reflecting on how harmonization reveals complexities of data generation in crises.

The study employs a semi-manual, five-step harmonization pipeline to integrate seven external COVID‑19 PHSM datasets into the CoronaNet taxonomy for policies before Sep 21, 2021.

Step 1: Automated taxonomy mapping

• Construct mapping between each external taxonomy and CoronaNet for: policy timing (start/end), initiator (country, ISO‑2 region where available), policy type and subtype, sources (URLs/PDFs), textual descriptions, and when possible geographic/demographic targets.
• Where one-to-one mappings were infeasible, adopt one-to-many mappings and adjust using keyword heuristics and ML text classifiers (for COVID AMP and WHO PHSM) to predict CoronaNet policy types from descriptions.
• Create and preserve unique identifiers; reformat where necessary (e.g., collapsing HIT‑COVID inbound/outbound border entries; synthesizing OxCGRT IDs from indicator, date, country, province when none provided).

Step 2: Basic cleaning and subsetting

• Standardize country and partially standardize ISO‑2 provinces; ensure consistent treatment for 430+ priority provinces (Brazil, China, Canada, France, Germany, India, Italy, Japan, Nigeria, Russia, Spain, Switzerland, USA).
• Exclude observations outside CoronaNet scope (e.g., US county/tribal-level COVID AMP entries, territories such as Greenland, USVI, Guam) and non‑PHSM policy areas (economic/financial measures).

Step 3: Automated deduplication

• 3a. Within-dataset deduplication: Special procedures for OxCGRT panel-to-event transformation remove repeated identical notes and most “no change” notes while retaining lengthy entries that might encode substantive changes. General dedup uses normalized description, country, province, and link; validated at 99% accuracy on a sample. Identified duplicates include ACAPS (246), WHO CDC (45), CIHI (1), COVID AMP (437), WHO EURO (304), HIT‑COVID (373), OxCGRT (5549).
• 3b. Across-dataset deduplication: Use type, type_sub_cat, country, province, target_who_what, date_start to group duplicates. Prioritize which record to keep by dataset quality and specificity: CIHI (Canada) > COVID AMP > WHO CDC/EURO > OxCGRT > HIT‑COVID > ACAPS; if single duplicate, keep entry with longest description. Identified 5,989 cross-dataset duplicates (validated ~74.5% true duplicates on a sample).
• 3c. Deduplication between CoronaNet and external datasets: No robust automated rule (best sample accuracy ~14%), so no automatic removal; manually reinstate any false positives identified during sampling.

Step 4: Pilot manual harmonization

• Pilot subsets across datasets and geographies (e.g., CIHI: SK, NB, AB, MB; HIT‑COVID: Indian states, Slovenia, Luxembourg; WHO CDC/EURO: Slovenia, North Macedonia, Estonia; COVID AMP: several US states; OxCGRT: Luxembourg; ACAPS: Bulgaria; CCCSL piloted but deprioritized due to taxonomy and source issues). Pilots refined mappings, uncovered pitfalls (dead links, vague descriptions), and established practical strategies (e.g., use Wayback Machine, attend to OxCGRT national entries that encode subnational policies).

Step 5: Manual harmonization at scale

• External observations are written to Google Sheets (“Data Harmonization Sheets”) with fields for IDs, dataset, description, timing, mapped and ML-predicted policy types, geographic/demographic targets, compliance, travel mechanism, and source links.
• 5a. Overlap assessment: Coders mark whether an external observation already exists in CoronaNet (Yes/No/NA) and record matched CoronaNet record_id when Yes.
• 5b. Harmonization assessment: For non-overlapping items, coders recode directly from original sources into CoronaNet via the standard Qualtrics workflow, labeling assessments as: Harmonized; Harmonized with additional research; Harmonized with new link; Harmonized with research + new link; Duplicated policy; Not a relevant policy; Link dead, no replacement.

Quality assurance and tooling

• Taxonomy maps and code for Steps 1–3 publicly available; standard operating procedures, workshops, manuals, and manager reviews guide coders. Automated checks and manager feedback correct inconsistencies. Progress metrics recorded (e.g., overlap and harmonization completion percentages).

• Scope and coverage gains

  • Identified 150,052 external PHSM observations (post-automated preprocessing) for potential harmonization into CoronaNet (Table 6). CoronaNet itself contains >180,000 observations (≈145,000 unique to CoronaNet at time of writing).
  • Approximately 83% of assessed external observations do not overlap with CoronaNet; ≈45% of non-overlapping observations can be recoded, implying ≈55,000 potential additional records.
  • As of reporting, overlap assessment completed for ≈53.65% and harmonization assessment for ≈36.46% of external records (Table 6).

• Data quality and completeness

  • Text description quality varies widely (Table 2). Average description length (chars): CoronaNet 359; WHO CDC 537; OxCGRT 329; WHO EURO 297; COVID AMP 227; HIT‑COVID 230; CIHI 254; ACAPS 172. OxCGRT has most short (<50 chars) descriptions (4,265); HIT‑COVID has most missing descriptions (>1,600).
  • End-date completeness improves post-harmonization for datasets that originally lacked them: ACAPS missing end dates drop from 100% (raw) to 51.72% (harmonized); HIT‑COVID from 100% to 55.61% (Table 4). OxCGRT’s apparent low missingness in raw (3.59%) increases to 39.06% upon harmonization due to panel-index structure masking granular end dates.
  • Source preservation: ~10.2% of external records rely on dead links with unrecoverable information; ~4.7% dead links were recoverable; ≈3% have missing links. WHO EURO and WHO CDC show higher dead-link issues; CIHI and COVID AMP have fewer (Table 5). CoronaNet stores PDFs for all entries.

• Duplication control

  • Within-dataset duplicates identified: OxCGRT 5,549; COVID AMP 437; WHO EURO 304; HIT‑COVID 373; ACAPS 246; WHO CDC 45; CIHI 1 (Table 7).
  • Across-dataset duplicates identified: ACAPS 1,909; WHO CDC 273; CIHI 22; COVID AMP 753; WHO EURO 519; HIT‑COVID 92; OxCGRT 2,421 (Table 8).

• Overlap and harmonization outcomes by source (Tables 11–12)

  • Overlap with CoronaNet (Yes rate): HIT‑COVID 34%; WHO CDC 24%; WHO EURO 20%; ACAPS 21%; COVID AMP 18%; CIHI 16%; OxCGRT 11%.
  • Among non-overlapping items, proportions assessed as: Harmonized (overall 36%, CIHI 80%, COVID AMP 50%); Duplicated policy (overall 25%, HIT‑COVID 34%, OxCGRT 30%); Not relevant (overall 21%, OxCGRT 30%); Link dead, no other found (overall 10%, WHO EURO 21%).

• Comparative assessment

  • The harmonization effort substantially improves temporal, spatial, and administrative-level coverage relative to any single dataset and outperforms the WHO PHSM aggregation in both scale and quality (notably source preservation and recoding rigor).

The study addresses the core challenge that no single tracker captured the full geography, scope, and timing of COVID‑19 PHSM with consistent quality. By mapping eight major datasets to CoronaNet’s richer taxonomy and manually recoding from original sources, the authors demonstrate that harmonization can markedly increase coverage, fill key metadata gaps (e.g., end dates), and improve source durability. The findings underscore that taxonomy heterogeneity and panel indices can conceal important policy details (timing, targets), which manual validation can recover. Although harmonization cannot eliminate all noise or missingness—especially for subnational policies and low-capacity contexts—the integrated resource enables more generalizable cross-national studies and more internally valid subnational analyses than any single source. The work also clarifies trade-offs: harmonizing to a fine-grained taxonomy minimizes information loss for overlapping concepts but excludes domains (e.g., economic measures) outside scope. The process highlights the social construction of data in emergencies, encouraging transparency via public taxonomy maps and reproducible code so that researchers can assess design choices and robustness across datasets.

The paper contributes a rigorous, semi-manual methodology and public tooling to harmonize large-scale COVID‑19 PHSM data across eight major trackers into the CoronaNet taxonomy. It demonstrates substantial gains in completeness (time, space, administrative depth) and data quality (end dates, source preservation, duplication control), surpassing prior aggregation efforts. While harmonization cannot fully resolve gaps—especially in subnational coverage and low-capacity settings—the resulting dataset provides the most comprehensive view yet of government PHSM during COVID‑19. The authors advocate continued resourcing to complete and extend harmonization beyond Sep 21, 2021 and recommend that future efforts document taxonomy changes, preserve sources (PDFs), and coordinate standards to facilitate cross-project comparability. The methodological blueprint can guide harmonization beyond PHSM to other social-science datasets.

• Scope constraints: Economic/financial policies (e.g., subsidies, rent support) are outside the CoronaNet taxonomy and thus not harmonized.
• Subnational gaps: Most trackers focus on national-level policy; even with harmonization, subnational coverage remains incomplete and often biased toward North America and Europe.
• Low state capacity contexts: Limited or non-digital government communications, reliance on media reports, and NGO/IO policy action complicate documentation and validation.
• Data dirtiness and inconsistency: Vague or missing descriptions, panel indices masking event dynamics (e.g., OxCGRT), missing or dead links, and taxonomy divergence create residual uncertainty.
• Manual process limitations: Despite training and QA, manual recoding at scale can introduce errors; complete error-free harmonization cannot be guaranteed.
• Temporal cutoff: Harmonization covers policies through Sep 21, 2021 (with some trackers continuing later); extending beyond this date requires additional resources.