Robust language-based mental health assessments in time and space through social media

The study addresses the challenge that population mental health monitoring relies heavily on self-reported surveys with limited spatial and temporal resolution (often annual, at state level), and lacks frequent measures for key conditions such as anxiety. Given the public health importance of depression and anxiety and their association with adverse outcomes, the authors propose leveraging social media language to generate high-resolution, language-based mental health assessments (LBMHAs). Drawing on psychometric principles, the research evaluates whether LBMHAs can reliably and validly track population-level depression and anxiety at finer scales (e.g., county-week). The purpose is to produce objective, scalable, and timely assessments that align with established measures and external criteria, enabling researchers and policymakers to detect geographic patterns and temporal changes, including responses to major societal events.

Prior work has used Twitter language to estimate population health outcomes (e.g., mortality, well-being, substance use, flu) mostly at coarse levels (states or limited counties). Psychometric frameworks emphasize assessing reliability and validity (convergent with accepted measures and external criterion validity). Ecological momentary assessment demonstrates value in short-timescale observations. Language-based assessments have shown predictive power over traditional surveys for psychological constructs and have improved via post-stratification to address selection biases. Studies have also leveraged language to predict clinical outcomes (e.g., depression questionnaires, PTSD trajectories) and personality, and have examined semantic drift and domain adaptation when transferring models across platforms and time. However, routine high-resolution monitoring of anxiety and depression remains limited in surveys, motivating the need for robust, fine-grained, language-based population monitoring.

Data: An updated County Tweet Lexical Bank (CTLB-19-20) of county-mapped Twitter posts spanning 2019–2020 was constructed. Users were mapped to US counties using GPS or first valid self-reported profile location (~93% accurate). The unprocessed dataset contained 2.7B posts from 2.6M users; after filtering, ~1B posts from 2.2M users (over 104 weeks) remained, covering 1,418 counties (~92% of the US population). Filtering removed non-English posts (langid), retweets, posts with hyperlinks, and duplicate messages. User and county-week inclusion thresholds: Users must post at least 3 times in a given week (3-UPT). Reliability analyses determined county-week thresholds for unique users (UT): reliability >0.8 at UT≥50 and ~0.9 at UT≥200. Counties failing to report for 10 consecutive weeks were dropped. To reincorporate sparse regions, a super-county binning strategy aggregated under-threshold county-weeks within a state using weighted means by reporting user counts; super-counties were then held to the same UT criteria. Linear interpolation filled missing county-weeks. To control seasonality, 2020 county-week scores were adjusted by subtracting 2019 means. Feature extraction and scoring: Words were tokenized using DLATK’s social media-aware tokenizer. Relative word frequencies per user-time were Anscombe transformed. Pretrained lexicon-based models for depression and anxiety (ridge regression-derived weighted lexica) were adapted from Facebook to 2019–2020 Twitter via target-side domain adaptation. Domain usage and relative frequency outlier filters removed words with divergent cross-domain usage; common US first names were dropped, yielding 5,469 tokens per lexicon plus an intercept. The lexicon scoring produced user-week depression (DEP_SCORE) and anxiety (ANX_SCORE) values scaled to 0–5. Post-stratification: Robust post-stratification weights were computed to correct selection biases (age, gender, etc.) estimated from language, with estimator redistribution, adaptive binning, and informed smoothing. County-week LBMHAs were computed as the post-stratified weighted mean of user-week scores within each county and time unit. Reliability assessment: Split-half reliability (R = 1 − Cohen’s d) was computed across spatiotemporal units (MSA/county/township × year/quarter/month/week/day). Repeated split-half reliability (RSR) guided minimum UT thresholds (RSR>0.7, 0.8, 0.9 mapping to thresholds such as UT 50 and 200). County-week was identified as the finest unit achieving high reliability. Convergent validity: Compared LBMHAs with Gallup COVID-19 Panel data (surveyed sadness and worry) using multilevel fixed-effects models across national, regional, and county levels (weeks/quarters). Gallup county-week data were reliability-filtered (≥0.7) to mitigate endogeneity and measurement noise. External validity and event sensitivity: Cross-sectional correlations with County Health Rankings (political, economic, social, and health variables) were evaluated, and agreement versus Gallup-survey correlations assessed. Temporal sensitivity was tested by comparing percent changes during major US event weeks (14 weeks in 2020) versus non-event weeks (38), with Cohen’s d and bootstrap CIs. Community-type analyses used American Communities Project categories. Data/code availability: County-week LBMHAs, toolkit, and analysis scripts are released via WWBP GitHub repositories.

- Reliability: High split-half reliability across space-time units; reliable measures achievable at county-week resolution. Reliability thresholds indicated UT≥50 yields >0.8 and UT≥200 ~0.9 reliability. Example ICC2 at county-year UT=50: Gallup Sadness 0.33 vs LBMHA Depression 0.97; at UT=200: Gallup 0.87 vs LBMHA 0.99. Reliability per unit (1−d) showed county-week ~0.921, township-week ~0.765, MSA-week ~0.986. - Coverage at UT thresholds: At UT>200, 35,969 county-weeks and 366 distinct counties (51 states); at UT>50, 73,240 county-weeks and 725 counties. - Convergent validity: Significant positive associations with Gallup at all granularities (p<0.01). Fixed-effects β: National-weeks: Depression β=0.763, Anxiety β=1.823; Regions-weeks: 0.759 and 1.817; Counties-quarters: 0.681 and 1.423; Counties-weeks: 0.410 and 0.343. Pearson correlations at nation-week: Depression–Sadness r=0.34, Anxiety–Worry r=0.67. - Temporal validity/event sensitivity: LBMHAs showed clear increases during major events in 2020: mean absolute increases of +23% (depression) and +16% (anxiety) in event weeks; non-event weeks showed mean decreases of 8% (depression) and 6% (anxiety). Nationally, both measures trended upward across 2020 relative to 2019. - External validity: Cross-sectional correlations with County Health Rankings variables were stronger for LBMHAs than survey measures on average. Agreement between LBMHA and survey correlations with external criteria: correlation-of-correlations r=0.67 (Anxiety–Worry) and r=0.34 (Depression–Sadness), both p<0.01. - Community differences: Exurbs exhibited the highest anxiety and depression among the five most represented American Community types, with modest overall differences. - Scale: Analyses used ~1B posts, 15.3B word instances from 2.2M users across >1,400 counties, enabling robust county-week monitoring.

The findings demonstrate that language-based mental health assessments can reliably and validly monitor population-level depression and anxiety at fine spatiotemporal scales (county-week), surpassing the granularity of traditional surveys. LBMHAs showed strong temporal alignment with known societal shocks (e.g., COVID-19 onset, murder of George Floyd) and converged with Gallup measures across multiple spatial and temporal resolutions, indicating they capture similar underlying constructs. External validity results suggest LBMHAs relate more strongly to political, economic, social, and health indicators than concurrent survey measures, underscoring their utility as population health indicators. Methodological advances—digital cohort aggregation, robust post-stratification, target-side domain adaptation, and reliability-informed thresholding—collectively improved generalizability and precision. Together, the results support the use of social media language to enable timely, geographically detailed monitoring of public mental health, informing research and policy and potentially guiding resource allocation during crises.

This study integrates best practices in language-based assessment, bias correction, and psychometric evaluation to deliver reliable, valid county-week estimates of depression and anxiety from social media. LBMHAs outperform conventional surveys in spatiotemporal resolution while maintaining moderate-to-strong convergence with survey measures and stronger associations with external community metrics. The authors release county-week data and open-source tools to facilitate reproducible research. Future work should extend and validate the pipeline across years, regions, and platforms; leverage transformer-based models to handle semantic drift and improve accuracy; explore quasi-experimental designs to uncover causal effects; and evaluate organizational and network-level applications for proactive mental health monitoring.

- Geographic sparsity: Many small-population counties lacked sufficient user volume and were aggregated into super-counties, reducing geographic specificity. - Bot and non-human content: Although filtering (no retweets, no URLs, duplicates) and user-level aggregation mitigate bot influence, full removal of non-human content is not guaranteed. - Temporal and domain scope: Analyses are limited to U.S. data from 2019–2020; language evolves (semantic drift), requiring revalidation and potential model adaptation in other years/contexts. - Platform dynamics: Changes in Twitter ownership, policies, and user populations may affect generalizability; applying the pipeline to other platforms (e.g., Reddit, Mastodon) needs validation. - Model class: Lexicon-based models were used for efficiency and prior validation; newer transformer models may improve performance but require rigorous validation at scale. - Clinical inference: LBMHAs reflect language correlates of symptoms and cannot determine individual clinical diagnoses or incidence. - Preregistration: The protocol was not preregistered; pretrained models were used, but analytic choices could introduce researcher degrees of freedom.