Mental health is a critical public health concern, with depression and anxiety affecting millions globally, significantly impacting quality of life and contributing to economic losses. Current monitoring methods, primarily relying on subjective, self-reported survey data, lack the necessary temporal and geographical resolution to effectively understand and address dynamic changes in population mental health. Annual or state-level estimates provide a coarse understanding, hindering timely interventions. This study aims to address this limitation by developing and validating a novel approach using social media data, specifically Twitter, to provide high-resolution, robust assessments of population mental health. The hypothesis is that analyzing the language used in geo-located tweets can provide reliable and valid estimates of depression and anxiety prevalence at the county-week level, surpassing the capabilities of current survey methodologies. The importance of this study lies in its potential to revolutionize population mental health monitoring, enabling more precise and timely interventions tailored to specific geographic areas and time periods.

Existing research on mental health predominantly uses self-reported questionnaires that offer limited temporal and spatial resolution, typically providing yearly estimates at the state level. While some studies have utilized social media data to predict population health statistics (mortality, well-being, substance use, etc.), they often focus on limited geographic areas or coarser resolutions. Ecological momentary assessment studies suggest that shorter-timescale observations reveal crucial insights into symptoms and correlates inaccessible through traditional methods. This study builds upon prior work demonstrating the potential of language-based mental health assessments (LBMHAs), improving upon previous limitations by incorporating advances in post-stratification techniques to address selection biases and applying psychometric principles to evaluate reliability and validity rigorously.

The study utilized the County Tweet Lexical Bank (CTLB-19-20), comprising ~1 billion tweets from 2.2 million geo-located users across the US in 2019-2020. Data preprocessing involved filtering out non-English tweets, retweets, posts with hyperlinks (to minimize bot influence), and duplicates. The team applied a 3-user posting threshold (3-UPT) and a 200-user threshold (UT) per county-week to ensure reliability. To address under-representation of certain counties, a super-county binning strategy was used, aggregating data from smaller counties into larger units while weighting based on population. Linear interpolation was performed to fill gaps in weekly data. Pre-trained lexical models for depression and anxiety were adapted to 2019-2020 Twitter language using target-side domain adaptation, removing words exhibiting different usage patterns between Facebook (source domain) and Twitter (target domain). Robust post-stratification weights were applied to adjust for biases in the Twitter sample, ensuring a better representation of the US population. Split-half reliability was assessed across different spatiotemporal resolutions to determine the optimal granularity for analysis. Convergent validity was evaluated by comparing LBMHAs with Gallup's COVID-19 Panel survey data using fixed-effects multi-level modeling. External validity was assessed by examining correlations between LBMHAs and various county-level socioeconomic and health indicators from the County Health Rankings.

The study found that reliable LBMHAs could be generated at the county-week level, exceeding the resolution of available surveys. Split-half reliability analysis revealed that county-week measurements with at least 50 unique users (UT=50) achieved a reliability exceeding 0.8, and with 200 unique users (UT=200), reliability exceeded 0.9. Convergent validity analysis using Gallup's survey data showed significant positive correlations between LBMHAs and self-reported sadness/worry, with correlations ranging from 0.34 to 1.82 (p<0.001) depending on the spatial and temporal granularity. External validity analysis demonstrated strong correlations between LBMHAs and county-level measures of health and socioeconomic status, which were generally stronger than those observed for the survey data. The LBMHAs exhibited temporal validity by showing a significant increase (23% for depression, 16% for anxiety) in the weeks following major events such as the murder of George Floyd and the COVID-19 pandemic declaration. Community-level analysis revealed that exurbs showed the highest levels of anxiety and depression among community types examined. The study also reported data on 1418 counties (representing ~92% of the US population) across 104 weeks. A detailed breakdown of data descriptives of the filtered CTLB (including counts of word instances, posts, unique words, users, and counties) is available in Table 1. The team made the dataset and the open-source toolkit publicly available.

The findings address the research question by demonstrating the feasibility and validity of using LBMHAs derived from social media data for high-resolution population mental health monitoring. The significantly higher reliability and improved external validity of LBMHAs compared to traditional survey methods highlight the potential of this approach to provide more accurate and nuanced insights into population mental health trends. The ability to track changes in mental health at the county-week level, coupled with the observed responses to major societal events, underscore the value of this approach for timely public health interventions and resource allocation. The observed correlations with socioeconomic and health indicators further support the ecological validity of the LBMHAs, suggesting that this method can effectively capture the broader context of mental health within communities. The study’s implications are far-reaching, potentially enabling more effective monitoring of mental health during crises, better understanding of the determinants of mental health disparities, and development of targeted interventions.

This study successfully demonstrated the feasibility and validity of using language-based mental health assessments derived from social media data to monitor population mental health at unprecedented spatiotemporal resolution. The results highlight the potential for social media data to improve the accuracy and timeliness of mental health surveillance, allowing for more effective resource allocation and targeted interventions. Future research could explore the use of more advanced language models (LLMs) to further enhance the accuracy and efficiency of LBMHAs, investigate the relationships between LBMHAs and clinical diagnoses of mental illness, and extend this methodology to other geographical contexts and populations. The release of the open-source toolkit facilitates wider adoption and contributes to advancing research in this field.

The study has several limitations. First, some smaller counties with insufficient data had to be combined into super-counties, potentially affecting the accuracy of results. Second, despite efforts to mitigate the influence of bots, some non-human accounts may still be present in the data. Third, the analysis is limited to US data from 2019-2020, and future studies are needed to evaluate generalizability to other time periods and countries. Fourth, the study uses lexicon-based models, and future research could explore the advantages of incorporating more sophisticated LLM approaches. Finally, the study does not directly link language use to clinical diagnoses of depression and anxiety.