Mental illness affects a significant portion of the adult population, yet early detection and prediction remain challenging due to infrequent clinic visits. Social media data, readily available and widely used, has been proposed as a potential source for identifying and predicting mental health conditions. Previous research suggests that language use on social media can reflect various mental health problems, including depression, eating disorders, and schizophrenia. However, the accuracy and specificity of these predictions are debated. This study aimed to determine the predictive power and specificity of language-based models trained on social media data for various mental health conditions. A key premise of this research lies in evaluating the clinical actionability and precision of such models, focusing on whether they accurately predict a specific mental illness over others. The study challenges existing literature by investigating whether language patterns associated with depression are unique to depression or common across various mental health conditions. The authors highlight previous work that points to shared variance between disorders, questioning the specificity of language-based indicators. Previous studies often compare depressed individuals to healthy controls, neglecting the crucial aspect of distinguishing between different psychiatric disorders. The use of self-reported diagnoses on social media as a definition of mental illness is also criticised, since individuals openly disclosing diagnoses online may not represent the larger population of undiagnosed individuals. Such self-reporting may lead to biased samples and inflate the apparent predictive power of social media language analysis.

The existing literature demonstrates a correlation between language use and depression. Depressed individuals frequently utilize first-person singular pronouns, obscenities, and express negative emotions in their writing and speech. This has been observed in various settings, including social media. However, the specificity of these language patterns to depression remains questionable, as similar patterns are present in other mental health conditions. Studies comparing depressed individuals to healthy controls lack the necessary specificity to determine whether these language patterns are unique to depression or indicative of broader mental health issues. While some studies have examined multiple mental health conditions, they often suffer from methodological limitations, such as using self-reported diagnoses or focusing on discussion forums that may not reflect everyday language use. There's also concern about the reliability of social media posts to reflect the true mental state of individuals, and the potential for circularity where the same content defining disorder status is used to study language use. Previous studies have shown discrepancies in accuracy when using less-biased methods that separate the content used to define disorder status from the content used to characterize language. This paper aims to overcome the limitations of past studies by utilizing validated clinical instruments for defining mental health status and directly examining the specificity of language patterns across multiple mental health conditions.

The study recruited 1006 participants (from an initial 1450) who completed nine self-report mental health questionnaires and consented to link their Tweets to the questionnaires. Participants were required to be at least 18 years old, have a Twitter account with at least five days of Tweets, and have at least 50% of their Tweets in English. They also had to pass an attention check. The researchers collected up to 3200 Tweets and likes from each participant’s account in the 12 months prior to survey completion. Preprocessing involved removing extraneous information such as @ symbols, hashtags, emojis, punctuation (except periods, exclamation points, and question marks), links, and non-alphanumeric characters. Tweets were aggregated into daily bins for analysis. Linguistic Inquiry and Word Count (LIWC) analysis was used to extract linguistic features from the Tweets. In addition to LIWC features, Twitter metadata such as number of followers, followees, replies, and an insomnia index (ratio of night versus day Tweets) were also included. An Elastic Net model was trained on depression severity scores using nested cross-validation (70% training, 30% test set). To assess the model's performance, the researchers calculated R-squared. To evaluate specificity, they applied the model trained on depression scores to predict the scores of other mental health conditions. Random label permutation was used as a null model to determine if the predictive ability of LIWC was above and beyond age and gender. Further analyses examined associations between the residuals of the depression model and Twitter usage metrics. In addition to the Elastic Net regression model, support vector machine (SVM) and random forest (RF) classification models were also employed to predict depression status (binarized from Zung Self-Rating Depression Scale) and compared to a model using keywords to identify depression on Twitter. Transdiagnostic dimensions ('anxious-depression', 'compulsivity and intrusive thought', and 'social withdrawal') were also used to train models. Finally, analyses explored the effect of the number of words per user and the type of Twitter data (Tweets, Retweets, Likes) on model performance.

The Elastic Net model trained on depression severity scores explained only 2.5% of the variance in depression symptoms in the held-out test set (R²=0.025, r=0.16). Adding age and gender slightly improved performance (R²=0.045, r=0.22). The depression-trained model also showed some predictive power for other mental health conditions: schizotypy, social anxiety, eating disorders, and generalized anxiety, demonstrating non-specificity of language features. The model did not predict alcohol abuse or impulsivity significantly. Univariate analyses revealed that several LIWC features associated with depression severity (e.g., word count, negative emotions, focus on present) were also associated with other aspects of mental health. Twitter metadata features, such as the number of Tweets and replies, were broadly associated with mental health, but not specific to one condition. Examining residuals of the depression model revealed no systematic associations with Twitter usage, suggesting that model failures were not due to differences in user engagement. Classification models (SVM and RF) yielded similar AUC values of around 0.59, indicating modest performance. A model trained on depression-related keywords performed much better (83.6% accuracy, 0.83 AUC), highlighting the potential bias of using keyword-based methods to define cases. Analysis of selection frequencies in models trained on individual questionnaires and transdiagnostic dimensions indicated a lack of specificity, even after removing shared variance. Only after controlling for shared variance across transdiagnostic dimensions could the researchers identify text features unique to each dimension.

The study's findings demonstrate that language use on Twitter is not sufficiently specific or accurate to predict individual mental health conditions. The modest predictive power of the models, even for depression, highlights the limitations of using social media data for individual-level predictions in a clinical setting. The lack of specificity of language features across multiple mental health conditions underscores the high comorbidity rates among psychiatric disorders. The relatively poor performance of models relying on keyword-based detection compared to those based on validated questionnaires underscores the importance of using robust ground truth measures of mental health. While the low predictive power of individual features is evident, it's not unusual within the field of mental health research. The small effect sizes observed are comparable to those reported in other areas, highlighting the complexity of mental health. The improved performance when including age and gender suggests the importance of considering multimodal data for a better predictive model. Analyzing the residuals of transdiagnostic dimensions is shown to be a promising approach to identify unique language patterns associated with specific mental health dimensions.

This study provides robust evidence against the use of social media data for individual-level predictions of mental health. The low predictive accuracy and lack of specificity highlight the limitations of current approaches. Despite the weak predictive power at the individual level, social media data can be useful in population-level studies and for testing theories that require large datasets. Future research should focus on combining multiple data sources (multimodal data) and employing more advanced analytical methods to improve prediction accuracy. Ethical implications regarding data privacy and potential misuse of predictions must also be carefully considered.

The study's reliance on self-reported questionnaires may introduce biases. The use of only Twitter data might limit generalizability to other social media platforms. The study's cross-sectional design restricts inferences about causality. The choice of LIWC features might have affected the model's predictive power, and other methods could potentially improve performance. The study's focus on predominantly English-speaking users could limit the generalizability of findings to other cultural contexts. Finally, the study assessed ‘trait’ mental health, and variations in ‘state’ features may have affected the results.