Identifying schizophrenia stigma on Twitter: a proof of principle model using service user supervised machine learning

Mental Health is frequently discussed on Twitter, and some service users may find a sense of community and a safe space for expression, support, and self-management information to help them cope with their mental health problems. But Twitter may be harmful through allowing the propagation of stigmatising attitudes and ideas, which can become part of the narrative around mental health conditions and those who suffer from them. Stigma has negative effects on people with mental health problems by making them less likely to seek help. The first stage of combating stigmatising attitudes is reliable identification, but it is difficult to police harmful and stigmatising tweets given the high tweet volume. Machine learning techniques could automatically identify and potentially block them or allow the targeting of online anti-stigma campaigns. Machine learning models have used social media data, for example, to identify symptoms of depression using the sentiment in a user's content. But models can be biased from the way data are collected (ascertainment bias), or as consequences of conscious or unconscious biases in human decision-making in the data used to train the models. Evaluation metrics (e.g., accuracy, false negatives) need to be acceptable to the community of users who will benefit from them. All these issues are important in classification and we have taken the view that the essential components for an acceptable model are: supervision of machine learning models to avoid bias, iterative modelling to identify the best performing model, and full involvement of the community who will use the technology to increase acceptability. This is a proof of principle study to understand if machine learning can use service user rated tweets to reliably automate the identification of new tweets as stigmatising. We chose to investigate stigma associated with schizophrenia because it is highly stigmatised on Twitter compared to other mental health or neurological disorders, and little is known about its prevalence on popular social media platforms.

Ethical framework: The study followed the Community Principles on Ethical Data Practices (CPEDP), involving people with lived experience of mental health services throughout conception, data curation, model development, and validation. Public, non-sensitive data were anonymized and stored on encrypted university servers; tweet texts were not published to avoid re-identification. Service user involvement: A national young person’s mental health advisory group (YPMHAG) advised that minimizing false negatives should be the primary evaluation metric. They also suggested potential features indicative of stigma. Data collection: Public tweets containing the keywords “schizophrenia,” “schizophrenic,” “psychosis,” “psychotic,” or “schizo” were collected in real time via Twitter’s Streaming API using Tweepy over five rounds totaling 48 hours between January and May 2018, yielding 13,313 tweets. Manual coding dataset: From 1,000 selected tweets, 746 English tweets were eligible and manually labeled by two service user researchers as stigmatising or not; interrater reliability kappa = 0.75. Final labels: 299 stigmatising (40%), 447 non-stigmatising (60%). Feature engineering and preprocessing: Text processing included removing stop words, lemmatization, vectorization, and additional engineered features (e.g., sentiment, subjectivity, length, punctuation, numerics). Model development: The labeled dataset (n=746) was split 80/20 (stratified) into training and test sets. Eight supervised models were trained using scikit-learn 0.17.1 with hyperparameter tuning via grid search: Random Forest, Gradient Boosting (tree ensembles), k-Nearest Neighbors (k=1–25), Naïve Bayes (var smoothing 1 to 1e-9), Support Vector Machines (linear, polynomial, sigmoid kernels; C ∈ {1,10,100,1000}; gamma ∈ {0.001,0.0001 for poly/sigmoid}). Bootstrapping/cross-validation was used to assess and mitigate overfitting without model modification during runs. Evaluation metrics: Primary metric was the number of false negatives, as defined with service users. Additional metrics included accuracy and area under the ROC curve (AUC), plus false positives. Model validation: Two stages using additional English tweets from the corpus. Blind validation: 1,000 unique tweets (922 eligible), split into two batches, each independently labeled by two service user researchers; model predictions compared via Cohen’s kappa and false negatives. Unblind validation: Another 1,000 tweets (797 eligible) labeled by one service user researcher who evaluated agreement with each model’s classification; kappa and false negatives computed. Big data analysis: The best-performing model by service user criteria (fewest false negatives and acceptable validation) was applied to all English tweets (n=12,145) to estimate stigma prevalence. Independent-samples two-sided t-tests compared sentiment and subjectivity between model-identified stigmatising vs non-stigmatising tweets; ANOVA examined country-level sentiment differences. Retweets were both included (to reflect user exposure) and analyzed separately to assess their impact.

- Interrater reliability for manual coding: kappa = 0.75 on 746 English tweets (299 stigmatising, 40%; 447 non-stigmatising, 60%). - Feature differences (service user-coded): Stigmatising tweets had more negative sentiment (non-stig mean = 0.037, SD=0.269; stig mean = −0.198, SD=0.247; t(744)=12.02, p<0.001, 95% CI 0.196–0.273), were more subjective (t(682.31)=−10.55, p<0.001), had fewer numeric characters (t(604.21)=3.17, p=0.02), fewer punctuations (t(706.82)=5.22, p<0.001), and were shorter (t(739.09)=9.581, p<0.001). - Model comparison on test set: Random Forest AUC 0.94 vs SVM linear AUC 0.92; SVM had fewest false negatives (3 vs 11) and one fewer false positive (10 vs 11); accuracy SVM 91% vs Random Forest 87%. - Blind validation (n=922): SVM kappa with service users ranged from 0.305 to 0.652 across raters/batches; false negatives 30–96; SVM classified 43% as stigmatising, matching an independent coder. Random Forest kappa 0.291–0.621; false negatives 45–112; classified 39% as stigmatising, matching an independent coder. - Unblind validation (n=797): SVM kappa = 0.667 (95% CI 0.616–0.718), 102 false negatives; Random Forest kappa = 0.614 (95% CI 0.561–0.667), 139 false negatives. SVM classified 42% as stigmatising; Random Forest 36%. - Big data application: Applying linear SVM to 12,145 English tweets identified 46.7% (n=5,676) as stigmatising. - Sentiment and subjectivity (SVM-identified): Stigmatising tweets were more negative in sentiment (t(12,143)=64.38, p<0.001, 95% CI 0.29–0.31) and more subjective (t(12,143)=58.37, p<0.001). Means: sentiment non-stig 0.08 (SD 0.25) vs stig −0.22 (SD 0.26); subjectivity non-stig 0.35 (SD 0.31) vs stig 0.66 (SD 0.27). - Retweets: Excluding retweets (n=6,168) did not change the pattern of results. - Geographic patterns: Some users lacked location (n=2,624, 21.6%). USA (n=4,958) had 47.6% stigmatising (n=2,700); Canada (n=933) 3.3% stigmatising; UK (n=1,357) 7.6% stigmatising. Sentiment of stigmatising tweets more negative in USA (mean −0.11±0.29) than Canada (0.02±0.31) and UK (0.01±0.28); ANOVA F(6100,2)=106.99, p<0.001.

The study demonstrates that supervised machine learning, developed and validated in partnership with mental health service users, can reliably identify schizophrenia-related stigma on Twitter. Prioritizing the reduction of false negatives, as requested by service users, led to selection of a linear SVM despite a slightly lower AUC than Random Forest. Validation exercises showed the SVM consistently produced fewer false negatives than Random Forest and aligned better with service users’ preference to be ‘too careful’ (overclassify as stigmatising) rather than ‘too lenient’ (miss stigmatising content). Applying the SVM to a large corpus revealed that almost half of English tweets referencing schizophrenia were stigmatising, a markedly higher prevalence than reported for some other conditions (e.g., Alzheimer’s disease). The findings support using ML to monitor stigma at scale, inform targeted anti-stigma campaigns, and evaluate their effectiveness in real time. The discussion also considers methodological and contextual factors: SVM susceptibility to noise and small training vocabularies can lead to overclassification; larger, more diverse training datasets and broader feature sets could improve generalization. Cultural and linguistic nuances, ambiguity in stigma interpretation, and variability across countries underscore the need for inclusive, context-aware model development with diverse service user input.

A service user supervised machine learning pipeline can effectively identify schizophrenia stigma on Twitter at scale, with a linear SVM minimizing false negatives and achieving strong validation. The high prevalence of stigmatising content highlights an urgent need for education and targeted online anti-stigma campaigns. Machine learning offers a practical means to monitor public discourse in real time and assess the impact of interventions. Future work should expand and diversify the labeled training data, incorporate cultural and linguistic contexts, iteratively retrain models over time, and continue deep engagement with service users to ensure models reflect their priorities and values.

- Potential sampling bias: Tweets were collected via Twitter’s Streaming API over limited 48-hour windows, which may not fully represent platform-wide activity. - Language restriction: Models were trained and applied only to English tweets; stigma expressed in other languages with cultural nuances was not captured. - Model bias and data noise: The linear SVM may overclassify tweets as stigmatising due to susceptibility to noise and a relatively small training vocabulary, risking broader generalizations to unseen data. - Human labeling variability: Despite good interrater reliability, stigma can be interpreted differently across individuals; ambiguity may affect ground truth. - Generalizability over time: The model was trained on data from early 2018; shifts in language use and social context may reduce performance on future data without iterative updates. - Limited feature coverage: Features derived from a small labeled set may not capture all relevant expressions of stigma, especially across diverse cultural contexts.