Angry by design: toxic communication and technical architectures

The proliferation of online hate speech is a growing concern. Current responses largely focus on two approaches: technical solutions like automated hate speech detection software, which struggle with the nuances of language and context, and human content moderation, which places a significant psychological toll on moderators. A third, often implicit, assumption is that hate speech originates from inherently hateful individuals. This study challenges these perspectives by proposing a design-centric approach. It argues that online platforms are designed environments that actively shape user behavior and interaction, and that the design of these platforms plays a significant role in fostering toxic communication. The research examines two prominent platforms, Facebook and YouTube, to analyze how their design choices contribute to the spread of hate speech and harmful online interactions.

Existing literature reveals the limitations of solely relying on technical solutions for mitigating online hate speech. Automated systems, while constantly improving, struggle to fully grasp the complex subtleties of human language and cultural context. The human cost of content moderation is also highlighted, with moderators experiencing significant mental health challenges due to exposure to graphic and hateful material. Additionally, a common misconception attributes online hate to inherently malicious individuals, disregarding the influence of platform design. This study builds upon existing research that emphasizes the impact of platform design on user behavior, drawing parallels to how urban design affects social practices. It leverages existing work on platform design, algorithmic bias, and the psychological impact of online interactions to support its analysis.

This study employs a design-centric methodology to analyze Facebook and YouTube. The research focuses on identifying key design elements – such as the News Feed on Facebook and the recommendation engine on YouTube – and analyzing their functionalities and affordances. The analysis examines how these elements shape user experience, encourage specific types of communication, and prioritize certain content over others. The study draws upon secondary literature from designers, platform users, and software engineers to supplement the analysis of platform features and their underlying logic. To gain a ‘vernacular’ perspective on the design and its practical impacts, the author conducted two unstructured interviews: one with a young social media user and another with a former online community manager. This qualitative data provides insights into how users perceive and interact with the platforms’ design features. This methodology is consistent with recent calls within the field for a more design-conscious approach to understanding online hate. The study incorporates insights from designers themselves, who have openly acknowledged the addictive and exploitative aspects of their creations, underscoring the potential of design to either exacerbate or mitigate negative online behaviors.

The analysis of Facebook reveals that its News Feed, driven by engagement metrics, prioritizes incendiary and divisive content. This creates a stimulus-response loop, where users are constantly exposed to outrage-inducing material and are readily enabled to share it. This leads to the normalization of outrage and a reduction in the barriers to expressing it online. The study finds that Facebook’s design facilitates the easy sharing and rapid propagation of such content, furthering its normalization. The analysis of YouTube focuses on its recommendation engine. This engine, designed to promote engaging content, often prioritizes controversial and polarizing videos, leading users down a path of increasingly extreme content. The system's dynamic nature and the autoplay feature create a gradual and almost imperceptible escalation towards more extreme viewpoints. The study notes the existence of a “pipeline” effect, where recommendations gradually lead users from mainstream to more radical content, contributing to the spread of harmful ideologies. Furthermore, YouTube's comment system, designed to reward any form of engagement (positive or negative), amplifies toxic and hateful comments, creating a hostile environment. The study also acknowledges a contrasting perspective suggesting that YouTube's role in radicalization might be overstated. This opposing view argues that YouTube mainly lowers the barriers to content production and distribution, leading to increased availability of niche and extremist material, effectively addressing the existing demand for such content rather than solely driving new demand.

The findings of this study support the central argument that platform design significantly contributes to toxic communication. The engagement-driven algorithms of both platforms prioritize incendiary content, creating feedback loops that amplify negative behaviors. The ease of sharing and the dynamic nature of recommendations on both platforms exacerbate these effects. The results highlight the importance of considering the ethical and social implications of platform design, emphasizing the need to shift from a solely engagement-focused approach to one that prioritizes user well-being and fosters more civil online interactions. While acknowledging an opposing perspective suggesting that platform design alone does not fully explain the rise of online radicalization, this study emphasizes that platform design provides powerful levers that influence the spread of extreme views and hate speech, thereby contributing significantly to the problem. The design of the platforms examined actively shapes the online environment, influencing not only the type of content users encounter but also the way they interact with it. This underscores the crucial role that design plays in shaping online behaviors and fostering either positive or negative online community dynamics.

This research demonstrates the significant role platform design plays in facilitating toxic communication. Both Facebook's News Feed and YouTube's recommendation engine, driven by engagement metrics, inadvertently amplify divisive and harmful content. The study suggests several design interventions, including altering content prioritization algorithms, introducing time delays before posting, incorporating humanizing prompts, broadening recommendation systems, and redesigning comment systems with reputation mechanisms. While acknowledging the challenges of implementing such changes within the existing corporate structures driven by monetization, the study emphasizes the possibility and necessity of redesigning platforms to prioritize user well-being and foster more inclusive and civil online environments. Future research should quantitatively analyze the influence of design on individuals across various demographics and apply the design-centric approach to additional platforms.

The study's scope is limited to two specific platforms, Facebook and YouTube, and its findings may not be generalizable to all online platforms. The qualitative data obtained from interviews is limited in scope, though it offers valuable insights. Furthermore, the precise extent to which platform design influences individual users, modulated by factors such as age, gender, class, or culture, requires further investigation. Finally, the study relies heavily on publicly available information and secondary sources, limiting the depth of insight into the proprietary algorithms used by these platforms.