Measuring social response to different journalistic techniques on Facebook

The study investigates how news framing on social media influences online social dynamics around a polarizing topic—migration in Italy. Social media enable direct access to vast, heterogeneous content streams whose distribution is shaped by algorithms attentive to user preferences, affecting opinion formation, policy communication, and public debate. Prior work shows users self-select like-minded content, cluster into echo chambers, and resist corrective information, with exposure to opposing views sometimes increasing polarization. Fact-checking and algorithmic interventions have shown mixed or limited effectiveness, and polarization itself is a key driver in online interactions. The research question is how different journalistic techniques and content types used by a major Italian newspaper (Corriere della Sera) affect user commenting behavior—specifically toxicity, criticism of the source, and stance on migration—on Facebook. Using an original dataset of 113 posts (21,618 comments) from March–December 2018, the study aims to assess whether content framing can reshape narratives and influence user responses, and to measure the additional role of social conformity in shaping comments.

The paper situates its inquiry within literature on online misinformation, polarization, and echo chambers. Studies show users prefer belief-congruent information and cluster into like-minded groups, amplifying confirmation bias and sometimes engaging with false content. Exposure to opposing views can increase polarization, and polarization may serve as an early signal for misinformation targets. Efforts to counter misinformation, including fact-checking and debunking, often show limited effectiveness and may backfire. Platforms like Facebook have partnered with fact-checkers, but definitional and methodological issues persist. Algorithmic and machine-learning approaches to credibility and misinformation detection exist, yet the broader dynamics of narrative framing and user behavior remain a challenge, motivating examination of how journalistic framing influences online reactions.

Ethics: Data were collected via the Facebook Graph API in accordance with platform terms. Following a February 5, 2018 policy change, only data available to page owners are accessible; users with privacy restrictions are excluded. All data are public, aggregated, anonymized, and user IDs were hashed (SHA-512).

Data and classification: The dataset comprises 113 Facebook posts by Corriere della Sera linking to articles on migration (March–December 2018), including all reactions and comments. Posts were classified along two dimensions:

• Journalistic Techniques (JT): News report; Article with context; Editorial; Constructive news; Data-driven pieces (fact-checking/analytic); Human interest—individual; Human interest—groups; Article with pop-culture references.
• Content Types (CT): Article; Article/infographics; Video/photo-gallery; Multimedia.

Comment annotation: Three Italian annotators used the Goldfinch platform to label 22,236 comments without post context for: (i) stance on migration (anti/neutral/pro), (ii) presence of toxic language (vulgar/aggressive/rude/disrespectful/unreasonable; excluding irony/sarcasm), and (iii) criticism directed at the media source (publisher/journalist/newspaper). Irrelevant/spam/non-Italian/empty comments were excluded, yielding 21,618 relevant annotated comments for analysis.

Statistical analysis:

1. Engagement and framing: To test whether engagement (likes, comments, shares) differs by CT/JT, Kruskal–Wallis tests were used due to right-skewed distributions, with null hypotheses that engagement distributions do not differ across CTs/JTs.
2. Association between framing and commenting behavior: Contingency tables and χ² tests assessed independence between CT/JT and each behavior (toxic vs non-toxic; critical vs non-critical; anti-migration vs non-anti). Cramér’s V quantified effect sizes.
3. Logistic regression models: Six multivariate logit models predicted the probability of a comment being toxic, critical, or anti-migration as a function of framing typologies (either CT dummies with baseline Article, or JT dummies with baseline News report) and social conformity variables defined as the percentages of previously posted comments on the same thread exhibiting the behavior: per_toxic, per_critical, per_anti. Because CT and JT are not mutually exclusive, models alternated between CT and JT sets. Sensitivity of p-values to sample size was considered per Lin et al. (2013), with CPS charts in Supplementary Materials.

Dataset and scope: 113 posts on migration by Corriere della Sera (March–December 2018) with 21,618 relevant annotated comments.

Engagement vs framing: Kruskal–Wallis tests failed to reject the null hypotheses (α = 0.05) that engagement distributions (likes, comments, shares) do not differ across CTs or JTs. Median-based comparisons showed:

• CT: Multimedia had the highest median likes (214) and comments (132); Article/infographics were most shared (median 77.5); Video/photo-gallery had the lowest medians (likes 155.5; comments 51; shares 38.5).
• JT: Editorials had the highest median likes (389); Articles with pop-culture references had the highest median comments (163); News reports had the highest median shares (74.5). Human interest (groups) was lowest across likes (78.5), comments (32.5), and shares (21).

Association between framing and commenting behavior: χ² tests indicated interdependence between framing (CT or JT) and commenting behaviors (p-value < 2.2e-16). Cramér’s V showed small effect sizes:

• Toxic-CT: V = 0.0879; Critical-CT: V = 0.0937; Anti-CT: V = 0.1123.
• Toxic-JT: V = 0.0936; Critical-JT: V = 0.1578; Anti-JT: V = 0.1723. JT-based framing showed stronger associations than CT-based framing.

Logistic regressions (selected results):

• CT framing: Video/photo-gallery was associated with more toxic comments than baseline Article (β ≈ 0.42, p<0.001), while Article/infographics was associated with fewer toxic comments (β ≈ −0.29, p<0.05). Multimedia and Video/photo-gallery elicited fewer critical comments (β ≈ −0.52 and −0.59, both p<0.001). Video/photo-gallery was less likely to elicit anti-migration comments (β ≈ −0.14, p<0.05).
• JT framing: Compared to News reports, several JTs drew significantly more criticism, including Articles with context, Articles with pop-culture references, Constructive journalism, Editorials, Focus on data, and Human interest stories (p<0.001 across most). Toxic comments were relatively lower for data-focused or constructive pieces, and relatively higher for articles providing deeper context. Pop-culture and editorial pieces elicited fewer anti-migration comments than News reports.

Social conformity: The likelihood that a comment would be toxic, critical of the source, or anti-migration increased with the proportion of preceding comments exhibiting the same behavior, indicating strong social conformity effects in comment threads.

Overall patterns: Visual formats (multimedia, video/photo) and factual news reports were associated with higher trust in the source, while editorial/opinion and data-driven pieces tended to receive more criticism; data-driven pieces were less likely to trigger toxic debate. Effects of framing on behavior were statistically significant but small, indicating framing is one of multiple factors shaping online responses.

The findings address the research question by demonstrating that news framing contributes to shaping online commenting behaviors on a polarizing topic. Although engagement distributions did not significantly differ across framing typologies, framing was significantly associated with comment civility, trust toward the source, and stance on migration. Visual content and factual news reports corresponded to higher trust in the media source, while editorial/opinion and data-driven formats elicited more criticism; data-driven formats also tended to reduce toxic exchanges. Crucially, social conformity exerts a strong influence: commenters are more likely to mirror the tone and stance of prior comments. The results underscore that while framing matters, its effect sizes are small relative to other determinants of user behavior in echo-chambered environments, suggesting that interventions considering both content format and social dynamics may be needed to foster healthier discourse.

This study contributes an empirical assessment of how journalistic techniques and content types influence online reactions to migration-related news on Facebook. Using annotated comments from 113 posts, the analysis shows that visual formats and news reports foster greater trust in the source, editorial/opinion and data-driven pieces are more often criticized, and data-driven formats are less likely to provoke toxic debate. Comment dynamics are also shaped by social conformity, with tone and stance trending toward previously visible comments. These insights highlight the joint roles of content framing and social influence in online discourse around divisive issues.

• Data access and privacy constraints: Due to Facebook API policy changes (February 5, 2018), users with privacy restrictions are not included; all data are public, aggregated, and anonymized, which may limit user-level inference.
• Scope: Analysis is limited to 113 posts by a single Italian news outlet (Corriere della Sera) on one topic (migration) over March–December 2018, which may limit generalizability to other outlets, topics, or periods.
• Effect sizes: Although associations between framing and commenting behaviors are statistically significant, Cramér’s V indicates small effects; framing is not the only nor the primary factor influencing individual behavior.
• Data availability: In accordance with EU GDPR, datasets are not publicly available; only an anonymized version may be requested from the corresponding author, potentially limiting replication.