"Why is this misleading?": Detecting News Headline Hallucinations with Explanations

The paper addresses the problem of hallucinations in automatically generated news headlines—cases where a headline is not supported by its underlying article. Given the scale of web news and the importance of concise, accurate summaries, such hallucinations pose a deployment challenge. Building a large, labeled dataset is difficult because hallucinations are relatively rare and require careful reading to annotate. The research question is how to detect headline hallucinations effectively with limited labeled data. The authors propose ExHalder, a framework that leverages transfer learning from natural language inference (NLI) and integrates natural language explanations to enhance detection accuracy and interpretability.

The related work covers four areas: (1) News Headline Generation: Earlier extractive approaches and more recent abstractive encoder–decoder methods can produce fluent headlines but are prone to hallucinations, hindering industrial deployment. (2) Hallucination Detection: Mitigation efforts include data cleaning and post-hoc classifiers; the present work falls in the latter, augmenting classifiers with explanation components. (3) Natural Language Inference (NLI): NLI predicts entailment/contradiction/neutrality and has large datasets, enabling transfer to tasks like factuality checking; prior NLI-based faithfulness measures for summarization do not leverage explanations. (4) Natural Language Explanations: Prior studies show explanations can aid learning, especially in low-resource settings, by adding supervision or training models to generate explanations; this work exploits explanations to improve transfer and provide human-readable justifications.

The proposed ExHalder (Explanation-enhanced Headline Hallucination detector) is a text-to-text framework built on an encoder–decoder architecture (T5). It comprises three components: (1) Reasoning Classifier: Input format "headline entailment: headline: <HEADLINE> article: <ARTICLE>" and output "<CLASS> because <EXPLANATION>", where <CLASS> ∈ {Entail, Contradict}. It predicts the label and generates a free-text rationale. (2) Hinted Classifier: Extends the input with the explanation as a hint—"... comment: <EXPLANATION>"—and outputs only the <CLASS>. By conditioning on the explanation, it focuses capacity on classification. (3) Explainer: Conditions on the known class appended to the input—"... <CLASS> because"—and generates the explanation text. Training strategy: To address limited in-domain labels, ExHalder uses (a) NLI-based pretraining with eSNLI and ANLI, mapping hypothesis→headline and premise→article, training all components with teacher forcing to produce labels and explanations; (b) Explainer-augmented training, where the learned explainer generates additional explanations per NLI example to expand supervision for training the reasoning and hinted classifiers; (c) Optional domain fine-tuning on a curated news headline hallucination dataset. Inference: Given ⟨article, headline⟩, the reasoning classifier produces a class probability and explanation; the explanation is concatenated to inputs for the hinted classifier to produce another class probability; a simple average combiner merges probabilities to produce the final prediction with the generated explanation. Implementation uses T5-11B via T5X; hyperparameters include batch sizes (128 pretraining, 64 fine-tuning), constant learning rate 1e-3, and explanation augmentation counts tuned (e.g., 1 during pretraining, 3 during fine-tuning).

- A new expert-curated news headline hallucination dataset was collected with 6,270 ⟨article, headline⟩ examples (train 5,190; validation 349; test 731); 1,934 labeled as hallucinated and 4,336 as entailed; 2,074 include rater explanations. - ExHalder achieves state-of-the-art performance across accuracy, precision, recall, and F1 on the news dataset, outperforming SVM, XGBoost, BERT, T5, and ablations (ExHalder-NoPT, ExHalder-NoEX, ExHalder-NoHC). NLI-based pretraining and explanation signals both contribute significant gains; removing the hinted classifier notably reduces accuracy and recall. - Explainer augmentation shows a sweet spot: using around 3–4 generated explanations per example maximizes performance; beyond that, explanation quality degrades and accuracy drops faster than recall. - Zero-shot transfer: Without in-domain training, ExHalder (and ExHalder-NoEX) outperforms prior best methods (ANLI-only T5 and Q2) on six TRUE benchmark datasets (MNBM, FRANK, QAGS, SummEval, FEVER, Vitamin-C), establishing new state-of-the-art accuracy. ExHalder’s explanations clarify subtle distinctions (e.g., busiest port vs one of the busiest) and support entailment reasoning (e.g., 61/100 implies <62%). - Case studies show ExHalder produces high-quality, human-readable explanations, can surface potential labeling errors, and reveals model failure modes (e.g., temporal reasoning like "March end" vs "March 31").

The study demonstrates that transferring entailment knowledge from NLI and leveraging natural language explanations materially improves detection of hallucinated headlines under limited supervision. The dual-classifier design uses generated rationales to inform a second-stage hinted classifier, boosting recall—critical for minimizing exposure of misleading headlines to users. Explainer augmentation enriches supervision, and zero-shot results show strong generalization across domains (summarization and fact verification) with faithful, interpretable explanations. These findings indicate that explanations serve both as auxiliary training signals and as user-facing justifications, bridging model performance and transparency.

The paper introduces ExHalder, an explanation-enhanced framework for detecting news headline hallucinations. By pretraining on NLI with explanations, augmenting training via a learned explainer, and combining reasoning and hinted classifiers, ExHalder achieves state-of-the-art supervised and zero-shot performance while producing human-readable justifications. Future directions include: learning a stronger combiner using validation data; integrating larger LLMs to improve zero/few-shot performance; extending to multilingual and multi-document scenarios; improving explanation readability and ensuring explanations are themselves entailed by sources; and enhancing temporal and logical reasoning to reduce specific error modes.

- The approach depends on large pretrained models (T5-11B), implying substantial computational resources and potential deployment costs. - Explanations from the explainer can vary in quality; generating too many per example degrades performance, indicating sensitivity to explanation quality. - The combiner uses simple probability averaging without supervised calibration; a trained combiner may be necessary for optimal aggregation. - Some reasoning failures persist (e.g., temporal equivalence like "March end" vs "March 31"), suggesting limited handling of nuanced temporal or lexical equivalences. - The collected dataset focuses on English and single-article grounding; multilingual and multi-document settings remain unaddressed and are left for future work. - Explanations are not constrained to be entailed by the source, raising a risk of unfaithful rationales; enforcing explanation faithfulness is future work.