"Why is this misleading?": Detecting News Headline Hallucinations with Explanations

The increasing prevalence of automated news headline generation necessitates addressing the issue of "hallucinations," where generated headlines lack support from the original news articles. This problem poses a significant challenge for deploying such systems at scale, as misleading headlines can negatively impact user experience. The rarity of hallucination cases and the need for careful human review make creating large, labeled datasets for training hallucination detection models difficult. This paper tackles this challenge by introducing ExHalder, a framework designed to detect headline hallucinations accurately and provide human-readable explanations for its predictions. The increasing demand for quick and efficient news consumption through headline generation highlights the critical need for accurate and reliable headline summarization. Abstractive methods, while improving headline quality, often generate headlines unsupported by the source article. This paper addresses the critical challenge of detecting these 'hallucinations' through the development of a novel framework. Early methods relied on extractive techniques, selecting and reorganizing words from article titles. However, more recent abstractive methods using encoder-decoder architectures generate headlines directly but still suffer from the hallucination problem, generating headlines that contradict or misrepresent the original article. ExHalder aims to solve this by utilizing transfer learning from NLI datasets and incorporating natural language explanations to enhance model accuracy and provide insights into the model's reasoning process. The infrequent occurrence of these hallucinations and the need for thorough human review to label them present a significant hurdle to building reliable detection systems. Therefore, the paper proposes a framework that can be effective even with limited labeled data.

The paper reviews related work in news headline generation, hallucination detection, natural language inference (NLI), and natural language explanation. In news headline generation, the shift from extractive to abstractive methods using encoder-decoder architectures is noted, along with the growing concern about hallucination. Existing approaches to hallucination detection focus on data cleaning or post-processing generated content. The paper highlights the relevance of NLI, as both NLI and headline hallucination detection involve determining if one text is supported by another. However, unlike previous studies using NLI models for summarization faithfulness, ExHalder leverages explanation information. The use of natural language explanations to improve machine learning models is also discussed, emphasizing their potential value in low-resource settings. The literature review establishes a strong foundation for the proposed work by highlighting the existing gaps and challenges in the field. Existing work on news headline generation primarily focuses on improving the quality and fluency of generated headlines. However, less attention has been paid to the problem of factual consistency between generated headlines and their source articles. Hallucination detection in natural language generation is a relatively new area, and existing methods often rely on large amounts of labeled data which are not readily available for the specific task of news headline hallucination detection. Natural language inference (NLI) has emerged as a powerful technique for measuring the semantic relationship between two sentences. Several large-scale NLI datasets have been created, and pre-trained models on these datasets have been successfully applied to various downstream tasks. The use of natural language explanations to improve the performance of machine learning models is gaining increasing interest. Explanations can provide valuable insights into the decision-making process of models and help improve model interpretability and explainability. By drawing on these related areas, the authors lay the groundwork for their proposed framework.

ExHalder, the proposed framework, consists of three key components: a reasoning classifier, a hinted classifier, and an explainer. The reasoning classifier takes an article-headline pair as input and outputs a class label ('Entail' or 'Contradict') along with a natural language explanation. The hinted classifier receives the article-headline pair and the explanation generated by the reasoning classifier, providing an additional prediction. The explainer component generates the natural language explanation. The framework addresses the challenge of limited labeled data by employing two key strategies. First, it leverages pre-training on large-scale public NLI datasets (eSNLI and ANLI) to transfer knowledge to the news domain. Second, it utilizes an explainer-augmented training approach where the explainer generates additional explanations to enrich the training data. The encoder-decoder architecture forms the backbone of these components. The encoder processes the input sequences (article and headline) creating vector representations. The decoder generates the output sequences: the class label and the explanation. For the reasoning classifier, the input is formatted as “headline entailment: headline: <HEADLINE> article: <ARTICLE>” and the output as “<CLASS> because <EXPLANATION>”. The hinted classifier takes the explanation as an additional input: “headline entailment: headline: <HEADLINE> article: <ARTICLE> comment: <EXPLANATION>” and outputs only the class label. The explainer takes as input the article-headline pair along with the class label to generate a free-text explanation. The training process involves first pre-training all three components using NLI datasets, where each example is a triple of <hypothesis, premise, label>. Then, explainer-augmented training uses the trained explainer to generate more explanations, enriching the original dataset. An optional fine-tuning step on news-domain specific data can further enhance the framework's performance. During inference, the reasoning classifier initially predicts and provides an explanation. This explanation is fed to the hinted classifier for another prediction. Finally, the two predictions are aggregated (in this case, averaged) to produce the final prediction with its explanation. The architecture uses a standard encoder-decoder approach, leveraging the power of transformer networks for encoding the input text and generating the class label and explanation. The use of pre-training and explainer-augmented training are crucial to address the limited amount of labeled data available for the headline hallucination detection task. The choice of combining predictions from the reasoning and hinted classifiers uses a simple averaging strategy but opens opportunities for more sophisticated combining methods in future work.

ExHalder achieves state-of-the-art performance across various metrics (Accuracy, Precision, Recall, and F1 score) on seven datasets, including one newly collected news headline hallucination dataset and six public ones. The NLI-based pretraining and explanation information significantly contribute to its performance. Ablation studies highlight the importance of each component. Removing the hinted classifier substantially decreases performance, particularly recall, demonstrating its value in refining predictions. Varying the number of explanations generated by the explainer reveals an optimal range (around 3-4) after which the quality of generated explanations decreases. The generated explanations are shown to be high-quality and human-readable, closely resembling human-written explanations and enabling the identification of potential annotation errors. ExHalder also provides explanations in cases where human raters did not provide comments. The evaluation is based on a newly collected dataset with over 6,000 labeled article-headline pairs, and six public datasets covering a wide range of domains and tasks, from natural language inference to fact verification. The results demonstrate that ExHalder's ability to transfer knowledge from NLI tasks and its use of explanations significantly improve performance. Across all tested datasets, ExHalder shows superior performance compared to existing methods, often by a considerable margin. The paper also performs extensive ablation studies to analyze the effect of each component of ExHalder on the overall performance. The experiments demonstrate the effectiveness of ExHalder in detecting headline hallucinations. Case studies show the quality of explanations, including instances where ExHalder identifies human annotation errors and provides explanations for the model's reasoning in cases of mispredictions. Even in zero-shot settings with no in-domain training data, ExHalder achieves state-of-the-art results, showcasing its strong generalization capabilities and the effectiveness of pre-training on NLI datasets.

The findings demonstrate the effectiveness of ExHalder in addressing the challenge of news headline hallucination detection, particularly in low-resource settings. The use of transfer learning from NLI datasets and the incorporation of natural language explanations prove highly beneficial. The high-quality explanations generated by ExHalder enhance model interpretability and provide valuable insights into its decision-making process. The ability to identify annotation errors suggests potential applications in improving the quality of labeled datasets. While simple averaging is used for combining predictions, this area can be further refined for improvement. The strong zero-shot performance highlights the robustness and generalizability of the framework. The results validate the key ideas behind ExHalder: transferring knowledge from related tasks and leveraging natural language explanations to enhance model performance and interpretability. The success of the framework in both supervised and zero-shot settings demonstrates the potential for broad applicability and significant impact on improving the reliability of automated news headline generation systems.

ExHalder offers a novel approach to automatically detect news headline hallucinations using limited labeled data. Its key contributions include a framework that leverages NLI datasets and generates human-readable explanations, a new dataset curated by news-domain experts, and state-of-the-art results on multiple datasets. Future research could explore more sophisticated methods for combining predictions, integrating large language models, expanding to multilingual settings, and improving explanation formatting and quality. Addressing multi-document headline hallucination is another promising area for future investigation. The work offers a valuable contribution to improving the quality and reliability of automated news headline generation.

The current implementation uses a simple averaging strategy for combining predictions from the reasoning and hinted classifiers. More sophisticated combination methods could potentially further improve accuracy. The quality of generated explanations, while generally high, can still be subjective and might be improved through more advanced techniques or by incorporating additional linguistic constraints. The study is primarily focused on English language news headlines. Extending the framework to support other languages would enhance its broader applicability. The dataset used was carefully collected but might not fully capture the diversity and complexity of real-world news headlines. Future work could benefit from a more comprehensive dataset representing a larger range of headline types and writing styles.