YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models

Understanding satire and humor is challenging for current Vision-Language models. This paper introduces three tasks—Satirical Image Detection (classify whether an image is satirical), Satirical Image Understanding (generate the reason the image is satirical), and Satirical Image Completion (given half of an image, select the correct other half from two options so the whole image is satirical)—and releases YesBut, a high-quality dataset of 2,547 images (1,084 satirical and 1,463 non-satirical) spanning multiple artistic styles to evaluate these tasks. Each satirical image depicts a normal scenario alongside a conflicting, funny, or ironic scenario. Despite VL models’ success on multimodal tasks like VQA and captioning, benchmarking shows they perform poorly on the proposed YesBut tasks in zero-shot settings by both automated and human evaluation. An additional set of 119 real satirical photographs is also released for further research.