Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

The research problem addressed in this paper is the challenge of open-vocabulary recognition, where the goal is to recognize and segment objects and scenes with unbounded categories, mirroring human capabilities. While existing methods rely on the generalization ability of text-image discriminative models, they often lack the spatial and relational understanding needed for panoptic segmentation. This paper hypothesizes that text-to-image diffusion models, trained on vast datasets, hold the key to overcoming this limitation. Diffusion models, known for generating high-quality images from text descriptions, have internal representations that correlate with real-world concepts. The paper explores the feasibility of using these models to create a universal panoptic segmentation learner.

The paper reviews related work in panoptic segmentation, open-vocabulary segmentation, and generative models for segmentation. Traditional panoptic segmentation methods operate under a closed-vocabulary assumption, limiting their capabilities to recognizing only categories present in the training set. Open-vocabulary segmentation methods, while addressing this limitation, typically focus on either instance segmentation or semantic segmentation. The paper distinguishes ODISE from these approaches by providing a unified framework for open-vocabulary instance and semantic segmentation. It also differentiates from prior works that rely solely on discriminative models, highlighting the superior performance of diffusion models in learning semantically meaningful representations.

The ODISE model consists of three main components: text-to-image diffusion model, mask generator, and mask classifier. The text-to-image diffusion model, when provided with an image and its caption, generates an internal visual representation that captures rich semantic information. This representation is then fed into the mask generator, which produces class-agnostic binary masks and their corresponding mask embedding features. The mask classifier, utilizing either category labels or image captions, associates each mask embedding with text embeddings of object category names to determine its category. The paper details the architecture of each component, emphasizing the use of an implicit captioner to overcome the need for explicitly captioned image data during training. The implicit captioner generates text embeddings directly from input images, enhancing the model's performance and generalization capabilities.

ODISE significantly outperforms the previous state-of-the-art on both open-vocabulary panoptic and semantic segmentation tasks. Notably, the model trained on COCO achieves 23.4 PQ and 30.0 mIoU on ADE20K, surpassing the previous best by 8.3 PQ and 7.9 mIoU. These results are attributed to the rich semantic representation learned from the text-to-image diffusion model, which proves superior to representations learned from discriminative models. The ablation studies demonstrate the importance of each component, highlighting the effectiveness of the implicit captioner and the fusion of diffusion and discriminative models for mask classification. Additionally, the paper investigates the impact of different diffusion time steps on the extracted visual representations, concluding that using the initial time step (t=0) yields optimal performance.

ODISE's success underscores the potential of text-to-image diffusion models in learning robust semantic representations for open-vocabulary recognition tasks. The model's ability to segment objects and scenes with unbounded categories establishes a new state of the art in this field. The findings suggest that diffusion models not only excel at generating high-quality images but also capture deep semantic understanding that can be effectively leveraged for other vision tasks.

ODISE introduces a novel open-vocabulary panoptic segmentation model that harnesses the power of text-to-image diffusion models. The model demonstrates superior performance compared to existing methods, showcasing the effectiveness of diffusion models in capturing semantic information. This work opens new avenues for utilizing the internal representation of diffusion models for various visual tasks, advancing open-vocabulary recognition and other computer vision applications.

The paper acknowledges the limitations of the current datasets, where ambiguous and non-exclusive category definitions can influence evaluation accuracy. Additionally, the use of pre-trained models based on web-crawled data may introduce potential biases in the internal representations.