Computer Science

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

J. Xu, S. Liu, et al.

Discover ODISE, an innovative open-vocabulary panoptic segmentation model that outperforms previous benchmarks in both panoptic and semantic segmentation tasks. This exciting research, conducted by Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello, showcases the potential of text-to-image diffusion models in enhancing semantic representations for diverse categories.

00:00

Playback language: English

Index

Abstract

This paper introduces ODISE, an open-vocabulary panoptic segmentation model that combines pre-trained text-to-image diffusion and discriminative models. It leverages the frozen internal representations of these models to perform segmentation of any category in the wild. ODISE surpasses prior state-of-the-art results on both open-vocabulary panoptic and semantic segmentation tasks, achieving significant improvements on datasets like ADE20K with COCO training. The model's performance highlights the potential of text-to-image diffusion models in learning rich semantic representations for open-vocabulary recognition tasks.

Publisher

arXiv preprint

Published On

Aug 24, 2023

Authors

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello

Related Publications

Explore these studies to deepen your understanding of the subject.

Education

Nurturing the reading brain: home literacy practices are associated with children's neural response to printed words through vocabulary skills

C. Girard, T. Bastelica, et al.

Medicine and Health

The 2022 Massive Open Online Course (MOOC) to train physiotherapists in the management of people with spinal cord injuries: a qualitative and quantitative analysis of learners’ experiences and its impact

J. V. Glinsky, J. Ilha, et al.

Chemistry

Structured information extraction from scientific text with large language models

J. Dagdelen, A. Dunn, et al.

Linguistics and Languages

The impact of text type on Chinese-English translation effort: an investigation with reference to translation entropy

X. Liu and Y. Cui

Listen, Learn & Level Up

Over 10,000 hours of research content in 25+ fields, available in 12+ languages.

No more digging through PDFs, just hit play and absorb the world's latest research in your language, on your time.

listen to research audio papers with researchbunny