Introducing SegGPT: A generalist model to perform arbitrary segmentation tasks on images or videos via in-context reasoning

Segmentation is one of the most fundamental challenges in computer vision, which seeks to find and reorganize important concepts such as foreground, categories, and object instances at the pixel level. Considerable progress has been made in recent years on various segmentation tasks such as foreground segmentation, interactive segmentation, semantic segmentation, instance segmentation, and panoptic segmentation. However, these expert segmentation models are limited to specific tasks, classifications, granularities, data formats, etc. New models need to be trained as they adapt to new environments, such as segmenting new concepts or objects in videos instead of photos.

Our goal in this work is to train a single model that can handle an infinite variety of segmentation tasks. This requires time-consuming annotation work and should be more sustainable for many segmentation jobs. The main problem lies in his two areas: (1) Incorporating vastly different data types into training, such as parts, semantics, instances, panoptics, people, medical images, and aerial images. (2) to create a generalizable training scheme that is flexible in task definition and can handle tasks outside its scope, unlike traditional multitask learning; To overcome these problems, researchers from Peking Academy, Zhejiang University, and Peking University introduced his SegGPT, a generalist paradigm for segmenting anything in context.

We integrate many segmentation tasks into generalist in-context learning frameworks and view segmentation as a general form of vision. This framework can handle different segmentation data types by converting them to the same image format. By using random color mapping for each data sample, the SegGPT training problem is expressed as an in-context coloring problem. The goal is to color only relevant areas such as classes, object instances, components, etc. by context. Employing a random color scheme forces the model to look at contextual data to do a specific job, rather than relying on a particular shade. This allows you to approach your training in a more adaptive and general way.

The rest of the training components remain the same when employing standard ViT and simple Smooth-l1 loss. Following training, SegGPT uses in-context inference to perform various segmentation tasks on photos and videos given several instances of object instances, materials, parts, contours, text, etc. This helps the model take advantage of multiple-example prompt scenarios. By tailoring customized prompts for special use cases, such as ADE20K semantic segmentation in the domain, SegGPT could easily work as a specialist model as well, without changing model parameters.

These are their main contributions.

(1) They demonstrate for the first time a single generalist model that can automatically complete a wide range of segmentation tasks.

(2) evaluate pre-trained SegGPT directly, i.e. without fine-tuning, for various tasks such as few-shot semantic segmentation, video object segmentation, semantic segmentation and panoptic segmentation;

(3) Both subjectively and statistically, their results demonstrate good skill in segmenting targets inside and outside their domain. Nevertheless, their work believes that general-purpose models may not be able to handle certain tasks, thus achieving new state-of-the-art results on all benchmarks or superseding existing specialized approaches. We do not promise to exceed.

check out paper, planand githubdon’t forget to join Our 19k+ ML SubReddit, cacophony channeland email newsletterWe share the latest AI research news, cool AI projects, and more. If you have any questions about the article above or missed something, feel free to email me. Asif@marktechpost.com

🚀 Check out 100 AI Tools in the AI ​​Tools Club