Accurate segmentation of multiple objects is essential for various scene understanding applications such as image/video processing, robot recognition, and AR/VR. The Segment Anything Model (SAM) was recently released. This is the basic vision model for extensive image segmentation. It was trained using billion scale mask labels. Using a set of points, bounding boxes, or coarse masks as input, SAM can segment different objects, components, and visual structures in multiple contexts. Its zero-shot segmentation capabilities caused a rapid paradigm shift as it can be used in many applications with just a few basic prompts.
Despite SAM’s excellent performance, SAM’s segmentation results still need improvement. Two key problems plaguing SAM: 1) Segmentation of thin object structures with rough mask boundaries is frequently omitted, as shown in Figure 1. 2) erroneous predictions, damaged masks or, in difficult cases, serious inaccuracies; This is often related to SAM’s tendency to misread thin structures, such as the kite line in the top right column of the figure. These errors severely limit the applicability and effectiveness of basic segmentation techniques such as SAM, especially for automated annotation and image/video editing jobs where highly accurate picture masks are essential.
Figure 1: C Compare prediction masks for SAM and HQ-SAM using a single red box or many point-on-object input prompts. With highly accurate boundaries, HQ-SAM produces more detailed results. In the rightmost column, SAM misinterprets the thin structure of the kite line and produces a number of errors resulting in broken holes in the input box prompts.
ETH Zurich and HKUST researchers propose an HQ-SAM that can predict highly accurate segmentation masks even in the most challenging conditions while retaining the robust zero-shot capabilities and flexibility of the original SAM ( (see Figure 1). They suggest slightly adjusting the SAM by adding parameters less than 0.5% to increase the ability for high-quality segmentation while maintaining efficiency and zero-shot performance. The general configuration of zero-shot segmentation is greatly hampered by directly tuning the SAM decoder or adding new decoder modules. Therefore, they propose that the HQ-SAM design fully retains the zero-shot efficiency and can be integrated and reused with currently learned SAM structures.
In addition to the original prompt and output tokens, a learnable HQ output token is created that feeds into the SAM’s mask decoder. The HQ output token and its associated MLP layer are taught to predict a high quality segmentation mask as opposed to the original output token. Second, the HQ output token works on an improved feature set to produce accurate mask information, rather than using only SAM’s mask decoder capabilities. It combines the mask decoder capabilities of SAM with the early and late feature maps of the ViT encoder to use global semantic context and fine-grained local features.
The full pre-trained SAM parameters are frozen during training, and only the HQ output tokens, associated 3-layer MLPs, and small feature fusion blocks are updated. Learning accurate segmentation requires a dataset containing accurate mask annotations of various objects with complex and intricate shapes. The SA-1B dataset contains 11 million photos and 1.1 billion of his masks that were automatically created using a model similar to SAM and will be used to train SAM. However, the SAM performance in Figure 1 shows that using this large data set has significant economic impact. I am unable to generate the required high quality mask generation that I am researching.
As a result, they created a new dataset, HQSeg-44K, consisting of 44K highly fine-grained picture mask annotations. Six existing image datasets are combined with highly accurate mask annotations to create HQSeg-44K spanning over 1,000 different semantic classes. HQ-SAM can be trained on eight RTX 3090 GPUs in less than four hours thanks to its smaller dataset and simple integrated design. They conduct rigorous quantitative and qualitative experimental studies to validate the effectiveness of HQ-SAM.
We compare HQ-SAM and SAM on a collection of nine different segmentation datasets from various downstream tasks. Seven of them are under zero-shot transfer protocols including COCO, UVO, LVIS, HQ-YTVIS, BIG, COIFT and HR. -SOD. This exhaustive analysis shows that the proposed HQ-SAM can produce masks with a larger aperture while having zero-shot capability compared to SAM. A virtual demo can be found on the GitHub page.
First high-quality zero-shot segmentation model by introducing negligible overhead to the original SAM
please check out Paper and Github. don’t forget to join 23,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email us. Asif@marktechpost.com
🚀 Check out 100’s of AI Tools at the AI Tools Club
Aneesh Tickoo is a consulting intern at MarktechPost. He is currently pursuing his Bachelor of Science in Data Science and Artificial Intelligence from the Indian Institute of Technology (IIT), Bhilai. He spends most of his time working on projects aimed at harnessing the power of machine learning. His research interest is in image processing and he is passionate about building solutions around it. He loves connecting with people and collaborating on interesting projects.
