Meta announced Segment Anything Model 2 (SAM2), a new machine learning model aimed at improving video segmentation capabilities. The announcement was made at SIGGRAPH by Meta CEO Mark Zuckerberg, joined by Nvidia CEO Jensen Huang.
Video segmentation improvements
Building on a previous model that focused on still images, SA2 extends these capabilities to video. The original model efficiently identified and outlined objects in images. The new iteration aims to replicate this effectiveness on video frames, a much more resource-intensive task. Zuckerberg highlighted potential applications in scientific research, such as studying coral reefs and natural environments. The model employs zero-shot learning, allowing it to identify objects even without prior examples.
Processing video data requires significant resources. SA2 aims to address these demands without taxing data centers. Meta plans to make SA2 freely available, mirroring the policy of the original model, and has already released a demo. In addition, Meta is providing the annotated database containing 50,000 videos that was used to train SA2. A separate database containing over 100,000 videos was also used, but will not be made public. We have contacted Meta for more details on these sources and their decision to keep parts of the dataset private.
Meta's Open AI Strategy
Meta has established itself in the open AI space with tools like PyTorch and models like LLaMa and Segment Anything, and Zuckerberg explained that the goal of open sourcing these models is strategic, not altruistic, to establish an ecosystem that will improve their effectiveness.
The first “Segment Anything Model” (SAM) was introduced in April 2023 as a foundational model for image segmentation and has been highly praised in the computer vision community. SAM 2 was trained using the new SA-V dataset, the largest publicly available dataset for video segmentation. The SA-V dataset contains 50,900 videos with 642,600 mask annotations for a total of 35.5 million distinct masks, which is 53 times larger than existing datasets. With nearly 200 hours of annotated video content, SA-V sets a new standard for training data.
Technical features
SA2 employs a Transformer-based architecture and incorporates a memory module that retains information about objects and previous interactions across video frames. This functionality allows the model to track objects through extended sequences and respond to user input. When applied to still images, the memory module is inactive and the model functions similarly to previous models.
During testing, SA2 achieved improved segmentation accuracy with one-third less user interaction compared to previous methods. Meta reports that the model surpasses current benchmarks for video object segmentation and also outperforms the original SAM model on image segmentation tasks, achieving six times speedups. Inference speed is 44 frames per second, very close to real-time performance.
But SA2 has limitations: it can lose track of objects after scene cuts or long occlusions, have difficulty tracking very fine details, and have trouble tracking individual objects within a group of similar moving objects. Researchers suggest that explicit motion modeling could help solve these issues.
