Meta releases I-JEPA, a machine learning model that learns high-level abstractions from images

Yann LeCun, Chief AI Scientist at Meta, has been talking for several years about deep learning systems that can learn world models with little or no human help. Now, Meta is gradually realizing that vision with the release of the first version of I-JEPA, a machine learning model that learns abstract representations of the world through self-supervised learning of images.

Initial tests have shown that I-JEPA performs well on many computer vision tasks. It is also much more efficient than other state-of-the-art models, requiring 10x less computing resources to train. Meta has open-sourced the training code and model, and he plans to present I-JEPA at next week’s conference on Computer Vision and Pattern Recognition (CVPR).

self-supervised learning

The idea of ​​self-supervised learning is inspired by how humans and animals learn. We acquire much knowledge just by observing the world. Similarly, AI systems should be able to learn through raw observation without the need for humans to label the training data.

Self-supervised learning is pervasive in some areas of AI, such as generative models and large language models (LLMs). In 2022, LeCun proposed a self-supervised model “Joint Predictive Embedding Architecture” (JEPA) that can learn important knowledge such as world models and common sense. JEPA differs from other self-supervised models in important ways.

>>Don’t miss our special issue: Laying the Foundation for Customer Data Quality<

Generative models such as DALL-E and GPT are designed to make detailed predictions. For example, parts of text or images are hidden during training, and the model tries to accurately predict missing words or pixels. The problem with trying to input all the information is that the world is unpredictable and the model is often stuck in the possible outcomes. This is why generative models fail when creating detailed objects such as hands.

In contrast, JEPA learns and predicts high-level abstractions, such as what a scene should contain and how objects relate to each other, instead of pixel-level details. I will try This approach makes the model learn the latent space of the environment, making it less error-prone and significantly less costly.

“By predicting representations at a higher level of abstraction instead of directly predicting pixel values, we hope to avoid the limitations of generative approaches and directly learn useful representations,” said researchers at Meta. wrote.

Aijepa

I-JEPA is an image-based implementation of the architecture proposed by LeCun. It uses an “abstract prediction target that potentially eliminates unnecessary pixel-level detail” to predict missing information, thereby allowing the model to learn more semantic features.

I-JEPA uses Vision Transformers (ViT) to encode existing information. This is a variant of the transformer architecture used in LLM, but modified for image processing. We then pass this information as context to the predictor ViT to generate a semantic representation of the missing part.

Researchers at Meta trained a generative model that creates sketches from the semantic data that I-JEPA predicts. In the image below, I-JEPA was given the pixels outside the blue box as context and predicted the content inside the blue box. The generative model then created a sketch of his I-JEPA predictions. The results show that the I-JEPA abstraction is consistent with the reality in the field.

I-JEPA does not produce photorealistic images, but it has many applications in fields such as robotics and self-driving cars, where AI agents must be able to understand their environment and process some highly plausible results. can be applied.

very efficient model

One of the obvious advantages of I-JEPA is memory and computational efficiency. The pre-training stage does not require the computationally intensive data augmentation techniques used in other types of self-supervised learning methods. The researchers found that using 16 of his A100 GPUs he was able to train a model of 632 million parameters in less than 72 hours. This is about a tenth of the time required by other techniques.

“Empirically, we found that I-JEPA learns strong off-the-shelf semantic representations without the use of handcrafted view extensions,” the researchers wrote.

>>Follow VentureBeat’s Coverage of Continuously Generated AI<

Their experiments show that I-JEPA requires far less fine-tuning to outperform other state-of-the-art models on computer vision tasks such as classification, object counting, and depth prediction. Using only 12-13 images per class, the researchers were able to fine-tune the model on the ImageNet-1K image classification dataset using 1% of the training data.

“By using a simpler model with a less stringent induction bias, I-JEPA is more applicable to a wider range of tasks,” the researchers wrote.

Given the high availability of unlabeled data on the Internet, models such as I-JEPA can prove to be of great value for applications that previously required large amounts of manually labeled data. The training code and pre-trained models are available on GitHub, but the models are released under a non-commercial license.