- executorch This is the Pytorch inference framework for edge devices developed by Meta with support from industry leaders such as ARM, Apple, Qualcomm and more.
- Machine learning (ML) models of running-on devices are becoming increasingly important to Meta's Family of Apps (FOA). These device models improve latency, maintain user privacy and enable offline functionality by retaining data on the user's devices.
- It presents some of the AI features of devices serving billions of people on Instagram, WhatsApp, Messenger and Facebook.
- These rollouts significantly improved the performance and efficiency of device ML models in Meta's FOA, and eased the research.
Over the past year, we've unfolded executorchFamily of Apps (FOA) for on-device inference on mobile and edge devices, and significant improvements in model performance, privacy enhancements, and latency for the previous device machine learning (ML) stack.
That's what executorch was Builds in collaboration with industry leaders Use Pytorch 2.x technology to transform your models into a stable, compact representation for efficient on-device deployment. Compact runtime, modularity, and scalability allow developers to easily select and customize components. Ensures portability across the platform, compatibility with Pytorch, and high performance.
By adopting Executorch, we have helped to enhance the user experience with products and services used by billions of people around the world.
Below are just a few of the various ML models of apps on Android and iOS devices supported by Executorch.
Enable cutouts on Instagram
Cropping It is one of the latest features of Instagram's creative expression and storytelling. People can convert photos and videos of their favorite moments into animation-based stickers that can be shared on reels and stories. I've migrated the cutout function on Instagram and executed it in Executorch Squeeze Samlightweight version Meta Segment Anything Model (SAM). On both Android and iOS, Executorch is significantly faster than the older stack, converting it to an increase in daily active users (DAUs) for Sutouts.
Improve the quality of WhatsApp videos and calls
WhatsApp needs to be usable and reliable regardless of the bandwidth of your network connection. To achieve this, we developed bandwidth estimation models for different platforms. These models help you detect and take advantage of available network bandwidth and optimize video streaming quality without compromising the smoothness of your video calls.
These models should be very accurate and run as efficiently as possible. By leveraging executorch, improvements in the bandwidth estimation model in performance, reliability and efficiency metrics were observed. Specifically, we reduced the apps rather than responsive (ANR) metrics, while significantly reducing model load times and average inference times. Along the way, we've further improved security assurance compared to the older Pytorch mobile framework. Fuzzing Testwhich involves providing invalid or random input to the program and monitoring exceptions. Using positive signals from these releases, several other important WhatsApp models are also migrating to Executorch, including those for noise cancellation and video enhancement of devices.
Delivery of device ML for end-to-end encryption of messenger
Messenger End-to-End Encryption (E2EE) We guarantee that no one other than you and the people you are talking to will see your message, even in meta. Executorch can enable E2EE in Messenger and continue to encrypt data transfers by moving server-side models and running on-device.
To enable E2EE, I migrated and deployed several models, including the on-device language identification (LID) model of the messenger. LID is a messenger model that detects the language of a given text and enables a variety of downstream tasks, including translation, message summary, and personalized content recommendations. With executorch, the lid on the device is significantly faster and saves server and network capacity.
To save Messenger's E2EE environment, I leveraged Executorch to move other messenger models to the on-device, including one to optimize video call quality (similar to WhatsApp's bandwidth estimation model) and one to optimize Image Sutout (similar to Instagram cutout). These shifts have increased infrastructure efficiency by freeing up capacity and allowing these capabilities to be expanded globally.
Facebook background music recommendations
Facebook employs a core AI model called SceneX that performs a variety of tasks, such as image recognition/classification, captioning, creating AI-generated backgrounds for images, and image safety checks. By shifting the scene to Executerch, we can now enhance people's Facebook stories by proposing background music based on images.
The executorch rollout has improved SceneX performance across boards, from low to high-end devices compared to older stacks. Several other models are at different stages of A/B testing, such as improving image quality and performing background noise reduction during calls.
Building the future of device AI with the executorch community
We hope that the results of using Executorch to see leverage to solve some of the challenges of ML on a large meta will encourage other industries. I encourage you to contribute to Executorch And share feedback about us github page. You can also join a growing community Executorch Discord Server.
We look forward to driving more innovation with Device ML and shaping the future of Device AI with our community.
