Netflix recently unveiled how its Machine Learning Platform (MLP) team is delivering an ecosystem around Metaflow, an open source machine learning infrastructure framework. By creating various integrations for Metaflow, Netflix can support hundreds of his Metaflow projects maintained by multiple engineering teams.
Metaflow's integration with Netflix's production systems allows you to move projects from prototype to production without incurring unsustainable operational overhead. The engineering team explains the key to success:
The ML and AI use cases we support are extremely diverse; […], we don't expect all projects to follow the same path from prototype to production. Instead, it provides a robust foundation layer with integrations with enterprise-wide data, compute, and orchestration platforms, and multiple paths to smoothly deploy applications into production. In addition to this, the team has built their own domain-specific libraries to support specific use cases and needs.
One of the integration examples provided is Metaflow's “Fast Data” library. Netflix hosts their main data lake as Apache Iceberg tables on S3 and uses Apache Spark for ETL. The Fast Data library enables fast, scalable, and robust access to the Netflix data warehouse by leveraging high-performance components of the Python data ecosystem. The library enables Netflix to collectively process terabytes of data and encode complex relationships between titles, actors, and other movie attributes to support a wide range of the company's business applications.
Metaflow High Speed Data Library (Source)
Maestro, Netflix's production workflow orchestrator, plays a key role in managing Metaflow projects in production. It supports scalability and high availability, and through event triggers it enables seamless integration with Metaflow flows and other systems. This integration allows Netflix engineers to support content decision-making and answer the question, “What content should Netflix bring to the service?”
Finally, for deployments that require APIs and real-time evaluation, Netflix offers Metaflow Hosting, an integrated model hosting service. Metaflow Hosting provides an easy-to-use interface on top of Netflix's existing microservices infrastructure, allowing data scientists to quickly move their work from experimentation to production-grade web services available via HTTP REST APIs with minimal overhead. I will make it possible.”
Using this integration, Netflix hosts and scales models to compute characteristics of various media assets. Once features are found, the consuming service can save them in a store for future use. An old talk provides an overview and details of this service.
Hosting and using media feature calculation models (source)
Netflix implemented the integration using Metaflow's extension mechanism. “Although this is public, it is subject to change and is not yet part of Metaflow's stable API.” They asked engineers in the Metaflow community Slack to discuss building additional extensions. We encourage you to contact us.
