Zilliz, a leading AI data infrastructure company and creator of Milvus, recently announced: Abia new storage engine that powers Zilliz Vector Lakebase, ships with Milvus 3.0. Loon is Lake-Native’s foundation for running real-time searches, large-scale discovery, and batch analytics simultaneously on a single copy of vector data, and the storage layer behind Zilliz Cloud’s evolution from a vector database to a unified data platform for AI.
Also read: AiThority interview with Matej Bukovinski, Chief Technology Officer at Nutrient
“Vector Lakebase is our answer to what happens after a successful Vector database.”
Vector Lakebase is built on the strict premise that one logical copy of vector data should serve all AI workloads (search, discovery, batch analytics in production) without copying or moving data between systems. The most difficult part is the storage layer. This is because the same dataset needs to behave like two systems at the same time. This means fast record-level lookups for service delivery and extensive scanning for analysis, all on inexpensive object storage. You also have to deal with constantly changing data as your team re-embeds, re-labels, and re-indexes the same records while improving the model.
“Searching for vectors is no longer the be-all and end-all. Vector Lakebase is our answer to what happens after a successful vector database,” said James Luan, co-founder and CTO of Zilliz. “Successful systems make continuous service and continuous discovery feel like part of the same machine, and that only works if the storage layer can provide a single copy of the data to all workloads. Loon is that storage layer.”
Storage built for evolving AI data
To make that possible, Loon treats vector datasets as real, physically heterogeneous objects, and is built on three ideas:
- Hybrid file format: Each type of column is stored in its appropriate format. Scalar and filter fields use Parquet for efficient scanning. Dense and sparse vectors use open Vortex format for fast, byte-accurate row-level reads on object storage. Additionally, raw videos, PDFs, and images remain in object storage and are referenced rather than copied to the database.
- Row ID arrangement: Columns separated into different formats still behave as one logical table, so new embedded models can be added as their own columns without rewriting the captions, metadata, and vectors that are already stored.
- Versioned manifest: A single source of truth defines the current version of a dataset (files, indexes, deletion logs, statistics), so serving clusters, on-demand compute, and external engines like Spark and Ray can all read and safely update the same dataset instead of maintaining separate copies.
Together, these allow one copy of data on object storage to feed many engines at once. In Zilliz’s internal testing of object storage, Loon’s Vortex-based layout resulted in approximately 135 times less data retrieved per record read compared to Parquet. This is the difference between low-latency services on cheap object storage being practical and impractical. And since the same data evolves on the fly, adding a new embedded model is a lightweight version update rather than a hundreds of gigabytes of rewrite.
This is the architecture behind Vector Lakebase. Real-time serving clusters stay fast and stable. Flexible on-demand computing performs discovery and batch analysis without impacting production. External collections index data in a customer’s own S3 or GCS bucket. Everything is on one semantic foundation, with no duplicate pipelines or ETL. This is the same Milvus and Zilliz Cloud foundation that over 10,000 companies and AI native teams are already building on, including MiniMax, OpenEvidence, Filevine, Exa, and Salesforce.
Also read: AI Systems – Interoperable AI Systems: Connecting models across platforms
[To share your insights with us, please write to psen@itechseries.com]
