Databricks is increasing its use of open data formats with significant advancements in Apache Iceberg™ within the Unity catalog. The company announced the general availability of Managed Iceberg, Iceberg v3, and Foreign Iceberg, positioning Unity Catalog as a comprehensive, production-ready solution for open lakehouses. This move signals an aim to improve data interoperability and governance across different engines.
Visual TL;DR. Open Data Lakehouse supports Databricks Unity Catalog. Databricks Unity Catalog achieves Iceberg GA. Iceberg GA includes five pillars. Iceberg GA enables interoperability and governance. Five pillars enhance interoperability and governance. Five pillars enable AI-driven optimization. Interoperability and governance enable production-ready solutions. AI-driven optimization contributes to production-ready solutions.
Open data lakehouse: The growing need for managed data access across diverse engines
Iceberg GA: Managed, v3, and Foreign Iceberg now generally available
Five Pillars: Open APIs, Federation, Access Control, Sharing, and AI Optimization
Interoperability and Governance: A unified view and consistent governance across your data assets
AI-driven optimization: Improving the performance of AI and agent applications
Production-ready solutions: Comprehensive and production-ready solutions for open lakehouses
Visual TL;DR
The platform currently boasts five core features designed to set it apart. Open APIs for engine flexibility, catalog federation for a unified view across disparate data assets, cross-engine access controls for consistent governance, zero-copy secure sharing, and performance optimization with AI. These capabilities are aimed at addressing the growing need for managed data access across a growing ecosystem of AI and agent applications.
Iceberg features generally available
The latest update brings a set of Iceberg features generally available and previewed. Managed Iceberg allows users to create, read, write, and optimize Iceberg tables directly within the Unity catalog, with features such as predictive optimization and liquid clustering that automate performance tuning. Iceberg v3 support is now native and includes delete vectors, row tracking, and VARIANT types across managed and external tables. External Iceberg support is also generally available, enabling governance and querying of externally managed Iceberg tables. Databricks is also introducing Iceberg-compatible materialized views and cross-engine attribute-based access control (ABAC) in beta.
Catalog federation is expanding with new connectors for Google Cloud Lakehouse and Palantir, in addition to existing integrations with AWS Glue, Snowflake Horizon, Hive Metastore, and Salesforce Data Cloud. The intent is to make Unity Catalog the central management layer for an organization’s entire Iceberg data estate.
Five pillars of interoperability
Databricks maintains that a truly open lakehouse catalog must provide more than basic metadata tracking. Unity Catalog addresses five core requirements: open APIs, federation, cross-engine governance, secure sharing, and continuous performance innovation.
With open APIs and automated credential sales, Iceberg-compatible clients, from Spark to DuckDB, can interact with tables in the Unity catalog without duplicating data or extensive storage permissions. The platform also provides credentials for federated Iceberg tables, enhancing secure access to externally managed data.
Catalog federation provides a single view across multiple catalogs, such as AWS Glue and Snowflake Horizon, and allows users to directly manage and query external Iceberg tables. This integrated approach simplifies the management of complex distributed data environments.
Cross-engine ABAC, currently in beta, extends fine-grained governance policies. Administrators define policies once in the Unity Catalog, which are then applied by the external Iceberg engine via the Iceberg REST catalog scanning API, ensuring consistent access control regardless of the query engine.
Delta sharing enhances zero-copy secure sharing and now fully supports Iceberg as both source and destination format. This enables secure live data sharing with Iceberg REST compatible clients without the need for data ingestion or copying. External Iceberg sharing is also in public preview and enables managed sharing of externally managed Iceberg tables.
Performance and format innovation is driven by AI. Predictive Optimization automatically adjusts table performance based on workload patterns, benefiting all engines accessing your data. The integration of Iceberg v3 features such as deletion vectors and row tracking aims to bridge the performance gap between Delta and Iceberg and enable interoperability without rewriting data.
The company sees a future where open lakehouse catalogs like Unity Catalog will be critical to managing data across systems, especially as AI and agent applications become more pervasive. According to Databricks, the integration of Iceberg v4 and Delta 5.0 into a unified metadata structure is expected to resolve the long-standing trade-off between interoperability and production-ready performance.
