On Wednesday, Databricks announced Mosaic Agent Brick and Lake Flow Designer, a new tool that makes it easy to develop AI applications using AI.
Agent Brick is an automation framework that not only attempts to develop agents with AI, but also addresses the problems faced by many developers when trying to move them into production. LakeFlow Designer, on the other hand, is a low-code/no-code environment for building extract, conversion, and load (ETL) pipelines for AI development.
Agent Brick is currently undergoing beta testing and previewed by the Lakeflow Designer. Both were introduced at Data + AI Summit, a Databricks user conference held in San Francisco.
These announcements help Databricks users apply AI and Genai applications to their own datasets, thereby gaining a competitive advantage.
According to BARC US analyst Kevin Petrie, the new features are likely to become generally available to Databricks users, given the simplification of AI development.
“These announcements help Databricks users apply AI and Genai applications to their own datasets, thereby gaining a competitive advantage,” he said.
San Francisco-based Databricks is a data management vendor that has grown over the past few years to include environments for users to build AI (Genai) and agent applications.
New Features
Agents are AI applications with inference capabilities and context awareness. Unlike chatbots that can only respond to user prompts, agents can autonomously express insights and perform tasks. Given the potential to make workers more informed and productive, agents are the latest trend in AI.
However, many companies struggle to move agents into production beyond development and testing. Approximately three-quarters of all AI projects have failed and is estimated to include agent AI.
According to Joel Minnick, Vice President of Marketing at Databricks, two main obstacles are high cost and low quality.
Contributing to the high cost and low quality of agents is the lack of a way to benchmark performance under development. This turns into a trial and error exercise, and lacks enough data to notify the application to avoid impeding hallucinations.
“We got that feedback [about cost and quality] A customer said, “Let's solve this.” “Agent Brick is the answer and offers customers a whole new way to build an agent system.”
Agent Bricks essentially automate the development and deployment of agents.
After the user provides a high-level description of the agent's purpose, the mosaic agent brick automatically generates task-specific ratings. They also generate LLM auditors to assess quality, create synthetic data that mimics the company's own data, and apply performance optimization techniques to manage development costs as the agent has the required amount of data.
Finally, this framework provides users with quality and cost-related choices. For example, one option could be an agent that gives 95% accuracy at a certain cost, while the second option would give 85% accuracy but 25% less for development.
Once that selection is made, DataBricks' model serving takes over and automatically produces the agent.
According to Petrie, not only does it help simplify development, but agent bricks are important for who is targeting them. The Databricks and Snowflake have been rivals for a long time. However, while Databricks has historically targeted trained professionals, Snowflake does not target technical users.
“Agent Brick strengthens DataBic's strategy to help data scientists build agent AI applications,” Petry said. “This contrasts with Snowflake, which focuses on providing advance agents and agent templates to unskilled data scientists.”
While Agent Brick aims to help Databricks users deploy agents, LakeFlow Designer is designed to provide non-technical users with some of the same data engineering capabilities that are only available to previously trained professionals.
Lowcode/nocode tools have long been able to perform data modeling and other data management tasks by non-technical users, but are usually limited in scope. Building complex data and AI pipelines with features such as CI/CD (Continuous Integration and Continuous Delivery) generally requires coding skills and other expertise. Furthermore, new demands such as increasing the age and scale of the pipeline are sometimes broken when placed on them.
As a result, data engineers have built pipelines and modified pipelines that may have been developed many years ago, and now they use old tools.
LakeFlow, introduced in June 2024, is a data engineering environment for code-first users.
LakeFlow Designer is a no-code environment that provides the same data engineering capabilities that LakeFlow offers experts, automatically translates natural language instructions into SQL code, and provides the same data engineering capabilities that apply data governance capabilities through the UNITY catalog. Additionally, Databricks Assistant provides AI-powered support, which ensures that queries are well structured, appropriate data is used, and errors are fixed along the way.
Unlike Agent Brick, which caters to a historical audience of Databricks trained experts, Lakeflow Designer is targeting different users, potentially allowing another worker to use the vendor's platform.
“LakeFlow Designer targets technically skilled users, data analysts who need to democratize access to corporate datasets,” he said. “This overlaps more with Snowflake's strategy.”
Like Agent Brick, according to Minnick, customer feedback provided Databricks with the driving force behind developing Lakeflow designers. In particular, it was the customer's desire to make business analysts more independent, while reducing the burden on trained engineers.
In addition to Agent Brick and Lakeflow Designer, Databricks introduced the following new features:
- Full support for the Apache Iceberg table in the Unity Catalog, including native support for the Apache Iceberg Rest Catalog API.
- It supports serverless graphics processing units currently being beta-tested.
- MLFLOW 3.0, an updated version of DataBricks' platform for managing the machine learning lifecycle, and other AI applications now commonly available.
- General availability of lake flows.
- LakeFlow's integrated user interface integrates different tasks previously.
- No-code data intake connector between LakeFlow and Google Analytics, ServiceNow, SQL Server, SharePoint, PostgreSQL, and Secure File Transfer Protocol.
Databricks Data + The new features announced during the AI Summit come a week after rival Snowflake also unveiled the scope of the new features aimed at improving the AI development environment.
According to Petrie, as competition continues to improve with AI after slow reactions to technological interests, rather than DataBricks does by integrating MLFLOW with other features, competition will be less competitive about who will provide a full AI development environment.
“It repeats over time so much that more AI teams can manage the entire lifecycle of their models and applications, from design to training, deployment, monitoring and optimization,” he said. “This iteration is important to adapt to changing business requirements and lessons learned.”
Looking ahead
Just as Agent Brick and Lakeflow Designers use AI to reduce the technical expertise needed to use the DataBricks platform, vendors' product development plans include the development of additional tools aimed at enabling more people within their organization to manipulate data and AI, according to Minnick.
“What we're constantly pursuing is that we keep our tech bar down to be successful,” he said.
According to Petrie, the focus is wise.
Databricks has a strong foundation of already trained professionals. One way to add more customers is to target fewer technical users with more tools, such as LakeFlow Designer, which offers the same functionality as other Databricks tools, but uses a simplified user interface with AI.
“Databricks recommends serving unskilled data scientists and developers by providing advance agents or agent templates,” says Petrie. “This will broaden the addressable market and help you take market share from competitors such as Snowflake.”
Eric Avidon is a senior news writer at Informa TechTarget and a journalist with over 25 years of experience. He covers analytics and data management.
