Data warehousing gave birth to business intelligence.
This is a point that analyst Mike Ferguson made to me in a memorable way when I interviewed him for a story celebrating Computer Weekly's 50th anniversary.
“We needed a data warehouse and we were targeting the BI market,” he said in 2016. “Until then, reports were based on green-and-white printed sheets spitting out of transactional database systems.”
In the 1990s and 2000s, data warehousing, ETL, and business intelligence software all worked together to bring about a dramatic shift in data analysis.
Computer Weekly is two years away from its 60th anniversary. A similar story can be said about the new generation of business intelligence. Call it what you like. Maybe it's Neo-BI, maybe it's Generative AI-driven business intelligence. But there's a logical parallel. What data warehouses were to business intelligence (Cognos, Business Objects, Microsoftategy, and relatively new vendors like Qlik, Tableau, Thoughtspot, etc.), 15 years of Hadoop data stores, data lakes, and even data lakehouses are to the new generation of BI and knowledge management that's just emerging. It's Generative AI-driven.
Databricks calls the “new type of business intelligence” that its data platform produces “AI/BI,” and the company recently announced it at the Data+AI Summit (previously known as Spark Summit) in San Francisco. One part of this is a conversational interface called “AI/BI Genie.”
The provider claims that its product “democratizes analytics and insights to everyone in an organization.”
Databricks AI/BI comes with complementary experiences: “Dashboard is an AI-powered, low-code interface for creating and distributing fast, interactive dashboards; Genie is a conversational interface for addressing ad-hoc and follow-up questions through natural language.”
Both are said to be powered by a composite AI system that “continuously learns usage across an organization's data stack, including ETL pipelines, lineage, and other queries.”
Composite AI
What Databricks co-founder and CTO Matei Zaharia and his colleagues call a “composite AI system” is one that “uses multiple interacting components to tackle an AI task, including multiple calls to models, retrievers, or external tools.” In contrast, an AI model is simply a statistical model, for example, a Transformer that predicts the next token in a text.
Just as the original Business Intelligence provided data in the form of reports to business professionals in finance, HR, operations, sales, etc., AI/BI targets a broader group of business users. Databricks was founded by academics and is well known as a technology provider beloved by data engineers, ML Ops engineers, data scientists, and other tech enthusiasts.
“A truly intelligent BI solution needs to understand the semantics and nuances specific to the business to effectively answer business users' questions,” said Ali Ghodsi, co-founder and CEO of Databricks, in a press statement announcing AI/BI.
“We believe this requires a different approach to BI software design than has been seen to date – one that puts AI systems at the center of the architecture, designed to leverage their strengths and compensate for their weaknesses to address the challenge of understanding and learning these nuances. The release of AI/BI is a step towards building just such a system.”
In the same statement, Felix Baker, head of data services at SEGA Europe, also voiced his support: “At SEGA, we aim to entertain the world with creative and innovative experiences, and data intelligence plays a key role in helping us achieve that goal. We use Databricks AI/BI to empower decision makers to ask ad-hoc questions about consumer behavior in real time, without relying on data experts to create dashboards and queries. Now, our team members can get deep insights into our game sales and gameplay data by simply asking questions in natural language.”
“AI/BI will enable the democratization of data, improve productivity and speed data-driven decision-making across SEGA.”
During our conversation at the event, Richard Tomlinson, Director of Product Marketing at Databricks, gave us some product context that's worth replicating.
He said, “We're trying to marry business intelligence with artificial intelligence. We've always offered a way for customers to create lightweight dashboards in Databricks as part of our SQL offering. More and more customers started using it and loved it, so we put a lot of engineering into it.”
“Then, with the Gen AI revolution, we started thinking, 'What if we could use the LLM to fundamentally redesign and reboot business intelligence?'
“Genie is a chat-like interface that's made up of five or six LLMs. When you ask a question, first it shows you what Genie is thinking, which is that all the LLMs have a little chat and figure out how to answer the question correctly. And another great thing about Genie is that if it's less than 95% sure of the answer, it's explicitly built to just say, 'I don't know. Can you explain?' What it doesn't know might be a business concept like churn rate. So you tell it what it is, and it will understand it and apply that logic the next time you ask it.”
It’s not just Databricks
One user I spoke to, on the condition of anonymity, spoke positively about Genie: “The whole concept of AI-enabled BI has been the holy grail for us for a long time.” He also noted that other providers are on the same path, because Databricks is not the only one trying to marry artificial intelligence with business intelligence, whether it's GenAI or traditional AI. I recently spoke with Karel Callens, CEO and co-founder of Luzmo, an embedded analytics software company based in Belgium.
His company has introduced what it calls an “embeddable AI insights component that delivers data-driven decision-making to a wide range of end users.” [Embedded] By incorporating this component into any workflow, it resides within the tools and applications your users use to generate business insights tailored to the data they interact with and their goals.
In a statement announcing this, Karens said: “The move from traditional business intelligence (BI) frameworks to more dynamic, AI-driven systems represents a major transformation in how organizations operate and make decisions, but for knowledge workers drowning in a sea of tools and information, these need to be integrated as a natural part of how they work.”
Viewpoint
The entire traditional business intelligence community is of a similar mindset, including Salesforce’s Tableau, Thoughtspot, Qlik, etc. More importantly, user organizations are seeking the “holy grail” of business intelligence powered by traditional and generative AI.
Recent research from Enterprise Strategy Group delves into the emerging convergence of AI and BI in their research report, “Unlocking the Power of AI in Analytics and Business Intelligence.” Report authors Mike Leone and Christian Perry say, “As the rate of business change often outpaces the speed at which data can be collected and analyzed, organizations need to help deliver timely, accurate insights based on the current state of the business. By empowering users to access and analyze data without requiring technical or coding expertise, AI helps democratize analytics across the broader business.”
The underlying motivation of democratizing analytics is something Databricks’ founders have consistently advocated for, and it appears to be evident in the company’s AI/BI products, but as the old analyst cliché goes, only time will tell if a broader business user base will adopt it.
For historical reasons, I will refrain from recommending the CW 50th anniversary article, “CW@50: Data Management – 50 Years of Searching for Business Value.”
Brian McKenna is a senior analyst in TechTarget's Enterprise Strategy Group, specializing in business applications, and was previously an editor at ComputerWeekly.
The Enterprise Strategy Group is a division of TechTarget, whose analysts have business relationships with vendors.
