Data management and analytics are now in an entirely new era, with AI dominating user interest and vendor product development, but AI readiness is key to helping organizations take advantage of cutting-edge capabilities.
In another era, the rise of self-service analytics required companies to modernize their data infrastructure and develop data governance frameworks that allowed employees to explore and analyze with confidence, while also restricting access to data based on employee roles.
Now, similarly, the age of AI is calling for organizations to modernize, according to Fern Halper, vice president of research at research and consulting firm TDWI. As a result, TDWI research shows that a top priority for organizations is supporting advanced analytics and preparing and making data available for AI models and applications.
“Organizations are working to prepare for AI as many see it as essential to digital transformation, competitiveness, operational efficiency and other business drivers,” Halper said July 10 at a virtual conference hosted by TDWI.
Ensuring readiness for the development and deployment of AI models and applications is a process, she continued, that includes proper data preparation, operational readiness including advanced data platforms, and proper AI governance.
AI Support
While technology and governance are important aspects of AI readiness, the process of developing and preparing for AI begins with organizational buy-in: Anyone who wants to use AI to surface insights and inform decision-making needs support from executive leadership that permeates the entire organization.
A new era of AI in data management and analytics began in November 2022 when OpenAI released ChatGPT, bringing significant improvements to generative AI capabilities.
Companies have long wanted to broaden the use of analytics because data-driven decisions drive growth at a higher rate than decisions made without data. But the complexity of analytical data management platforms, the need for coding to perform most tasks, and data literacy training to interpret the output have stagnated the adoption of analytics for nearly two decades. Only about a quarter of employees within an organization regularly use data in their workflows.
Generative AI has the potential to change this by enabling true natural language processing that has never been possible with tools developed by analytics and data management vendors. Additionally, generative AI tools can be programmed to automate repetitive tasks, reducing the burden on data engineers and other data professionals.
As a result, many vendors are making generative AI a focus of their product development, building tools like AI assistants that work with a company's data to enable natural language query and analysis. At the same time, many companies are making generative AI a focus of their development, building models and applications that can be used to generate insights and automate tasks.
Still, getting executives to recognize the importance of generative AI can sometimes be an effort, Halper said.
“If organizations aren't committed to this, nothing will work,” she said.
Halper went on to point out that while this new era is only two years old, the commitment is an ongoing process, noting that only 10% of respondents in the TDWI survey have a clear AI strategy in place, and another 20% are in the process of implementing one. Additionally, less than half of respondents said their leadership is committed to investing in the necessary resources, including data operations staff and other talent needed to work with the necessary tools.
To gain executive buy-in, you need to demonstrate existing problems that AI capabilities can solve and show potential outcomes like cost savings or increased growth.
“Your organization needs to know what it needs from AI,” she says. “It's best to understand the business problem you're trying to solve with AI and think about how to get there.” [the need for AI] In a way that business leaders can understand, so you can show them how you're going to measure the value you're getting from AI. This takes some work, but it's necessary to get business stakeholders on board.”
Here are seven benefits that businesses can potentially realize from using generative AI.
Foundation
With organizational support, AI readiness starts with the data that underpins every model and application.
Models and applications trained on high-quality data will produce high-quality results. Models and applications trained on low-quality data will produce low-quality results. Furthermore, the more high-quality data available to train an AI model or application, the more accurate it will be.
This results in a need for structured data, such as financial and transactional records, that have traditionally informed analytical reports and dashboards, as well as unstructured data, such as text and images, that often go unused.
A modern data platform is needed to access unstructured data in addition to structured data, and transform that unstructured data to make it discoverable and usable, as well as to apply generative AI by combining it with large-scale language models such as ChatGPT and Google Gemini.
A 20-year-old data warehouse simply doesn't have the necessary technology (e.g., computing power) to handle AI workloads, and neither do on-premises databases.
“Organizations are concerned about future-proofing their environments to meet the demands of AI data availability and increased workload speed, power and scalability,” Halper said.
Cloud data warehouses, data lakes, and data lakehouses can handle the volumes of data needed to inform AI models and applications, which is why spending on cloud-based deployments is increasing while spending on on-premise deployments is decreasing.
But this is just the beginning: trusted data needed to be AI-ready remains a problem, with less than half of those surveyed by TDWI saying they have a trusted data foundation in place.
Automation can help, according to Halper: By using data management and analytics tools that use AI to automate data preparation, organizations can improve the quality of their data and the reliability of their insights.
Data ingestion, integration, pipeline development, and curation are complex and labor intensive. Because machines are much faster than humans, tools that automate these processes improve efficiency. They also improve accuracy. No individual or team can vet every possible data point, which may number in the millions, but machines can be programmed to do so.
“Automation can play a key role in data mapping accuracy, job processing and automating workflows,” says Halper. “What we're seeing most often is automating and enhancing data classification and data quality.”
For example, AI-powered tools such as data observation platforms are used to scan data pipelines and identify problem areas.
“The use of these intelligent tools is important,” Halper said. “Organizations are realizing that they need to look for tools that can help them: [data readiness for AI]There are several tools that organizations can leverage as their data volumes continue to grow.”
Governance
Data quality, the right technology, and organizational support are still not enough to ensure that companies are ready to develop and deploy AI models and applications.
To prevent organizations from leaking confidential information, violating regulations, or acting without proper due diligence, they need to develop guidelines that limit who can access certain AI models and applications and how they can use them.
Many organizations are preparing for AI because they see it as essential for digital transformation, competitive advantage, operational efficiency, and other business drivers.
Fern HalperVice President and Senior Director of Advanced Analysis at TDWI Research
When self-service analytics platforms were first developed to empower business users to interact with data in addition to the IT teams that historically oversaw all data management and analysis, organizations needed to develop a data governance framework.
Done properly, these data governance frameworks can protect enterprises from harm while enabling confident self-service analytics and decision-making. As the use of AI models and applications becomes more widespread within the enterprise (especially generative AI applications that allow more people to engage with data), a similar governance framework must be put in place for their use.
“For AI to be successful, it needs governance,” Halper said.
AI requires new types of data, such as text and images, and it requires the use of different platforms, such as data warehouses and data lakes, vector databases that enable discovery of unstructured data, and search-enhanced generative pipelines to train models and applications with relevant data.
Governance therefore includes diverse data and multiple environments to address AI readiness, Halper said. Governance should also include monitoring different types of AI, including generative AI, to determine whether the output is harmful or inaccurate, or if there is bias in the model or application.
“The future starts now, and there's a lot to think about,” Halper says. “Data management is going to continue to be a long journey in terms of managing new data with AI and beyond. Organizations need to think strategically about the future and not get caught off guard.”
Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with over 25 years of experience, focusing on analysis and data management.
This website uses cookies to ensure that you get the best experience on our website. OkRead More
Privacy & Cookies Policy
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are as essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
