Join C-suite executives in San Francisco July 11-12 to hear how leaders are integrating and optimizing their AI investments for success.. learn more
Data is critical to any business, but the growing volume of information and complexity of pipelines inevitably create roadblocks.
A new survey of 200 data professionals working in the United States found that data downtime — the period during which company data is missing, inaccurate, or inaccessible — is the number of quality incidents. Considering the surge in fires and firefighting efforts, it has almost doubled from the previous year. Team time.
The study, commissioned by data observability firm Monte Carlo and conducted by Wakefield Research in March 2023, aims to help organizations collect as much data as possible to build downstream AI and analytics applications for their businesses. It highlights significant gaps that need to be addressed as they race to grab assets. – Critical functions and decisions.
“As data grows and becomes more complex, so does the potential for data corruption. As data becomes more and more integral to an organization’s revenue-generating operations, so does the catch rate of data incidents. It means that business users and data consumers are more likely to discover incidents that data teams missed,” Lior Gavish, Monte Carlo co-founder and CTO, told VentureBeat.
event
transform 2023
Join us July 11-12 in San Francisco. There, he shares how management integrated and optimized his AI investments to drive success and avoid common pitfalls.
Register now
Data downtime factors
The study attributed increased data downtime to three key factors, essentially: increased number of incidents, increased time to detect incidents, and increased time to resolve problems. thinking about.
Of 200 respondents, 51% say they typically witness 1-20 data incidents per month, 20% report 20-99 data incidents, and 27% report at least 100 each month of data incidents. This consistently surpasses last year’s figures, with the average number of monthly incidents witnessed by the organization increasing from 59 in 2022 to 67 this year.
As cases of bad data continue to grow, teams are spending more and more time finding and fixing problems. Last year, 62% of her respondents said it usually took an average of 4 hours or more to detect a data incident, but this year that number has increased to 68% of her.
Similarly, 63% of respondents said resolution typically takes four hours or more after an incident is discovered, up from 47% last year. Here, the average time to resolution of data incidents has decreased from 9 hours for him to 15 hours for him year over year.
It’s the manual approach that’s to blame, not the engineer
It’s all too easy to blame a data engineer for failing to ensure quality or taking too long to fix a problem, but understand that the problem is the task at hand, not the talent. is important. As Gavish points out, engineers are not only dealing with high volumes of high-speed data, but also ever-changing approaches to how data is produced from sources and consumed by organizations. is not always in your control.
“The most common mistake teams make in this regard is relying solely on manual static data testing. You have to anticipate all the ways the data can go bad and write tests, which takes a lot of time and doesn’t help,” he explains.
Instead of these tests, teams should consider automating data quality by deploying machine learning monitors to detect data freshness, volume, schema, and distribution issues wherever they occur in the pipeline. Yes, the CTO said.
This gives enterprise data analysts a holistic view of data trust in critical business and data product use cases in near real-time. Additionally, if something goes wrong, the monitor can send an alert so your team can address the issue quickly and before it impacts your business.
Sticking to the basics remains important
In addition to ML-driven monitoring, teams should also follow certain basics, starting with focus and prioritization, to avoid data downtime.
“Data generally follows the Pareto Principle, where 20% of datasets provide 80% of business value, and 20% of those datasets (not necessarily the same) have 80% of data quality issues. Be able to identify high-value or problematic datasets and recognize when they change over time,” says Gavish.
Additionally, tactics such as creating service level agreements (SLAs) for data, establishing clear ownership, creating documentation, and conducting postmortems can also help, he added.
Today, Monte Carlo and Bigeye are positioned as major players in the rapidly maturing AI-driven data observability space. Other players in this category are start-ups such as Databand, Datafold, Validio, Soda and Acceldata.
That said, it is essential to note that teams do not necessarily have to deploy a third-party developed ML observability solution to ensure quality and reduce data downtime. You can also choose to build it in-house if you have the time and resources required. According to Monte Carlo Wakefield research, it takes an average of 112 hours (about two weeks) to develop such a tool in-house.
While the market for specific data observability tools is still developing, research by Future Market Insights shows that the broader observability platform market will grow from $2.17 billion in 2022 to $5.55 billion by 2032. The dollar is expected to grow at a CAGR of 8.2%.
VentureBeat Mission will be the digital town square for technical decision makers to gain knowledge and transact on transformative enterprise technologies. Watch the briefing.
