Solutions Review’s premium content series is a collection of contributed articles written by industry experts in the enterprise software category. In this feature, Quantexa’s Chief Product Officer, Dan Higgins, shares the secrets of increasing AI-driven business value. Clean data.
Years of global conflict, growing economic uncertainty, shifting consumer expectations and accelerating digital transformation are putting organizational leaders under tremendous pressure to deliver results. But delivering results quickly is becoming more difficult, putting organizations and business leaders alike under pressure to make faster and more accurate decisions. In a Gartner survey, 65% of respondents said they feel they are making far more complex decisions today than they were two years ago. Meanwhile, 53% of them said they feel more pressure to explain or justify their decisions. This is a clear sign that there is a missing link between the rush to automate and understanding exactly what is computerized and why.
This is where organizations often turn to technologies such as artificial intelligence (AI) and machine learning (ML) to support decision making. However, the biggest challenge we recognize is that organizations often implement these technologies without due consideration of their contextual importance. AI and ML need contextual information to make effective predictions and decisions. It’s important to remember that automation challenges are not just complex lines of code. The quality of the input data also plays an important role.
Solving the Chicken-Egg Problem: Data Quality and AI Business Value
AI and ML applications depend on the quality of incoming data. This is why data scientists focus on working with trusted and transparent data to create better performing AI algorithms. For example, when building a classifier to distinguish between pictures of cubic zirconia and pictures of diamonds, a data scientist would ideally have an input image dataset certified by a jeweler. If you didn’t get this, the next best place to find this could obviously be online. But here comes the challenge of typos and mislabeling.
There is also the challenge of inconsistent data entry, as a single entity may be referenced using different names. For example, using my name, I Daniel John Higgins can appear as DJ Higgins, Dan Higgins, Mr. Higgins, and so on. The same applies to companies, which may be referred to by their full full name or shortened name.
It is important that the algorithm can recognize and learn different names and formats. This is especially difficult given the scale of the data and the number of similarly named entities. The challenge is compounded by the number of individuals and organizations sharing the same name. Understanding this scenario and its implications is called context. To make this kind of distinction, the algorithm must be able to learn all the different names and forms.
Unlock the power of data to transform your business
Less than half (42%) of global IT decision makers trust the accuracy of their organization’s data. It also reveals that he has one duplicate in eight US customer records, according to a new study from Quantexa. This means that a huge number of organizations cannot distinguish me as DJ Higgins, Dan Higgins, or Daniel John Higgins.
Data is critical to the success of digital transformation initiatives that use data to improve operational efficiency, improve customer value, and create new avenues for revenue generation. In fact, we’ve never had more data than this. Therefore, even though data can serve as an organization’s greatest strength, it can also serve as its greatest barrier to transformation efforts.
It is estimated that businesses recently invested a staggering $1.3 trillion (USD) towards digital transformation. Not surprisingly, a whopping 70% of these efforts fall short as companies prioritize investments in other technologies over the data culture needed to support their intended purpose.
This challenge will continue to grow as these efforts snowball to generate more than 5x more data points, further compounding the problem of complexity across the industry.
And in some industries, such as banking and financial services, organizations are in a situation where they can fall victim to this data “context gap.” This is a direct result of duplicate data sets related to the same customer spread across different CRM and other management tools and systems. It could be a simple duplicate error, but the impact on insight is huge. For example, if a customer’s name is misspelled by one letter in one system, but misspelled in another system, the organization can , you are more likely to consider them to be two distinct entities. This is just the nature of siled data. Without context, it’s nearly impossible to deploy any kind of meaningful analysis, completely hampering the decision-making process.
Getting a 360-degree view of your customers in a scalable way requires more than just looking through archives and manually finding duplicates. Manual data management is not only time consuming and tedious, it is also highly prone to human error. Traditional methods of doing this, such as master data management (MDM), have historically been ineffective at identifying and connecting these “lost links” to individual customers. To do this effectively, organizations need to introduce a new category of products that do entity resolution.
Entity resolution brings focus to rich context
Leveraging advanced AI and machine learning models, Entity Resolution effectively connects, standardizes, and parses data to consistently identify similar entities. This is accomplished by grouping related records, thereby creating a set of attributes and labeled links for each entity. In contrast to the traditional method of record-to-record matching utilized in MDM systems, entity resolution allows organizations to introduce new entity nodes that serve as key connection points for linking real-world data.
This enables more accurate and efficient data linking, including the ability to match high-value external data sources such as corporate registry information, which were previously difficult to link reliably. According to the same Quantexa survey, today only 27% of organizations worldwide are using entity resolution technology to manage their data and make informed decisions.
Due to the prevalence of duplicate data in various databases and data storage systems, entity resolution is critical for decision-making intelligence, helping businesses avoid making decisions based on inaccurate or incomplete data. .
All Roads Lead to Clean Data
We all know how sensitive organizations are to the importance of using data to enhance decision making. Breaking down data silos requires companies to parse swamps of duplicate and redundant data, which can have ramifications for decision-making efficiency.
and accuracy. This leads to wasted resources across data, IT, and business teams, and hinders a company’s ability to quickly identify risks and provide top-notch customer service. As such, intelligent decision-making ultimately requires a foundation of data.
Prior to joining Quantexa, Dan spent over 20 years at EY in various leadership roles. As a former member of the Global Consulting Investment Committee, he also helped shape the company’s platform, product and asset strategy. Prior to joining EY, Arthur Andersen, Prudential, He Financial, Technology Solutions, He Company, and Oxford Solutions, held various leadership, project management and development roles.
