When artificial intelligence (AI)-powered predictive and prescriptive analytics first promised to give businesses a crystal ball to predict the future, many companies were dazzled by the allure of beautiful, interactive graphs and rushed to invest in the technology without properly understanding the fundamentals.
For many of them, the AI project was a disappointment.
A few years ago, Gartner predicted that by 2022, 85% of AI projects will deliver erroneous results due to biases in the data, algorithms or the teams managing them. A 2019 white paper by Pactera Technologies in collaboration with Nimdzi Insights also reflected this high failure rate.
Despite the challenges that organizations have experienced, AI has become central to most organizations’ digital transformation efforts.
According to PricewaterhouseCoopers' third annual AI Forecast report, published last month, businesses still believe in the potential of AI: More than half of U.S. respondents are increasing their AI investments following the COVID-19 crisis, but 76% of organizations are barely breaking even on their AI investments.
The survey found that a quarter of participants reported that AI adoption was growing, up from 18% the previous year.
AI needs data
For analytics to be effective, the data you use must be clean, accurate, and relevant. Data analysts have traditionally spent the majority of their time (up to 70%) discovering and preparing data rather than developing data science models.
Similarly, for AI in analytics to be effective, the right foundations must be in place. It is well-known that AI is inherently dependent on data. The challenge for many organizations is the identification, access, and availability of datasets relevant to their use cases, resulting in AI models not always producing the expected results.
For analytics to work, the data you use must be clean, accurate, and relevant.
In South Africa, AI is the next logical step following the current wave of machine learning adoption. However, because machine learning (ML) is also being deployed to draw conclusions and predictions based on example data sets, it needs the same foundation that AI needs: high-quality, readily accessible data.
If the dataset used is of poor quality and does not reflect the target market (i.e. it is non-specific or generalized), prediction accuracy will suffer and business decisions will be made based on inaccurate data.
As the demand for insights increases and AI enters the analytics space, organizations should consider preparing a catalog of timely, high-quality data that enables data scientists to quickly review data sources, define algorithms, and derive better insights for business use cases.
The right data in the cloud
Where the data is stored is another important consideration. In the past, many organizations have migrated entire big data lakes from data centers to the cloud, believing that this would result in cost savings. However, not only did these savings not come true, but analysis often became more complex.
The cloud provides an ideal environment for analytics because of its scale, elasticity, redundancy, and accessibility across the enterprise, but you shouldn't store all your data in the cloud – just the right data.
Approaches to cloud migration projects are often focused on “moving data to the cloud” rather than starting with a top-down approach and focusing only on the data relevant to the business problem.
Ingesting and storing unmanaged, uncurated data causes cloud platforms to incur huge costs as unnecessary and irrelevant data is moved and processed in the cloud, which poses a significant challenge to the “extract and load now, transform later” approach traditionally taken in big data projects.
When embarking on such an endeavor, one of the key success factors is to focus on a set of high-value use cases on which to base your AI/ML models. These use cases are typically based on the business problem that needs to be addressed.
Data is often collected from a variety of sources, but the key is to identify the data that is relevant to these high-value use cases to ensure business problems are being solved, cloud platform costs are being managed, and AI models are not just leading to technical benefits.
When an enterprise embarks on a cloud migration project, it is important to consider a cloud data management strategy in conjunction with the cloud migration planning process.
The use of technologies such as data integration platforms, data catalogs, data quality and data security solutions is important in this step as they ensure that data management principles are carried out in a controlled and standardized manner.
Once this data foundation is set, future efforts in the cloud can follow a recipe for data curation that leads to better AI models and insights.
