Data scientists believe that business metrics are more important than technical metrics, but in reality they focus more on technical metrics. This derails most projects. So why?
Eric Siegel
Predictive AI It offers incredible possibilities, but has a well-known track record. Outside of Big Technology and a few other large companies, most initiatives cannot be deployed and do not recognize value. why? Data experts are not capable of selling deployments to businesses. The technical performance metrics they typically report are not consistent with business goals and do not mean anything to decision makers.
To plan for stakeholders and data scientists alike, Sales and Green Light Prediction AI Developmentthey need to establish and maximize the value of each machine learning model in terms of business outcomes such as profit, savings, or any KPI. Only by measuring values can a project actually pursue value. And only by guiding business and data experts on the same value-oriented page can initiatives move forward and deploy.
Why are business metrics so rare in AI projects?
Given their importance, why are business metrics so rare? Research shows that data scientists know better, but generally do not adhere to it. We rank business metrics as the most important, but in reality Focus more on technical indicators. Why do they usually skip beyond such important steps – calculating potential business value – a lot to the end of their own projects?
That's a very good question.
The industry doesn't stay in this rut for psychological and cultural reasons alone, but they contribute to the factors. After all, it's Gauche and “on the nose” who talk about money. Data professions feel compelled to exercise their expertise and stick to traditional technical indicators that demonstrate them. This doesn't just make them smarter. Terminology is a common way for every field to protect its own existence and salary. There is also the common but misguided belief that non-questions do not truly understand quantitative reports of predictive performance and are only misunderstood by reports intended to be spoken in simple business language.
But if they were the only reason, “cultural inertia” would have succumbed to years ago given the enormous business victory when the ML model successfully unfolds.
Reliability challenges: business assumptions
Instead, this is the biggest reason. Business value forecasts must be based on specific assumptions, which is why we face reliability questions. It is not enough to estimate the values the model captures in the deployment. The calculation must prove its reliability. This is because it depends on business factors that are affected by changes and uncertainties, such as:
- Each financial loss False positivesuch as when the model flags legitimate transactions as invalid. For example, in a credit card transaction, this can cost around $100.
- Each financial loss False negativeetc., when the model fails to flag an incorrect transaction. For example, a credit card transaction may cost the transaction amount.
- The two factors affecting the above mentioned costs. For example, credit card fraud detection could reduce the costs of each undetected fraudulent transaction if the bank has fraud insurance or if the bank's enforcement activities recover downstream fraud losses. In that case, each FN's cost could only be 80% or 90% of the transaction size. When we estimate the model's expansion values, there is room for wiggling in the percentage.
- Decision boundarythat is, the percentage of eligible cases. For example, are the top 1.5% transactions where the model appears to be the most scam are likely to be blocked, or should the top 2.5% be listed? That percentage is Decision boundary (I'll decide next Decision Threshold). This setting tends to be less and less attention-grabbing, but it often has a greater impact on the value of a project than model or data improvements. That setting is a business decision driven by business stakeholders and represents the basis for defining exactly how the model is used for deployment. By turning this knob, the business can balance the trade-off between the main bottom line/monetary value of the model with the number of other KPIs and the number of false negatives.
Establish prediction reliability despite uncertainty
The next step is to make existential decisions. Do you avoid forecasting business value of ML values entirely? This prevents the can of worms from opening. Or, given the need to calculate potential rises to achieve ML deployments, do you see it as an issue that must be addressed in ML evaluation? If that's not yet clear, my vote is for the latter.
To address this reliability question and establish trust, we need to explain the effects of uncertainty. Try different values at extreme ends of the uncertainty range. That's how you interact with your data and reports. Find it How important is uncertainty? And whether somehow it has to be narrowed down to establish a clear case of deployment. Only through insights and intuition about how different these factors make can a project establish reliable forecasts of potential business value, thereby ensuring deployment is achieved.
