A surprisingly easy way to double the profits of your AI models is to make decisions based on expected values rather than predicted scores. This section explains how to do this using fraud detection as an example.
getty
I would like to share with you what I consider to be a true breakthrough. Predictive AI. This is the most underutilized, yet surprisingly simple, way to multiply the value you get from your machine learning projects.
Typically, when companies deploy predictive AI models, drive each decision Use raw model scores. For example, a model may indicate that a particular transaction has a 30% chance of being fraudulent. To make decisions, companies typically Compare that probability to a fixed threshold50%, etc., to decide whether to block the transaction.
But here comes the paradigm shift. Rather than using raw model scores to make decisions, model deployment requires the following: Drive each decision based on expectations.
This very rare practice is technically and conceptually a simple “no-brainer.” Let’s use a specific example of payment card fraud detection to explain how it works.
Raw risk score flaws
Imagine you are performing fraud detection in a bank. Predictive models output a risk score (probability) for each single transaction.
If you base each decision purely on raw risk level, you would treat a $100 trade exactly the same as a $5,000 trade. However, a $5,000 fraudulent transaction results in far more loss than a $100 transaction. For large transactions, the downside of undetected fraud is even greater. But on the other hand, the cost of “false positives” that incorrectly block legitimate transactions and inconvenience customers typically do not increase at the same rapid rate, even for large purchases. At a given risk level (predictive model score), the upside can quickly outweigh the downside when deciding whether to block a higher value transaction.
With this in mind, many banks and companies currently use clunky “if-then” business rules that set different risk thresholds for different transaction sizes. They might say, “If your trades are over $1,000, lower your risk tolerance.” However, these rules are individual and strict. They do not help optimize the “numbers game” of fraud detection because they treat all potentially wide-ranging transactions the same.
Fix: Decisions based on expected values
Maximizing profits requires a smooth and continuous approach. Rather than using hard and fast rules, just do a little elementary school math before applying the decision threshold.
Step 1: Use machine learning models to calculate the probability of fraud.
Step 2: Multiply that probability by the size of the transaction.
This simple multiplication gives us the expected value. This is, in this context, the amount of money you would expect to save by blocking the transaction.
Let’s look at mathematics as an example.
- Transaction A: According to this model, there is a 20% (0.2) chance that the transaction is fraudulent. The transaction amount is $100. Multiplying 0.2 by $100 gives us the expected value of blocking this transaction: $20.
- Transaction B: According to this model, the chance of fraud is only 5% (0.05). The transaction amount is $5,000. Multiplying 0.05 by $5,000 gives us the expected value of blocking this transaction: $250.
Step 3: Instead of setting thresholds based on raw percentages (risk), probability), draw the decision threshold based on: monetary Expected value.
Setting the decision threshold to an expected value of $15 blocks both transactions above. Transaction B is technically “low risk” at only 5%, but the larger size of the transaction should reduce your risk tolerance. You can fully understand this by basing your decision on an expected value of $250.
Visualize the impact on your bottom line
To truly understand why this is a huge benefit to your business, you need to look at the following: profit curve (or similarly, the savings curve).
A benefit curve (savings curve) that shows the amount of money saved due to fraud detection in relation to the proportion of the riskiest transactions that are blocked. The Y-axis represents the total “dollar savings” and the X-axis represents the “percentage of blocked transactions” moving from highest risk on the left to lowest risk on the right. This view, calculated across an assumed pool of 10 million transactions, assumes that a correctly blocked fraudulent transaction (true positive) saves the bank money on that transaction, and a legitimate transaction that is incorrectly blocked (false positive) costs $75.
Eric Siegel
A standard deployment based solely on the model’s risk score would prioritize transactions from left to right based on their likelihood of being fraudulent. If you save money by blocking the riskiest trades, the curve will rise and eventually reach the peak. After the peak, the curve declines. This is because you start blocking too many legitimate transactions and the cost of false positives (which annoy your customers) eats up your savings.
This curve looks pretty good. The maximum amount is over $20 million. But what if we sorted the cases and prioritized them based on expected value instead?
Comparison graph showing the original curve and the new “expected value” curve. The peak of the new curve is much higher on the Y-axis (greater savings) and shifted further to the left on the X-axis (fewer transactions are blocked).
Eric Siegel
What a difference! When you switch to the expected value approach, two surprising things happen to this chart.
- The peak is even higher: Maximize savings over making decisions based solely on raw model scores.
- The peak moves to the left. This is the best news of all. This means you can reach significant maximum savings while blocking fewer transactions.
By giving higher priority to high-value transactions, commerce does not need to be disrupted as much to maximize costs. Only the few transactions on the far left that most deserve intervention need to be blocked.
Beyond fraud detection
This technique doesn’t just apply to credit card fraud. It can improve a variety of predictive AI projects with different case sizes.
For example, a nonprofit fundraiser can determine who receives a high-ticket advocacy campaign by multiplying a donor’s likelihood of giving by their estimated financial capacity. The expected value approach has also been proven to dramatically improve churn modeling, target collection, and credit scoring decisions.
Although this approach is rare, it is impactful and simple. I mean, I’ve basically never heard other people in the industry bring this concept up. And the senior practitioners to whom I present this idea respond, “Oh yeah! That’s a great idea! I’ll try it.”
So let’s take a look at some of the current efforts in predictive AI. By taking this simple step of moving from pure model probabilities to estimated monetary values, you can double the benefits of deploying machine learning models.
To watch a YouTube video that visually details this approach, go to click here.
