As companies deploy artificial intelligence tools and both the volume and personalization of offers increase, a core challenge remains.
“How do you decide which AI tool or AI agent is best?” asked Professor Alexandre. belloniWestgate Distinguished Professor of Decision Science at Duke University’s Fuqua School of Business.
“How do you understand the added value of a new customer experience or the return on investment of a new ad?”
According to bellonithe answer lies in testing and measurement. He shared his insights in a recent talk on Fuqua’s LinkedIn page.
Why A/B testing is still important
Companies routinely use A/B testing to determine the causal effects of decisions and tools. Comparing the results will reveal whether the new approach is working.
“A/B testing allows you to quickly evaluate failures.” belloni Said. “You don’t have to change the whole system without knowing if it actually works. You can delay the decision until you know it’s the right thing to do.”
Successful A/B tests are numerous and have a long history. belloni The 2008 Obama presidential campaign used A/B testing to find the best mix of ads and said it saw a significant increase in registrations and donations. In another example, a travel agency randomly assigned some employees the option to work from home to measure the impact on productivity.– find very uneven results.
When A/B testing isn’t good enough
But while A/B testing remains the gold standard, it’s not always viable. belloni he pointed out.
One limitation arises when participants can choose to participate or not participate in treatment, skewing the results. For example, a company that offers customers to sign up for a credit card.
Suppose you are interested in evaluating the impact of these credit cards on customer purchases. Because participants in the treatment group can reject the offer, the accepting group is self-selected, making it difficult to measure the impact of the card on purchases. belloni Said.
“The group of customers who accept your offer may be quite different from the group as a whole.” belloni He explained. “That means you can’t make a fair comparison with a control group.”
“This selection bias can significantly skew the results,” he said.
The second challenge arises from interference between groups, where individuals influence each other.
belloni It noted that vaccination programs seek to assess infection rates in treatment groups that received the vaccine and control groups that did not receive the vaccine. In this example, unvaccinated people in the control group who happened to be surrounded by vaccinated people may have a lower risk of infection simply because the vaccinated group blocks, or interferes with, the results of the untreated group.
The same is true for ride-hailing platforms, where incentives offered to drivers in one geographic region can affect ridership for drivers in another region, skewing the results of an experiment.
How AI assesses causality
This is where artificial intelligence, especially machine learning, comes in handy for causal inference. belloni Said.
Machine learning models can estimate the likelihood that each customer will accept your offer. By reweighting data based on these trends, companies can adjust for selection bias and make fairer comparisons, he said.
By predicting which customers are most likely to accept a credit card offer, the company can identify the true impact of the credit card itself, rather than simply measuring the behavior of customers who are already inclined to spend more.
“We can build machine learning models to assess trends. This allows us to undo unfair comparisons and measure causal relationships. By combining these trend estimates with machine learning estimates of the impact itself, we can also improve our estimates,” he said.
Supporting organizations in complex environments
Causal inference is useful for organizations in complex real-world environments, but may require more sophisticated randomized experiment designs. belloni Said.
“For example, in many business environments it is very practical to monitor users at different points in time,” he said. “In such cases, users can be switched between treatment and control at different times to reduce network impact and interference.”
And while big tech companies have been investing in these tools and many other technologies for years, small businesses could also benefit, he added.
“Some companies that are starting A/B testing today don’t need a fine-tuned AI system to gain an advantage,” he said. “While these tools are relatively inexpensive, they also offer great value to small and medium-sized businesses.”
