Limited explanations of AI may benefit consumers: Tepper study

Modern artificial intelligence (AI) algorithms are often referred to as “black box” models, meaning that their inputs and operations are invisible to users and other stakeholders, making their decisions difficult to interpret. Explainable AI (XAI) is a set of techniques that seeks to address the lack of interpretability and trust in AI by explaining AI decisions to customers.

Many experts believe that regulating AI by mandating fully transparent XAI would improve societal welfare, but new research from the Tepper School of Business challenges this idea, concluding that such regulation could lead to suboptimal outcomes for both the companies and consumers who use AI.

The study, conducted by researchers at Carnegie Mellon University and the University of Southern California, has been accepted for publication in the journal Marketing Science.

“Consumer advocacy groups are increasingly calling for an XAI mandate that would increase AI regulation and transparency,” Behnam Mohammadi said.Left photo), a doctoral student at Carnegie Mellon University's Tepper School of Business and co-author of the study.

“Companies are under pressure from legislators and customers to adhere to accountable AI practices, but little is known about the economic impacts of XAI. We dove deep into the complexities of XAI regulation to learn about the impacts on competition between companies and social welfare.”

The use of AI models for decision-making has increased significantly in recent years, with billions of dollars being spent worldwide on this technology. However, a key challenge is the interpretability of AI decisions and predictions.

While early models were easily interpretable, modern methods (such as deep neural networks) feature opaque decision systems that make many people reluctant to adopt algorithms that they cannot easily interpret, control, or trust. This is particularly problematic when black box models produce biased results (such as showing fewer ads for high-paying jobs to women than to men, or failing to recognize non-White faces).

XAI is a class of techniques that aims to create human-explainable “glass box” models while maintaining predictive accuracy. The goal is to enable people, including non-technical experts, to understand, trust, and effectively govern emerging AI systems. Along with calls for these types of techniques from consumer activists, XAI is gaining traction in the healthcare, retail, media and entertainment, and aerospace and defense industries.

Do customers and businesses want different things from XAI? If so, how should the rules for XAI be set? Researchers focused specifically on the insurance industry, which uses AI to determine pricing, and studied situations where two large companies dominate the market. The study found that where there are no rules, businesses and customers often expect different levels of explanation from the AI.

“Sometimes, a full explanation is not good for the customer. They might prefer that a company not explain everything. That might lead to a better product,” said Nikhil Malik, an assistant professor of marketing at the University of Southern California's Marshall School of Business and co-author of the study.

The study concludes that partial explanations may be better for both consumers and companies. In fact, the authors were surprised to find that full explanations may be disadvantageous for consumers: Consumers may prefer that one or both companies set a lower XAI than a full XAI, which would incentivize the companies to offer higher quality products.

Their key finding is that, counterintuitive as it may be, regulating AI products to provide full descriptions is not the recommended regulatory strategy. Instead, an optimal XAI policy could contribute to societal welfare by allowing companies to offer flexible policies on optional XAI or to differentiate XAI levels.

“Based on our findings, we urge policymakers to consider a more nuanced approach when developing XAI regulations,” Tim Dardenger noted.Right photo“A one-size-fits-all policy for all markets, especially one that mandates full accounting, may not produce the desired results,” said Matthews, an associate professor of marketing and strategy at the Tepper School and a co-author of the study.

Kannan Srinivasan, a co-author of the study and a professor of management and marketing at the Tepper School, noted that as AI plays a central role in enterprises, many solutions have been proposed to mitigate the risks associated with it.

“Transparency is seen as a mechanism to mitigate potential bias in AI algorithms. But our analysis finds that this is quite likely not the case,” Srinivasan said.