In 2019, Apple’s credit card business came under fire for recruiting women. 1/20th The amount of credit given to the husband. When she filed her complaint, an Apple representative reportedly told her: It’s just an algorithm. ”
Decisions are increasingly made today by such opaque and unexplainable algorithms, often with similarly problematic results. From credit approvals, to customized product and promotion recommendations to re-start readers, to fault detection for infrastructure maintenance, organizations across a wide range of industries are investing in automated tools to understand how their decisions are made. Decisions are often made with little or no insight into what is going on.
This approach creates real risks. Research shows that lack of explainability is one of executives’ most common concerns about AI, with a significant impact on user trust and willingness to use AI products, not to mention safety. is shown.
However, despite the downsides, many organizations continue to invest in these systems. Because decision makers assume that unexplainable algorithms are inherently better than simple explainable algorithms. This perception is known as a trade-off between accuracy and explainability. Technology leaders have historically assumed that the better humans can understand algorithms, the less accurate they will be.
white box and black box
Specifically, data scientists distinguish between so-called black-box AI models and white-box AI models. White-box models typically contain a few simple rules, presented as, for example, decision trees or simple linear models with limited parameters. Due to the small number of rules and parameters, the processes behind these algorithms are usually understandable to humans.
In contrast, black-box models use hundreds or thousands of decision trees (known as “random forests”) or billions of parameters (similar to deep learning models) to inform the output. To do. According to cognitive load theory, humans can only understand models with a maximum of about 7 rules or nodes, making it functionally impossible for an observer to explain the decisions made by a black-box system. But does that complexity necessarily increase the accuracy of black-box models?
Debunking the trade-off between accuracy and explainability
To explore this question, we run black-box models and white-box models on a broad array of about 100 representative datasets (known as benchmark classification datasets) across domains such as pricing, medical diagnostics, and bankruptcy. We conducted a rigorous and extensive analysis of how the box model works. Anticipation and buying behavior. We found that black-box and white-box models produced equally accurate results for nearly 70% of the dataset. In other words, there is often no trade-off between accuracy and explainability, and a more explainable model can be used without sacrificing accuracy.
This is consistent with other emerging research exploring the potential of explainable AI models, as well as our own experience working on case studies and projects with companies across different industries, geographies and use cases. For example, COMPAS, a complex black-box tool widely used in the U.S. justice system to predict the likelihood of future arrests, is not as accurate as simple predictive models that look only at age and criminal record. has been repeatedly proven. Similarly, the research team created a model that predicts the likelihood of loan defaults that are simple enough for the average bank customer to easily understand. We found that the accuracy of that model was less than 1% lower than the equivalent black-box model (a difference within the margin of error).
Of course, there are cases where black-box models are still beneficial. However, considering the downsides, our research suggests that companies need to take several steps before adopting a black-box approach.
1. The default is a white box.
As a rule of thumb, a white-box model should be used as a benchmark to assess whether a black-box model is necessary. Organizations should test both before choosing a model type. If the performance difference is not important, you should choose the whitebox option.
2. Know your data.
One of the main factors in determining whether a black-box model is necessary is the data involved. First, decisions are driven by data quality. If the data is noisy (that is, it contains a lot of false or meaningless information), relatively simple white-box techniques tend to work well. For example, we spoke to analysts at Morgan Stanley who commented on very noisy financial data sets that “companies are undervalued, underperformed recently, and oversized. I’ve found that simple trading rules like “buy stocks if not” work well.
Second, the type of data also influences the decision. For applications that contain multimedia data such as images, audio, and video, the black box model may provide better performance. For example, we worked with a company developing an AI model that allows airport officials to predict security risks based on images of air cargo. They found that the black-box model was more likely to detect high-risk shipments that could pose a security threat than the equivalent white-box model. These black-box tools have saved inspection teams thousands of hours by focusing on high-risk shipments, and significantly improved the organization’s performance on security metrics. Even in equally complex applications such as face detection in cameras, vision systems in self-driving cars, facial recognition, image-based medical diagnostic devices, illegal/harmful content detection, and most recently, generative AI tools like ChatGPT and DALL-E. , the box approach may be advantageous or may be the only viable option.
3. Know your users.
Transparency is always important for building and maintaining trust, but especially for sensitive use cases. If a fair decision-making process is paramount to users, or if some form of procedural justice is a requirement, then it makes sense to prioritize explainability, even if the data lends itself to a black-box approach. It may be true. Or if the hard-to-explain model turns out to be slightly more accurate.
For example, in areas such as recruitment, allocation of organs for transplantation, and legal decisions, choosing a simple, rules-based, white-box AI system reduces risk to both organizations and their users. Many leaders discover these risks the hard way. In 2015, Amazon discovered that its automated candidate screening system was biased against female software developers. Meanwhile, his AI benefits fraud detection tool in the Netherlands was shut down in 2018 after critics accused it of being a “massive and fraudulent service.” An opaque black hole. ”
4. Know your organization.
Whether an organization chooses white box AI or black box AI also depends on the organization’s level of AI readiness. If your workforce tends to have low trust and understanding of AI, and your organization is less digital, it may be best to start with simpler models before moving on to more complex solutions. This usually means implementing a white-box model that everyone can easily understand, and only considering black-box options once your team is comfortable with using these tools.
For example, we worked with a global beverage company that launched a simple white-box AI system to help optimize their employees’ daily workflows. The system provided limited recommendations, such as which products should be advertised and how much different products should be restocked. Then, as organizations matured in their use of and trust in AI, administrators began testing whether more complex black-box alternatives could offer advantages in any of these applications.
5. Know your regulations.
In certain areas, explainability may be a legal requirement rather than a nice-to-have. For example, in the United States, the Equal Credit Opportunity Act requires financial institutions to be able to explain to loan applicants why credit has been denied. Similarly, the European General Data Protection Regulation (GDPR) suggests that employers should be able to explain how candidate data has been used to inform hiring decisions. White-box models are the only option if your organization is legally required to be able to explain the decisions made by AI models.
6. Explain the unexplainable.
Finally, of course, there are situations where black-box models are arguably more accurate (as was the case with 30% of the datasets tested in our study) and are acceptable for regulatory, organizational, or user-specific concerns. For example, applications such as computer vision for medical diagnostics, fraud detection, and cargo management all benefit greatly from the black-box model, and the legal or logistical hurdles it poses are more manageable. tend to. In such cases, if an organization decides to deploy opaque AI models, it should take steps to address the reliability and safety risks associated with lack of explainability.
In some cases, it is possible to develop explainable white-box proxies to roughly clarify how the black-box model arrived at its decisions. Even if this description isn’t completely accurate or complete, it can go a long way in building trust, reducing bias, and increasing adoption. In addition, a deeper (albeit imperfect) understanding of the model will help the developer to further refine the model and bring more value to these companies and their end users.
In other cases, the organization may actually have very limited insight into why the model makes such decisions. Even when high-level explanations are not possible, leaders can prioritize transparency in talking about the model internally and externally, candidly recognize and address risks.
***
Ultimately, there is no one-size-fits-all solution for implementing AI. All new technologies carry risks, and choosing how to balance those risks against potential benefits depends on your specific business context and data. However, our research shows that, in many cases, simple, interpretable AI models can mimic black-box alternative models without compromising user trust or enabling decision-making with hidden biases. is shown to work.
