Prateek Tripathi
There has been a recent surge in AI applications, and the usefulness of this technology seems limitless. But with growing usefulness comes inevitably regulatory responsibility. Governments and policymakers around the world are scrambling to put in place regulatory mechanisms and frameworks. This sense of urgency is understandable, given the disruptive and potentially dangerous nature of this technology. But before we can discuss regulation, a more important question needs to be answered: How does AI work?
The “black box” problem
AI has been around for a long time but has mostly lurked in the background. It was the emergence of generative AI models, specifically ChatGPT, that really got things moving and brought the technology to center stage. This gave rise to Microsoft's Bing Chat, Google's Bard, and various other so-called “chatbots.” All of these generative AI systems are based on large-scale learning models (LLMs), which fall under the category of machine learning (ML).
ML proceeds in three steps. First, there is an algorithm that prescribes a set of steps. Second, the algorithm learns to identify patterns after examining vast amounts of “training data”. Once the algorithm has sifted through enough data, an ML model like ChatGPT can finally be deployed. If this process sounds familiar, it’s because deep learning is essentially inspired by theories of human intelligence. Just as part of human intelligence relies on learning by example and then extrapolating that to new experiences, AI learns in a similar way. But just as we can’t remember the exact example that led us to understand a particular concept, AI can’t tell us which particular data or input led to a particular decision. Thus, AI essentially functions as a “black box” – that is, we input an input and get a particular output, but we can’t look into the code or logic of the system that produced the output. As a result, the exact reasons why an LLM behaves the way it does, and the underlying mechanisms behind its operation, are often unknown even to the creators of the LLMs themselves.
LLM is inherently a costly endeavor and requires processing large amounts of data. This is the main reason why industry has overtaken academia in creating machine learning models over the past decade. The difference is that academia has been very proactive in releasing the source code of their models, whereas companies have not. The code for applications such as OpenAI's ChatGPT has not yet been made public, and is unlikely to be made public in the future.
Any of the three components of an ML system can be hidden or put into a black box. Often, the algorithms are publicly known, making them less effective if put into a black box. Therefore, AI developers often put their models into a black box to protect their intellectual property. Another approach software developers take is to hide the data used to train the model, i.e., putting the training data into a black box.
Consequences of a Black Box Approach
Using a black box approach leads to a number of problems: It masks potential flaws in the datasets used to train the AI model. This further creates a lack of accountability. For example, say an ML model determines that a person does not qualify for a bank loan. If the algorithm used is inside a black box, the person cannot find out why they were denied and therefore, effectively cannot correct the reasons.
The black-box approach makes ML models inherently unpredictable and difficult to correct if they produce undesirable results. This can have potentially deadly consequences, especially in the military. We've already seen this happen: For example, the U.S. Air Force reportedly conducted a simulated test in which it ordered an AI-enabled drone to destroy an enemy's air defense system, then attacked anyone who interfered with the command.
The Fundamental Problem with AI Regulation
Historically, a central problem in regulating emerging technologies such as the Internet was the unpredictability that arose from the fact that we had no way of knowing how society would use them; yet we had a solid understanding of how they worked. The problem with AI is twofold. As mentioned above, it is not just the problem of uncertainty about how the technology will evolve, but the lack of a fundamental understanding of how it works that further complicates the issue. When it comes to ChatGPT, for example, society is essentially following a black box approach, because we are unaware of its inner workings.
A fundamental requirement for regulating any technology is a good understanding of how it actually works. When it comes to AI, and generative AI in particular, this is a fundamental problem. OpenAI's recent furor amid ethical concerns over the rapid advancement of the technology further supports this point. Regulation cannot be effective unless the subject in question is fully understood. The uncertainty and mystery surrounding AI is largely due to the massive ignorance about how it actually works, and this needs to be rectified as soon as possible if future regulation is to be effective.
The EU AI law passed earlier this year spells out the need for transparency and accountability, especially for high-risk AI systems, but it is unclear who is directly responsible for implementing these obligations, and to what extent. The provisions regarding training data are also quite vague, leaving them open to misuse by big tech companies like OpenAI.
Prerequisites for AI Regulation: Disclosure of the Black Box
Until recently, ML models were used in low-risk applications like online advertising and web search, and their inner workings were not of much importance. But the recent boom in generative AI has brought it to nearly every aspect of our lives, making it essential to open the hood and peer inside the black box.
Interpretable models offer a more transparent and perhaps more ethical alternative to black box models. These are also known as “glass box” models. An AI glass box is a system where the algorithm, training data, and model are all visible to everyone. Additionally, the field of “explainable AI” (XAI) is working to develop algorithms that are not necessarily glass boxes, but are at least more understandable to humans. XAI techniques such as LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (Shapley Additive Explanations) are some of the tools being used to increase the interpretability of AI systems.
There is a widespread belief that the most accurate ML and deep learning models must be inherently unpredictable and complex. This assumption has been proven wrong on several occasions, such as the Explainable Machine Learning Challenge in 2018. Interpretable and glass box models have been shown in several different cases to be just as effective as black box models. While the EU AI law is a step in the right direction and regulators are increasing their scrutiny of Big Tech companies, more work needs to be done in this direction. It is difficult and highly impractical to make AI regulation effective if the technology in question is hidden behind a black box, whether the training data or the algorithm itself.
- About the author: Prateek Tripathi is a research assistant at Observer Research Foundation.
- Source: This article was published by the Observer Research Foundation
