Fairness tools catch AI bias early

Machine learning software helps institutions make important decisions, such as who gets a bank loan or the area where police should patrol. However, if these systems have bias, even small systems can cause real harm. For example, certain groups of people can be underestimated in training datasets, and learn that machine learning (ML) models can increase bias, leading to unfair outcomes such as loan denials for prescription management systems and higher risk scores.

Researchers at Carnegie Mellon University's Computer Science School (SCS) created Fairsense to help developers deal with inequities in ML systems before the harm occurred. Currently, most fairness checks consider the system at a particular time, but ML models are being trained, adapted and modified. FairSense simulates these systems in an environment over a long period of time to measure inequity.

“The key is to think about the feedback loop,” says Christian Kestner, associate professor in the Department of Software and Social Systems (S3D). “There may be small biases in models, like slight discrimination against gender and race. When unfolds, models create effects in the real world. It discriminates against people. It often involves fewer opportunities, less money, getting into prison.

“So it may be small at first, but it can be effective in the real world and lead to a vicious cycle where bias grows as the model learns again.”

In “FairSense: Long-term Fairness Analysis of ML-enabled Systems,” SCS researchers investigated how fairness changes as these ML systems are used over time. They focused on testing these systems in a dynamic environment rather than in a static state.

To use FairSense, developers provide information about machine learning systems, models of the environment used, and metrics that indicate fairness. For example, in a bank, the system could be software that predicts the creditworthiness of the applicant and makes loan decisions. The environmental model includes relevant information from the applicant's credit history and how credit scores are affected, and the fairness metric can be parity between the different groups of people approved for the loan.

In addition to Kestner, the team included S3D's Yining Sheing and associate professor Eunsuk Kang, a doctoral student. Sumon Biswas from Case Western Reserve University also participated in the study. The study was presented at an international conference on software engineering earlier this year.

“It simulates how equity changes over the long term after the system is deployed,” she said. “Observing the increase in inequity over time, the next step is to be able to identify core factors that influence this equity and actively address these issues by helping developers address these issues.”

ML-enabled systems are deployed in a variety of complex situations that are not always predictable, so FairSense can capture and simulate that uncertainty in an environmental model. For example, loan lending can have no control over your credit score renewal or new loan applicants, which can affect how the system works over time. FairSense simulations generate a wide range of possible scenarios based on these variables, allowing developers to identify factors such as credit score thresholds and other parameters.

“A lot of the software we build can have a negative impact on people,” Kang said. “The systems we build have social implications. Those who build these systems should think about issues that may arise not only now but over time.

“What potential bad things can happen when building and deploying systems? I hope that reading these papers will encourage software developers to think more broadly by creating and actively addressing these types of issues before they can be deployed in the real world.”

Researchers will expand Fairsense's work to continuously monitor the fairness of ML systems and develop tools to explain how these systems become unfair.