Around 90% of employers are using AI to some extent in recruitment, but there is virtually no research on how this is impacting job candidates.
Image: GenAI. For illustrative purposes only.
In one of the first studies to analyze AI recruiting tools, researchers at Stanford University found that algorithms made racially biased decisions in many job applications. “A lot of previous research has shown racial bias in hiring when people make decisions,” said co-author Dan Jurafsky, Jackson Eli Reynolds Professor of Humanities in the College of Humanities and Professor of Computer Science in the College of Engineering. “We were surprised that an AI system that uses game-based ratings to rank people was still biased against Black and Asian applicants.”
The researchers also found evidence that some candidates were repeatedly rejected from multiple jobs. This suggests that some candidates may be locked out because companies are all relying on algorithms created by the same vendor. The researchers presented their findings at the ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT) in Montreal on June 27.
The rise of AI tools in recruitment
Job listing websites and the expansion of remote working have made it easier to apply for more jobs, but it can also be difficult for candidates to stand out among the ever-growing pile of applications. For example, in 2024, Google received more than 3 million job applications for approximately 20,000 roles.
Many employers contract with third-party AI vendors to help screen candidates. AI-based tools promise not only to manage large volumes of applications, but also to reduce human biases that can hurt some job seekers. But this shift also means that many companies’ screening decisions are left to a relatively small number of AIs.
The study’s authors wondered how this “algorithmic monoculture” might impact the application process. “Many different employers are using recruiting AI tools, sometimes using the exact same tools or tools built by the same vendor. We were interested in seeing what kind of results that produced,” said lead author Rishi Bommasani, a senior research fellow at Stanford University’s Institute for Human-Centered Artificial Intelligence.
To find out, the research team used a dataset from Pymetrics. This dataset consists of more than 4 million applications submitted for approximately 2,000 positions between 2018 and 2022. After initially applying for a job, applicants were redirected to Pymetrics’ game-based assessment, which aims to measure soft skills such as risk tolerance, focus, and generosity. Based on the score, the algorithm divides candidates into “recommended” and “not recommended” categories.
The researchers looked for evidence of racial bias using an application that included demographic information. They used a standard set by the U.S. government called the “four-fifths rule.” If a group’s recommendation rate for a position is less than 80% of the most recommended group, it’s a red flag for potential discrimination.
When the researchers first examined the data, they asked whether the entire application was within this standard. Overall, I found that to be the case. “There may be some bias, but it doesn’t rise to the level of legal concern,” Bommasani said.
However, when we calculated the percentage of group recommendations for each individual job opening, a new picture emerged. They found that 15 percent of Asian applicants and 26 percent of Black applicants applied for jobs where AI tools appeared to be biased against their racial group. The selection algorithms for these jobs recommended Asian and Black candidates less than 80% of the time, compared to key groups dominated by white candidates. The researchers calculated that 40,000 additional applications from Asian and black candidates would have been recommended if racial groups had been chosen at equal rates.
“I didn’t expect this at all,” Bommasani said. Especially since a preliminary analysis of the entire application showed no significant bias. “Some companies believe that AI will make decision-making fairer,” he added. “That’s not necessarily what our results suggest.”
The researchers also looked at how often applicants who apply for multiple positions are rejected by everyone, an outcome known as “systemic rejection.” We found that 4% of applicants who applied to 10 positions using game-based ratings were rated “Not Recommended” by AI for all positions. This rate was higher than would be expected if companies were making their own decisions about whether to proceed with the application.
“The AI algorithms we studied were much more likely to act in the same way and reject people outright than if companies were acting individually,” Jurafsky said. “This suggests that this kind of monoculture, where all algorithms are the same, can cause problems.”
The study found that 15% of Asian applicants and 26% of Black applicants applied for jobs where AI tools appeared biased.
Make recruitment tools fair and transparent
It’s no secret that human recruiters can introduce bias into job decisions, and research has shown this for decades. New research shows that AI can also make biased decisions, even when judging seemingly neutral criteria such as gameplay scores.
“We still don’t understand what kinds of algorithms have such disparate effects on different groups of applicants, and we don’t know what causes these disparities,” Jurafsky said. “What we need is continued research. We can’t fix disparities if we don’t know what causes them.”
This result reveals how disparities can be hidden simply by looking at the average promotion rate of applicants across all job categories. “One of the lessons from this study is that it’s always important to disaggregate, because there can be a lot of complexity hidden by the average,” said co-author Percy Liang, a professor of computer science.
The findings also highlight the need for independent research into such third-party tools. But hiring data like the one the team used tends to be kept private by companies, hampering such scrutiny. New policies requiring AI companies to share data could help make the hiring process more transparent. “Without policy, it is highly unlikely that further research will be conducted on the impact of AI and employment,” Bommasani said. “There’s just no way to get the data.”
The results also show that employers, who are ultimately responsible for preventing discrimination, should question the vendors they hire for AI-based screening to ensure they have verified that their algorithms are not discriminatory, Bommasani said. “There is a clear incentive for companies to internalize this and make more sophisticated procurement decisions.”
—
Jurafsky is also a professor of linguistics. The study’s authors include Sarah Bana, a digital fellow at Stanford University’s Digital Economy Institute and an assistant professor of management science at Chapman University. Kathleen Creel is an assistant professor of philosophy and computer science at Northeastern University.
Funding sources for the research included the National Science Foundation and a Stanford Lieberman Fellowship.
This post originally appeared on Stanford News and was republished here with permission.
Review by Irfan Ahmad.
Read more: AI cannot replace mental health therapists. But here’s where the difference could be
[ad_2]
Source link
