Not all companies have the size and skills of Intuit's credit Karma, but the company's data science head has advice on where others can start to come up with their own AI governance frameworks.
Credit Karma can use Intuit's Genos AI operating system using a catalog of AI models, agents and software development tools. With the help of Genos, the Credit Karma team recently created a multi-agent system that automatically checks AI output before it can reach production. These form the technical basis for the AI compliance initiative led by Madelaine Daianu, senior director of data science and engineering at Credit Karma, but these efforts began with practical human collaborations that other companies could emulate as they had to devise their own coordinated approaches.
Break and break the response generated in LLM to the internal red team, learn from it and develop a thorough custom evaluation framework for use cases.
Madeleine DianeCredit Karma, Senior Director of Data Science and Engineering
“It's very important to find a balanced act between innovation and safety, compliance, or anything related to them, and we took a step to slow down a bit before we could run and move faster,” Diane said. “Breaking the responses generated by LLM to the internal red team, learning from them and developing a thorough custom evaluation framework for use cases.”
At Credit Karma, Red Teams has come up with a five-stage evaluation framework for AI governance, breaking through large-scale language model (LLM)-driven workflows and identifying weaknesses.
The framework stages include:
Response quality and accuracy
AI safety including bias detection
Compliance is primarily when presenting credit card and loan information to customers on the platform, along with contractual expectations of credit karma partners.
Data Origin and Accuracy
System metrics such as cost and delay
“In this framework, compliance is where we had to be extremely innovative. [manually] Check out the LLM overview. “For example, credit cards, you need to express the benefits of a mapped card with the maximum accuracy to your partner brand. But to do that, you had to extract the fields from, say, a summary relating to charges and fees.”
That's how multi-agent systems have emerged. A specialized AI agent checks each specific data field in the LLM-generated summary to ensure that presentations to users follow the partner brand. In this and other stages of this evaluation framework, LLM is also used to determine the overall quality of responses from a group of agents.
However, these models were trained with human feedback from Credit Karma's customer success team, which is still running spot checks. According to Daianu, AI agents reapply that assessment process to a new summary, making it up to 50 times faster.
However, when evaluating AI tools, it is also important not to overuse them, Daianu said.
“We use genai as a judge for some elements of our framework, especially for compliance, but not anywhere,” she said. “For AI safety, traditional machine learning can be used. It's important that it doesn't fit too well with genai. It's important because it often gives better accuracy, better explanability and isn't a black box.”
Beth Pariseau, senior news writer at Informa TechTarget, is an award-winning veteran of IT journalism covering Devops. Any hints? Please email her Or reach out @pariseutt.
