Over the past year, Silicon Valley has become obsessed with “token maxing,” or the idea that maximizing AI usage (and therefore token consumption) should improve worker productivity.
Charles Holive is measuring something entirely different.
The chief AI officer of investment banking at BNP Paribas CIB, one of Europe’s largest banks, told Business Insider that his team is tracking token consumption, but that’s not the first thing he’s looking at.
“We’re trying to move away from vanity metrics, billions of tokens a day,” Holive told Business Insider at Mistral AI’s summit in Paris last week.
“We strive to make sure that what we track is results, not vanity metrics,” he added.
Rather than asking employees how many AI tokens they have used, Holive starts with a different set of questions.
“What did you do that you hadn’t done before? How quickly did you do it?” he said.
Results beyond tokens
His comments come as some U.S. companies are beginning to question whether rising AI costs are yielding meaningful benefits.
Amazon recently shut down its internal AI-powered leaderboards after employees reportedly started completing tasks just to improve their rankings.
Uber Chief Operating Officer Andrew McDonald has publicly questioned whether rising AI costs are leading to more useful products, while GitHub recently moved Copilot to usage-based pricing as AI prices continue to rise.
For the AI projects he oversees at BNP Paribas CIB, Holive says he starts with a clear vision of how much revenue and productivity they can generate.
His team then creates KPIs and monitors progress against those goals on a monthly or quarterly basis.
That doesn’t mean his team is completely ignoring Token. Holive said the bank still has a dedicated team monitoring token consumption to track costs.
“We will focus on token consumption because we need to manage costs,” he said.
Beyond the leaderboard
Holive’s approach echoed comments made by other executives at the summit.
Amit Kapur, chief AI and transformation officer at Tata Consultancy Services, told Business Insider that companies should focus on “business outcomes” and “business impact” rather than “just the token as a single item.”
Antoine Pichot, director of innovation, digital and data at La Banque Postale, told Business Insider that the bank evaluates AI projects based on factors such as efficiency gains, customer satisfaction, employee satisfaction and financial impact.
Still, Holive said the use of tokens can still be useful as a measure of adoption, especially in software engineering.
But he said his approach differs from companies that focus on AI usage metrics.
“We didn’t say, ‘Use the tools, good luck,'” he says. “We did the opposite of that,” he said.