Silicon Valley has spent the past year turning its use of AI into a scoreboard.
However, when we asked four executives how they measure the return on investment of AI, At the Mistral AI summit in Paris last month, I made a surprising realization. No one started by talking about how many AI tokens their employees were using.
Charles Holive, chief AI officer at BNP Paribas CIB, said his team is focused on outcomes rather than what he calls “vanity metrics.”
“We are trying to move away from vanity metrics of billions of tokens a day,” he said. “We strive to make sure that what we track are outcomes, not vanity metrics.”
Holive said that instead of asking how many AI tokens an employee has used, ask, “What did you do? Didn’t you do it before? How quickly could you do it?”
Antoine Pichot, director of innovation, digital and data at La Banque Postale, told Business Insider that the bank evaluates AI based on whether it increases employee efficiency, improves customer service and provides value for money.
Amit Kapur, chief AI and transformation officer at Tata Consultancy Services, one of the world’s largest IT services and consulting firms, said the focus is on whether AI is improving performance, rather than the number of tokens employees spend..
Meanwhile, Sujay Bhattacharyya, executive managing director and global head of digital workplace services and Mistral AI at NTT Data, which helps large enterprises implement AI tools from multiple AI providers, said customers are increasingly looking beyond the number of tokens to the overall cost and business value of AI projects.
“tokenmaxxing” backlash
Mistral’s comments at the AI Summit come as some U.S. companies are starting to move away from “token maxing,” the idea that increased use of AI and increased consumption of tokens automatically translates into increased productivity.
Last month, Amazon shut down its internal AI-powered leaderboards after employees started completing tasks just to improve their rankings.
Uber Chief Operating Officer Andrew McDonald has publicly questioned whether increased AI spending is creating more useful products, saying it remains difficult to draw a direct line between token consumption and customer value.
Meanwhile, OpenAI, Anthropic, and GitHub have all moved to usage-based pricing for enterprises, putting more pressure on employees at customer companies to prove they can meaningfully benefit from increased usage of AI.
Moving away from tokenmaxxing does not mean that companies have stopped tracking AI token usage.
Holive said the team at BNP Paribas CIB continues to monitor token consumption to manage costs and measure adoption.
But the executives I spoke to in Paris all seemed to agree on one point. That said, while tokens may show how much AI is being used, they don’t necessarily tell us whether it’s producing a measurable return on investment.