Box CEO Aaron Levy says tech companies won’t be the last ones to see their AI budgets balloon.
“Of course, this starts with engineering. We already know that developers can run multiple agents in parallel or run projects overnight, but this ultimately impacts other knowledge work as well,” Levie writes about X.
As examples, Levie cited legal and sales as two areas that could become large-scale token users.
The token determines how AI is measured and how its consumption is priced. These are units of text, words, or parts of words that are essentially the building blocks of large-scale language models that power popular chatbots like OpenAI’s ChatGPT, Anthropic’s Claude, or xAI’s Grok. Importantly, the token is not a flat fee, and AI companies tend to charge higher fees when they use cutting-edge models or ask more complex questions.
Levie’s post was in response to NVIDIA CEO Jensen Huang’s observation that he would be “very alarmed” if engineers who are paid $500,000 do not use AI tokens worth at least half of their salary.
“At the end of the year, I’m going to ask that $500,000 engineer how much he spent on tokens. If he says $5,000, I’ll copy something else,” Huang said on Thursday’s episode of the “All In Podcast.”
“This fundamental concept and trend will become very real, as workers who leverage AI properly will only consume more of it,” Levie said.
“Their computing budgets only increase monotonically over time,” he writes.
It’s not just the workers, Levy said. Agents can run at any time, making them potentially the largest consumers of tokens.
“These are not chatbot workflows that answer simple questions, but agents that run and process incredible amounts of data at scale, generating all new forms of information,” he wrote.
Not everyone is happy with their AI computing budget. Venture capitalist Chamath Palihapitiya said he had to tell his startup software company, 8090, to stop using Cursor, a popular AI coding tool, after seeing how much the company was spending on tokens.
“Since November 2019, our costs have more than tripled,” Palihapitiya said on a previous episode of “The All In Podcast.” “If you add up the huge inference costs we pay to AWS, the Cursor costs, and the Anthropic costs, we’re just spending millions of dollars.”
Levy said businesses “need to figure out how to budget for this.”
“It will likely not become an IT budget item over time, but will eventually be owned and allocated by the enterprise,” he wrote. “Perhaps the CFO will end up in charge of AI :-)”