Box CEO Aaron Levie says he’s not concerned about the number of tokens his company’s engineers are using.
“Right now I’m like, ‘Oh, I should probably waste a bunch of tokens because that means I’m trying something new,'” Levy said on a recent episode of “The A16z Show.”
Levie’s view echoes the current situation in much of Silicon Valley when it comes to the use of agent AI. Nvidia CEO Jensen Huang recently said he would be “very alarmed” if an engineer making $500,000 didn’t spend $250,000 worth of tokens. Some companies, such as Meta and OpenAI, encourage “Tokenmaxxing” by displaying leaderboards of their largest users.
Review previously talked about how companies need to realize that the use of tokens is not just limited to their own engineering teams.
Tokens are a way for large language models to split words into numeric input and numeric output. AI providers, including OpenAI, charge fees based on the number of tokens used.
Engineers and businesses are facing a big question: how to deploy AI agents. By design, AI agents perform more sophisticated prompts and operate for longer periods of time, resulting in increased token usage. Levie said all engineers are working on this issue.
“You have to make a decision: Do you want it to be a long-running prompt? Does it need to be a long-running agent?” he said. “Do you want to parallelize it? Are you okay with wasted tokens?”
Levie said these issues won’t be resolved “until we actually find a way to build data center capacity.” At that point, tokens could become cheaper as AI providers no longer have to worry about overextending their finite computing. Already, AI companies like Anthropic have put policies in place to deal with peak usage.
Agent AI, on the other hand, isn’t just about using tokens, Levie says. He said CFOs and CIOs are “running around trying to figure out” whether their companies’ current IT and integration policies will work in the age of agent AI.
“Not from a performance standpoint, but how do you reconcile the fact that you’re probably accessing this system 10,000 times an hour or so?” he said. “But how do you make sure that people don’t accidentally move files from one folder to another while the agent is just running amok and someone else is trying to perform a write operation and someone else is trying to delete something?”