How CIOs evaluate AI models

Machine Learning


According to , Anthropic led the way in enterprise AI adoption in the US, securing 34.4% of the market compared to OpenAI’s 32.3%. Ramp 2026 AI Index In May. Among first-time AI companies, Anthropic wins about 70% of head-to-head matches.

“Anthropic was an early winner among people who were already experimenting with AI in their companies,” said Ara Karazian, chief economist at Lamp. These early adopters are a good leading indicator of where the rest of the market is going, he said. ”

This change is a milestone in the AI ​​platform wars. But for CIOs making purchasing decisions, leaderboards are never the main event.

“I would be cautious in a direct confrontation,” said Phil Leslie, chief technology and innovation officer at Cornerstone Research, an economic and financial consulting firm that supports high-stakes litigation. “The differences between the major Frontier models are real, but narrow and constantly changing. A more useful question is not ‘Which model is best for this quarter?’ but ‘Which setup can I switch to as Frontier changes?'”

Related:Security is slowing down autonomous AI. How are CIOs responding?

That pragmatism is widespread. as AI model As competition intensifies, CIOs say they’re less focused on picking winners and more focused on building win-win architectures.

Security first, everything else second

Before we talk about performance, price, and features, CIOs say, AI platforms must pass security first and a governance test.

“Our work relies on strict customer confidentiality, so a platform must clear several non-negotiable hurdles before it can be considered,” Leslie said. “Client data is never used to train the model, interactions are never exposed to human review, and the data remains on U.S. infrastructure. These constraints define the feasible set; everything else is a choice within that.”

Eric Pace, head of AI at Cox Business, the commercial services division of Cox Communications, agrees.

“Safety is non-negotiable for us,” Pace said. “Given the amount of critical infrastructure we manage, we have to start with risk and consider whether the solution meets our security, legal, data privacy, and governance requirements. If it poses a risk that we’re uncomfortable with, it’s not worth pursuing.”

At Lowenstein Sandler, a national law firm, the hurdles are similarly high. “Security and confidentiality is not one factor among others,” said Maureen Norton, the company’s chief information and innovation officer. “They are teeth This is a threshold test. ”

The model is not Hori

With security and governance requirements met, CIOs say performance is important, but not in the way that much vendor marketing suggests. The differences between the major models are small and constantly changing. Betting on today’s benchmark winners is a short-term play at best.

Related:Outsourcing contracts were not built for AI. CIO is currently being renegotiated

“The durability advantage was never in the model. The model is the easy part,” said Jeremy Brook, a partner at West Monroe, a management and technology consulting firm. “The advantage lies in how quickly an enterprise’s data assets, context, workflows, controls, and signals can be turned into action.”

Cornerstone Research intentionally built a stack that is model agnostic. “The frontier is moving too fast to connect our architecture to a single vendor. The risk of lock-in is real, and gaps that seem decisive now could close within two quarters,” said Leslie.

Several CIOs say they operate a portfolio of AI platforms tailored to different use cases, rather than picking a single winner.

Multiple models to play different roles

“We don’t approach it as picking a single winner,” Lowenstein Sandler’s Norton said. “We think they’re occupying separate lanes rather than competing for one seat.”

This approach is becoming the norm. “The continuous model of leapfrogging has allowed companies to accept that the rate of change is only accelerating,” said West Monroe’s Brooke. “Enterprises are no longer focused on the ‘smartest model’ but on modular platforms that reduce switching costs as new solutions emerge.”

Related:The real power behind OpenAI’s new jalapeño chip

Ramp’s Kharazian sees the same pattern in data spending. “The evaluation is moving from ‘Which AI vendor should I use?'” “Which model should perform this task and at what cost?” he said. “This will push enterprises toward multivendor setups, routing, open source models, and inference platforms.”

freedom within the framework

Developer enthusiasm is the driving force behind AI adoption. But CIOs say bottom-up demand works best when AI is present Guardrail in place.

Cox Business’s Pace describes this approach as “freedom within the framework.” The company offers a set of managed model options across on-premises and cloud environments, and within that framework, teams have the flexibility to choose the best fit for their use case.

“Developer preferences play an important role, especially in model selection,” Pace said. “Importantly, all of this takes place within a broader governance framework, which we see as an enabler rather than a constraint.”

A similar trend is seen at Cornerstone Research. “Our data scientists are strong and opinionated. They perform their own evaluations and have real-world views on model performance,” says Leslie. “That input is essential.”

However, a complete assessment also requires IT departments to assess security and review legal confidentiality requirements. These reviews are not done in isolation. “Leadership’s job is not to override the technical judgment that comes from the team,” Leslie argued. “Just to make sure the whole assessment actually happens.”

At Lowenstein Sandler, Naughton frames this as a design principle. “The healthiest version of this is for governance to set boundaries and the front lines to drive priorities,” she said.

Watching the Meter: How CIOs Manage Consumption

Governance solves one problem. Costs create new costs.

“AI is the fastest growing spending category we’ve ever seen,” said Ramp’s Kharazian. “The average company is spending 13x more on tokens than in January 2025.”

Usage-based pricing allows CIOs to know exactly what they’re spending, but not necessarily what they’re getting in return. As adoption grows, CIOs are developing new disciplines to manage consumption without suppressing value.

West Monroe’s Brooke said the first move is to tie spending to results. “You can’t manage what you can’t attribute to,” he said. “So before trying to reduce spend, disciplined companies will track which teams, workflows, and increasingly who’s driving spend, in relation to units of business value.”

Pace takes a value-first approach to cost management for Cox Business. “When AI helps solve years of backlogs in weeks, the conversation moves from controlling costs to asking what more is possible,” he said.

Karazian sees three trends emerging:

  • visibility: Separate subscriptions, APIs, and inference platforms.

  • Fitting the model and task: Adapt cheaper models to simpler tasks.

  • control: Set alerts, set spending limits, and track attributes before your bill arrives.

“Most companies are not slowing down their AI adoption,” Pace said. “They’re trying to make a profit without getting away with the bill.”

What CIOs want now: Evaluate AI models from output to action

A year ago, CIOs were wondering if AI tools would actually work. That question has almost been resolved.

“Our standards are increasingly geared toward integration depth, security posture, governance suitability, and more agent capabilities—tools that perform actions rather than just generate text,” Norton said. “Bars have gone from novelty to durability.”

When an AI system can take actionThe evaluation will completely change.

“Once a system can search the web, call external services, and write and run its own code, you have to ask a different class of questions,” says Cornerstone’s Leslie. “What guardrails are there? How do we stop this virus from doing what it does within our environment?”

Old tests were about what the model produced. “The new test is also about what the model does,” Leslie said. “That’s a really difficult question.”

Model selection remains important. But CIOs also need to build an architecture that can adapt when the leaderboard flips again.





Source link