For the CIOs who spoke in last week’s talk, the race to scale AI across the enterprise is more of a marathon than a sprint. Momentum AI Conference in New York City.
According to the committee, which includes CIOs from Whirlpool, Cleveland Clinic, and Duke Energy, AI pilots need time to demonstrate their capabilities, prove reliability, and drive end-user adoption. Rapidly advancing AI for speed is counterproductive, according to a discussion moderated by Alexander Puutio, an adjunct professor at Harvard and Columbia universities.
In fact, panel members said taking the time to identify business outcomes, gain buy-in from employees, and measure the effectiveness of AI pilots took priority over deploying AI quickly.
Panelist Priya Ponnapalli, senior vice president of engineering at Scale AI, an AI infrastructure and software company, said an important first step in moving from pilot to production is recognizing the difference between consumer AI and enterprise AI.
“If you have a consumer-facing chatbot and it’s wrong 5% of the time, that’s curiosity. But if you’re an enterprise agent, and it’s wrong 5% of the time, that’s a big liability,” Ponnapalli says.
When integrating agents into critical areas such as medical devices and insurance claims processing, the margin of error must be low, she said.
She also pointed to the need to identify very clear and measurable business outcomes when using AI agents. Deploying agents requires a rigorous evaluation-driven approach, which is different from evaluating models against benchmark datasets, Ponnapalli said.
Important differences include understanding that agents often have prompts, policies, tools, and orchestration logic that need to be evaluated, as well as the environment in which they operate. In an enterprise, this may mean operational APIs that use databases and file systems.
“You really need an evaluation strategy that tests your agents end-to-end,” she said.
It is also important to have a well-designed evaluation that shows the agent’s performance and provides confidence that it can be moved into production. This is all aimed at improving the agent over time, Ponnapalli added.
Whirlpool CIO talks about AI change management challenges
“I honestly think the biggest challenge with scaling is not about technology, it’s about change management,” said panelist Daniel Brown, senior vice president and chief information officer of consumer electronics company Whirlpool.
Brown said he has been driving digital transformation for more than a decade and change management is at the core of his efforts.
Whirlpool uses agent AI models to predict appliance demand. The models use a variety of inputs to generate estimates, but as technology evolves, Brown said it can become difficult to determine inventory output based solely on such models.
To cover its bases, Whirlpool took a multi-layered approach that included traditional processes. on top of the agent model. “We’re running both at the same time, which gives business users the ability to trust their data,” she said.
Change management should also include conversations with employees to gain buy-in for implementing resources that will benefit the organization, Brown said. “We have peer-to-peer discussions as we extend the same model to other parts of the business,” she said. “It’s not like an engineer comes in and says, ‘This is the model we want you to use.'”
Priya Ponnapalli of ScaleAI and Duke Energy’s Richard Donaldson of Momentum AI. (Joan Pierre S. Rouss/InformationWeek)
Cleveland Clinic CIO: “Slow is smooth and smooth is fast.”
Early on in an AI pilot, it’s important to clarify which questions the tool is intended to answer for your organization, says Sarah Hatchett, senior vice president and CIO at Cleveland Clinic. This will determine whether the project moves forward or not.
This requires understanding what metrics are. What will be the impact of AI? and whether the organization is ready for the changes that this implementation will entail.
“I think we need to design the pilot in a way that answers these specific questions,” Hatchett said.
She cited the mantra often heard in military circles: “Slow is smooth and smooth is fast” to explain carrying out operations methodically and efficiently rather than hasty actions that can delay desired results.
While you may want to keep up with market movements, Hatchett cautioned against rushing. “There is a risk of firing [AI] When you put it out there, without having that discipline up front, you fall into a gray area where things seem to be working just fine,” she said.
cleveland clinic is AI tool that listens to outpatients’ voices Consult with your doctor and prepare notes in the format required by your healthcare provider. There was great demand for this, but Cleveland Clinic didn’t jump on it without careful review, she said.
“We evaluated five different vendors with this capability over time and set a specific time period to evaluate this,” Hatchett said.
Cleveland Clinic selected vendors based on the quality of the output and physician acceptance of the tool’s note-taking capabilities, she said.
Once the clinic decided to scale up the pilot, more than 6,000 providers began using the tool within four months, she added. Approximately 80% of physicians participating in the system continue to use this tool on a daily basis. “If you take the time to understand how it works in your environment, it’s a great introduction,” Hatchett said.
Duke Energy CIO: Expanding AI pilot requires employee buy-in
Exploring AI pilots may mean taking a big leap into unknown possibilities, but it’s important to remember that pilots may impact some people, said Richard Donaldson, senior vice president and CIO of utility company Duke Energy. That may require some ingenuity. “We’re getting them used to the output of the AI and handling the AI,” he said.
Donaldson likened the importance of adoption within an organization to the early days of software such as Excel and Lotus 1-2-3. Back then, one person would understand the functionality of the software and share that knowledge with another colleague.
“If we bring all our employees together, we have 26,000 employees, and we can get used to it. [of AI]”And then they realize that instead of eliminating what these tools can do, they improve what they can do, and then all of a sudden these use cases start to ignite.”
Still, keeping an organization’s employees interested in new technology can be a challenge for CIOs. Identifying and communicating the business outcomes of AI pilots is key to gaining long-term employee buy-in. The value of piloting doesn’t have to lie solely in cost savings. Donaldson said safety, customer satisfaction and product reliability could be improved.
He recommended that pilots be “prescriptive” about what they deliver and decide how to measure it in terms of end-user pain points, which could require a very different approach to solve. “Think about your users. It’s different for each user group,” he said.
