Not every AI project is a winner.
Therefore, CIOs must apply the “fail fast” principle to their operations. AI initiativesif a promising idea doesn’t work, decide as soon as possible.
That’s easier said than done. of MIT report The State of AI in Business 2025 found that 95% of the 153 senior leaders surveyed had “zero profits.”
To understand how CIOs decide when to halt an AI project, we asked two IT leaders. What are the specific red flags that indicate an AI pilot has become a sunk cost and needs to be retired? Both identified clear signs that a project is going off track.
-
Sujin Baerstockchief information technology officer at Great Day Implementations, a direct-to-consumer home improvement company, said missed milestones were a red flag to change course, and that careful advance planning made abandoning the AI project virtually non-existent for her.
-
ed clarkThe CIO of California State University, which serves nearly 500,000 students, said stagnant progress and weak implementation are clear indicators that a project is faltering, and it’s important for leaders to be aware of these signs so they can reallocate resources to more promising initiatives.
Below are Beerstock and Clark’s answers to our questions, edited for clarity and length.
Soo-Jin Behrstock, Chief Information Technology Officer, Great Day Improvements
Beerstock: “First of all, what does success look like?”
“When approaching an AI initiative, I always start with: What does success look like and how do we measure it?
“For example, if you’re using AI for sales or marketing forecasting, you start with a small sample of data that you’re familiar with. Based on that, you have a good idea of what the output should be. If the output isn’t going in the right direction, you usually know something is wrong. It could be the data, the process, or the model.
“From there, we set short milestones, usually every few weeks, to see if we are getting closer to the results we defined with measurable results.
“If we weren’t [getting closer to the outcome]then pivot or postpone. I don’t think about promoting AI just by saying I’m doing it. If success isn’t clearly defined or you can’t measure progress toward success, that’s a red flag.
“I don’t know about killing. [an AI initiative] Unless you decide that it is not suitable for your business. ”
“Change direction to success”
“One of the things I feel like is that some developers can get paralyzed in their analysis in terms of how AI should work, so timelines and budgets are extended. But if incremental milestones aren’t being met, you have to think about what needs to change to be successful.
“Let me give you an example. Right now, we’re working on using AI predictive modeling. We’re taking a small sample of data that we’re familiar with and measuring what the output looks like, so we can say, ‘Here’s what’s actually good. This works.’ Then we can add more data and measure whether the model is working correctly or if we need to pivot.
“In such cases [where we need to pivot]You may not have the right resources or skills, so you may need to partner with a consulting firm to help. ”
The value of being “very intentional”
“I’ve never had to be in a position where I can say, ‘Let’s kill it.’ But if something I thought made sense for the business turns out to not make sense later on, I might do that. But I’ve never been in that situation yet, because everything was very intentional. I’m very careful about setting expectations up front and defining success. So if I haven’t hit a milestone, it’s usually [because of issues] We’re data- and process-centric, so we just need to figure out where we need to adjust and pivot. ”
Ed Clark, California State University System CIO
Clark: List of red flags
“In my opinion, a red flag is when a pilot loses a clear path to creating strategic value for the organization.
“Another red flag is when the team gets stuck in a loop, when the same status updates come back but no progress is being made, when they see the same slides or the same hurdles, when they hear, ‘We’re almost there,’ but nothing is happening and there is no deliverable. That’s when you know this is stuck.
“The other thing to watch out for is when you have low adoption rates, when you put something out there that everyone says, ‘Oh, this is going to be really cool,’ but no one uses it.
“Also, if executive sponsorship runs out, that’s something I would look for as well.
“And the other signal that really matters, and this happens all the time, is when the vendor is creating the core functionality of the platform. [that’s similar to the AI project you’re developing]. We do not intend to compete with these vendors.
“And what ultimately happens, and this especially happens with artificial intelligence, is when the technology advances so quickly that the original use cases that you were looking forward to are now obsolete.
“Any of those could be red flags.”
Find the reason behind red flags
“We have to think about why some projects end up in the red.
What is being asked of you may be beyond what your team can accomplish. Next, you need to determine: [the AI project] Is it a good enough idea to pursue — do I want to pursue it and bring in outside resources to make it happen, or is it a pilot where the team is fine with just practicing and learning, or is management excited about it but not actually wanting to meet with us? I don’t Don’t pursue it because you don’t care anymore.
“All the money and effort you’re spending could be spent on something else that can accomplish your organization’s objectives.”
Pilot who failed to establish
“Let me give you a specific example. One of the things we always look at is affordability, and we thought that by creating an AI overlay, we could make open textbooks, or free textbooks, more accessible to students. [functions as] Private tutor.
“So we tried to pilot this, but it didn’t work out. It was frustrating because adding this support system was a way to make public textbooks more useful for students.”
“We found that the general faculty didn’t like open textbooks because they didn’t come with the materials they wanted. So, even though it was a great idea that would help us fulfill our mission and advance our strategic goals, and even though management thought it was great at the beginning, we had to cancel the idea.”
What the team learned from project cancellation
“It really hurt to make that call because I think our students are putting in a lot of effort.” [collectively] Hundreds of millions of dollars are spent on textbooks each year. However, we learned a lot by working on that project. For example, if you really want to do this, you’ll need to make sure it’s multilingual and can handle mathematical symbols. We learned something useful for the community that can be applied elsewhere. ”
