Measuring the impact of our AI investments in IT at Microsoft

As an IT organization, we need to understand which of our AI investments are creating business value for Microsoft. We need to know how that value shows up, whether we can measure it, if we can trend it, and how we can use what we learn to make better decisions for the company.

That’s why, as part of our broader approach to AI at Microsoft, we—Microsoft Digital, the company’s IT organization—are building a framework to measure the impact of the AI investments we’re making on behalf of the company.

“If we want to measure the business impact of AI, the conversation quickly moves toward identifying the agents or AI efforts that are driving the most value and satisfying business outcomes. We know that those conversations can be complex, so we use a value measurement framework to capture and assess the signals we have available.”

Don Campbell, principal group technical program manager, Microsoft Digital

Our framework is helping us move from AI enthusiasm to AI accountability. It creates a common way for us to talk about value across our different initiatives, teams, and business processes. It also helps us ask a harder, more specific question every time we assess the impact of AI at Microsoft Digital: If AI saves time, reduces costs, improves quality, or lowers risk, what changes are we making to take advantage of that?

We don’t have the full answer yet—we’re still improving the way we measure. Some of our signals are instrumented, some rely on strong hypotheses, and some need better telemetry. But we’re not waiting for perfect results to start learning.

“If we want to measure the business impact of AI, the conversation quickly moves toward identifying the agents or AI efforts that are driving the most value and satisfying business outcomes,” says Don Campbell, a principal group technical program manager in Microsoft Digital. “We know that those conversations can be complex, so we use a value measurement framework to capture and assess the signals we have available.”

Building a framework for AI business value

AI value doesn’t show up the same way everywhere. One investment we make might help employees complete a task faster, while another might improve quality, reduce risk, increase coverage, or lower operational costs. Some of that value can be measured directly, while in other cases it starts as a hypothesis that needs to be tested. That range is why we needed a common framework instead of a single metric.

Our value measurement framework helps our Microsoft Digital teams answer three basic questions before and after they build:

• What kind of value do we expect this AI investment to create?
• How will we measure that value?
• What will we do with what we learn?

We organize the answers to those questions around six value areas:

Our framework doesn’t require every AI investment that we make to create value in all six areas. In fact, that rarely occurs. A support automation scenario might focus primarily on productivity, employee experience, and cost avoidance. A security scenario might involve risk reduction, vulnerability coverage, or the ability to address more issues than a team could handle manually. A process quality scenario might relate to measures that are specific to a particular workflow and be harder to roll up into a single number.

“Measurement sources vary based upon the type of AI initiative. The six value areas give us a framework for measurement, but we need to observe and collect information where measurement starts to become practical. Teams need to understand the processes affected, including how long work takes, how many resources are involved, and what the workflow looks like before and after AI.”

David Laves, director business programs, Microsoft Digital

The framework creates consistency in how we talk about value, while giving teams room to measure what actually matters for their scenario. Some measures roll up easily, including cost savings, time savings, and certain risk measures. Others are more specific to the process being improved. Those measures still matter because they help business owners understand whether AI is changing the work in a meaningful way.

“Measurement sources vary based upon the type of AI initiative,” says David Laves, a director of business programs in Microsoft Digital. “The six value areas give us a framework for measurement, but we need observe and collect information where measurement starts to become practical. Teams need to understand the processes affected, including how long work takes, how many resources are involved, and what the workflow looks like before and after AI.”

Our framework also helps us make better investment decisions. Before we commit to an AI scenario, we can map the opportunity to the value areas that matter most, estimate the value we think it can create, and decide what needs to be measured. After implementation, we can compare results against the baseline, review the data with the right business and AI owners, and adjust the work based on what we’re seeing.

That last step is critical. Our framework creates a way for us to create an operating rhythm around AI value. It helps us take a promising scenario and prove its worth (or lack thereof) by establishing expected value, evaluating what is being measured, and deciding what to change because of the results.

It’s a continual evolution of business value measurement that prioritizes progress over perfection, and it’s helping ensure that our AI approach stays grounded in the outcomes that are most meaningful to our business.

Turning measurement into an operating rhythm

A framework only matters if teams use it to make decisions. For us, that means moving measurement out of one-off reporting and into a regular management cadence. We track our highest-business-value AI initiatives by priority and business function, then review the KPIs that show whether those investments are creating the value expected.

Some KPIs roll up cleanly. Our cost savings, time savings, and certain risk measures can be summarized across initiatives and discussed at a leadership level. Other KPIs stay closer to the individual scenario because they’re tied to a specific workflow, process, or business outcome. We need both. Our rollup metrics help our leaders see broad progress, while the scenario-level metrics help our teams understand what’s changing inside their work.

“We actually create monthly targets and end-of-year targets for every top AI-enabled initiative,” Campbell says. “Then we basically reiterate every single month with our leadership team to look at the value we’re driving and have conversations about it.”

That monthly rhythm helps us proactively manage AI value. If a measure is trending positively, we look at what’s working and where else the pattern might apply. If a measure is off track, teams can dig into the supporting data, review the assumptions, and decide whether they need to adjust the solution, the measurement, or the operating process around it.

This process supercharges prioritization. Our team here in Microsoft Digital has a large set of AI opportunities, and not every idea can move at the same pace. By mapping initiatives to value areas, estimating expected impact, and tracking results over time, we can have a more grounded conversation about where to invest, where to scale, and where to keep learning.

That discipline becomes more important as AI moves deeper into business processes. We don’t want teams to measure only adoption or usage if the real goal is a better business outcome. Usage matters, but it doesn’t tell the whole story. A tool can be used often and still fail to improve the process it was meant to change.

Applying the framework: Global Support

Consider the following example of applying the framework from our Global Support team. This team is currently examining how AI can help automate specific pieces of the ticket management process.

“In Global Support, the processes that often matter most from a value perspective are the ones with high repetition. If a process runs thousands of times a month and can operate autonomously, without human input, that’s where AI can deliver meaningful, measurable impact.”

David Finney, principal program manager, Microsoft Digital

As part of this effort, we examined a support process that depended on manual follow-up. In this process, after a Global Support team member marks an issue as resolved, the team waits for the user to confirm that the ticket can be closed. If the user doesn’t respond, the agent must follow up once a day for up to three days. After the third attempt, the agent simply closes the ticket.

This user flow gave us a practical way to test the value framework. It has repetition, because it runs often; it has autonomy potential, because the steps are deterministic and rule-driven; and it has a clear time-savings opportunity, because a human agent spends time checking the ticket, writing the follow-up, and sending the message. It also has a measurable implementation effort, because the data exists in the ticket process but the solution still needs to integrate with ServiceNow.

“In Global Support, the processes that often matter most from a value perspective are the ones with high repetition,” says David Finney, a principal program manager in Microsoft Digital. “If a process runs thousands of times a month and can operate autonomously, without human input, that’s where AI can deliver meaningful, measurable impact.”

Finney estimated that about 5,000 tickets a month execute this process. Because each ticket can require up to three follow-ups, that can create up to 15,000 manual email follow-ups a month. At about three minutes per follow-up, that’s roughly 750 hours of productivity spent on one small piece of the process each month.

The framework helps us look at that work through both value and effort. On the value side, we can evaluate repetition, autonomy potential, and time savings. On the effort side, we can assess whether the data exists, how complex the solution is, whether engineering work or system integration is required, and how long implementation may take.

“We started to evolve our conversation to, ‘So what?’” Campbell says. “You saved money, you saved hours. What did you do with it? Where did the actual business outcome sit?”

Don Campbell, principal group technical program manager, Microsoft Digital

That structure matters because a high-value opportunity still needs the right implementation path. In this case, the process is part of the ticket workflow, so the needed data exists. The complexity comes from integrating the automated agent with ServiceNow so it can interact with the ticket, check whether the user responded, send follow-ups, and resolve the ticket according to the defined process.

Instead of trying to automate all of support at once, the team identifies specific subprocesses where AI has a clear role and the value can be measured. “It’s taking a sort of bite-sized approach to AI rather than trying to solve for everything in one big go,” Finney says.

That’s the kind of practical example the framework is designed to surface. It helps us find work that’s frequent enough to matter, structured enough for automation, and measurable enough to prove whether the AI investment changed the process.

Moving from savings to reinvestment

Measuring value starts the next conversation. If an AI investment saves time, reduces cost, increases coverage, or improves quality, we need to know what happens next. The number is relevant, but the business outcome is more important.

“We started to evolve our conversation to, ‘So what?’” Campbell says. “You saved money, you saved hours. What did you do with it? Where did the actual business outcome sit?”

That’s the harder part of AI value measurement. A team might use AI to reduce time spent on repetitive work, but the real value depends on how that recovered capacity gets used. In some cases, the reinvestment path is clear. A team can point to more programs delivered, more backlog reduced, more issues reviewed, or faster service delivery.

In other cases, the value is harder to trace. Some AI improvements return small amounts of time to many employees. Those minutes matter, but it’s difficult to prove exactly where each person reinvested them.

We’re careful when it comes to measuring ROI. We know our leaders will ask for it, and it belongs in the broader value conversation. But we don’t want ROI at the center of the story before we have the right cost model, telemetry, and approved data to support it.

For now, we’re focused on the operating discipline: Define expected value, baseline the current state, instrument the AI-enabled process, track results, review the data, and act on what we learn. That discipline is teaching us a number of practical lessons:

• Measurement needs to be built into the design of the AI investment, not added after launch.
• Teams need a baseline for the current process, so they can compare it with the AI-enabled process.
• Teams need to pick measures that fit the scenario.
• Data must have clear ownership, because uncertain data weakens the conversation with business owners and leaders.

Consistency matters as much as the metric itself. When our teams review value on a regular rhythm, they can see trends, test assumptions, and adjust the solution or the process around it. Some measures will be mature, and others will be directional. Some will need better instrumentation. The point is to keep improving the quality of the measurement while keeping the conversation focused on business value.

We’re continuing to build our value measurement muscle across Microsoft Digital. We’re not looking for one perfect formula for every AI investment. Instead, we’re creating a repeatable way to define value, measure it, review it, and use it to guide our next action.

As our AI investments and overall strategy mature, that framework helps us stay honest about what we know, clear about what we still need to learn, and focused on the outcomes that AI is designed to improve.