How to measure the use of advanced AI in the workplace

A groundbreaking study of 1.4 million real-world workplace interactions with artificial intelligence reveals teachable differences between everyday and advanced AI use, and provides organizations with a concrete roadmap to identify and scale high-impact AI capabilities.

Key findings:

• AI is everywhere, but the impact is not: The most sophisticated users don’t just “prompt better”; work With AI, it gets even better. Differentiators are observable behaviors. This means framing the problem, directing the model’s approach, and iterating instead of accepting the initial output.
• A measurable and teachable blueprint for successfully leveraging AI. The researchers evaluated approximately 30 behavioral signals and found sophisticated usage tracking with patterns such as regression to AI over time, persistent improvements, ambitious demands, and intentional tool/model selection.
• What can talent leaders actually do with this? The findings provide organizations with practical ways to coach high-impact AI collaboration as an employee competency, rather than a perk or tool deployment.

A joint study by US audit, tax and advisory firm KPMG LLP and the McCombs School of Business at the University of Texas at Austin identified clear and observable patterns in how influential users frame problems, guide AI inferences, and apply AI across the complex tasks that KPMG applies internally and in its client-facing work. This study harvard business review.

Researchers spent eight months studying KPMG LLP’s back-office operations to analyze how people are using AI in the workplace. The most successful users of AI, or the “sophisticated” users in this study, were not simply the users who used AI the most or those with the highest technical skills. Rather, they excelled at patterns of engagement with AI to frame problems, direct the AI ​​model’s approach to tasks, and apply AI across the business.

What is advanced AI utilization?

To move beyond assumptions about what “appropriate” AI use looks like, KPMG LLP collaborated with McCombs’ Shulkin School of Accounting faculty members Zach Kowaleski, Nick Holman, and Jamie Schmidt to analyze the behavioral signals embedded in real-world AI interactions, assessing more than 30 characteristics of prompting behavior across months of usage data, including task complexity, prompting techniques, and repetition patterns.

“We weren’t looking for abstract power users,” said Schmidt, the McCombs Professor of Accounting and director of the C. Aubrey Smith Center for Auditing Education and Research. “We were looking for someone who understood how to think with models, not just question them.”

It wasn’t experience or technical know-how that separated the best users. This research approach reveals consistent differences in how a small group of sophisticated users engage with AI over time.

See sophisticated AI in action

Sophisticated users have shaped the way AI approaches problems by treating AI as a reasoning partner and asking models to take on specific roles and perspectives. Provide concrete direction and examples. Show AI how to reason about tasks. Ask the model to explain how it arrived at its response. and provide ongoing feedback. Rather than accepting the initial output, they refined the model’s work through multiple interactions and applied it to the most complex and ambitious tasks.

They also set boundaries, specify structure, articulate clear goals, and delegate cognitively demanding tasks across brainstorming, analysis, technical guidance, and problem solving. For these users, AI was used as a general cognitive tool rather than a narrow productivity tool.

Importantly, these actions left visible, measurable patterns that organizations can observe. Advanced usage was strongly correlated with four signals: how often users returned to the AI, how tenaciously they refined the output, how ambitious their initial requests were, and how intentionally they chose tools and models.

“The gap between everyday and advanced AI use is not hidden in the prompts themselves, but in the patterns of engagement. And when those patterns are made visible, they can be recognized, discussed, and expanded upon.” said Anu Puvada, KPMG Studio Leaderled the company’s investigation. “Repetition makes ambition a reality, ambition drives strategic tool selection, and repeating successes strengthens engagement.”

How KPMG uses insights to upskill its workforce

KPMG implemented a company-wide training and development initiative to help employees develop the more sophisticated skills and behaviors identified in the survey. Approximately 5% of users consistently exhibit these behaviors across months of usage data, providing a clear, data-backed signal of what effective human-AI collaboration actually looks like.

“We recognized early on that access to AI alone would not lead to better outcomes.This is a challenge that many organizations are still grappling with. ” Steve Chase, global head of AI and digital innovation at KPMG, said:. “That’s why we intentionally introduced AI-enabled tools, training programs, and routines to help visualize and anticipate effective behavior, and to teach better problem framing, stronger AI oversight, and purposeful repetition.”