Rethinking Productivity Measurement in the Age of AI

Executives are focused on how generative artificial intelligence will impact employee productivity. As chief strategy officer at DevSecOps software company GitLab, I spend a lot of time talking to customers about the impact AI will have on software development.

Organizations have largely overcome fears around AI and are now working to make it scalable and sustainable. However, many executives need help quantifying AI's impact on productivity. In a recent GitLab survey, more than half of executives (57%) said measuring developer productivity is key to business growth, and 51% feel their methods for measuring developer productivity are flawed, or that they would like to measure it but don't know how.

So what does “productivity” mean in this context? How should executives view and measure the impact of generative AI on their development teams?

Integrating AI into an organization's workflow leads to improved business outcomes, building strategic capabilities, and improving competitive advantage. Developers play a pivotal role in all three aspects. Finding meaningful ways to measure the impact of AI on developer productivity in these areas is essential to connecting AI to business outcomes and unlocking its strategic value.

Measuring beyond power

Traditional metrics like lines of code, code commits, and tasks completed often overlook key aspects of software development like problem solving, teamwork, and innovation, which are crucial in assessing business impact. Understanding AI contributions requires more than just aggregating time, team dynamics, and tasks. These metrics must be linked to tangible business outcomes like user adoption, revenue, and customer satisfaction. It's also important to keep in mind that business outcomes may vary across companies and projects.

Tracking the overall project completion time and maintaining a comprehensive view of the development pipeline is crucial. This includes monitoring deployment frequency, change lead time, and service recovery time, giving you a holistic view of project efficiency. Additionally, evaluating team metrics is also important. Colleague support, working environment, work engagement, and collaboration have a significant impact on employee turnover and productivity.

Developers spend only about 25% of their workday writing code. The rest of their time is spent fixing errors, resolving security issues, or updating legacy systems. Using generative AI to automate these tasks allows developers to use their expertise more effectively and focus on creativity and complex problem-solving. This not only drives innovation, but also increases job satisfaction. Performance reviews, turnover, and internal customer satisfaction surveys are valuable tools for tracking these improvements.

Additionally, AI is crucial in predicting development bottlenecks and automating routine tasks, leading to more predictable release cycles and faster time to market. Gen AI improves code reviews and creates comprehensive test scenarios to make code more reliable and reduce bugs, leading to higher software quality and customer satisfaction. Gen AI can quickly and accurately customize software to match user feedback, ensuring products more effectively meet customer needs and expectations.

These AI-driven improvements can be measured through customer feedback, service requests, analyst and peer reviews, and overall market performance, providing a clear picture of AI’s contribution to business goals.

Strategic choices to help developers

Recognizing that the impact of generative AI on developer productivity will also impact business performance, strategic capabilities, and the competitive edge of the enterprise, executives must make strategic choices about adopting AI to empower their development teams.

1. Empower developers as decision makers: Give developers decision-making power over which AI tools will increase their sense of ownership and engagement, empowering them to decide how to integrate AI into their work.
2. Iteratively adapt: Encourage a culture of experimentation and iteration with AI tools. Allow development teams to go through a trial-and-error phase to understand how AI best fits into their processes. Support them in the long-term gains while they may experience short-term productivity hits as they adapt to new tools.
3. Beware of bad habits: AI can help less experienced developers write code faster and improve their skills, but it can also unintentionally teach poor coding practices, which development team leaders need to monitor closely.
4. Deploying AI for long-term transformation: Think of AI not as a temporary fix, but as a transformational tool that can fundamentally change software development. By aligning their AI strategy with their long-term business goals, companies can ensure sustainable growth and leadership in a technology-driven market.

Get the most out of what you measure

Developer productivity is multi-faceted – not just about task completion and time management, but also team dynamics, problem-solving skills, and more. To truly understand how developers contribute to business value, management needs to take a more holistic view.

A recent survey revealed that an acceleration is underway, with 69% of respondents saying software is being shipped at least twice as fast as it was a year ago, yet only 26% said they were implementing AI.

Forward-thinking executives need to consider how AI tools can improve the quantity of work and the quality of business outcomes, so that companies can not only measure AI's true potential, but also empower themselves to make the most of it.

Ashley Kramer is chief marketing and strategy officer at GitLab Inc. This article was written for SiliconANGLE.

Image: SiliconANGLE/Ideogram