A recent internal experiment at Westpac found that generative AI-assisted software engineers were 46% more productive, with no loss in code quality, compared to a control group who performed the same task exclusively manually. bottom.
Westpac’s chief technology officer, David Walker, said the AI coding experiment was conducted in the bank’s Growth Labs division, which has been conducting AI experiments for “the last few years.”
Westpac’s Growth Lab, an extension of its innovation unit, Co.Labs, has been working on generative AI for “a little over two years,” he said.
Walker said he wants to test the impact of generative AI on software development and whether it helps or hinders programmers and their deliverables.
The experiment brought together 60 engineers who were “randomly divided into four groups.”
“One of them was a control group. That control group basically had to be hand-coded and had to do what they normally do,” Walker said.
The other three groups were given generative AI tools from Microsoft, Amazon, and OpenAI and given “about 3-4 hours” to familiarize themselves with the basics.
Each team was then given seven tasks across different coding languages, consisting of challenges such as extracting and exporting data, writing unit tests, and transforming data.
Walker said the tests took place “over several days,” adding that “the headline results were pretty impressive.”
“There is an overall 46% increase in productivity in terms of generative AI tools supporting the coding of the three teams given these tools compared to the control team that just coded the normal way. I get it, it’s great,” he said.
The code quality didn’t noticeably decrease with the productivity gains.
“We scanned all of our code for vulnerabilities and looked at maintainability and reliability perspectives, all the key metrics we look at when looking at code quality, and we didn’t see any degradation. said Walker. .
He added, “We looked at the average time it took hand-coding teams to actually complete a task, and it was 3.5 times longer than the time it took other teams to complete their tasks.” rice field.
The bank also evaluated the impact of generative AI assistance on developers with different levels of experience.
Walker said 83% of young engineers were “shocked” and appreciated “there was help to help people in the early stages of their careers.”
On the other hand, more experienced engineers found the tools to handle the “heavy tasks” and “allow them to focus on the more complex aspects of the software.”
Walker said the experiment was “short and sharp,” but the results were “great.”
Feedback gathered from attendees included comments such as, “I’m so excited to be incorporating these tools into my daily workflow,” and others saying, “Just ask the AI the right questions and it will do what I want.” I was able to get a working Python code that does exactly that.”
“It’s very hard to write Python code if you’ve never done it before. We’re developers here, but you probably have Java or other code experts. [generative AI] It’s about working on it,” Walker said. [it has the potential to deliver] It gives very strong results. ”
mesh
Walker said Westpac’s in-house engineering platform, called “mesh,” is where the expected experimental results can be applied.
“We had [the mesh platform] This system has been in development for about five years now and is where our engineers build user interfaces, APIs, microservices and all kinds of technical components to build applications,” said Walker. .
“We do about 40 percent of our development in that environment, so we want to start there.
“This is our environment, and we build a lot of our internal code. Our website, our mobile banking app, and a lot of the applications we build ourselves are all built on that environment.”
He said the platform is “already a very productive environment for our engineers” and that new AI capabilities are expected to “enhance it even further.”
“What we want to do is eliminate all the effort, eliminate all the noise, and just let them do what they want,” Walker said.
“mesh environment” [generative AI] Once we hooked this up, about 40 percent of our engineers had access to this feature out of the box, and it looks like the first team is already using it. So we have already started. ”
Walker added that Westpac is “starting to heat up the engine” and gaining momentum for much of the underlying work it has been developing over the years.
“In my view, engineers are the first-class citizens of the bank,” he said, adding that he is excited to provide them with generative AI tools.
“They are people who really want us to be able to act as efficiently and effectively as possible,” he said.
Walker said generative AI is “a big advance, [engineers] thing [to make] Being able to help them work more efficiently, have more fun, and grow career-wise is huge. ”
Cassist and AI
Walker said the AI coding experiment is separate from Westpac’s work with conversational AI company Kasist, which it is using to develop its finance-specific Large Language Models (LLMs). .
Westpac said in April that the partnership would help create a more secure and accurate LLM than ChatGPT.
Walker said public LLMs “are trained on the Internet, so they can say anything and can be misleading.”
“We can’t afford that…that’s where Cassisto comes in,” he says.
“Kasisto trained a large language model for the financial sector.
“Kassisto’s models are trained very specifically about banking conversations, that’s why they specialize in that area, and that’s why we know vaults. We will never mislead our customers or staff or do the wrong thing.”
