National Disability Insurance Agency staff are using machine learning to help draft plans for NDIS participants, documents obtained by Guardian Australia reveal.
Documents about the NDIA’s use of AI, released under the Freedom of Information Act, show that 300 staff members took part in a six-month trial of Microsoft’s Copilot AI starting in January last year.
The agency said Copilot uses generative AI, but it is only used for NDIA email, meetings and other non-customer-facing tasks, not for participant planning.
But before the Copilot trial began, the NDIA was already using a type of AI, namely machine learning, to create draft budget plans for participants, according to documents.
Machine learning is defined as “a subset of AI that involves the use of algorithms to learn from data and make predictions and decisions without being explicitly programmed.”
The NDIA said NDIS staff made all final decisions regarding the plan, and its AI policy document from April 2024 states that “AI tools must not access participant records” without explicit permission from the Chief Information Officer and authorization under the NDIS Act.
A briefing paper prepared for the Senate estimates for 2023-2024 states:
“This algorithm is only used to make recommendations, and decisions are made by real representatives.”
“Machine learning recommendations will be used to assist participants by speeding up initial analysis and provide participants with faster resolutions and service improvements,” the document continues.
Sign up: AU breaking news email
The June 2024 report said staff were more productive when writing documents and emails by “interpreting NDIA policies and generating purpose summaries.”
Overall NDIA staff reported a 20% reduction in task completion time during the Copilot trial and a 90% satisfaction rate, including hearing-impaired staff who reported positively on the use of live transcription during meetings.
The report said challenges faced by the tribunal included staff concerns about the findings of the Robodebt Royal Commission into automated decision-making and concerns about AI being used to reduce staff numbers.
The end-of-trial report notes that one of the risks of using Copilot is accidental data leaks, but said authorities will put in place robust access controls, regular audits and training for employees.
Dr Georgia Van Torn from the University of New South Wales, who has written about the impact of algorithmic decision-making in the public sector, said machine learning and data-driven approaches often fail to “address complexity and nuance”.
“I don’t think that’s necessarily a bad thing, especially in relatively simple cases. But… you can’t expect a machine learning approach to predict the kind of support someone will need if they don’t fit neatly into the box. And most people do, right?”
Van Toorn added that machine learning also has a “black box” problem. This means that it is difficult for humans to know what data points a machine is using, what weights it is giving it, and what biases it is assuming as it learns and makes decisions.
“I think there’s an assumption that because it’s data-driven, it’s accurate and personalized… But I think the important part in this case is that the humans in the loop need to understand the limitations.” [of the technology] …and exercise your own discretion and judgment.
After newsletter promotion
“And to do so at the right time, you need the right training and support.”
Van Tone said it was important to note that the NDIA document clearly states that decisions about aid programs are made by humans.
But she cautioned that there is a lot of evidence of what researchers call “automation bias,” or how people are influenced by AI recommendations when making decisions.
“Planners may have time constraints or pressure to deliver a certain amount of plans to meet KPIs, or there may be pressure across the NDIA to reduce the number or cost of plans,” she said.
“The risk is that if planners make their job faster or easier, they may be more likely to rely on algorithms and follow recommended plans instead of using their own judgment or listening to NDIS participants.”
Dr Stevie Lang Howson, an NDIS participant and disability advocate, said her “biggest concern” was whether staff were trained, equipped and given the time to “meaningfully plan for their individual needs”.
“These are actually people’s lives. It’s how many times people can go to the bathroom… It’s how often they can get out of the house, it’s whether the wheelchair you’re sitting in is too small and causes pain… These things are so serious and so important that they need to be crafted carefully and transparently and in a way that reflects people’s individual needs.”
An NDIA spokesperson said AI is not used in systems that “interact directly with participants or providers or are used to make decisions about NDIS funding or eligibility.”
“Participants will use the information and evidence provided by participants in accordance with the NDIS Act to make decisions regarding their NDIS funding,” the spokesperson said.
The federal government on Wednesday announced a government-wide AI plan for the use of generative AI in public services. Finance Minister Katie Gallagher said the plan would give all civil servants access to generative AI tools and provide training and guidance on how to use them safely and responsibly.
