OpenAI, which developed both ChatGPT and Whisper, told RNZ that “coping with hallucinations is an area of ongoing research.”
photograph: NurPhoto via Jonathan Ra/NurPhoto/AFP
The project was canceled due to severe abuse by police officers who were trialling the generative AI, but was restarted six months later because officers used unapproved AI models.
The police high-tech crime group has been using approved Whisper AI in investigations for nine months since the project was restarted with additional controls.
The testing and use of the AI system was revealed in an Official Information Act document released to RNZ.
Officers tested Whisper’s ability to translate and transcribe non-English audio for six months until last March.
The documentation states that Whisper is 45% inaccurate in Maori and Pacific languages and cannot be used in these languages.
However, it was scrapped in March last year because police officers were prohibited from using it to transcribe English during trials.
“The abuse of justice demonstrated the limits of relying solely on behavioral regulation,” the document states.
Police reinstated the device in September and told RNZ it could only be used by trained and authorized personnel and its output would not be used as evidence.
hallucination
This use of AI in law enforcement investigations is controversial around the world, with some critics saying the stakes are too low to be reliable. But supporters, including companies selling AI, say it will save employees countless hours of paperwork.
New Zealand Police have banned generative AI (GAI) until mid-2024 and said they would take extreme caution in introducing any system.
A Cornell University study in May 2024 found that Whisper AI produced fabricated hallucinations, including racial comments, violent rhetoric, and imagined treatments, 1 percent of the time.
OpenAI, which developed both ChatGPT and Whisper, told RNZ that “coping with hallucinations is an area of ongoing research.”
Whisper was an open source research model released in 2022.
“As we have consistently said publicly, voice recognition systems are not perfect and must be carefully evaluated depending on the intended use case,” a company spokesperson said on Tuesday.
The company continues to improve its models, developing new models like GPT-Realtime-Whisper, which it says has fewer hallucinations and is more accurate in a variety of languages.
The trial began six months before police introduced their first generative AI policy in March 2025. It began three months after police announced there were no plans to ban the use of GAI.
A spokesperson for the Muslim community said that while they trusted police and supported the use of technology, it was clear that Whisper was being used against ethnic groups.
“We should have been consulted,” said Abdul Razak of the Federation of Islamic Associations.
“The trial has been terminated.”
The court concluded that AI was “efficient and valuable to Intel.”
However, the OIA document also includes a statement from Police Headquarters’ Technical Oversight Group, which also says that “significant abuse was observed (English audio was processed despite restrictions)”.
“Not approved for continued use. Trial ended.”
In the same month, a public presentation by police on the March 2025 GAI stated that misuse or error could have a significant impact on the public.
There were also internal warnings that the use of AI could cause irreparable damage, including to the reputation of law enforcement, so a high degree of due diligence was needed from the start.
“This is even more important given there is no specific legal framework for the use of AI in New Zealand,” he warned.
“Unapproved Substitutes”
Whisper made headlines again in technology watchdog circles in September 2025.
“The memo provided a call for reconsideration of Whisper AI following its previous termination,” the minutes state.
“We observed strong operational demand. Turning off the tool increased the risk of staff using unapproved alternatives.”
Detective Chief Inspector Keith Borrell of New Zealand Police’s National Criminal Investigation Group did not mention abuse in a statement to RNZ.
Borrell noted the value of Whisper’s investigative work, saying, “The tool has been temporarily turned off while test results are analyzed, additional safety measures are put in place, and guidance is updated.”
Police did not respond to questions.
“I’m not aware of any serious issues.”
Borrell said police thought Whisper was the best model to test.
Regarding US reports of fabricating non-existent content, the statement said: “To date, we are not aware of any significant issues arising from the inaccuracies inherent in automatic translation or transcription.”
Currently, its users must complete mandatory AI awareness training and be aware of the “limitations of this type of tool.” Recordings primarily in Te Reo Māori or Pacific languages were not permitted.
“Text has been added to the output to make it clear that the records should not be relied upon for critical decision-making or evidentiary purposes without verification by appropriately qualified persons,” Borrell said.
Police policy introduced in March 2025 states that if GAI is used in the draft, the final human-reviewed recording must be disclosed to the court if it is used as evidence.
Police policy requires a six-monthly audit of the use of GAI.
RNZ requested to see these audits of Whisper AI.
Borrell said late Tuesday that police have not conducted an audit of Whisper’s use.
“Although there has been no formal audit since last year’s trial and evaluation, TranScriptor [as police call the model] It is managed under strict usage management.
“This service is subject to continuous monitoring to ensure reliable operation, system performance is continuously monitored and all usage is logged to the user for future auditing purposes.”
The police document emphasizes establishing a “continuous evaluation cycle” and communicating openly with the public about the use of AI.
The Ministry of Justice said it is considering the use of transcription technology for court proceedings.
“As do New Zealand Police, [sic] “We are also advancing the use of Speech to Text for transcription requirements, and both agencies are in regular contact to share and learn from our respective approaches,” the spokesperson said.
Comes with tender out.
Sign up for Ngā Pitopito Korero. Our daily newsletters, hand-picked by our editors, delivered straight to your inbox every weekday.
