A major field trial on integrating generative artificial intelligence (AI) into the country’s justice system revealed on Friday that the use of AI solved an additional 1,848 cases a year, more than 6% more than the average.
A research paper on the introduction of generative AI assistants in Pakistan’s trial courts, titled “Tomorrow’s Courtrooms: Evidence from the Nationwide Deployment of Generative AI,” published on Tuesday, said: “This increase does not appear to have come at the expense of a decline in quality.”
According to the paper’s authors, researchers Sultan Mehmoud, Christophe Gossman, and Elliot Asch, the custom assistant: Judge GPT — was a chatbot based on OpenAI’s GPT-4 family of models. It was customized to Pakistan’s context and beta-tested “intensively” with the Federal Judicial Academy before being rolled out for use by 1,559 judges serving in 118 courts.
The findings showed that the tool was most effectively implemented when combined with targeted training for judges on its use. The trial found that introducing assistants alongside targeted training solved an additional 1,848 cases per year, an increase of 6.3% above average.
“Judges who received access to AI along with targeted training on the use of the tool were more likely to adopt AI, use it more intensively, and continue to use it over time,” the study found. “Their attitudes toward AI have also changed. They expect tools and targeted training to improve their productivity.”
It added that targeted training on the implementation of AI tools has clearly found that its use has shifted to tasks where language models are “likely to be more useful”, such as text improvement, and away from open-ended legal questions, where “validating answers is costly”.
The trial involved “approximately half of the nation’s judges and 80% of the district courts,” and 1,559 judges were randomly divided into three groups for the study.
One group was given access to JudgeGPT with targeted training on how to use the tool in judicial work, and the other group was given access to the tool with only “general training” on technology and law. The control group received general training without assistant access.
Results were measured by a baseline survey of judges’ attitudes toward generative AI, followed by another survey approximately three months after implementation measuring post-treatment perceptions and expected productivity gains. record of Judge GPT We also accessed the platform to measure uptake and determine what tasks judges assigned to the tool.
Additionally, we accessed administrative court records at the local level to investigate whether assigning judges to AI tools would facilitate local case resolution. Judicial opinions before and after the trial were also used to assess the impact on the quality of the writing and “whether the AI changed the writing’s attitude towards gender and religion.”
The results showed that while the post-treatment opinions of the “treated” judges contained more sentences classified as AI-generated than those of the control group, there was little evidence that this led to a decline in the quality of the sentences.
“If anything, AI is having a positive impact on quality ratings.” The study also notes that “there is little evidence of a systematic shift in pro-Muslim or gender bias in judicial language.”
Judges were primarily using the tool for legal research and writing support, but targeted training found that its use had shifted to “limited support tasks such as text improvement and summarization rather than full-text generation.”
“These usages lend themselves well to large-scale language models and are more likely to sustain judicial institutions,” the study added.
In its conclusion, the report said: Judge GPT We found that the use of AI was increasing, but sustained engagement was highly dependent on targeted training.
“We are not researching AI to replace judges,” the paper said. “We’re studying this as a tool that could potentially change how judges perform repetitive parts of their jobs.”
“AI is therefore not a panacea for law enforcement agencies facing persistent backlogs. However, when tools are built around relevant legal materials and combined with training that directs their use towards appropriate tasks, it can be a practical tool to improve national capacity.”
In April, the National Judicial Policy-Making Commission (NJPMC) formally published national guidelines for the use of AI in the judiciary.
Key highlights of the guidelines include a human-centered approach that ensures that AI supports, rather than replaces, judicial decision-making and that judges remain the “final arbiters.”
Similarly, it focuses on accountability and accountability, promotes ethical and transparent use, and provides strong safeguards against bias.
