NEW DELHI: The Supreme Court on Thursday stressed the need to regulate the use of artificial intelligence in judicial proceedings, noting that untested artificial intelligence-generated material can undermine the administration of justice.Directing the Bar Council of India (BCI) to set up a committee to consider the use of AI in legal practice and trials, the court said there was a clear need for legal safeguards. It also called on both courts and tribunals to exercise caution when using such technology.The court said that while AI can assist in adjudication, it should not replace human reasoning or decision-making. He stressed that human oversight must remain central to all stages of the judicial process.The court expressed concern about the use of fabricated or non-existent precedents generated by AI, warning that such material could have a negative impact on the integrity of the judicial system. The court described the judicial use of false, non-existent, AI-generated hallucinatory precedents as “invisible, insidious, and devastating” and likened it to “the release of methyl isocyanate (a highly toxic, flammable, and volatile organic compound) in the field of law and justice.”This observation comes as the Supreme Court is quashing the orders of the National Company Law Tribunal (NCLT) and the National Company Law Appellate Tribunal (NCLAT) in the Essel Infra Project bankruptcy case after it was found that the NCLT had relied on a non-existent fake AI-generated hallucinatory precedent while deciding the issue. The court found that such reliance undermined the integrity of the judicial process and required a new review of the dispute.
Why the Supreme Court set aside the Essel Infrastructure Project order
A bench of Justices PS Narasimha and Alok Alade was hearing an appeal filed by suspended director Pooja Ramesh Singh against the NCLT’s decision to admit Essel Infrastructure Project to corporate insolvency proceedings on the basis of a petition filed by Jammu and Kashmir Bank under Section 7 of the Insolvency and Bankruptcy Code.The insolvency proceedings arose out of alleged default in connection with a Rs 200-crore credit facility extended by Jammu and Kashmir Bank to Pan India Utilities Distribution Company Limited. The loan was backed by a corporate guarantee from Essel Infrastructure Projects and a mortgage on land at Gorai in Borivali, Mumbai.The Mumbai Bench of the NCLT had in August 2024 allowed the insolvency petition against default claims of Rs 8,743 crore and the NCLAT had subsequently upheld the order.However, the Supreme Court found that the NCLT had relied on fabricated and non-existent AI-generated judgments as precedents in reaching its conclusion.Calling such reliance unacceptable, the court said, “Courts must adopt a zero-tolerance mode for creating, quoting, or using AI-generated precedents without verification. Citing such judgments without verification is an offense on the part of advocates.”Additionally, the court said it would be equally gross negligence if a judge relied on AI-generated false or hallucinatory material when deciding a case.“We have no hesitation in declaring that such a decision is not a decision in the eyes of the law, regardless of whether such material is directly or indirectly related to the decision-making. Even if a large amount of false or hallucinatory material enters the decision-making process, such a decision violates the sanctity of the judgment and should be disregarded,” the court said.The Supreme Court, therefore, set aside both the NCLT and NCLAT orders and directed the court to pass a fresh verdict based on the facts of the case.The court clarified the scope of its ruling, saying it did not consider the technical reasons behind AI hallucinations or how to eliminate them. These problems are aimed at engineers and scientists. Judiciary authorities were concerned that false, non-existent AI-generated material would be used as precedent.
