In his recent annual letter on the judiciary, Chief Justice John Roberts warned the legal community against over-reliance on AI. “The use of AI requires caution and humility,” he wrote, adding that human judges are unlikely to be removed anytime soon, if at all. This agreement was notable as it recognizes the growing presence of AI in the judicial field and highlights significant shortcomings of AI in law.
“AI in legal practice is no longer a question of whether, but how. Daniel E. HoProfessor of Law at Stanford Law School and Senior Research Fellow at the Stanford Human-Centered AI Institute (HAI). “The ‘how’ question is more difficult: how do we leverage AI without compromising the professional and ethical obligations that are at the heart of legal practice?”
Information is downstream from institutions.
Hey, Stanford professors. Julian Nyarko and chris manningand HAI Managing Director Vanessa Parli are the editors. special edition of Proceedings of the National Academy of Sciences (PNAS) Explore the future of AI and justice. This issue includes the following papers: Neil Guhaa recent graduate of the Stanford CS Ph.D./J.D., and currently an associate professor at Columbia University School of Law, co-authored with Ho, Nyarko, and Manning, “There are no free benchmarks: An institutional perspective on legal AI benchmarks.” Ho said the problem stems from how legal AI systems are currently tested and validated, a stage in the AI development process better known among computer scientists as “benchmarking.”
“Legal AI is missing something we say” readability“We know surprisingly little about the performance of legal AI systems, the types of mistakes they make, and the likelihood of them making mistakes,” Ho said. The consequences can be severe, with more than 1,700 lawsuits involving illusory facts, events, and laws. ”
This field needs who, whatand how Benchmark’s. Guha describes this as an “institutional view” of benchmarking. “A benchmark is a series of decisions about the configuration of metrics, data, and methodologies. While there has been much discussion in computer science about benchmark design, there has been little analysis of the underlying institutional dynamics in environments where benchmarks are expensive and data is sensitive. After all, someone You need to benchmark and find out that someone is operating under clear incentives and constraints. ”
Guha points out that an institutional perspective can go a long way in explaining why legal AI is not readable. “When we consider the diverse actors in the legal AI ecosystem, including developers, companies, academics, and public watchdogs, it begins to become clear why there are widespread challenges around AI transparency.”
Benchmarking for institutions
In their paper, the authors suggest that overcoming readability issues with legal AI requires serious consideration of institutional design. “The key is to recognize the institutional constraints in society,” Ho explains. the beginning From there, identify how the benchmarking process should be structured. ”
The authors separate their recommendations based on the amount of resources available to tackle the benchmark effort, which they represent as high, medium, and low resource configurations. For example, high-resource settings may utilize public institutions that can participate in benchmarking in an independent and neutral manner based on their expertise. Ho and his co-authors note that one of the candidate institutions, the National Institute of Standards and Technology (NIST), is already modeling this approach in the context of facial recognition. Conversely, resource-poor settings require a more targeted approach, focusing on settings with marginal benefits. Any Benchmark information is maximum. This could include areas of law such as bankruptcy law or child custody, for example, which are the areas where individuals are most likely to seek advice from popular consumer chatbots.
This paper is just one of many. special features In this issue, PNAS approaches emerging or underrepresented research areas through an interdisciplinary perspective. As a top interdisciplinary institution, Stanford HAI was invited to put together this special issue. Topics include collective licensing of copyright works for training, whether AI regulation should target adopters or developers, how fairness in machine learning should address recent changes to the Equal Protection Doctrine, using legal interpretation to align AI systems with human values, and using machine learning to map federal common law as a network. All works are subject to rigorous peer review and require full transparency regarding conflicts of interest.
