This article is taken from GTDT Practice Guide: Legal Technology. Click here for the full guide.
Few words in the legal industry generate the same level of excitement as ‘Artificial Intelligence’, or more specifically ‘Machine Learning’, when it comes to what the future holds for lawyers. However, discussions often remain in a rather abstract sphere, barely scratching the surface of what Machine Learning actually means for the legal industry, how it works and – from a law firm or legal department perspective – what is needed to create substantial value for clients. Efficiency gains, probably the benefit that first comes to mind for many of us, are just the beginning. This article explains the technology behind Machine Learning and how it is, and can be, used by lawyers in different industries. It also explores ways to unlock the true potential beyond efficiency gains and identifies potential challenges to be overcome.
Machine Learning
Definition
To automate a specific task, software developers traditionally draw up what they refer to simply as ‘rules’. To this end, they work with the relevant business team to write so-called user stories, which describe the requirements from the end user’s perspective. These informal descriptions are then converted into rules, which are then implemented in a computer program. During a subsequent testing phase, these rules are manually tweaked and adjusted until the desired output is obtained. Traditional computer programs thus consist of a series of well-defined rules that are executed one after the other, starting with the input data and ending, all being well, with the sought-after output.
Machine Learning (ML) takes a completely different approach. Instead of being provided with predefined rules, ML models are given training examples, each consisting of both input data and the desired output. On this basis, ML models are instructed to autonomously predict suitable output from input data. At the beginning of the training phase, a model will produce incorrect results, but by learning from these errors and updating its internal parameters, the model’s predictions gradually approach the desired output.
The prime example of ML models is artificial neural networks. Their name derives from the fact that they are loosely modelled after the human brain. The network of neurons that comprise a human brain is mimicked by nodes arranged in interconnected layers. The more layers are stacked on top of each other, the more powerful the model is. Such deep neural networks can grasp complex patterns, but because of the large number of parameters, or coefficients, the model must optimise, they require large amounts of data to train. Besides neural networks, a wealth of other powerful ML models exist, such as support vector machines, regression models, conditional random fields, clustering algorithms and decision trees. A key advantage of these models is that they usually have much fewer parameters than neural networks and for that reason require less training data to learn a certain task.
In ML, a distinction is made between supervised and unsupervised learning. In supervised learning, the training data is labelled (ie, the input data is annotated (highlighted) or assigned to a certain category). As this is typically done manually, labelling data forms the bottleneck for the application of Machine Learning. In unsupervised learning, the data is unlabelled. Unsupervised learning is mainly used to detect patterns in the input data, which are subsequently used by a second ML model to solve a specific downstream task using supervised learning.
Since unsupervised learning does not involve human labelling, the idea behind deep learning is to take as much data as possible to pre-train a deep neural network. The larger the model and the more unlabelled data is thrown at it, the more capable it becomes in carrying out a specific downstream task. This implies that the amount of labelled data needed to prepare the pre-trained model for the task becomes smaller, narrowing down in some cases to just a few examples – a phenomenon known as ‘few-shot learning’.
ML hype in the legal market
‘AI will replace lawyers’ was one of the claims made in the early 2010s, when the first ML applications for the legal market were offered on a wider scale. This narrative was typical of the hype cycle phenomenon that accompanies the introduction of new technologies. In keeping with Gartner’s Hype Cycle,2 ML in the legal industry has gone through the first three stages of ‘innovation trigger’, ‘peak of inflated expectations’ and ‘trough of disillusionment’. It has now entered the ‘slope of enlightenment’. While the narrative has shifted from competition to collaboration, where ML is perceived as a supporting technology, the introduction of ChatGPT seems to have triggered the replacement narrative once again. However, it is fair to say that there are certain areas of the legal industry, particularly legal research, contract review, e-discovery or investigations, where the use of ML has become standard practice. What makes the introduction of ML to the legal market somewhat challenging is the common perception that there is no room for error (ie, a solution is only considered useful if it delivers results that are 100 per cent correct – all the time). Naturally, this is also not guaranteed when tasks are performed by lawyers. First, lawyers are not aware of their own accuracy or precision scores that would give the audience – or team members – an idea of how well a given lawyer performs or how accurately the work has been carried out. Second, the numbers produced by ML models give the lawyers using the technology a sense of control, leading to technology being subjected to a higher level of scrutiny. However, it would be wrong to overlook the huge benefits that an accuracy level of, say, 90 per cent can bring if ML is embedded in the overall workflow in the right way.
Large language models
Of particular interest to the legal market are language models. In its simplest form, a language model predicts the next word based on previous words in a sentence. In recent years, so-called large language models (LLMs) have emerged. Adopting ideas from machine translation,3 researchers devised a neural network architecture, called Transformer,4 that is especially adept in carrying out this task. A crucial ingredient in this architecture is the so-called attention mechanism. It aids the language model in predicting the next word by directing it to those parts of a sentence already partly built that are relevant for that prediction. It was established that the larger the model and the text corpus used for training, the more capable these models are in carrying out downstream tasks such as document classification, text summarisation, questions and answers (Q&As) and text generation. In other words, the lawyers’ adage that ‘a lot helps a lot’ also applies to LLMs. They are therefore trained on huge amounts of data (including Wikipedia, the Book Corpus of more than 10,000 books of different genres and Common Crawl, which contains billions of internet pages). And although the computational costs for training LLMs are immense, it can be cast as an unsupervised learning task by randomly masking words in a sentence and instructing the model to predict the masked words.
Having taken inspiration from machine translation, the original Transformer architecture consists of two parts: an encoder and a decoder. When, for instance, a German sentence is to be translated into English, the source sentence must first be encoded. The decoder then takes the encoded German sentence as input to decode it and generate the English translation in a second step.
Since the introduction of the original Transformer model, a variety of modified models were proposed that differ in the details of how training is carried out. It was also established that the two parts of the original Transformer architecture can be used as stand-alone and provide state-of-the-art results in different downstream tasks. For example, encoder models such as BERT5 excel at information retrieval, whereas decoder or generative models such as GPT-3 and ChatGPT, which was built on top of GPT-3, excel at text summarisation and generation.
Utilisation in the legal field
Since 2012, the field of Machine Learning has grown tremendously, primarily through deep learning, with applications in speech recognition, machine translation, internet search (both text and image), object recognition, and semi-autonomous driving, among others. Whereas only a few research groups worldwide were working on ML before 2012, the number has grown exponentially since. At the same time, a variety of high-quality ML libraries as well as frameworks and pre-trained models have been developed and made available as Open Source, such as scikit-learn, PyTorch and Detectron2 (Meta), TensorFlow (Google) and Hugging Face. This democratisation of ML means that the latest innovations can be adopted and used within your own organisation or are available through start-ups and other providers (ie, off the shelf). Within the legal domain, ML models have surfaced mostly through SaaS solutions. Accordingly, many organisations of all sizes are faced with the question of build versus buy.
Build or buy
Build
To make ML work in your own organisation, the task to be automated must be well-defined, and sufficient high-quality labelled data must be available. If these two prerequisites are not satisfied, the final ML model will perform poorly. More specifically, if the training data is of poor quality, or the task is not well defined, the model will perform accordingly – ’garbage in, garbage out’. Insufficient training data can lead to so-called overfitting, meaning that the model does not perform well on new data that differs only slightly from the training data.
The self-deployment of ML models requires different technical skills, depending on whether they need to be trained or fine-tuned, and on how they can be accessed:
- Some pre-trained models, such as the DeepL Translator, need no further training and can simply be accessed through a web application.
- Other pre-trained models, such as GPT-3,6 need just a few explicit examples (few-shot learning) to carry out a certain task, now known as ‘prompt engineering’, and can be accessed through an application programming interface (API). These models may need the help of a data scientist or someone with similar programming skills to operate them.
- Finally, ML models that need to be trained from scratch, such as a model to classify contract types, or deep neural networks that need to be fine-tuned, such as a pre-trained object recognition model to recognise a certain type of document, require ML engineers and a suitable IT infrastructure.
Whatever approach is taken, it is important to rigorously evaluate the model performance on a test dataset before deployment. It should be recognised that ML models are never perfect. Humans are not perfect either, but mistakes made by ML models tend to be perceived as graver than those made by humans. If perfect results are required, a human validation of the model’s predictions is necessary. A validation station is essential for this task. The more intuitive and convenient the user interface (UI) of the validation station, the more efficient and reliable this part of the workflow will be. If, after testing, the decision is made to deploy the model, a final challenge is to integrate it into the existing workflow.
Buy
Typically, off-the-shelf solutions in the legal market build on ML models (eg, conditional random fields) and combine them with a UI for validating the results generated by the underlying ML models (eg, Kira Systems). Most of these off-the-shelf solutions offer advanced UIs and include basic workflow features, such as document tagging, work allocation among team members, basic dashboards and integration for various virtual data rooms (VDRs), where documents are made available. In addition, some solutions allow you to further train the models to be used for use cases beyond what the software offers out of the box. For example, the user may train models for specific clauses or to classify and compare legal arguments, work with additional languages, etc. The ‘labelling’ of the data is carried out through an intuitive UI. Most solutions support such labelling directly on the document, highlighting a few examples. The threshold is around 30–50 examples of the type of information that the solution should automatically identify in a wider universe of documents.
While deploying self-developed ML models requires specialists, working with many off-the-shelf solutions also comes at a cost: most off-the-shelf solutions do not disclose what technology they use nor what data was used to train their models, with the latter information being crucial for trusting the results produced. For example, when using a specific pre-trained model to classify clauses in contracts, it is not entirely clear on which documents the model was trained (eg, financial, lease or IT contracts). However, the language and type of clauses vary widely across different legal areas and specialities. Thus, an evaluation is required for each deployment before the information is relied upon and validated in detail. Also, to the extent data scientists or ML engineers are involved, it may become difficult for them to understand which underlying model has been used, making it even more difficult to systemise issues with the model. Therefore, relying on the results of such an off-the-shelf tool is comparable to relying on a result generated by ChatGPT (ie, without having any information about the sources or origin of the output information). Furthermore, integration options are often very limited. To upload documents into the solution, few direct integrations with VDRs are available to date. APIs are basic, making it difficult to transfer the results to another digital platform for review, post-processing and ultimately for publishing in the required reporting formats. Review of the results is either limited to the entire product or may be exported to Excel or Word documents.
Conclusion
While the build approach requires trained specialists, there are several off-the-shelf solutions available at relatively low cost that allow lawyers to venture into the application of Machine Learning for various purposes. As the use cases become more sophisticated, the more the results need to be integrated into a complex workflow, and the greater the technology stack and mature know-how is within an organisation, the stronger the case for starting to deploy and develop ML models within an organisation. Notably, with developments such as Microsoft expanding access to ML models created by Open AI (including ChatGPT) and Google launching Bard, specialists in ‘prompt engineering’ of these powerful tools will soon be working hand in hand with lawyers.
Challenges to using ML in the legal market
The following challenges are faced by users, particularly in the legal market, when envisaging, adopting or working with ML.
Lack of data
A major challenge to applying ML in the legal market is the lack of large datasets that can be used for supervised learning. The underlying reasons are manifold: confidentiality obligations, competition to harness know-how within larger law firms, lack of sharing and open-source culture compared to the software world, major legal publications published by a few main publishing houses and not accessible to the public. Fortunately, this is slowly changing. For example, in 2021 the CUAD7 dataset of over 500 English legal contracts was introduced, which contains 13,000 examples of highlighted, salient text covering 41 categories relevant for contract review. The dataset can be used for training an extractive Q&A model. The preparation of such labelled datasets is resource-intensive, costly and far from trivial. The CUAD dataset is estimated to carry a price tag of about US$2 million. And despite the great effort the responsible team undertook in training, instructing and supervising the law student annotators, a subsequent paper8 trying to explain the mediocre performance of models trained on the dataset noted various issues that seem to impair the quality of the dataset.
Labelling of data
Provided there is sufficient data available, the labelling of data is more of a legal than a technical challenge. While lawyers usually tend to agree on labels for extracting key points of interest (KPIs) such as, for exanple, the name of a company, the name of a court, the address of a plaintiff, a date, etc, the question of labelling a commercial clause in a contract is more challenging: where does the clause stipulating the ‘term’ end and where does the ‘renewal’ clause begin? Looking at one contract might be easy, but analysing and setting ‘labels’ for multiple contracts usually proves more difficult. Lawyers frequently face the question what it really is on which they wish to train the machine. What shall the machine ultimately deliver? Shall it deliver the entire text of a clause (ie, technically classify a clause), or shall it deliver only the relevant KPI? Quality control needs to be rigorous for any labelling of data.
Absence of large data sets
To be able to apply ML in the absence of large labelled datasets, two different strategies are suggested:
- The first is to use ML models that have significantly fewer parameters than deep neural networks and therefore require less labelled data to train.
- A second strategy to overcome the limited amount of labelled training data is to first train a neural network with unlabelled data and then use a small labelled dataset to fine-tune the network for the specific task. Alternatively, any other ML model can be used in the final step to perform the downstream task. The rationale for unsupervised pre-training is that it allows the network to autonomously detect important patterns in the dataset that are helpful in performing different tasks. Since unsupervised training of models is generally very slow, this step requires large but unlabelled datasets. This is the approach taken by LLMs, as discussed above.
Perfect is the enemy of good
Another typical challenge can be overcome culturally and is rooted with either (1) unattainable expectations towards ML or (2) a strong, unfounded scepticism. Working with ML requires a basic understanding of how to use any ML-powered solution in a workflow, which is comparable to working in a team. There is no reason to expect that an ML model will replace entire teams or even just individual lawyers, judges and let alone the entire legal profession. However, an ML-powered solution can significantly reduce manual efforts and typically allows working with massive amounts of data more quickly and gaining insights without the need for weeks of manual review. As is the case with manual work, not all results will be correct. Lawyers will need to adopt, embrace and establish appropriate processes and workflows to ensure the necessary controls are in place.
Technological maturity and risk appetite
The use of ML, particularly in the context of self-deployed models, requires a certain maturity of the overall technology stack within an organisation. Making use of ML is computation-intensive and typically requires cloud resources and other up-to-date technical infrastructure. Moreover, access to data is a sensitive topic within the legal industry, as stakeholders are highly regulated and held to the highest confidentiality and information security standards. Besides, the development and deployment of ML is a continuous testing and readjustment of what works and what does not, which renders it difficult to quantify expenditure beforehand. This in turn requires a certain level of risk appetite by any organisation venturing into this field. Some of these concerns may be successfully mitigated by licensing an SaaS product.
Overview of utilisation in the legal market
To assess the current uptake and use of ML specifically targeted at the legal sector, the following provides an overview of challenges and possibilities for different user groups: courts (including the criminal justice system), law firms and legal departments, as well as consumers and the general public (access to justice).
Courts
Several court systems around the world have added ML technology to their repertoire. Chatbots, for example, are used to answer questions from the public about standard operating procedures, manuals and other existing information resources with a high degree of accuracy. The US Judiciary Information Agent, for example, provides answers with an alleged accuracy of 75 per cent.9 Courts also use ML to facilitate legal research. In China, an ML-based programme is connected to the system of every judge. It automatically checks court cases for references, generates legal documents and is supposed to change judgments tainted by human error, if necessary.10 Similarly, a software called Xiao Zhi 3.0 (‘Little Wisdom’) takes care of repetitive tasks, analysing case material and verifying information from databases.11 China’s Supreme People’s Court has even recently mandated that all Chinese courts develop systems by 2025 that use Artificial Intelligence (AI) to assist in judicial supervision and management.12 In India, the Supreme Court Portal for Assistance in Court Efficiency facilitates the judge’s research by processing relevant facts and relevant laws. ML is also used to translate legal documents. In India, the Supreme Court’s Vidhik Anuvaad software translates legal documents into local languages.13 Some tools even help determine the urgency of cases – as does Prometea in Colombia’s Constitutional Court.14
In Germany – as in most other European civil law jurisdictions – the use of ML is practically limited to ’soft AI’,15 which assists with the efficient handling and pre-processing of data relevant for a court proceeding or expert statement, records decisions through speech and image recognition, provides simulations or analyses patterns. At the same time, there are several non-public initiatives in Germany to develop solutions for analysing existing decisions and inferring what a decision could look like in the case at hand.
In the criminal court systems, ML is leveraged to predict repeat offences or future crimes. Tools such as US COMPAS, UK HART or Chinese System 206 are meant to predict the likelihood of future offences by a suspect (eg, for police custody decisions). Similarly, PredPol is used in the US to guide police patrols by predicting crime hotspots. These predictive tools have been heavily criticised due to significant concerns about bias.16
Law firms and Legal departments
Law firms and legal departments are commonly using ML tools to facilitate legal research. Subscription software such as ROSS, Westlaw Edge or Casetext leverage Natural Language Processing to recommend relevant case law or resources.
Another common ML use case is tools that extract and classify relevant information from documents, allowing contract reviews to be more focused and much more efficient. Popular off-the-shelf tools include Kira Systems, Luminance, Ravn ACE, EigenTechnologies or eBrevia, to name only a few. Most of these solutions classify information such as key clauses in contracts and search for KPIs in documents.
Investigations or e-discovery exercises are typically conducted using predictive analytics. Such technology-assisted review (TAR) processes are typically carried out in solutions such as Relativity, Gravity or CasePoint, depending on the level of automation required to deal with the mass of information. The number of files and documents to be analysed regularly runs into the thousands and millions. Such larger exercises are regularly accompanied by ML engineers and data scientists, and are systematically approached with large review teams to improve the underlying ML. The review teams work in a validation station where documents are reviewed, and where the findings of the ML model deployed are highlighted. If a finding is correct, reviewers confirm; if it is wrong, reviewers reject the proposed result. These findings are then used to train the ML model, which is subsequently deployed to the entire dataset under review.
ML can also be used to predict legal outcomes. Popular examples include Lex Machina and Ravel Law. Both predict court decisions and case outcomes by mapping how cases are related and how judges tend to rule. Regularly, the application of such solutions is limited to jurisdictions and areas of law where the required information is available.
Moreover, several solutions exist to analyse data for specific use cases, which can help lawyers sift through and analyse information. A popular example is Litigate.ai, which can create chronologies of cases based on an automated analysis of documents. There are also several ML-based tools that improve efficiency and automate tasks such as timekeeping (eg, Zero Apollo) and document management (eg, Clio, HighQ).
Consumers and the public
For the user group of consumers or the broader public in general ML-powered services have brought fundamental changes to the access to justice in certain fields by making it substantially cheaper to enforce one’s rights (eg, for compensation for airline or train delays, overcharged rent or banking fees). In addition, automated dispute resolution software has been available for years.17 Since 2014, Dutch consumers have been using the Rechtswijzer online service for divorce matters and complaints about online purchases. The comparable Canadian tool Smartsettle ONE18 allegedly settled a three-month dispute in less than an hour. In 2015, a ‘Robot Lawyer’ was released in the shape of a smartphone application called DoNotPay. The chatbot has originally been created to contest parking tickets – and is said to have successfully done so in 160,000 cases in London and New York in 2016, adding up to around US$4 million of avoided fines. The app now provides further legal services, such as annulling a marriage or filing a restraining order at a three-monthly fee of US$36. Consumers can also use ML to review and draft documents such as non-disclosure agreements. An example is the British tool LISA, launched in 2016. ChatGPT may bring further change. In India, for instance, the pilot project Jugalbandi aims to train GPT-3 to answer questions regarding government entitlements, such as eligibility for affordable housing.19
Case study: mass claims
While the use cases of contract analysis, investigations and e-discovery are relatively well explored, and several off-the-shelf solutions are available for these purposes (see above), the following case study focuses on mass claims, drawing on the approach taken for several ML-based solutions developed in the Freshfields Lab for specific mandates. The background and details provided are intended to stimulate a discussion on other use cases beyond mass claims.
For the purposes of this article, ‘mass claim’ shall refer to any claim made outside of court or in court, where multiple claimants assert similar claims (through collective action such as class actions, model declaratory actions or other such actions that may exist across jurisdictions) against a single respondent or several respondents that are economically or legally associated.
The ideal use case
Managing mass claims on behalf of a respondent can be an ideal use case for an ML-based solution due to the following factors:
- The amount of available data is typically high, because of the large number of claims (as well as the material presented for such claims).
- Only a few KPIs need to be lifted from the documents provided (eg, the claimants’ details, types of relief sought, amounts in dispute) to inform the type of legal redress or other next steps.
- Handling the mass claims data, including analysing documents, compiling lists, managing court appointments, tracking cases, gathering information to develop a strategy based on exposure is hardly achievable without any level of structured data analysis and data capture processes. ML is one of the main building blocks to grapple with larger amounts of data, provide insight in near real-time and work efficiently.
In a specific use case relating to mass claims, the Freshfields Lab deployed the following solution focused on ML components:
Step 1: Recognition of document type
A first challenge in parsing an incoming claim is to recognise the type of document by its main page, as it mostly contains all relevant metadata and key information. Rather than relying on the textual content of the main page, we found that the distinct layout of this page is better suited for detecting it. With the help of a pre-trained object recognition model, which is a deep neural network, this task is readily carried out by fine-tuning the model on a dataset of about 300 examples.
Step 2: Digitisation of documents
Once documents are scanned, a next step consists in applying optical character recognition (OCR) to extract text from the scans. This task involves neural networks either of the sequence-to-sequence or the convolution type. The former, which converts a sequence of visual symbols into a sequence of machine-readable characters, produces the best results for scans of pages with running text and a simple layout. The latter, which is rooted in computer vision, is better suited for pages with a complicated layout such as a business letter because, for example, it groups the lines in an address field together. With the extracted text at hand, the next step consists of splitting the text into sentences, since a sentence forms a convenient unit of context.
Step 3a: Extraction of KPIs
When searching for a specific KPI, we iterate over all sentences and look for clues as to whether the relevant KPI is specified in a sentence. If this is the case, we focus on that sentence and apply pattern recognition to extract the actual KPI and, if necessary, apply post-processing to cast it into standard format. In addition to extracting the KPI, we also record its exact location within the document.
Step 3b: Classification of content
Arguments put forward or facts alleged in claim statements or pre-litigation correspondence are automatically matched to already known arguments by using semantic similarity. This classification is used to generate the correct legal response by selecting the corresponding text blocks.
Step 4: Validation
Since the extracted KPIs must be 100 per cent correct for case handling, they are uploaded together with their locations (within a document) to a validation station. There, the KPIs are summarised and displayed in a convenient manner for review. If reviewers click a specific KPI, they are automatically taken to the location in the document where that KPI is specified. Reviewers can thus easily determine whether the extracted KPI is correct or not.
For any suggested text block corresponding to an argument or fact advanced as part of a claim, the reviewer may choose whether it is correctly selected or if manual rework is required to produce the correct response.
The validation station is also equipped with editing and search functionalities so that changes to the extracted results can be easily made, and other aspects of the content might be investigated. After the KPIs have been validated, they are imported into our case management system.
Step 5: Claim management platform
In a final step, the entire dataset (including the input documents themselves) is stored and maintained through a tailored claim management platform, where a new case file is created or the relevant data is automatically added to an existing case file. The claim management platform is the central record for any claim-related activity such as reporting in interactive dashboards (accessible to the client and other stakeholders), computation of the financial exposure and risk in near real-time, management of the claim lifecycle and automated production of documents. The foundation for all such steps is laid by deploying ML.
Unlock the potential
In light of the challenges described above, the following key elements will – in our view – prove crucial for the success of unlocking the potential of ML in the legal market.
Collaboration
The key to success is to assemble an interdisciplinary team equipped with the right skills – lawyers with subject matter expertise, ML engineers, data specialists, software developers and project managers – and to provide a setup that enables smooth collaboration within such a cross-functional team.
Understanding the value add of ML and validation
It is essential to start building institutional know-how of what ML can achieve when embedded in a business process. This can best be achieved by starting with isolated use cases, without significant time pressure or pressure on the quality of the results. This allows for an iterative approach on both the ML capabilities (develop-test-develop-test-develop-…), but equally the design of a manual process of validating the results (including feeding back regarding the ML capabilities). A good example could be analysing contract repositories for key parameters and clustering the information accordingly. Based on this experience, in particular the limitations of ML and the setup required for validation, a process may be designed for more complex solutions.
Visualisation as a key driver
The presentation of the results of an ML-assisted review is another important element of the overall value add for stakeholders. Providing access to the metadata of ML such as processing status, number of automatically processed documents so far, pages automatically reviewed, etc ensures additional buy-in due to the more transparent and understandable process. The data obtained in (near) real-time and visualised may be used for pivoting results, testing hypotheses, steering the prioritisation of next steps or identifying cluster risks early on. In other words, ML’s potential is not limited to pure efficiency gains, but can unlock an entire new way of working for the legal industry and beyond.
