Manual evaluation of transaction monitoring models is time-consuming and error-prone, and mistakes can lead to large fines. To get around this, banks are increasingly turning to automated machine learning.
Regulators increasingly expect banks and financial institutions to be able to demonstrate the effectiveness of their transaction surveillance systems.
As part of this process, banks need to evaluate the models they use and validate (and document) that they can perform the tasks. Financial institutions that fail to maintain sufficiently effective anti-money laundering programs are often subject to huge fines, some totaling over US$1 billion.
US Department of Justice (DoJ) Deputy Attorney General Lisa Monaco said she expects companies to invest in strong compliance programs while announcing the recent fines against Danske Bank. And it could be a one-way ticket to a multi-billion dollar guilty plea.
Such threats are putting additional pressure on smaller banks and financial institutions. Larger institutions often struggle less for legions of data scientists, but model validation and evaluation can be a burden for players with limited resources.
What is a model?
In the United States, banks typically monitor transactions using a rule-based system of parameters and thresholds. A general rule is to detect the value of a transaction over a period of time or an increase in the volume or value of a transaction. Alerts are triggered when sufficient conditions are met.
Even in its simplest incarnation, regulators view such systems as models. According to the supervisory guidance OCC 2011-12, a model is defined as a quantitative approach to processing inputs and producing reports. In practice, a typical rule-based transaction monitoring system contains multiple layers of rules.
Regardless of complexity, banks need to manage model risk appropriately. There are three main types of model risk that banks need to consider.
- specification – Does the model produce output of the expected quality? Is it useful?
- implementation – Is the model implemented as designed? For example, is the data consistent with the specified source/quality?
- application – Is the model properly used and interpreted?
These are easy questions to ask, but can be very difficult to answer. The OCC’s supervisory guidance stipulates that banks should manage model risk in the same way as other types of risk. This includes “substantial analysis by an objective and informed stakeholder who can identify model limitations and assumptions and produce appropriate changes.”
The guidance states that banks should ensure that their models are performing as expected in line with their design goals and business use. It defines the key elements of an effective validation framework as follows:
- Assessment of conceptual soundness, including developmental evidence.
- Continuous monitoring including process validation and benchmarking.and
- Result analysis including backtesting.
corporate compliance
Regulators continue to raise the bar as the United States seeks to limit access to sanctioned countries and individuals and to crack down on financial crime in general.
Since 2018, the New York State Department of Financial Services has required the board or senior executives to submit an annual “compliance survey” demonstrating the effectiveness of a financial institution’s transaction surveillance and sanctions filtering programs.
Taking this a step further, the DoJ announced in 2022 that it is considering a requirement to certify the design and implementation of compliance programs for the Chief Executive Officer and Chief Compliance Officer. As the war in Ukraine drags on and geopolitical tensions persist, the potential costs of non-compliance only increase.
Model regulation is under these broad requirements for effective risk management. The approach taken by banks to evaluate models varies on a case-by-case basis, but the general principles apply equally.
Similarly, the frequency of model evaluations should be determined using a risk-based approach. This is typically driven by significant changes in an institution’s risk profile, such as mergers and acquisitions, or expansion into new products, services, customer types, or geographic areas. However, regulators increasingly expect models to be evaluated every 12 to 18 months.
Model evaluation challenges
Rules-based models are being asked to do more as the nature and volume of financial transactions evolve. Models are becoming more complex (but less effective) as new threats emerge. Unfortunately, many fail to do so.
Models are often confusing black boxes that few people in an organization understand. Over the years, changes in data feeds, scenario logic, system functionality, and staffing can result in incomplete or inaccurate documentation of how the model works. All of this can make valuation very difficult for smaller banks. First-time evaluations are almost certainly time consuming, expensive, and potentially flawed.
But the challenges never go away. The changes in consumer behavior that were accelerated during the pandemic will continue. Banks and financial institutions have digitized their operations, significantly expanding the range of online services and payment methods. Consumers are also showing willingness to switch to challenger banks with digital-first business models.
These changes created more vulnerabilities. Competitive pressure is straining compliance budgets, while the expansion of online services increases his AML failure opportunities. To keep up, financial institutions must respond quickly and flexibly to new threats.
Improving model evaluation with automated machine learning
This model evaluation process can be optimized using automated machine learning (AutoML). This allows continuous (or short-cycle) evaluation of models in a standardized process, leading to higher quality evaluations. In contrast, manual approaches are time consuming and error prone.
AutoML models take huge data sets and learn from the behaviors encoded in that data to uncover patterns that indicate evidence of money laundering. The rapidly changing AML regulatory landscape, coupled with increasing numbers of transactions and customers, leaves little room for the traditional manual, project-by-project approach. As such, the industry is increasingly turning to more disruptive approaches: models trained on good customer behavior. The results of this non-traditional method combined with AutoML will help banks adapt to new realities and stay ahead of nearly all new criminal patterns.
