Additionally, the following best practices can help you create ML models that are less susceptible to catastrophic failures.
Integrate more data sources when possible and increase the frequency and granularity of data collection. For example, increasing the frequency and granularity of data collection from weekly to daily or daily to hourly can shorten learning cycles and improve understanding of changes in data segments.
For example, when predicting financial markets, models need to be trained not only on the ups and downs of the past few years, but also on black swan events such as the Black Monday stock market crash of 1987, the dot-com bubble burst of 2000, and the global financial crisis of 2007-2008. And of course, health crises like coronavirus are the perfect catalyst for building better ML models.
Create a log of the event and how it is represented in the data, documenting the reason and expected results. It is important for data scientists to resist the temptation to completely exclude 2020 data from predictive models due to anomalies and instead rely on 2019 and 2018 data.
It is true that governments and regulators are deploying blunt instruments to stimulate the economy, creating abnormalities in consumer behavior. However, these anomalies should not be ignored and the model should be trained to guide them, optionally incorporating dummy variables to capture and account for anomaly periods.
While it is unlikely that economies will face such severe global impacts any time soon, regime change is more frequently encountered in regional contexts. For example, closures due to severe weather or natural disasters can have significant local impacts and are far more common. Therefore, it is important to train the model through anomalies (or abnormal situations) to understand the business impact, duration, and recovery rate. This allows you to make smarter decisions at scale and in less time.
Acknowledge the existence of bias. Bias can arise from ML models when they are tasked with implementing policy decisions as a result of non-representative training datasets. Although it is not the subject of this paper to go into further detail, it is important to explore the intentional and unintentional consequences of bias on ML algorithms.
For example, if the input data used to train a model is more representative of certain groups of people based on gender, race, or other factors, predictions from the model may also be systematically biased toward those groups. Therefore, these biases can lead to decisions that can negatively impact certain groups of people, even if they are not explicitly modeled by developers.
