While environmental scientists are increasingly using vast amounts of artificial intelligence models to predict weather and climate change, new research by MIT researchers shows that larger models aren't always better.
The team demonstrates that in certain climate scenarios, a much simpler, physics-based model can generate more accurate predictions than modern deep learning models.
Their analysis also reveals that benchmarking techniques commonly used to evaluate machine learning techniques for climate prediction can be skewed by natural variations in data, such as variations in weather patterns. This will lead someone to believe that if the deep learning model is not, they will make more accurate predictions.
Researchers have developed a more robust way of evaluating these methods. This shows that, while simple models are more accurate when estimating regional surface temperatures, deep learning approaches are the best choice for estimating local rainfall.
They used these results to enhance simulation tools known as climate emulators. This can quickly simulate the effects of human activity on future climates.
Researchers see it as a “warning story” about the risks of deploying large-scale AI models for climate science. Deep learning models have shown incredible success in domains such as natural language, but climate science includes proven physical laws and set of approximations, and the task becomes a way to incorporate them into AI models.
“We are trying to develop models that will help decision makers with the kinds of things they will need in the future when choosing climate policy. While using the most recent, big machine learning models on climate issues may be appealing, this research is about stepping on the problem and really thinking about the problem. Director of the Center for Earth, Atmospheric, Planetary Science (EAPS) and Sustainability Science Strategy.
Serine's co-author is lead author Björn Lütjens, a former EAPS Postdoc, currently a research scientist in IBM research. Senior author Raffaele Ferrari, professors of Oceanography at EAPS, Cecil and Ida Greene, and co-director of the Lorentz Center. Duncan Watson Paris, assistant professor at the University of California, San Diego. Serin and Ferrari are also co-investigators who bring calculations to the Climate Challenge project, and this study emerged. Paper appears today Advances in Modeling the Earth System Journal of Advances.
Comparison of emulators
The Earth's climate is so complex that it could take weeks to run cutting-edge climate models to predict how pollution levels will affect environmental factors such as temperature.
Scientists often create simpler approximations of climate emulators, faster and more accessible artistic climate models. Policymakers can use climate emulators to see how alternative assumptions about greenhouse gas emissions will affect future temperatures and help develop regulations.
However, emulators are not very useful when making inaccurate predictions about the local impacts of climate change. Deep learning is becoming more and more common in emulation, but few studies have investigated whether these models perform better than proven approaches.
MIT researchers conducted such studies. They compared a traditional method called Linear Pattern Scaling (LPS) with a deep learning model using a general benchmark dataset for assessing climate emulators.
Their results showed that LPS is superior to deep learning models in predicting almost all the parameters tested, including temperature and precipitation.
“Large AI methods are very appealing to scientists, but rarely solve whole new problems, so implementing existing solutions requires examining whether complex machine learning approaches will actually improve,” says Luchens.
Some initial results appeared to fly in the face of researcher's domain knowledge. Because these data do not follow a linear pattern, a powerful deep learning model should be more accurate when predicting precipitation.
They found that high amounts of natural variation in the implementation of climate models could lead to poorly functioning deep learning models with unpredictable long-term vibrations like El Niño/La Niña. This causes the benchmark score to be distorted in favor of the LPS and averages these vibrations.
Building a new evaluation
From there, researchers have built new assessments that contain more data to address natural climate change. With this new evaluation, the deep learning model performed slightly better than LPS due to local precipitation, but LPS was even more accurate in temperature prediction.
“It's important to use modeling tools that are right for your problem, but to do that, you need to set up the problem in the right way to begin with,” says Serine.
Based on these results, researchers incorporated LPS into a climate emulation platform to predict local temperature changes for various emission scenarios.
“We don't argue that LPS should always be a target. There are still limitations to that. For example, LPS does not predict variability or extreme weather events,” adds Ferrari.
Rather, they hope that their results highlight the need to develop better benchmarking techniques.
“The improved benchmarks for climate emulation allow us to explore issues that are currently very difficult to tackle, using more complex machine learning methods, such as aerosol effects and extreme precipitation estimation,” says Luchens.
Ultimately, more accurate benchmarking techniques can help you ensure that you make decisions based on the best available information.
Researchers hope others will build on their analyses, perhaps by studying climate emulation methods and additional improvements to benchmarks. Such studies can explore new variables such as drought indicators and impact orientation indicators such as wildfire risk, or local wind speed.
The study is part of the MIT Climate Grand Challenges team, funded in part by Schmidt Sciences, LLC, and “bring calculations to climate challenges.”
