Nvidia scientists have teamed up with the Lawrence Berkeley National Laboratory (Berkeley Lab) to release a machine learning tool called the giant ensemble (Hens) that brings supercomputer class predictions, but due to their significantly lower computing power and cost. Available as open source code or as off-the-shelf models, it predicts low-level, impactful events, from long heat waves to 100-year hurricanes. The technology helps climate scientists, city officials and emergency managers quickly test scenarios and update response plans with minimal computing resources.
A two-part study published in the journal Development of geoscience models27,000 years of data, introduces a method called Hens, one of the largest and most reliable ensembles of available weather and climate simulations.
Using NVIDIA PhysicsNemo, researchers trained global weather models to improve chicken methodology using open source Python frameworks and Makani open source frameworks.
“The 27,000-year simulation is a gold mine for studying statistics and drivers of extreme weather phenomena,” said Ankur Mahesh, co-author of the study and a graduate student researcher in the Earth and Environmental Sciences Region at Berkeley Lab. “This large sample size is exactly a scale we didn't see before.”
Research shows that chickens can predict weather faster than other methods, taking minutes instead of hours. It also extends the forecast window to predict future extreme weather events from 6 hours to 14th at a resolution of 15 miles (25 km). This helps researchers study weather patterns at high resolution for decades and identify new clues leading up to extreme events.
“With Hens, there's the luxury of posting the low-Japanese, impactful and extreme events that have been predicted over many years and decades, rather than a single short-term event,” says Bill Collins, a senior scientist in the Global and Environmental Sciences Region at Berkeley Love and a professor at UC Berkeley.
This new approach also requires much less energy and human time than other methods, saving energy by reorganizing the model with new data (methods that ensure accuracy).
Training Hens: PhysicsNemo and 40 Years of Climate Data
Hens employs an AI model trained using 40 years of ERA5 data using PhysicsNemo. After being trained, the model offers a much cheaper computational approach for prediction, Shashank Subramanian, a machine learning engineer at the National Energy Research and Science Computing Centre (NERSC), said Mahesh was the co-author of the study that developed and tested the training and evaluation workflow.
“Hens is a game changer. To this day, generating an ensemble of 1,000 or 10,000 members of a simulation has simply not been practical due to prohibited calculations and data storage costs,” said Michael Pritchard, director of climate simulation research at NVIDIA and professor at UC Irvine. “Thanks to the careful work of calibrating this team's new AI simulation technology, it is suitable for purposes of generating large ensembles with realistic heat wave counterfactuals with a wider completion than traditional numerical simulations.”
How can hens be used to improve the accuracy of weather forecasts?
To understand the scope of future weather outcomes, National Weather Services performs multiple different simulations, or “ensemble members.” Each has slight changes to the initial conditions. These numerical models are based on physics laws, such as mass conservation, momentum conservation, energy conservation. There is a lot of confidence in these physics-based simulations, but they are very computationally expensive because they require a supercomputer.
Due to this cost, traditional weather models only have 50 ensemble members. Finding extreme weather requires disrupting the initial conditions of the model thousands of times and hundreds of supercomputing times.
The researchers used chickens to create 7,424 ensemble members based on initial weather conditions from each day of the summer of 2023. It was the hottest and recorded at the time. This has 150 times more members than the previous model can do.
“This allowed us to get a better estimate of the tail of the distribution and understand the extreme events that could have occurred that summer,” Mahesh said.
The predictions made by chickens have uncertainty that are ten times smaller than those of traditional models. It can catch 96% of rare, but serious, extreme weather events that other models usually overlook. Together, these strengths allowed the team to create a vast dataset of approximately 27,000 years of climate data (20 petabytes).
During a rigorous verification experiment at NERSC, Mahesh and Team weighed ensemble predictions of a wide range of diagnostic metrics, showing that Hens is very close to the gold standard.
What's next?
In future work, Mahesh said the team plans to study the 27,000 years of simulation in the hopes of uncovering new insights into the drivers behind the devastating heat waves, hurricanes and atmospheric rivers that have devastated communities in recent years. They also aim to further reduce the calculation requirements for running chickens.
NERSC is the office of Berkeley Lab's DOE Science User Facility. This work was supported by the DOE Science Bureau.
