Simulation surrogates are stable and adapt to new scenarios

Surrogates are becoming an essential tool in engineering to speed up computationally intensive simulations, but they often suffer from poor performance when there is a conflict between training and real-world deployment conditions. Anna Zimmel, Paul Setinek, and Gianluca Galletti collaborated with colleagues from the 1ELLIS unit, LIT AI Lab, and Machine Learning Institute in Linz, Austria, to develop a new Test Time Adaptation (TTA) framework to address this critical problem. Their work introduces a method based on preservation of D-optimal statistics that provides maximum information, allowing stable adaptation and parameter selection for high-dimensional unstructured regression problems, and is a significant advance over existing TTA techniques primarily designed for low-dimensional classification. Demonstrating up to 7% off-variance performance improvements with minimal computational overhead on benchmarks such as SIMSHIFT and EngiBench, this study is the first systematic demonstration of effective TTA for high-dimensional simulation regression and generative design optimization.

Scientists are grappling with a major hurdle in engineering design: ensuring the accuracy of computer simulations in the face of unexpected scenarios. Accurate and fast simulations rely on “surrogates”, which can be degraded by new data. Smart adaptive technology now allows these critical tools to remain reliable even when conditions change during use.

Scientists are increasingly deploying machine learning surrogates to speed up complex engineering simulations, but significant challenges arise when these models encounter conditions different from the original training data. Changes in the distribution, such as unseen geometry or configuration, can lead to significant performance degradation and impede prediction reliability.

Test time adaptation (TTA) offers a potential solution by allowing models to be adjusted during use, but current TTA methods are primarily designed for simple tasks with clear visual patterns and structured output. This limitation causes instability when applied to high-dimensional unstructured regression problems frequently encountered in engineering simulations.

Researchers have now developed a new TTA framework that addresses this instability by preserving and utilizing statistics that provide the most information, and in particular by adopting a D-optimization approach that selects the most relevant data. This method allows stable adaptation and automatic selection of optimal parameters during testing. Integration with pre-trained simulation surrogates improves out-of-distribution performance by up to 7% with minimal additional computational cost.

To the best of the authors’ knowledge, this is the first systematic demonstration of effective TTA for high-dimensional simulation regression and generative design optimization validated using the SIMSHIFT and EngiBench benchmarks. Neural surrogates have become an essential tool for accelerating partial differential equation (PDE) simulations across numerous scientific and engineering fields.

Although these surrogates perform well when test conditions match the training data, their accuracy often decreases when faced with changes in unseen configuration, geometry, material properties, or structural dimensions. This problem is particularly acute in industrial environments where the design optimization process produces configurations that exceed the initial training range.

Access to original training data is often limited by portability and proprietary issues, and zero-shot adaptation and automatic model selection require model- and task-independent approaches. Dealing with distributional changes is a central theme of several research fields, including domain adaptation, domain generalization, meta-learning, and active learning.

Test time adaptation (TTA) adapts the model during inference without requiring source data or incurring significant computational overhead, and thus stands out as an approach particularly suitable for engineering tasks where rapid adaptation is required and the distribution of the target domain is unknown in advance. Although TTA has been proven to be effective in areas such as medical image processing and object detection, its application to high-dimensional regression problems remains largely unexplored.

Improved performance of stable adaptation when testing simulations across different robot datasets

Across all simulation datasets, Stable Adaptation at Test-Time for Simulation (SATTS) consistently outperforms existing Test-Time Adaptation (TTA) techniques, improving out-of-variance performance by up to 7% with minimal computational overhead. Specifically, on the rolling model dataset, SATTS achieves a root mean square error (RMSE) of 0.545±0.019 compared to 0.566±0.020 for SSA and 1.825±0.002 for Tent, showing a clear advantage in prediction accuracy.

The motor model shows even more significant improvement, with SATTS matching Oracle’s performance with 0.109±0.003 RMSE, while the source model significantly underperforms with 0.109±0.001 and Tent with 1.132±0.032. For the shaped model, SATTS achieves an RMSE of 0.157±0.001, which is slightly improved over the source model’s 0.161±0.001 and significantly better than the SSA’s 0.215±0.005.

The heatsink model presents a more nuanced picture, with SATTS achieving 0.738±0.004 RMSE, a slight decrease from the source model’s 0.747±0.001, but a notable improvement compared to Tent’s 0.876±0.001. These results, averaged over 20 TTA runs, establish SATTS as a new baseline for adapting simulation surrogates to unseen conditions. Visual inspection of equivalent plastic strain (PEEQ) predictions for hot-rolled samples further confirms the effectiveness of SATTS, showing that it successfully corrects systematic underprediction in the deformation zone and improves physical consistency with ground truth data.

Additionally, analysis of the EngiBench generative design optimization task revealed comparable success. For the Beams2D model, SATTS achieves a COMP score of 118.8±12.409, slightly better than the source model’s 123.7±17.854 and SSA’s 119.4±4.586. Meanwhile, for the HeatConduction2D model, SATTS achieves a COMP score of 0.537±0.491, again outperforming the source model. 0.577±0.561, SSA 0.712±0.615. The Proxy A Distance (PAD) value quantifies the mismatch between the source and target domains and correlates with performance improvement. Datasets with higher PAD values ​​consistently show improvements with adaptation, enhancing the method’s ability to deal with significant distributional changes.

D-optimal adaptation of simulation surrogates for out-of-distribution generalization

D-optimal experimental design strategy underpinned the methodology used to adapt pre-trained simulation surrogates to unseen data distributions. This technique is borrowed from statistics and selects the smallest set of data points that maximizes the gain of information about the behavior of the surrogate model. Rather than randomly selecting samples for adaptation, the researchers carefully curated a subset that best constrained the model’s parameters, improving stability and accuracy in the face of out-of-distribution inputs.

This is in contrast to typical test-time adaptation methods, which often struggle with the high dimensionality and complex relationships inherent in engineering simulation. Initial work involved establishing the baseline performance of several existing surrogates trained on benchmark datasets, namely SIMSHIFT and EngiBench. These benchmarks represent a variety of engineering problems and enable a thorough evaluation of the adaptive framework.

Once baseline performance was quantified, the D-optimal statistic was calculated from a representative set of training data. These statistics capture important characteristics of a surrogate’s input/output mapping and provide a concise overview of its behavior. Then, during testing, the incoming data points were compared to the stored D-optimal statistics.

This comparison guides the selection of appropriate adaptation parameters, effectively shifting the surrogate’s predictions to better match the new data distribution. This process avoids complete retraining and provides a computationally efficient solution for adapting to changing conditions. By focusing on informative statistics, this method avoids the instability common in high-dimensional regression problems, where small perturbations can lead to large errors.

Stabilize surrogate models against unexpected changes in engineering design optimization

Scientists are increasingly relying on computer simulations to design everything from aircraft wings to heat sinks, but these simulations are often time-consuming and computationally expensive. To speed up processing, engineers build “surrogate” models, or fast approximations of the full simulation. However, these surrogates flinch when faced with a slightly different scenario than the one used during training, a problem known as a distribution shift.

This mismatch can render the surrogate useless and negate the time saved. Adapting these substitutes to new conditions has proven surprisingly difficult. Existing methods designed for simple tasks such as image classification struggle with the high-dimensional and unpredictable outputs of engineering simulations. Here, a new approach demonstrates how to stabilize this adaptation process by intelligently storing and applying important statistical information.

By focusing on the most informative data points, the system can adapt to unseen conditions with minimal computational overhead, improving performance on benchmark tests by up to 7%. This work represents the first systematic demonstration of effective “test-time adaptation” for complex simulated regression and opens the door to generative design optimization, where algorithms automatically explore a myriad of possibilities.

Although the current implementation relies on a specific network architecture, the basic principle of using carefully chosen statistics to guide adaptation may be applicable to a wider range of surrogate models. Further research should address the limitations of this approach in the face of extreme distributional changes and explore ways to automate the selection of these “informative” statistics. Once these challenges are resolved, we can expect a future in which simulations are not only faster, but also much more adaptable and reliable.