MIT researchers have significantly accelerated design optimization for complex engineering challenges, achieving solutions 10 to 100 times faster than traditional methods in initial tests of benchmarks such as power system optimization. The team redesigned Bayesian optimization, a widely used technique, by integrating an underlying model trained on tabular data to automatically identify the most important variables affecting performance. This allows algorithms to efficiently refine solutions without continuous retraining, an advantage that is particularly valuable for demanding fields such as materials development and drug discovery. “There may be 300 design criteria for a car, but not all of them are the main drivers of the best design when trying to improve safety parameters,” explains Rosen Yu, a graduate student in computational science and engineering and lead author of the study to be presented at the International Conference on Learning Representations.
Bayesian optimization powered by tabular underlying models
This innovation addresses a critical bottleneck in design processes, such as vehicle crash testing, where evaluating potential solutions is resource-intensive and time-consuming. The core of the progress lies in replacing traditional retrained surrogate models within Bayesian optimization with generative AI systems designed specifically for tabular data. Unlike models that require continuous retraining, this foundational model is pre-trained on a huge dataset of structured information and can be quickly adapted to new applications. This reusability is a key advantage, allowing engineers to apply algorithms to different problems without starting from scratch. Importantly, this algorithm does more than just speed up existing methods. Intelligently prioritize which variables to investigate. By identifying highly influential variables, such as the size of the front crumple zone and its impact on safety ratings, the system avoids wasting computational effort on less influential parameters.
The method did not outperform the baseline algorithm in all scenarios, such as robot path planning, but the researchers believe this is due to limitations in the model’s training data. Faez Ahmed, associate professor of mechanical engineering, said the study “suggests a broader shift in using fundamental models not just for perception and language, but as algorithmic engines within scientific and engineering tools.”
Foundation model identifies key design variables
Engineers facing complex design challenges are routinely faced with a daunting number of variables that prevent efficient optimization. Traditional methods are challenging when evaluating countless combinations of parameters, especially in areas such as power grid management and vehicle safety. Bayesian optimization provides a structured approach to overcome these complexities, but it relies on iterative retraining of surrogate models, which creates a significant computational bottleneck, especially as the number of variables increases. MIT researchers addressed this limitation with a new application of the underlying model: integrating a large-scale AI system pre-trained on an extensive dataset into a Bayesian optimization framework. The innovation is centered around a “tabular foundational model,” which Rosen Yu described as “like the ChatGPT of spreadsheets,” which can process and predict results based on structured tabular data common in engineering applications. Unlike traditional surrogate models that require periodic recalibration, this pre-trained model provides reusability and greatly accelerates the optimization process. The team’s approach goes beyond simply applying models. Use that power to identify the most influential design variables.
“Modern AI and machine learning models have the potential to fundamentally change the way engineers and scientists create complex systems. We have come up with one algorithm that can not only solve high-dimensional problems, but is also reusable, so it can be applied to many problems without having to start everything from scratch.”
Rosen Yu, graduate student in computational science and engineering and lead author of a paper on the technique
Achieved 100x speedup on engineering benchmarks
The team, led by computational science and engineering graduate student Rosen Yu, integrated a pre-trained tabular base model into a Bayesian optimization algorithm, dramatically accelerating the process of finding optimal solutions to multifaceted problems. This advancement solves a long-standing problem. Evaluating all possible configurations in areas such as power grid optimization and vehicle design is often prohibitively expensive and time-consuming. Unlike traditional surrogate models that require continuous retraining, this underlying model leverages an existing knowledge base and eliminates significant computational burden. This reusability is especially useful when migrating between different design scenarios, since the model does not have to be rebuilt from scratch each time. The team demonstrated the effectiveness of their approach on 60 benchmark problems, including power grid design and vehicle crash test simulations, consistently outperforming five state-of-the-art optimization algorithms. A key element of system efficiency is the ability to intelligently prioritize design variables.
By identifying the variables that have the greatest impact on performance, the algorithm avoids wasting computational resources on less influential parameters. Although the method did not perform well in all scenarios tested, and robotic path planning proved to be its current limitation, the researchers are already exploring ways to further enhance the performance of the tabular base model and extend the technique to even higher-dimensional problems, potentially finding applications such as warship design.
“The approach presented in this study, which uses a pre-trained base model in conjunction with high-dimensional Bayesian optimization, is a creative and promising way to reduce the large amounts of data required for simulation-based design. Overall, this work is a practical and powerful step toward making advanced design optimization more accessible and easily applied in real-world settings.”
Wei Chen, Wilson Cook Professor of Engineering Design and Chair of Mechanical Engineering at Northwestern University
Extend high-dimensional optimization problems with reusable algorithms
The ability to quickly optimize complex designs is becoming increasingly important across all industries, and a team at MIT has developed an algorithm to accelerate progress in fields ranging from power grid management to vehicle engineering. The researchers tackled this challenge by rethinking Bayesian optimization, a method for finding optimal configurations for complex systems, and integrating it with new applications of artificial intelligence. This pre-training allows you to adapt your model to different applications without having to rebuild it from scratch for each new scenario. A key advance lies in the algorithm’s ability to intelligently prioritize variables. Recognizing that not all design criteria have the same impact, the system identifies the most important features that impact performance.
“A car may have 300 design criteria, but not all of them are key factors in the best design when trying to improve safety parameters. Our algorithm can smartly choose the most important features to focus on.”
