Researchers are tackling complex multi-objective optimization challenges that require balancing competing goals with limited resources and only “black box” access to the problem itself. Leonard Papenmeier from the University of Münster and Petru Tighineanu from Robert Bosch GmbH, together with their colleagues, present a new approach called SMOG (Scalable Meta-Learning for Multi-Objective Bayesian Optimization) that leverages historical data from similar tasks to accelerate the process. This work is important because it uniquely combines meta-learning and Bayesian optimization for multi-objective problems to provide a scalable solution that explicitly models correlations between objectives and propagates uncertainty in a principled manner, potentially leading to significant improvements in efficiency and performance across a variety of applications.
This innovative approach propagates uncertainties in historical data to surrogate models in a principled manner, improving prediction confidence. After conditioning on metadata from related tasks, SMOG generates a closed-form target task prior enhanced by a flexible residual multi-output kernel that captures the complex relationships between objectives. As a result, the resulting surrogate model is seamlessly integrated with standard multi-objective Bayesian optimization acquisition functions to streamline the optimization process.
This study reveals a probabilistic framework that fills the gap in existing methods by meta-learning correlated multi-objective Gaussian process surrogates while preserving principled Bayesian uncertainty propagation. The resulting target task surrogate can be easily incorporated into standard MOBO pipelines, such as those using hypervolume-based acquisition optimizations, to achieve consistent propagation of metadata uncertainty to the target task. Experiments demonstrate that SMOG effectively leverages historical data to accelerate multi-objective optimization in scenarios where the tasks are related but heterogeneous. This study establishes a way to efficiently exploit correlations between objectives, which is particularly valuable when evaluation is expensive and data are lacking. This research opens new possibilities for applications in industrial process tuning, materials discovery, and machine learning system design. Balancing competing goals is important to achieve optimal performance and resource utilization. This innovative model builds a pre-structured joint Gaussian process across both the metatask and the target task, allowing us to propagate metadata uncertainty to the target surrogate in a principled manner. In this study, we adopted a hierarchical parallel training strategy to increase scalability. This caching mechanism enables efficient reuse of learned information and reduces the computational burden of subsequent optimization tasks.
Scientists leveraged this surrogate model within a standard multi-objective Bayesian optimization acquisition function to seamlessly integrate pre-trained meta-learning into existing optimization pipelines. SMOG’s core innovation lies in its ability to model relationships between multiple objectives simultaneously. The team designed a flexible residual multi-output kernel to enhance a priori target tasks and provide a more nuanced understanding of objective correlations. This structure avoids the pitfalls of treating objectives individually, which can lead to wasted information if evaluation is costly.
Experiments demonstrate that this approach effectively integrates historical data, propagates uncertainty well, and allows faster convergence to a Pareto-optimal solution. Additionally, the study prioritized Bayesian uncertainty awareness to ensure that the model takes into account the uncertainties inherent in both the historical data and the target task. This is achieved through full Bayesian processing of all task data, providing a robust and reliable framework for meta-learning in the low-data domain. The resulting surrogate model provides a principled method for efficient multiobjective Bayesian optimization and balancing exploration and exploitation, which is critical to achieving significant performance improvements across a variety of applications.
SMOG improves early multiobjective Bayesian optimization performance
Scientists developed SMOG, a scalable meta-learning algorithm for multi-objective black-box optimization problems, and demonstrated improved sample efficiency in the first Bayesian optimization iteration. This study focused on leveraging observations from related tasks and modeling correlations between tasks to construct an informative target task posterior distribution. Two-objective Hartmann benchmark experiments revealed that while non-metadata approaches initially struggle, SMOG achieved significant initial speedup and outperformed Ind. -ScaML-GP, which lacked the ability to model task correlations. Specifically, SMOG achieved a normalized hypervolume of approximately 0.4 with 30 BO iterations, outperforming Ind. -GP and MO-GP, which remained below 0.3 with the same number of iterations.
Results show that SMOG consistently performs well across a variety of benchmarks, including the four objective Hartmann benchmarks, where it maintained competitive overall performance. On the Terrain benchmark with two objectives, averaged over three target tasks and 50 random restarts, SMOG achieved a normalized hypervolume of approximately 0.6 over 30 BO iterations, comparable to Ind. -ABLR. Further analysis of the four-objective Terrain benchmark shows that SMOG’s solid performance remains competitive despite the reduced effectiveness of Ind. -ABLR. The team measured hypervolume (HV) as a key metric and evaluated the Pareto front built from the solutions found in each BO iteration.
A study on HPOBench’s protein structure task showed that SMOG significantly outperformed Ind. -GP and MO-GP alongside Ind. -ABLR, achieving higher initial solution quality. According to the data, SMOG and Ind. -ABLR have the strongest performance on this benchmark, demonstrating the effectiveness of metadata utilization. Measurements confirm that SMOG’s ability to model task correlations contributes to its superior performance, especially in scenarios where related tasks provide valuable information. This study documents that methods that utilize metadata consistently outperform methods that do not utilize metadata, highlighting the importance of incorporating prior knowledge into the optimization process. The researchers observed that although MO-TPE was initially competitive, it was unable to maintain its performance against SMOG and other metadata exploitation techniques. This work introduces a robust algorithm that is competitive on all benchmarks, demonstrating its versatility and potential in real-world applications, while other methods struggle on at least one problem.
