Graphalocbench enables flexible multi-objective policy learning with diverse preferences

Researchers are tackling the challenge of creating adaptable artificial intelligence that can balance multiple, often competing goals through preference-conditional policy learning in multi-objective reinforcement learning. Zhiheng Jiang of the University of California, Los Angeles, Yunzhe Wang, Ryan Marr, and Ellen Novoseller, Benjamin T Files, and Volkan Ustun of the USC Institute for Creative Technologies have introduced a new benchmark, GraphAllocBench, designed to measure progress in this field. By providing a realistic and scalable resource allocation environment inspired by urban management, this benchmark addresses critical gaps in current evaluation methods that often rely on simplified scenarios. In addition to offering a diverse set of problems, GraphAllocBench introduces new metrics to assess how well an AI system adheres to specified settings, promising to accelerate the development of more flexible and intelligent multi-objective decision-making systems.

GraphAllocBench provides a diverse problem suite with different objective functions, preference conditions, and high dimensional scalability, allowing for more robust evaluation of PCPL methods. This discovery paves the way for applying graph-based techniques, such as graph neural networks, to complex high-dimensional combinatorial assignment tasks. CityPlannerEnv models resource allocation as a sequential problem expressed through a bipartite resource demand dependency graph, reflecting the challenges faced in real-world urban planning.

This environment simulates how cities allocate limited resources, respond to diverse and often conflicting demands, reduce congestion, foster economic growth, and promote sustainability while adapting to changing preferences. Agents in CityPlannerEnv incrementally allocate resources by adding or removing productions as per demand, pursuing custom-defined goals based on user preferences. This incremental allocation mechanism is inspired by the multi-step Colonel Blot game, which is a model for competitive resource allocation on a graph. The benchmark includes test problems that span a variety of optimization challenges, including difficult objective functions, non-convex Pareto fronts, sparse observation spaces, complex dependency structures, and high-dimensional graph observations. Through rigorous experiments, the researchers demonstrated that GraphAllocBench not only challenges existing MORL techniques, but also provides a platform for developing and evaluating more sophisticated PCPL algorithms capable of handling complex real-world resource allocation problems.

CityPlannerEnv and GraphAllocBench benchmarks are under development

In this study, we designed this environment to address the limitations of existing multi-objective reinforcement learning (MORL) benchmarks, which often lack realism and scalability. Agents in the environment allocate resources by adding or removing productions in response to each demand at each time step, pursuing custom-defined goals tailored to user preferences. This work pioneered an incremental allocation mechanism inspired by the multistep Colonel Blot game, a model for competitive resource allocation on a graph. The team implemented the dependencies between resources and demands as a bipartite graph, allowing goals to be defined across any subset of production.

PNDS captures configuration consistency directly and the OS complements widely used hypervolume metrics. The results show that graph-based techniques, such as graph neural networks, are particularly suited for complex high-dimensional combinatorial assignment tasks within this framework. The innovative methodology enables a more nuanced evaluation of PCPL algorithms, facilitating progress in adapting policies to arbitrary tradeoffs at runtime.

GraphAllocBench evaluates preference-conditioned multiobjective reinforcement learning algorithms

Results show that PCPL policies trained using Proximal Policy Optimization (PPO) on GraphAllocBench exhibit varying performance depending on the problem structure. For problems 1a-c, which are characterized by rapid changes in the reward signal, the team recorded significantly higher variance in the Pareto front and a worse approximation of the Pareto front compared to the baseline with a smooth objective function. Testing has proven that non-convex Pareto fronts like Problems 1c and 3b pose challenges for accurate representation, even with smooth Chebyshev scalarization. Measurements confirm that for unbalanced goals, where one goal receives more sparse rewards, the quality of the Pareto front median approximation decreases, as exemplified in Problem 2c.

In this work, we recorded instances where the agent falls into a local optimal trap where the PCPL agent collapses into a local solution and fails to generalize to the global Pareto front, as observed in Problem 2b. Problems 6a-c involve a large number of resources and productions, and the scientists implemented a heterogeneous graph neural network (HGNN) to represent complex demands and resource dependencies. The HGNN-based feature extractor utilizes multiple graph attention networks, one for each node type (demand, resource, unallocated), with stacked layers and residual connectivity. The researchers hope to show that flexible pooling techniques, such as mean/max pooling and attention pooling, can capture global information more efficiently than MLP and improve performance on complex combinatorial assignment tasks.

GraphAllocBench evaluates preferential learning in resource allocation scenarios

This benchmark addresses gaps in existing evaluations, which are often limited to simple tasks and fixed environments, by providing a more realistic and scalable platform for testing algorithms. In particular, when the PPO algorithm was equipped with heterogeneous graph neural network feature extraction capabilities, it significantly outperformed standard MLP baselines, especially on large and complex graphs. The authors acknowledge that current benchmarks focus on specific dependency structures and lack features such as efficiency metrics and environmental uncertainty. Future research aims to address these limitations by incorporating richer dependencies, simulating events such as natural disasters, and evaluating risk-aware decision making. GraphAllocBench thus establishes a versatile and scalable testbed for advancing research in preference-aware multi-objective policy learning and provides a valuable tool for developing algorithms capable of handling complex combinatorial assignment tasks.