How Pasta works
A large, diverse set of interaction data is required to effectively train AI agents to adapt to the individual preferences of users. However, it is difficult to collect this data from real users due to several factors, including user privacy. To address this, we trained the pasta using a two-stage strategy that combines real human feedback with large-scale user simulations.
First, we collected high-quality basic datasets with continuous interactions of over 7,000 evaluators. These interactions included a large multimodal model of Gemini Flash and a prompt extension generated by the corresponding images generated by the stable diffusion XL (SDXL) T2I model. We then trained a user simulator designed to generate additional data that replicates real human choices and preferences using this first seed of authentic priority data.
At the heart of our method is the user model, which consists of two important components. 1) a utility model that predicts the degree to which users prefer a set of images, and 2) a selection model that predicts the set of images to be selected when presented in several sets. We built the user model using a pre-trained clip encoder and added user-specific components. The model was trained using the expectation maximization algorithm. This allows you to simultaneously learn the details of your user preferences, while also discovering potential “user types,” i.e. clusters of users with similar preferences (those who tend to prefer animals, scenic views, or abstract art).
A trained user simulator provides feedback and explicit settings for the generated images, allowing you to make choices from the proposed set of images. This allows for the generation of trajectories of over 30,000 simulated interactions. Our approach is not just about creating more data. This provides a controlled environment for exploring a vast range of user behaviors so that pasta agents can be trained to effectively collaborate with users.
