Pinterest has developed its own AI text-to-image generation process, but Pinterest's approach is a bit different from what you'll see in other apps.
As outlined in a new brief from Pinterest's engineering team, Pinterest's “Canvas” model aims to provide generated options for product backgrounds without altering the main focus: the product shot itself.
This requires a bit more training. Most large-scale language models are designed to create images based on descriptions by matching text notes from other images with the actual visual output. However, most product photos don't provide context within the caption, so the team at Pinterest had to come up with a new way to separate background and foreground and make it easy to interact with the tool with simple commands.
According to Pinterest:
“Training the Pinterest Canvas gives us a strong base model that understands what objects look like, their names, their general configuration in a scene, etc. However, as mentioned above, our goal is to train a model that can visualize or reimagine real-world ideas and products in new contexts.“
So, conceptually, Pinterest wants to use its existing product image database to establish common framing, placement, and background types to better handle AI background generation requests.
It's a complex approach, but Pinterest has built a system that allows it to do this with great precision.
“[We] We use a segmentation model to separate the foreground and background to generate product masks. Existing text captions typically ignore the background and only describe the product, but the background is important to guide the background inpainting process. Therefore, we incorporate more complete and detailed captions from the visual LLM. At this stage, roller All UNet layers allow for fast and parameter-efficient fine-tuning. Finally, we easily fine-tune our model on a curated set of high-engagement promotional product images to guide the model towards an aesthetic that resonates with Pinners.“
So while the system is specifically designed to generate backgrounds based on existing pin images, Pinterest also aims to tailor the model to specific visual styles to further simplify creation.
Ultimately, brands will be able to input the style they like based on a common description, and Pinterest's system will provide product photo options with that aesthetic.
It's an interesting concept, and one that Pinterest is already testing with select advertising partners.
This is a great way to add variety to your pin images and make your products stand out with different design approaches.
You can learn more about Pinterest's approach to AI background generation here.
