- Facebook Reels goes beyond metrics like likes and watch time to improve personalized video recommendations by directly leveraging user feedback.
- our new User True Interest Survey (UTIS) Modelhelps you surface more niche, high-quality content and increase engagement, retention, and satisfaction.
- We’re exploring advanced AI to enhance personalization, tackle challenges like sparse user data and bias, and make smarter, more diverse recommendations.
- Our paper “Integrate user research feedback to improve personalization of large ranking systems‘ explains this work in detail.
Providing personalized video recommendations is a common challenge for user satisfaction and long-term engagement on large social platforms. At Facebook Reels, we’ve worked to close this gap by focusing on “interest matching,” which ensures that the content people see truly aligns with their unique tastes. By combining extensive user research with recent advances in machine learning, we have been able to better understand and model what people really care about, significantly increasing both the quality of recommendations and overall user satisfaction.
Why true profits matter
Traditional recommendation systems often rely on engagement signals and heuristics such as likes, shares, and watch time to infer a user’s interests. However, these signals are noisy and may not fully capture the nuances of what people are actually interested in or want to see. Models trained solely on these signals tend to recommend content with high short-term user value as measured by watch time and engagement, but do not capture the true interest that is important to a product’s long-term usefulness. To fill this gap, we needed a more direct way to measure users’ perceptions of content relevance. Our research shows that effective interest matching goes beyond simple topic matching. It also includes factors such as audio, production style, mood, and motivation. By understanding these aspects accurately, you can provide recommendations that feel more relevant and personalized, encouraging people to return to your app more often.
How to measure user perception
To validate our approach, we launched a large-scale, randomized survey within the video feed, asking users, “How well does this video align with your interests?” These surveys were rolled out across Facebook Reels and other video surfaces, allowing us to collect thousands of contextual responses from users every day. As a result, the previous interest heuristic Accuracy of identifying true profit is 48.3%highlighting the need for a more robust measurement framework.
By weighting responses and correcting for sampling and nonresponse bias, we built a comprehensive dataset that accurately reflects real user preferences and leveraged direct, real-time user feedback beyond implicit engagement signals.
Framework: User True Interest Survey (UTIS) model
Every day, a certain percentage of users watching a session on the platform are randomly selected and shown a single-question survey: “How closely does this video match your interests?” On a scale of 1 to 5. This survey aims to collect real-time feedback about the content viewed by users.
The main candidate ranking model used in the platform is a large-scale multi-task, multi-label model. Trained a lightweight UTIS alignment model layer Analyze based on collected user survey responses using the main model’s existing predictions as input features. The survey responses used to train the model were binarized and the variance of the responses was denoised to facilitate modeling. Additionally, new features have been designed that use object functions to capture user behavior, content attributes, and interest signals to optimize user interest match predictions.
The UTIS model is designed to output and interpret the probability that a user will be satisfied with a video, allowing you to understand the factors that contribute to a user’s interest matching experience.
Integration of UTIS model into the main ranking system
We have experimented and deployed several use cases of the UTIS model with ranking funnels. All of them demonstrated success in improving tier 0 user retention metrics.
- Late Stage Ranking (LSR): UTIS is deployed in parallel with the LSR model and provides additional input functionality to the final value equation. This allows you to fine-tune the final ranking stage to incorporate your true interests while balancing other concerns.
- Early ranking (search): UTIS is used to aggregate survey data to reconstruct a user’s true interest profile and predict affinity for a particular user-video pair. This allows us to re-rank the user’s interest profile and source more candidates related to the user’s true interests. Additionally, large sequences based on user-to-item retrieval models are aligned using knowledge distillation-based goals trained on UTIS predictions from LSR as labels.
The UTIS model score became one of the inputs to the ranking system. Videos predicted to be of high interest will receive a slight boost, while videos predicted to be of low interest will be demoted. This approach yielded the following results:
- Increased distribution of high-quality, niche content.
- Reducing recommendations based on low quality and general popularity.
- Increase likes, shares, and follow rates.
- Improved user engagement and retention metrics.
Since we started this approach, we have observed solid offline and online performance
- Offline performance: The UTIS model has improved accuracy and reliability over the heuristic rule baseline.. Precision increased from 59.5% to 71.5%, precision increased from 48.3% to 63.2%, and recall increased from 45.4% to 66.1%. These advantages demonstrate the model’s ability to help accurately identify users’ interest preferences.
- Online performance: Extensive A/B testing with over 10 million users confirmed these improvements in real-world environments. The UTIS model consistently outperformed the baseline and increased user engagement and retention.. Specifically, we saw a +5.4% increase in high survey ratings, a -6.84% decrease in low survey ratings, a +5.2% increase in total user engagement, and a -0.34% decrease in integrity violations. These results highlight the effectiveness of our model in improving the user experience and matching users with relevant interests.
Future work towards recommendations of interest
By integrating survey-based measurement and machine learning, we create more engaging and personalized experiences, delivering content on Facebook Reels that feels tailored to each user and encourages repeat visits. While survey-driven modeling has already improved recommendations, important opportunities for improvement remain, including better serving users with sparse engagement histories, reducing bias in survey sampling and distribution, further personalizing recommendations for diverse user cohorts, and increasing recommendation diversity. To address these challenges and continue to improve relevance and quality, we are also exploring advanced modeling techniques such as large-scale language models and more detailed user representations.
read the paper
Integrate user research feedback to improve personalization of large ranking systems
