With the digitization and continued advancement of broadcast technology, sports fans have come to expect a dynamic viewing experience. Whether the game is streamed live or recorded for on-demand viewing, viewers are increasingly demanding standout elements in their coverage of their favorite teams, from captivating graphics and data insights to timely replays and highlights. The stakes are even higher if the match in question is one of the most anticipated matches in the sports world. With this in mind, FOX has built and introduced a new AI-powered Catch Up with Highlights feature ahead of his 2022 FIFA World Cup Qatar™ broadcast in the United States.
Catch Up With Highlights is an end-to-end solution that leverages Amazon’s open source Media Replay Engine (MRE) framework for automating video clipping and replay generation, and Amazon Web Services (AWS) infrastructure and solutions to ingest broadcast streams, identify precious game moments through AI-trained computer vision, and encode delivery streams with metadata across user devices. AWS already underpins Fox Corporation’s media workflows, allowing the development team to launch a complex new feature in about eight weeks.
Phil Martin, Vice President of Data Products and Machine Learning (ML) Strategy at FOX, who led the development and initial implementation of Catch Up With Highlights, said: Additionally, we have rallied around the idea of revolutionizing fandom. We believe sports should be fun and accessible for everyone, so we designed our platform to make it easy to participate and have fun. The highlight catch-up feature is a great example of this philosophy. We understand the level of play at an event like the World Cup is exceptional and wanted our feature to reflect that standard. To achieve this, we have developed cutting-edge AI technology to create a highlighting engine that can dynamically recreate the game’s story in real-time, highlighting key moments and events as they unfold. This allows users to stay up to date with the action and experience the excitement of gaming wherever they are. Ahead of this effort, AWS has been a trusted partner to us, and together we have taken another step forward in leveraging data to revolutionize fandom on the world’s biggest sports arena. ”
At launch, the functionality ran seamlessly with little human input required. Martin and his team built Catch Up With Highlights to continuously optimize the highlights recap video as the match progressed, retelling his story more than 20 times game after game with event-driven updates. This approach has allowed viewers to access a comprehensive recap of the match so far via his FOX Sports app on their connected TV or mobile device. Incremental improvements were rolled out based on performance, such as adding thumbs up or downs. Enhanced events such as saved shots and penalty kicks. In the end, 731 wrap-up videos were released autonomously across 61 games.
The “Catch Up With Highlights” workflow includes AWS Lambda event-driven serverless computing and computer vision with Amazon Rekognition. Sports data is stored in an Amazon DynamoDB NoSQL database and piped to the MRE. Live video is delivered quickly to viewers via AWS Elemental MediaConnect, and videos are stored in Amazon Simple Storage Service (Amazon S3) and Amazon ElastiCache for caching live clips.
“Football is often described as a ‘beautiful game’ because of its fluid, artistic style of play, but objectively it’s simply a sport where you score more goals than your opponent,” Martin explained. “But training a model and developing logic based on a game without a set pattern is never easy. Adding to the complexity is the first publicly released platform capability to ingest multiple simultaneous streams, truncate clips for inference, contextualize and optimize each segment, and transcode and deliver directly to a CMS as soon as an event occurs.”
Additionally, the team had limited reference information from event organizers. Instead, we used audio cues and camera angles to train a computer vision model. Clips are automatically assembled using sophisticated elements such as transitions and fades to black to provide fans with a quality viewing experience. Now, Martin looks forward to replicating the “Catch Up With Highlights” playbook for other sports broadcasts due to its proven, scalable approach.
“We are very bullish on this space and have already expanded it to other sports venues,” said Martin. “Fans can experience this feature today during broadcasts of Major League Baseball (MLB), the United States Football League (USFL) and the FIFA Women’s World Cup™. Catch Up With Highlights will be on football broadcasts this fall with more personalized content. The possibilities are endless and we are ready. ”
