“Firefly debuted last March. It's only been around for a little over a year, but it's incredible to see the acceptance of our basic model. We have models, we have vector models, we have design models and we are working on video models, which will be brought to market this year,” Subramaniam told IBC365 on the eve of the MAX event in Battersea. Subramaniam leads the team that advances Adobe's digital images, photography, video, and design strategy. “Excitement, enthusiasm and recruitment from the community [towards Firefly] It's great and all of its development was done through an open public beta. ”
The Firefly model integrates AI image generation into Adobe's creative applications and is demonstrated in MAX running in Adobe Photoshop, Lightroom, Illustrator, InDesign, and Adobe Express. But most interesting to IBC365 readers is his sneak preview of a new video model that promises his AI-powered post-production capabilities in Premiere Pro.
Adobe introduced its Generative Extend feature, which lets you extend clips by adding frames for improved editing, longer shots, and smoother transitions in a smooth video presentation. AI tracking and masking features allow you to add or remove objects in your footage. Adobe says this will allow editors to quickly remove unnecessary items, change actors' costumes, and add set decorations such as paintings on desks or realistic flowers. . It may also cause a bit of concern for the VFX team.
read more: Power of Color Symposium 2024: Skin tone, open gate photography, post AI
AI text-to-video will now be able to generate entirely new footage from prompts or reference images directly within Premiere Pro. Adobe suggests that you can use these clips to generate ideas, create storyboards, and create B-roll to augment live-action footage.
The future of third parties
Adobe isn't the only company working on AI video research. In fact, the preview shows Open AI and Runway's third-party video generation model integrated to generate B-roll, with Pika Labs' generative model working with the Generative Extend tool to add seconds to the end of the shot. It was shown how to do it. .
Does this mean that Firefly will handle some aspects of content generation while leaving some of the heavy lifting required for video generation, especially B-roll, to third-party players?
“Sometimes certain models are a little bit better at coming up with ideas,” Subramaniam says. “You never want to use that output in production just to get the creative juices flowing and think things through. The idea is to.
“[Firefly models] It's great, it's commercially safe, and the community loves it, but we think introducing more models would create a better world for creativity.
“Exactly how third-party mechanisms are enabled from customer endpoints remains to be determined,” she added. “At this point, we are excited about the state of the technology and are looking to explore this as a research foray with these partners and take it from there.”
safety first
The tag “commercially safe” comes up often in conversation. That's not surprising. There are many questions about what content commercial AI models are trained on, how input data can bias and distort results, and whether the content produced is original human work. There has been intense debate.
“We want to have a dialogue about those fears. Understand them. Share our approach,” Subramaniam says. “We've been very open, whether the actual technology is being rolled out through public beta or not releasing it until we feel it's ready. We've been very open about just sharing what we're working on with the video model. But we were able to elicit the feedback, “What do you want to see?” What are you worried about? ”
“The majority of people I talk to want a commercially secure model, and what we offer is completely secure, from how you set up and train the model to how you deploy the model. We're in control and we can guarantee it. We provide coverage. This is a core part of our approach to Firefly.”
“It's not lip service,” she continues. “From the beginning, it's about how we build this model and how we deploy these workflows.
Part of this approach is content credentials, which are tamper-proof metadata that can be applied to assets during export or download. These provide greater transparency regarding the origin and history of the assets to which they apply. These are available through the Content Authenticity Initiative (CAI) and will be coming to Premiere Pro this year, according to Subramaniam.
“We are very excited about the adoption of content credentials and collaboration around content authenticity efforts,” she says. “We have all the right vectors to advance AI innovation in a responsible way and encourage other companies to follow suit.”
UX and transferable skills
A few months ago, Subramaniam spoke at the Television Academy about AI and the creation of television media and entertainment. “This is a very important topic, especially behind the scenes of the guild strikes,” she says now. “What I find encouraging is that these industries are starting to understand both the power and impact of AI and generative AI. He not only withdrew, but also said, “Please tell me more.'' And how can I get help? And what do we need to know? And please help us understand this so that we can use this technology in a more informed way. ”
This ties in with what she revealed about the user experience (UX) that Video Model's product team is currently planning.
“You need to make sure that the people using Premiere understand the model they're using and the impact of that model,” she explains. “You can see that we're going down the path of guiding the user to say, 'What are you trying to do? Are you trying to come up with more ideas? Okay, so here's a set of models that are good for this. ” Or if you're writing for production, be aware of the models you can choose from and understand what you're actually trying to create. [deploy]. That's exactly the type of UX element we're looking at. ”
As Subramaniam points out, many video editors also use Photoshop, and we suspect Premiere's AI UX will be similar to other Creative Cloud tools.
“We are very cognizant of the fact that there is a lot of customer movement between these applications,” she says. “Photoshop is incredibly powerful and important to video workflows, whether it's the actual video workflow or just the still images you create to promote your video. Our design and AI teams work together to , whether it's Firefly.com, Pro apps or Adobe Express, we try to have some continuity with these concepts, meaning you don't have to relearn the handle.
wild wild west
Subramaniam emphasized that all content generated in the preview video is the actual output from third-party models, not just the Firefly video model. If you have experience generating AI image content from scratch, especially in the early days of AI models, you may have noticed that the results are often not very good. When I suggested that video could be a similar possibility, Subramaniam acknowledged, “The video model is definitely new territory.”
“The image model has really matured. Now everyone is thinking about more difficult media like video and audio,” she added. “We're part of that cutting-edge research. We've quietly developed some features around auto-dubbing, allowing us to use audio models to translate very short clips into different languages. These are very simple and easy to use examples. [for production] We need to globalize at a demanding pace and scale. So we're always pushing the boundaries of research, but we're also thinking clearly about how to incorporate it into applications. ”
Though she admits it's still early days and “it will take a long time for text-to-video conversion to improve,” Subramaniam said the reaction to previews has been “incredibly positive.” says.
“We chose a workflow that professional video editors use,” she said. “Generative Extend, for example, allows you to extend a video clip by a few frames. [In production]if you miss a shot, you've missed a shot, costing you a lot of time and money. [fix in post]. So people have been saying, “Christmas has come.” Please bring it. ”
“Add and remove is also a big hit, especially with people who are new to Rotobrushing in After Effects. The time-saving aspect is huge,” she added.
Is it an interdisciplinary program?
The use of instant replacement in commercials and marketing promotions is obvious, but what was the reaction of TV drama and movie editors?
“Whether you're creating content for social media or longer-term content, professional video editors can spend hours of their day adding, deleting, or enhancing clips. “I'm excited that Adobe understands that they can spend a lot of money on commercial-like pieces,” agrees Subramaniam. “We haven't heard from people yet about how they use it on a daily basis in broadcasting, which I think is coming, but that's also why we're sharing this. We want to get the community excited. So that you can participate in our show.'' Beta. We'll use that feedback and work with them to improve and enhance our innovations during the actual release. ”
After Effects is clearly the next target for Firefly.
“We're definitely thinking about it,” she admits. “We love all of our video applications, not just Premiere. We talked a little bit about revolutionizing audio with generative AI and bringing that to other applications. Our video platform is robust. We are revolutionizing that with generative AI.”
“The pace of innovation and research is very fast,” Subramaniam adds. “We are very excited about the quality levels we will be able to achieve as we bring our commercially safe video models to market this year.”
Deepa Subramaniam has worked at Adobe for more than a decade, first as a computer scientist and then as the lead creator of Creative Cloud. She also advised technology startups as vice president of product and design at Kickstarter, co-founder of Wherewithall, and was director of product for the Hillary for America campaign.
read more IBC Accelerators 2024: Final eight projects announced
