June 29, 2023
Searching for cute cat videos on the internet is not difficult. But if you want to find a video of a cat chasing a dog in the city on a sunny day, it becomes even more difficult.
Now, the Mizzou Engineering team has developed a new system that uses image captions to search for video clips of specific objects or scenes. Associate Professor Praveen Rao and former PhD student Arun Zakaria gave an overview of the method at the Society of Computing Machinery (ACM) conference earlier this month.
“People watch a lot of videos,” Rao said. “With such a large video database, we need a system that allows users to efficiently search for videos of interest. We built a search system to search for video clips.”
Current video retrieval methods are based on deep learning, a form of artificial intelligence, that extracts image features in videos. Its deep learning captures relationships between scenes and objects.
Because AI can already identify objects and scenes and provide captions for them, Rao and his team chose to base the system on text rather than images.
The team previously developed a similar system known as QIK for image search. The system uses AI to automatically generate captions for photos and uses these captions for image search.
“We took over the previous system and used the same principles to build a video search system,” he said.
Here’s how it works: A given video clip is split into a small number of frames representing the entire video. For example, if you want a video of a man walking to a car, getting in a car, and driving away, the system will convert it to three frames of him representing the context of the entire video (man walking, getting in, driving). split into
When a user issues a query requesting such videos, the video with all three components is ranked highest, and the video of a man just walking is ranked lower.
“This is a different approach from what the community has done so far,” Rao said. “That’s why this job is so interesting.”
Further research is needed to expand online search efforts, but Rao said the program could be adapted to specific areas such as defense, healthcare and e-commerce.
“What we’ve built is a local system that companies with collections of videos they want to search can use,” says Rao. “Creating a large system will require more research and development, but the concept can be applied to any number of videos.”
Click here for details computer science At Mizo Engineering.
