Editor’s note: This post AI on A blog series exploring the latest technology and real-world applications in agent AI, chatbots, and co-pilots. This series also focuses on NVIDIA software and hardware that power advanced AI agents. These form the basis of AI query engines that gather insights and perform tasks to transform everyday experiences and reshape industries.
Today’s computer vision systems are great at identifying what’s happening in physical spaces and processes, but they lack the ability to explain the details of a scene and why it’s important, or to reason about what will happen next.
Agent intelligence powered by Vision Language Models (VLM) helps bridge this gap, giving teams quick and easy access to critical insights and analytics that connect text descriptors with spatio-temporal information and the billions of visual data points captured by the system every day.
Here are three approaches that organizations can use to enhance their legacy computer vision systems with agent intelligence:
- Apply dense captions to searchable visual content.
- Enhance your system alerts with detailed context.
- Use AI inference to summarize information from complex scenarios and answer questions.
Make visual content searchable using high-density captions
Traditional convolutional neural network (CNN)-based video search tools have limited training, context, and semantics, making manually gathering insights tedious and time-consuming. CNNs are tailored to perform specific visual tasks, such as finding anomalies, and lack the multimodal ability to convert what they see into text.
Businesses can embed VLM directly into existing applications to generate highly detailed captions for images and videos. These captions transform unstructured content into rich, searchable metadata, allowing for much more flexible visual searches that are not limited to file names or basic tags.
For example, UVeye, an automated vehicle inspection system, processes over 700 million high-resolution images every month to create one of the world’s largest vehicle and component datasets. By applying VLM, UVeye transforms this visual data into structured condition reports that detect and locate subtle defects, changes, or foreign objects with the highest accuracy and reliability.
Visual understanding powered by VLM adds critical context and ensures transparent and consistent insight into compliance, safety, and quality control. UVeye detects 96% of defects compared to 24% using manual methods, allowing early intervention to reduce downtime and control maintenance costs.
Relo Metrics, a provider of AI-powered sports marketing measurement, helps brands quantify the value of their media investments and optimize spend. By combining VLM and computer vision, Relo Metrics goes beyond basic logo detection to capture context, such as a courtside banner displayed during the final shot of a game, and convert it into real-time monetary value.
This contextual insight feature highlights when and how a logo appears, especially in high-impact moments, giving marketers a clearer picture of return on investment and how to optimize their strategies. For example, Stanley Black & Decker (which includes the DeWalt brand) has traditionally relied on end-of-season reports to evaluate sponsor asset performance, limiting timely decision-making. Stanley Black & Decker uses Relo Metrics for real-time insights to adjust billboard positioning, saving $1.3 million in potential sponsored media value.
Enhancing computer vision system alerts with VLM inference
CNN-based computer vision systems often generate binary detection alerts, such as yes or no, true or false. Without VLM’s inference capabilities, false positives and missing details can occur, leading to costly mistakes in safety and security and loss of business intelligence. Rather than completely replacing these CNN-based computer vision systems, VLM can easily extend them as an intelligent add-on. By overlaying VLM on top of a CNN-based computer vision system, detection alerts are not only flagged but reviewed in context to explain where, how, and why the incident occurred.
To enable smarter urban traffic management, Linker Vision uses VLM to verify important urban alerts such as traffic accidents, flooding, and downed power poles and trees due to storms. This reduces false positives and adds important context to each event, improving real-time city response.
Linker Vision’s agent AI architecture includes automated event analysis from over 50,000 diverse smart city camera streams, enabling cross-functional remediation. This means coordinating actions across teams such as traffic control, public works, and first responders when an incident occurs. All camera streams can be queried simultaneously, allowing the system to quickly and automatically turn observations into insights and trigger recommendations for next-best actions.
Automated analysis of complex scenarios with Agentic AI
Agentic AI systems can process, reason, and answer complex queries across video streams and modalities, including audio, text, video, and sensor data. This is possible by combining VLM with inference models, large-scale language models (LLM), search augmented generation (RAG), computer vision, and speech transcription.
Essentially integrating VLM into your existing computer vision pipeline can help you validate short video clips of important moments. However, this approach is limited by the number of visual tokens that a single model can process at once, resulting in surface-level answers without long-term context or external knowledge.
In contrast, the entire architecture built on agent AI enables scalable and accurate processing of long-duration multichannel video archives. This provides deeper, more accurate, and more reliable insights beyond surface-level understanding. Agent systems can be used for root cause analysis and analysis of long inspection videos to generate reports with time-stamped insights.
Levatas uses mobile robots and autonomous systems to develop visual inspection solutions that improve the safety, reliability, and performance of critical infrastructure assets such as utility substations, fuel terminals, rail yards, and logistics hubs. Levatas used VLM to build a video analytics AI agent to automatically review inspection footage and create detailed inspection reports, dramatically accelerating traditional manual and time-consuming processes.
For customers like American Electric Power (AEP), Levatas AI integrates with Skydio X10 devices to streamline inspections of power infrastructure. Levatas will enable AEP to autonomously inspect utility poles, identify thermal issues and detect equipment damage. Alerts are sent instantly to the AEP team when issues are detected, allowing for rapid response and resolution, ensuring a reliable, clean and affordable energy supply.
AI game highlighting tools like Eklipse use VLM-powered agents to enrich video game live streams with captions and index metadata to quickly query, summarize, and create polished highlight reels in minutes. This is 10x faster than traditional solutions and leads to a better content consumption experience.
Powering agenttic video intelligence with NVIDIA technology
For advanced search and inference, developers can use multimodal VLMs such as NVCLIP, NVIDIA Cosmos Reason, and Nemotron Nano V2 to build metadata-rich indexes for search.
To integrate VLM into computer vision applications, developers can use the Event Reviewer feature in NVIDIA Blueprint for Video Search and Summarization (VSS), which is part of the NVIDIA Metropolis platform.
For more complex queries and summarization tasks, VSS blueprints can be customized to allow developers to build AI agents that access VLM directly or use VLM in conjunction with LLM, RAG, and computer vision models. This enables smarter operations, richer video analytics, and real-time process compliance that scales with your organization’s needs.
Click here to learn more about NVIDIA Powered agent video analysis.
Subscribe to NVIDIA’s Vision AI Newsletter to stay informed. join the community Follow NVIDIA AI linkedin, Instagram, × and facebook.
please explore VLM Technology Blogand Self-paced video tutorials and live streams.
