Contextual intelligence for big data and AI agents

Machine Learning


An agent’s intelligence is only as good as the context it can reason about. Today, that context is data lakes, data warehouses, lakeside house,database,stream, And then Organizational knowledge that has never been documented before. We want to trust the decisions made by our AI agents. can’t do it Occurs until the agent understands the context. Imagine what would be possible if you provided a way for agents to securely access the context they need to make decisions they trust.

This is the AWS Summit in New York City. It was announceare doing series of innovation deliver intelligence For big data and AI agents.

AWS Context (coming soon)

In today’s keynote, we introduced AWS Context, a new service that automatically maps relationships between existing data into a knowledge graph and provides agent search. This gives AI agents in your organization access to data relationships, business rules, and domain knowledge managed at runtime. Data stewards and curators manage graphs through an intuitive console experience, review and promote inferred relationships to production, and enrich with domain-specific knowledge such as business definitions and usage rules.

AWS Context extends the same knowledge graph technology that powers Amazon Quick. At Amazon Quick, hundreds of thousands of users interact every day with a production knowledge graph that catalogs datasets, dashboards, and metadata, and learns from usage patterns to make every interaction smarter. This graph already processes millions of requests per day. With AWS Context, you extend what was your personal knowledge graph into an organizational knowledge graph, a shared, managed context layer available to agents and applications within your organization. Existing Amazon Quick users can benefit immediately. When AWS Context is enabled, Quick agents have access to a broader enterprise knowledge graph, including relationships between systems, business rules, and curated context, beyond what a single user’s personal graph can provide. AWS Glue Data Catalog, Amazon SageMaker Unified Studio, and AWS Lake Formation are integrated with Knowledge Graph, so teams can manage the Knowledge Graph with business rules and permissions and add new context automatically with AI assistance or explicitly through manual curation.

Key elements of the context layer are exposed to Amazon S3 in Apache Iceberg format, so customers are free to use any Iceberg-compliant tool to consume metadata and build against AWS Context based on open standards. There is no infrastructure to provision or acquisition pipelines to build, and customers can start collecting and curating context for their agents with just a few clicks in the AWS Management Console.

Let’s take a closer look at the features behind it.

Context to learn from how agents work

AWS context become wiser The more the agent uses it, the more the agent queries the graph. observe Which sources produce the correct results, which joining paths the agent relies on, and which carefully selected rules are applied. We rank sources by actual usage and share what we learn across the organization, so when one agent discovers the correct join path or resolves a schema ambiguity, other agentss pick it up andoutside need be human Re-curatoe graph.

Open and portable design

AWS Context exposes all key metadata from structured and unstructured sources in Apache Iceberg format in Amazon S3 tables, so you can query the context and build, audit, or migrate downstream systems on top of it using Amazon Athena, Amazon Redshift, Apache Spark, or any Iceberg-compatible engine.

AWS Context is designed to connect to third-party catalogs, so you can bring context from non-AWS systems into the same graph. Agents run queries through the agent search API and MCP tools, whether built on Amazon Bedrock AgentCore, deployed on Amazon EKS, or running on an MCP-compatible framework. The context is queryable, portable in Apache Iceberg format, and completely yours.

Identity aware and managed by default

Deploying agents into production raises governance issues. This means being able to show exactly what data can be accessed and who has access to what. AWS Context Answer both by Create any query conscious of identity. each call is designed to It inherits the calling user’s IAM and Lake Formation permissions, so the agent can see and traverse relationships only for that identity. is authorized to access. Access is via identity, so all interactions are auditable. Security and compliance teams can use the same controls they already rely on to see what agents have accessed and with what privileges.

AWS Glue Data Catalog Business Context and Semantic Search (Preview)

Today, we are also announcing a preview of Business Context and Semantic Search in the AWS Glue Data Catalog, providing context and tools that make it easier for humans and AI agents to discover and understand data. Customers can now enrich Glue tables, views, and columns (including those supported by S3 tables) with business descriptions, glossary terms, and custom metadata and associate them with skill assets that provide additional data context stored outside of the catalog. Business context is indexed alongside technical metadata in Glue Data Catalog, so customers can use the new Glue Search API to find data faster by business meaning, and AI agents can make inferences based on trusted definitions rather than guessing context.

We’re also excited to offer previews of skill assets in the Glue Data Catalog. Data producers can now create skill assets, a new asset type that references URIs to files (such as AI skills, guide markdown files, team runbooks, etc.) hosted anywhere in S3, Git repositories, wikis, and more. Associating a skill asset with a data asset allows agents to get additional context and instructions to work with specific data, gradually, without having to retrain all agents one prompt at a time. For example, a skill URI location can point to a team’s repository that contains data usage details such as domain-specific documentation and processes. Things like granularity and scope, common query patterns and best practices, and usage rules (when to use the data, join keys and required filters).

Skill assets help AI agents find the right data to use within your data assets, but that’s only half the problem. The agent also needs to know how to use it, such as what filters to apply before aggregating the data, join paths to follow, and notes that don’t appear in the technical schema. The AWS Agent Toolkit currently includes default skills that help AI agents work with the Glue Data Catalog and other features such as Amazon Athena and S3 tables. Many companies have unique skills developed by their data teams. First, developers can connect Any MCP-compatible agents use the fully managed remote AWS MCP to access AWS service skills, or install the aws-data-analytics plugin for Claude Code, Cursor, and Amazon Kiro to ask agents to search for data, perform analysis, or build applications on top of that data using AWS or other custom skills. Agents built using the AgentCore harness can access all AWS skills in the AWS Agent Toolkit with one line of code. This enables agents to quickly leverage AWS service expertise and best practices.

Amazon S3 Annotations (official release)

To make it easy for customers to add their own custom contexts to their data lakes, we are announcing the general availability of Amazon S3 annotations. This is a new way to attach rich, queryable business context directly to S3 objects and store that context in S3 Iceberg tables. Customers have long used object tags and user-defined metadata to describe objects in S3, and these remain a good tool for operational tasks such as access control and small pieces of information that are set at upload time. But as customers build agents around their data, they want to attach much more metadata. They want to create and evolve at scale a rich context that agents can read and act on. S3 annotations provide that functionality in an open data format. Each object stored in S3 can contain up to 1 GB of context. Annotations are mutable, allowing the context to evolve as the data changes. S3 annotations exist with S3 objects in S3 storage. This means that S3 annotations travel with the associated S3 object through copy and replication operations and are deleted when the object is deleted. With annotations, you don’t have to build, synchronize, or protect a separate metadata database from becoming stale.

Annotations are made queryable through S3 metadata. When you enable annotation tables on your bucket, all annotations automatically flow into the fully managed Iceberg tables. All objects can be queried using Amazon Athena, Amazon Redshift, or any Iceberg-compatible engine, and the agent can discover annotations in natural language through the S3 Tables MCP server.

and Amazon S3 annotationsallows you to attach rich business context directly to S3 objects and query them at scale, so agents can find what they need without having to build a separate metadata system.

Context is a data lake for AI agents, and these innovations are building the foundation of knowledge and intelligence for AI agents working with data across organizations and enterprises of all sizes.


About the author

Mylan Thomsen Bukovec

Mai-Lan Tomsen Bukovec, Vice President of Technology at AWS, leads Amazon Cloud Data Services, which millions of AWS customers rely on for digital transformation, business analytics, machine learning, generative AI, and next-generation customer experiences. With over 25 years of experience in the technology industry, Mai-Lan is a pioneer in helping customers leverage cloud-based technology to transform their businesses.



Source link