Exploring Mantle's Zero Operator Access Design

At Amazon, our culture built on honest and transparent discussions of growth opportunities allows us to focus on investment and innovation to continually improve the standards of our ability to deliver value to our customers. Earlier this month, I had the opportunity to share an example of this process in action with Amazon Bedrock's next-generation inference engine, Mantle. As generative AI inference and workload fine-tuning continue to evolve, we need to evolve how we best deliver inference to our customers, which led to the development of Mantle.

As we began rethinking the architecture of our next-generation inference engine, we made raising the bar for security a top priority. AWS shares our customers' unwavering focus on security and data privacy. This has been at the heart of our business since the beginning, and a particular focus since the early days of Amazon Bedrock. We knew from the beginning that generative AI inference workloads presented an unprecedented opportunity for our customers to leverage the potential value of their data, but with that opportunity comes the need to ensure the highest standards of security, privacy, and compliance as our customers build generative AI systems that process their most sensitive data and interact with their most critical systems.

As a baseline, Amazon Bedrock is designed with the same operational security standards found across AWS. AWS has always used a least privilege model for operations. In this model, each AWS operator has access to only the minimum set of systems necessary to perform their assigned tasks, and only when that privilege is needed. All access to systems that store or process customer data or metadata is logged, monitored for anomalies, and audited. AWS protects against any actions that override or circumvent these controls. Additionally, Amazon Bedrock never uses your data to train your models. Model providers have no mechanism to access customer data because inference occurs only within an Amazon Bedrock-owned account that model providers cannot access. This strong security posture is a key factor in helping customers unlock the potential of generative AI applications on sensitive data.

With Mantle, we've raised the bar even further. Following the AWS Nitro System approach, we designed Mantle from the ground up to be Zero Operator Access (ZOA), intentionally eliminating technical means for AWS operators to access customer data. Instead, systems and services are managed using automation and secure APIs that protect customer data. Mantle provides no mechanism for AWS operators to sign into the underlying computing system or access customer data such as inference prompts and completions. Interactive communication tools such as Secure Shell (SSH), AWS Systems Manager Session Manager, and serial console are not installed anywhere in Mantle. Additionally, all inference software updates must be signed and verified before being deployed to the service, ensuring that only approved code runs on Mantle.

Mantle uses the recently released EC2 Instance Authentication feature to configure a hardened, constrained, and immutable computing environment for customer data processing. Mantle's service of processing model weights and performing inference operations based on customer prompts is further backed by the high assurance of cryptographically signed authentication measurements through the Nitro Trusted Platform Module (NitroTPM).

When a customer calls a Mantle endpoint (for example, bedrock-mantle.[regions].api.aws) Customer data (prompts), such as those that provide the Responses API in Amazon Bedrock, are encrypted as they exit the customer's environment through TLS and all the way to the Mantle service running on ZOA. Throughout the flow and in Mantle, neither AWS, customer nor model provider operators have access to customer data.

I'm looking forward to it

Mantle's ZOA design embodies AWS' long-term commitment to customer data security and privacy. This focus has enabled teams across AWS to invest in further raising the bar on security. At the same time, we made the basic confidential computing capabilities that we use internally at Amazon, such as NitroTPM Attestation, available to all customers on Amazon Elastic Compute Cloud (Amazon EC2).

We're not done here. We continue to invest in strengthening the security of our customers' data and are committed to providing them with transparency and assurance about how we achieve this.