Accelerate enterprise AI development with Weights & Biases and Amazon Bedrock AgentCore

This post was co-authored by Thomas Capelle and Ray Strickland of Weights & Biases (W&B).

The adoption of generative artificial intelligence (AI) is accelerating across the enterprise, evolving from simple underlying model interactions to sophisticated agent workflows. As organizations move from proof of concept to production deployment, they need robust tools to develop, evaluate, and monitor AI applications at scale.

This post shows how you can use Amazon Bedrock's Foundation Model (FM) and the newly released Amazon Bedrock AgentCore with W&B Weave to help you build, evaluate, and monitor enterprise AI solutions. We cover the entire development lifecycle, from tracking individual FM calls to monitoring complex agent workflows in production.

Overview of W&B Weave

Weights & Biases (W&B) is an AI developer system that provides comprehensive tools for training, fine-tuning, and leveraging underlying models for companies of all sizes in a variety of industries.

W&B Weave provides an integrated suite of developer tools to support every stage of your agent AI workflow. This allows you to:

• Tracing and monitoring: Track large language model (LLM) calls and application logic to debug and analyze production systems.
• Systematic iteration: Adjust and iterate on prompts, datasets, and models.
• experiment: Try different models and prompts in the LLM Playground.
• evaluation: Use custom or pre-built scorers with comparison tools to systematically evaluate and enhance application performance. Collect user and expert feedback for real-world testing and evaluation.
• guardrail: Protect your applications with content management, rapid safety, and other safeguards. Use custom or third-party guardrails, including Amazon Bedrock Guardrails, or W&B Weave's native guardrails.

W&B Weave can be fully managed by Weights & Biases in a multi-tenant or single-tenant environment, or deployed directly into a customer's Amazon Virtual Private Cloud (VPC). Additionally, W&B Weave's integration into the W&B development platform provides organizations with a seamlessly integrated experience between model training/fine-tuning workflows and agent AI workflows.

To get started, subscribe to the Weights & Biases AI development platform through AWS Marketplace. Individuals and academic teams can subscribe to W&B at no additional charge.

Tracking Amazon Bedrock FM using W&B Weave SDK

W&B Weave seamlessly integrates with Amazon Bedrock through the Python and TypeScript SDKs. Once you install the library and patch your Bedrock client, W&B Weave automatically tracks LLM calls.

!pip install weave
import weave
import boto3
import json
from weave.integrations.bedrock.bedrock_sdk import patch_client

weave.init("my_bedrock_app")

# Create and patch the Bedrock client
client = boto3.client("bedrock-runtime")
patch_client(client)

# Use the client as usual
response = client.invoke_model(
    modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 100,
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ]
    }),
    contentType="application/json",
    accept="application/json"
)
response_dict = json.loads(response.get('body').read())
print(response_dict["content"][0]["text"])

This integration automatically creates versions of your experiments and tracks configuration, giving you complete visibility into your Amazon Bedrock applications without changing your core logic.

Experiment with Amazon Bedrock FM at W&B Weave Playground

W&B Weave Playground accelerates rapid engineering with an intuitive interface for testing and comparing Bedrock models. The main features are:

• Direct prompt editing and message retry
• Compare models side by side
• Access from trace view for rapid iteration

First, start experimenting by adding your AWS credentials to your Playground settings and choosing your preferred Amazon Bedrock FM. This interface allows rapid iteration at prompts while maintaining full traceability of experiments.

Amazon Bedrock FM rating by W&B Weave rating

W&B Weave Evavals provides specialized tools to effectively evaluate generative AI models. Using W&B Weave Evaluation with Amazon Bedrock, users can efficiently evaluate these models, analyze output, and visualize performance across key metrics. Users can use W&B Weave's built-in scorers, third-party or custom scorers, and human/expert feedback. This combination provides a deeper understanding of trade-offs between models, including differences in cost, accuracy, speed, and output quality.

W&B Weave has a great way to track evaluations using the Model & Evaluation class. To set up an assessment job, customers can:

• Define a list of datasets or dictionaries containing a collection of samples to evaluate.
• Create a list of scoring functions. Each function has a model_output and optionally other inputs from the sample, and must return a dictionary containing the scores.
• Define an Amazon Bedrock model using the Model class
• Evaluate this model by calling Evaluation

The following is an example of the settings for an evaluation job.

import weave
from weave import Evaluation
import asyncio

# Collect your examples
examples = [
    {"question": "What is the capital of France?", "expected": "Paris"},
    {"question": "Who wrote 'To Kill a Mockingbird'?", "expected": "Harper Lee"},
    {"question": "What is the square root of 64?", "expected": "8"},
]

# Define any custom scoring function
@weave.op()
def match_score1(expected: str, output: dict) -> dict:
    # Here is where you'd define the logic to score the model output
    return {'match': expected == model_output['generated_text']}

@weave.op()
def function_to_evaluate(question: str):
    # here's where you would add your LLM call and return the output
    return  {'generated_text': 'Paris'}

# Score your examples using scoring functions
evaluation = Evaluation(
    dataset=examples, scorers=[match_score1]
)

# Start tracking the evaluation
weave.init('intro-example')
# Run the evaluation
asyncio.run(evaluation.evaluate(function_to_evaluate))

Evaluation dashboards visualize performance metrics and help you make informed model selection and configuration decisions. For detailed guidance, see our previous post on evaluating LLM summaries using Amazon Bedrock and Weave.

Amazon Bedrock AgentCore Observability Enhancements with W&B Weave

Amazon Bedrock AgentCore is a complete set of services to more securely deploy and operate high-performance agents at enterprise scale. It provides a more secure runtime environment, workflow execution tools, and operational controls that work with popular frameworks such as Strands Agents, CrewAI, LangGraph, LlamaIndex, and many LLM models, whether from Amazon Bedrock or external sources.

AgentCore has built-in observability through the Amazon CloudWatch dashboard that tracks key metrics such as token usage, latency, session duration, and error rates. It also tracks workflow steps, showing which tools were called and how the model responded, providing essential visibility for debugging and quality assurance in production.

Using AgentCore and W&B Weave together allows teams to use AgentCore's built-in operational monitoring and security foundation while also using W&B Weave when it aligns with their existing development workflows. Organizations that have already invested in a W&B environment can choose to incorporate W&B Weave's visualization tools along with AgentCore's native functionality. This approach gives teams the flexibility to use the observability solution that best fits their established processes and preferences when developing complex agents that chain multiple tools and inference steps.

There are two main approaches to adding W&B Weave observability to AgentCore agents. Either using the native W&B Weave SDK or integrating via OpenTelemetry.

Native W&B Weave SDK

The simplest approach is to use W&B Weave's @weave.op decorator to automatically track function calls. Initialize W&B Weave with your project name and wrap the functions you want to monitor.

import weave
import os

os.environ["WANDB_API_KEY"] = "your_api_key"
weave.init("your_project_name")

@weave.op()
def word_count_op(text: str) -> int:
    return len(text.split())

@weave.op()
def run_agent(agent: Agent, user_message: str) -> Dict[str, Any]:
    result = agent(user_message)
    return {"message": result.message, "model": agent.model.config["model_id"]}

AgentCore runs as a Docker container, so add W&B Weave (for example, uv add Weave) to your dependencies to include it in your container image.

OpenTelemetry integration

For teams already using OpenTelemetry or needing vendor-neutral instrumentation, W&B Weave directly supports OTLP (OpenTelemetry Protocol).

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

auth_b64 = base64.b64encode(f"api:{WANDB_API_KEY}".encode()).decode()
exporter = OTLPSpanExporter(
    endpoint="https://trace.wandb.ai/otel/v1/traces",
    headers={"Authorization": f"Basic {auth_b64}", "project_id": WEAVE_PROJECT}
)

# Create spans to track execution
with tracer.start_as_current_span("invoke_agent") as span:
    span.set_attribute("input.value", json.dumps({"prompt": user_message}))
    result = agent(user_message)
    span.set_attribute("output.value", json.dumps({"message": result.message}))

This approach routes traces to W&B Weave for visualization while remaining compatible with AgentCore's existing OpenTelemetry infrastructure. When using both AgentCore and W&B Weave together, teams have multiple options for observability. AgentCore's CloudWatch integration provides tracing of agent inference and tool selection while monitoring system health, resource utilization, and error rates. W&B Weave provides visualization capabilities that display execution data in a format familiar to teams already using a W&B environment. Both solutions provide visibility into how agents process information and make decisions, allowing organizations to choose the observability approach that best fits their existing workflows and configurations. This two-tier approach allows users to:

• Monitor production service level agreements (SLAs) through CloudWatch alerts
• Debug complex agent behavior with W&B Weave's Trace Explorer
• Optimize token usage and latency with detailed execution breakdowns.
• Compare agent performance across different prompts and configurations

The integration requires minimal code changes and your existing AgentCore deployment can be maintained and scaled according to agent complexity. Whether you're building a simple tool invocation agent or orchestrating a multi-step workflow, this observability stack provides the insights you need to iterate quickly and deploy with confidence.

See previous post for implementation details and complete code example.

conclusion

In this post, we demonstrated how Amazon Bedrock's FM and AgentCore can be combined with W&B Weave's comprehensive observability toolkit to build and optimize enterprise-grade agent AI solutions. We considered how W&B Weave can power every stage of the LLM development lifecycle, from initial experimentation in the playground to systematic evaluation of model performance and, ultimately, production monitoring of complex agent workflows.

The integration of Amazon Bedrock and W&B Weave provides several important features.

• Automatically track Amazon Bedrock FM calls with minimal code changes using W&B Weave SDK
• Test prompts and compare models for rapid experimentation using W&B Weave Playground's intuitive interface
• Systematic evaluation with custom scoring functions to evaluate different Amazon Bedrock models
• Comprehensive observability of AgentCore deployments using CloudWatch metrics provides more robust operational monitoring, complemented by detailed execution traces.

To get started:

• Request a free trial or subscribe to the Weights &Biases AI development platform through AWS Marketplace
• Install W&B Weave SDK and start tracking Bedrock FM calls by following the code examples
• Experiment with different models in the W&B Weave Playground by adding your AWS credentials and testing different Amazon Bedrock FMs.
• Set up an evaluation using the W&B Weave evaluation framework to systematically compare the performance of models for your use cases.
• Enhance your AgentCore agent by adding W&B Weave observability using the native SDK or OpenTelemetry integration

Start with a simple integration to track Amazon Bedrock calls and gradually adopt more advanced features as your AI application becomes more complex. The combination of Amazon Bedrock and W&B Weave's comprehensive development tools provides the foundation you need to build, evaluate, and maintain production-ready AI solutions at scale.

About the author

James Yee I am a Senior AI/ML Partner Solutions Architect at AWS. He spearheads AWS' strategic partnerships in emerging technologies and leads engineering teams to design and develop cutting-edge collaborative solutions in generative AI. He enables field and technical teams to seamlessly deploy, operate, secure, and integrate partner solutions on AWS. James works closely with business leaders to define and execute collaborative go-to-market strategies to drive growth for cloud-based businesses. Outside of work, I enjoy playing soccer, traveling, and spending time with my family.

ray strickland He is a Senior Partner Solutions Architect at AWS, specializing in AI/ML, Agenttic AI, and Intelligent Document Processing. He enables partners to deploy scalable generative AI solutions using AWS best practices and drives innovation through strategic partner assistance programs. Ray collaborates across multiple AWS teams to accelerate AI adoption and has extensive experience in partner assessment and enablement.

thomas capel I'm a machine learning engineer at Weights & Biases. He is responsible for keeping the www.github.com/wandb/examples repository live and up to date. We're also building content about MLOPS, W&B applications to industry, and fun deep learning in general. Previously, he used deep learning to solve short-term predictions for solar energy. He has a background in urban planning, combinatorial optimization, transportation economics, and applied mathematics.

scott juan I am Alliance Director at Weights & Biases. Prior to joining W&B, he led numerous strategic partnerships at AWS and Cloudera. Scott studied materials engineering and is passionate about renewable energy.