Deploy QWEN models using Amazon Bedrock Custom Model Import

We look forward to announce that Amazon Bedrock Custom Model Import supports Qwen models. You can now import custom weights for QWEN2, QWEN2_VL, and QWEN2_5_VL architectures, including models such as QWEN 2, 2.5 Coder, QWen 2.5 VL, and QWQ 32b. If you don't need to take your own customized QWEN models to Amazon Bedrock and manage infrastructure or model servings, you can deploy them in a fully managed serverless environment.

This post covers how to deploy a QWEN 2.5 model using Amazon Bedrock custom model imports, making it accessible to organizations looking to use the latest AI capabilities within their AWS infrastructure at an effective cost.

Qwen model overview

Qwen 2 and 2.5 are a large family of language models available in a wide range of sizes and specialized variants to suit a variety of needs.

• General language models: A model with a range of 0.5B to 72B parameters with both a generic task base and an instructional version
• Qwen 2.5-Coder: Specializing in code generation and completion
• Qwen 2.5-math: Focusing on advanced mathematical reasoning
• Qwen 2.5-VL (Vision Language): Enable image and video processing functions, multimodal applications

Overview of importing Amazon Bedrock custom models

Amazon Bedrock Custom Model imports allow you to import and use customized models along with existing basic models (FMS) via a single serverless, integrated API. You can access imported custom models on demand without the need to manage the underlying infrastructure. Accelerate the development of generated AI applications by integrating supported custom models with native Amazon bedrock tools and features such as the Amazon Bedrock Knowledge Bases, Amazon Bedrock Guardrails, and Amazon Bedrock Agent. Importing Amazon Bedrock custom models is generally available in the US East (N. Virginia), US (Oregon), and Europe (Frankfurt) AWS regions. Next, we will explore how to use the QWEN 2.5 model in two common use cases: as a coding assistant and for image understanding. QWEN2.5-CODER is a cutting-edge code model that matches the matching features of proprietary models like the GPT-4O. It supports over 90 programming languages ​​and is excellent at code generation, debugging and inference. QWen 2.5-VL brings advanced multimodal functionality. According to Qwen, Qwen 2.5-VL is skilled in not only recognizing objects such as flowers and animals, but also analyzing charts, extracting text from images, interpreting document layouts, and processing long videos.

Prerequisites

Before importing a QWEN model with Amazon Bedrock Custom Model Import, make sure it exists as follows:

1. Active AWS account
2. Save QWEN model files Amazon Simple Storage Service (Amazon S3) bucket
3. Enough permissions to create an Amazon bedrock model import job
4. We have confirmed that your area supports importing Amazon Bedrock custom models

Use Case 1: Qwen Coding Assistant

This example shows how to build a coding assistant using the QWEN2.5-Coder-7B-Instruct model

1. Hugging your face, search and copy the model ID qwen/qwen2.5-coder-7b-instruct.

I'll use it Qwen/Qwen2.5-Coder-7B-Instruct For the rest of the walkthrough. We have not demonstrated the fine-tuning procedure, but you can also tweak it before importing.

2. Use the following command to download a snapshot of the model locally: The Python library for hugging your face provides a utility called Snapshot Download for this.

from huggingface_hub import snapshot_download

snapshot_download(repo_id=" Qwen/Qwen2.5-Coder-7B-Instruct", 
                local_dir=f"./extractedmodel/")

Depending on the model size, this can take a few minutes. Once complete, the Qwen Coder 7B model folder will contain the following files:

• Configuration File: include config.json, generation_config.json, tokenizer_config.json, tokenizer.jsonand vocab.json
• Model File:4 safetensor Files and model.safetensors.index.json
• document: LICENSE, README.mdand merges.txt

3. Upload and use the model to Amazon S3 boto3 Or the command line:

aws s3 cp ./extractedfolder s3://yourbucket/path/ --recursive

4. Start the import model job using the following API call:

response = self.bedrock_client.create_model_import_job(
                jobName="uniquejobname",
                importedModelName="uniquemodelname",
                roleArn="fullrolearn",
                modelDataSource={
                    's3DataSource': {
                        's3Uri': "s3://yourbucket/path/"
                    }
                }
            )
            

You can also do this using Amazon Bedrock's AWS Management Console.

5. Select on the Amazon Bedrock console Imported models In the navigation pane.
6. choose Import the model.

7. Enter the details including a Model name, Import the job nameand the model S3 location.

8. Create a new service role or use an existing service role. Next, select the import model

9. After selecting Import The console must display the status as an import when the model is imported.

If you are using your own role, add the following trust relationships as explained when creating the service role for model import:

Once the model is imported, wait for the model inference to be ready before chatting with the model via the playground or API. In the following example, we add Python It prompts the model to output Python code directly and lists the items in an S3 bucket. Don't forget to use the appropriate chat template and enter the prompt in the required format. For example, you can use the code below to get a chat template suitable for any model that hugs your face.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")

# Instead of using model.chat(), we directly use model.generate()
# But you need to use tokenizer.apply_chat_template() to format your inputs as shown below
prompt = "Write sample boto3 python code to list files in a bucket stored in the variable `my_bucket`"
messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

Be careful when using invoke_model The API requires that the imported model uses the entire Amazon resource name (ARN). You can find the model ARN in the bedrock console by going to the imported model section and viewing the model details page, as shown in the following image.

Once you're ready to infer the model, you can call the model using the bedrock console or the chat playground in the API.

Use Case 2: Understanding QWEN 2.5 VL Images

QWEN2.5-VL-* provides multimodal functionality that combines vision and language understanding in a single model. This section shows you how to deploy QWEN2.5-VL using an Amazon Bedrock custom model, and imports and tests the image understanding feature.

Import QWEN2.5-VL-7B to Amazon Bedrock

Download the model from Huggingface Face and upload it to Amazon S3.

from huggingface_hub import snapshot_download

hf_model_id = "Qwen/Qwen2.5-VL-7B-Instruct"

# Enable faster downloads
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

# Download model locally
snapshot_download(repo_id=hf_model_id, local_dir=f"./{local_directory}")

Next, import the model into Amazon Bedrock (via console or API):

response = bedrock.create_model_import_job(
    jobName=job_name,
    importedModelName=imported_model_name,
    roleArn=role_arn,
    modelDataSource={
        's3DataSource': {
            's3Uri': s3_uri
        }
    }
)

Test the vision feature

Once the import is complete, test the model with image input. The QWEN2.5-VL-* model requires the proper formatting of multimodal inputs.

def generate_vl(messages, image_base64, temperature=0.3, max_tokens=4096, top_p=0.9):
    processor = AutoProcessor.from_pretrained("Qwen/QVQ-72B-Preview")
    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    
    response = client.invoke_model(
        modelId=model_id,
        body=json.dumps({
            'prompt': prompt,
            'temperature': temperature,
            'max_gen_len': max_tokens,
            'top_p': top_p,
            'images': [image_base64]
        }),
        accept="application/json",
        contentType="application/json"
    )
    
    return json.loads(response['body'].read().decode('utf-8'))

# Using the model with an image
file_path = "cat_image.jpg"
base64_data = image_to_base64(file_path)

messages = [
    {
        "role": "user",
        "content": [
            {"image": base64_data},
            {"text": "Describe this image."}
        ]
    }
]

response = generate_vl(messages, base64_data)

# Print response
print("Model Response:")
if 'choices' in response:
    print(response['choices'][0]['text'])
elif 'outputs' in response:
    print(response['outputs'][0]['text'])
else:
    print(response)
    

Once images of cat examples (such as the following image) are provided, the model will accurately explain important features such as the cat's location, fur color, eye color, and general appearance. This demonstrates the ability to process visual information in the QWEN2.5-VL-* model and generate descriptions of related texts.

Model response:

This image features a close-up of a cat lying down on a soft, textured surface, likely a couch or a bed. The cat has a tabby coat with a mix of dark and light brown fur, and its eyes are a striking green with vertical pupils, giving it a captivating look. The cat's whiskers are prominent and extend outward from its face, adding to the detailed texture of the image. The background is softly blurred, suggesting a cozy indoor setting with some furniture and possibly a window letting in natural light. The overall atmosphere of the image is warm and serene, highlighting the cat's relaxed and content demeanor. 

Pricing

You can use Amazon Bedrock Custom Model Import to host FMs along with Amazon Bedrock, using the weights of custom models within Amazon Bedrock for supported architectures, providing them in a fully managed way in on-demand mode. Importing a custom model does not charge to import a model. You will be charged for inference based on two factors: the number of active model copies and the duration of their activity. The billing occurs in a 5-minute increment starting from the first successful call of each model copy. Pricing per minute varies based on factors such as architecture, context length, space, computing unit version, and other factors, and is layered by model copy size. The custom model required for hosting depends on the model's architecture, parameter count, and context length. Amazon Bedrock automatically manages scaling based on usage patterns. If there is no 5 minute call, scale it to zero and scale as needed, but this may include a cold start latency of up to 1 minute. If the inference volume consistently exceeds the concurrency limit of a single copy, an additional copy is added. Maximum throughput and concurrency during import are determined during import based on factors such as input/output token mix, hardware type, model size, architecture, and inference optimization.

For more information, see Amazon Bedrock Pricing.

cleaning

To avoid continuous fees after completing the experiment:

1. Use the console or API to remove imported QWEN models from Amazon Bedrock custom models.
2. Optionally, if you no longer need an S3 bucket, remove the model file from the S3 bucket.

Remember that importing Amazon Bedrock custom models is not billed to the import process itself, but it is billed to use and storage of the model's inference.

Conclusion

Amazon Bedrock Custom Model Import helps organizations benefit from enterprise-grade infrastructure, while also using powerful public models, particularly Qwen 2.5. The serverless nature of Amazon Bedrock eliminates the complexity of model deployment and operational management, allowing teams to focus on building applications rather than infrastructure. Amazon Bedrock offers a production-ready environment for AI workloads, including auto-scaling, pay-per-user pricing, and seamless integration with AWS services. The combination of QWEN 2.5's advanced AI capabilities and Amazon Bedrock Managed Infrastructure provides the optimal balance of performance, cost, and operational efficiency. Organizations can start and scale up with smaller models when needed, while still fully controlling the deployment of their models and benefiting from AWS security and compliance capabilities.

For more information, see the Amazon Bedrock User Guide.

About the author

Ajit Mahareddy It is an experienced product with over 20 years of experience in product management, engineering and market. Prior to his current role, AJIT led AI/ML products to major technology companies such as Uber, Turing and eHealth. He is passionate about advancing generative AI technology and promoting real-world impact with generative AI.

Shreyas Subramanian A leading data scientist, helping customers by using generative AI and solving business challenges using AWS services. Shrayas has a background in large-scale optimization and ML, and augmentation learning to accelerate ML use and optimization tasks.

Yang Yang Chang He is a senior Generated AI Data Scientist at Amazon Web Services, working as a Generated AI Specialist on cutting-edge AI/ML technologies, helping customers use Generated AI to achieve the desired results. Yanyan graduated from Texas A&M University with a PhD in Electrical Engineering. Outside of work, she loves to travel, work out and explore new things.

Dharinee Gupta He is the Engineering Manager at AWS Bedrock and focuses on enabling customers to seamlessly utilize open source models via serverless solutions. Her team specializes in optimizing these models to provide the best cost-performance balance for their customers. Prior to her current role, she gained extensive experience in authentication and authentication systems on Amazon and developed a secure access solution for Amazon's offering. Dharinee is passionate about making advanced AI technologies accessible and efficient for AWS customers.

Lokeshwaran Ravi I'm a senior deep learning compiler engineer at AWS and specializes in ML optimization, model acceleration, and AI security. He focuses on improving efficiency, reducing costs, and democratizing AI technology by creating a safe ecosystem, making cutting-edge ML accessible and impactful across the industry.

June won He is the leading product manager for Amazon Sagemaker Jumpstart. He focuses on making Foundation models easy to discover to help customers build generative AI applications. His experience on Amazon also includes mobile shopping apps and last mile delivery.