In recent years, rapid advances in artificial intelligence and machine learning (AI/ML) technologies have revolutionized many aspects of digital content creation. One particularly exciting development is the emergence of video generation capabilities, providing unprecedented opportunities for businesses across a wide range of industries. This technology allows you to seamlessly combine to create short video clips that can create longer, more complex videos. The potential applications of this innovation are vast and widespread, and are committed to transforming the way businesses communicate, market and engage with their audience. Video generation technology presents countless use cases for businesses looking to enhance their visual content strategies. For example, e-commerce companies can use this technology to create dynamic product demonstrations and showcase items from multiple angles and in a variety of contexts. In the area of education and training, organizations can generate educational videos tailored to specific learning goals and quickly update content as needed without re-entering the entire sequence. With customized messaging and visuals, marketing teams can create personalized video ads at scale, targeting different demographics. Additionally, the entertainment industry can benefit greatly from its ability to quickly prototype scenes, visualize concepts, and help create animated content. The flexibility provided by combining these generated clips with longer videos opens up even more possibilities. Companies can create modular content that can be quickly relocated and reused for a variety of displays, viewers, or campaigns. This adaptability not only saves time and resources, but also allows for a more agile and responsive content strategy. As we dig deeper into the possibilities of video generation technology, it becomes clear that its value goes far beyond just convenience, providing transformative tools that can drive innovation, efficiency and engagement across the enterprise landscape.
In this post, we will explore how to implement a robust AWS-based solution for video generation using the Cogvideox model and Amazon Sagemaker AI.
Solution overview
Our architecture uses AWS managed services to provide a highly scalable and secure video generation solution. The Data Management Layer implements three purpose-specific Amazon Simple Storage Service (Amazon S3) buckets (input video, processed output, and access logging).
For computational resources, use AWS Fargate for Amazon Elastic Container Service (Amazon ECS) to host Streetmlit web applications and provide auto-scaling capabilities for serverless container management. Traffic is efficiently distributed through application load balancers. The AI Processing Pipeline uses Sagemaker AI Processing Jobs to process video generation tasks and separates intensive calculations from the web interface for cost optimization and improved serviceability. User Prompt feeds the Cogvideox-5B model for high quality video generation, creating an end-to-end solution that balances performance, security, and cost-effectiveness.
The following diagram illustrates the solution architecture.
Cogvideox model
Cogvideox is an open source, cutting-edge text-to-video generation model that can produce 10 seconds of continuous video with 16 frames at a resolution of 768 x 1360 pixels. This model effectively converts text prompts into coherent video narratives and addresses the general limitations of previous video generation systems.
The model uses three important innovations:
- Improve compression efficiency and video quality using 3D Variation Autoencoder (VAE) that compresses videos along both spatial and temporal dimensions
- Expert Transformers with adaptive layered lams that enhance intertext alignment through deeper fusion between modalities
- Progressive training and multi-resolution frame pack techniques that allow for longer, coherent videos with important motion elements
Cogvideox also benefits from an effective intertext data processing pipeline with a variety of preprocessing strategies and specialized video captioning methods, contributing to higher generation quality and better semantic alignment. Model weights are publicly available, allowing you to access implementations in a variety of business applications, including product demonstrations and marketing content. The following diagram shows the architecture of the model.
Quick reinforcement
To improve the quality of video generation, this solution offers options to enhance user-provided prompts. This is done by instructing the Large Language Model (LLM) to get the user's initial prompt in this case, add additional details and expand it, creating a more comprehensive description of video creation. The prompt consists of three parts.
- Role Section – Defines the purpose of AI when enhancing video generation prompts
- Task section – Specifies the instructions that must be executed at the original prompt
- Prompt Section – Where the original user input is inserted
By adding more descriptive elements to the original prompt, the system aims to provide richer and detailed instructions to a video generation model, which could result in a more accurate and visually appealing video output. This solution uses the following prompt template:
"""
Your role is to enhance the user prompt that is given to you by
providing additional details to the prompt. The end goal is to
covert the user prompt into a short video clip, so it is necessary
to provide as much information you can.
You must add details to the user prompt in order to enhance it for
video generation. You must provide a 1 paragraph response. No
more and no less. Only include the enhanced prompt in your response.
Do not include anything else.
{prompt}
"""Prerequisites
Before deploying a solution, make sure you have the following prerequisites:
- AWS CDK Toolkit – Install AWS CDK Toolkit globally using NPM.
npm install -g aws-cdk
This provides the core functionality for deploying infrastructure as code to AWS. - Docker Desktop – This is required for local development and testing. Before deploying, make sure that container images can be constructed and tested locally.
- AWS CLI – You must install the AWS Command Line Interface (AWS CLI) and configure it with the appropriate credentials. This requires an AWS account with the required permissions. Configure using the AWS CLI
aws configureIt has an access key and a secret. - Python environment – You must install Python 3.11+ on your system. It is recommended to use a virtual environment for isolation. This is required for both AWS CDK infrastructure and streamlined applications.
- Active AWS Account – Sagemaker service quota request must be filed in ML.G5.4XLARGE to process the job.
Deploy the solution
This solution has been tested with us-east-1 AWS Region. To deploy, complete the following steps:
- Create and activate a virtual environment.
python -m venv .
venv source .venv/bin/activate- Install infrastructure dependencies.
cd infrastructure
pip install -r requirements.txt- Bootstrap for the AWS CDK (if it's not already completed in your AWS account):
cdk bootstrap- Deploy the infrastructure.
cdk deploy -c allowed_ips="[""$(curl -s ifconfig.me)'/32"]'To access the Streamlit UI, select the Streamliturl link in the AWS CDK output log after a successful deployment. The following screenshot shows a streamlined UI that can be accessed via a URL.
Basic video generation
Complete the following steps to generate the video:
- Enter a natural language prompt in the text box at the top of the page.
- Copy this prompt to the text box at the bottom.
- choose Generate a video Create a video using this basic prompt.
Below is the output from a simple prompt “A bee on a flower.”
Improved video generation
For high quality results, complete the following steps:
- Enter the first prompt in the text box at the top.
- choose Enhance the prompt Send the prompt to Amazon Bedrock.
- Wait for Amazon Bedrock to expand the prompt to a more descriptive version.
- Check the expansion prompt that appears in the bottom text box.
- Edit the prompt further if necessary.
- choose Generate a video Starts the processing job in Cogvideox.
Once the process is complete, the video will appear on the page using the download option. Below is an example of the extension prompt and output:
"""
A vibrant yellow and black honeybee gracefully lands on a large,
blooming sunflower in a lush garden on a warm summer day. The
bee's fuzzy body and delicate wings are clearly visible as it
moves methodically across the flower's golden petals, collecting
pollen. Sunlight filters through the petals, creating a soft,
warm glow around the scene. The bee's legs are coated in pollen
as it works diligently, its antennae twitching occasionally. In
the background, other colorful flowers sway gently in a light
breeze, while the soft buzzing of nearby bees can be heard
"""
Add an image to the prompt
If you want to include images in the text prompt, complete the following steps:
- Complete the text prompt and optional extension steps.
- choose Include images.
- Upload the photo you want to use.
- With both the text and image prepared, select it Generate a video Starts the processing job.
Below is an example of a previous extension prompt with the included images.
To see more examples, see the Cogvideox Gallery.
cleaning
To avoid incurring fees, clean up the resources you created as part of this post.
cdk destroy
Considerations
Although current architectures serve as effective proof of concept, several enhancements are recommended for production environments. Considerations include improved job management and reliability, and improved error handling and monitoring capabilities, with the implementation of an API gateway with AWS Lambda backrest endpoints and improved interfaces and authentication, as well as improved queue-based architectures using Amazon Simple Keue Service (Amazon SQS).
Conclusion
Video generation technology has emerged as a transformative power in the creation of digital content, as demonstrated in a comprehensive AWS-based solution using the Cogvideox model. By combining powerful AWS services such as Fargate, Sagemaker, and Amazon Bedrock with innovative, rapid-fire systems, we have created a scalable and secure pipeline that can produce high-quality video clips. The architecture's ability to handle both text-to-video-image generation, coupled with a user-friendly streamlined interface, makes it an invaluable tool for businesses across the sector, from demonstrations of e-commerce products to personalized marketing campaigns. As featured in the sample video, this technology has impressive results in opening new paths for creative expression and the production of efficient content at scale. This solution is a glimpse into the future of visual storytelling and digital communication, as well as technological advancements.
For more information about Cogvideox, see Cogvideox on Face. Try the solution yourself and share your feedback in the comments.
About the author
Nick Biso I am a machine learning engineer at AWS Professional Services. He uses data science and engineering to solve complex organizational and technical challenges. Additionally, he will build and deploy AI/ML models in the AWS cloud. His passion extends to his tendencies towards travel and his diverse cultural experiences.
Natasha Chill I am a cloud consultant at the Generating AI Innovation Center, specializing in machine learning. With a strong background in ML, she is currently focused on developing proof-of-concept solutions for generation AI within Geneaiic, innovation and applied research.
KateRine Feng I'm a cloud consultant for AWS Professional Services within the Data and ML team. She has extensive experience building full stack applications for AI/ML use cases and LLM-driven solutions.
Zinzhao Feng I am a machine learning engineer at AWS Professional Services. He focuses on architecture and implementation of large-scale generation AI and classic ML pipeline solutions. He specializes in FMOPS, LLMOPS, and distributed training.
