The cloud has many well-known benefits, with the most notable on-demand scalability and high reliability. Both are ideal features for hosting AI workloads. However, according to a recent Business Application Research Center (BARC) report, only 33% of AI workloads are hosted in the public cloud. On-premises and hybrid environments divide the remainder almost evenly, with the slimmest edges (34%) on the premises.[1]
Certainly, the cloud could be the right choice for some AI workloads. If an enterprise needs to serve users in heterogeneous locations with low latency, the public cloud global infrastructure can provide that use case well. Also, many IT professionals prefer to use hyperscaler pre-built AI services and large language models to eliminate the complexity of model deployment, scaling and maintenance.
However, as many of them have discovered, there are also many good reasons to maintain AI workloads in your facility. First of all, AI workloads are well-known for their resource intensiveness. If your model takes longer than expected, or if you need to train multiple iterations, cloud-based graphics processing units can run more than $100 per hour, but you can quickly get massive overruns. Similarly, if you need to transfer large datasets from the cloud, the output fee can further increase the cost, and the time required to move data can be extended for the project's timeline. Also, given the demand for intense computational resources in AI models, low network latency is important to achieving real-time inference, and shared cloud resources may not provide the required level of consistent performance.
Finally, many AI applications process sensitive information such as trade secrets and personally identifiable information that falls under strict regulations governing data usage, security, and location. Because you don't control the underlying infrastructure, public clouds can have a difficult level of compliance and security that you need.
“The dynamics of the market are driving buyers' interest in on-premises solutions,” said Sumeet Arora, Chief Product Officer of Teradata.
Of course, building an AI-Ready infrastructure on the premises is not an easy task. While on-premises solutions have full control over compliance and security, these tasks remain challenging, especially when custom integrations with multiple tools. Additionally, on-premises solutions must maintain complex infrastructure with the power, speed and flexibility to support the high demands of AI workloads.
Fortunately, the market has matured to the point where a tightly integrated, ready to run AI stack became available. This eliminates complexity while enabling compliance, security and high performance. A good example of such a pre-integrated stack is Teradata's AI factory, which extends Teradata's AI capabilities from the cloud to make them available on-site.
“Teradata is a clear leader in this environment and has a proven foundation that makes AI meaningful and reliable. Its first-class speeds, predictable costs, and integration with golden data records,” continues Arora. “Teradata AI Factory builds on these strengths in a single solution for organizations using the ONPREM infrastructure to gain control, meet sovereignty needs and accelerate AI ROI.”
This solution provides seamless hardware and software integration and removes the need for custom setup and integration. And since it's all pre-integrated, users don't need to get multiple approvals for different toolsets. As a result, organizations can expand their AI initiatives faster and reduce operational complexity.
Many practitioners build native searched generation (RAG) use cases and pipelines to prefer on-premises solutions. Teradata AI microservices with NVIDIA provide native RAG functionality for intake and search, integration, embedding, re-ranking, and guardrails. Users can query all their data in natural language, providing faster, more intelligent insights at scale. This comprehensive solution enables scalable and secure AI execution within the company's own data center.
While Cloud offers scalability, global access, and infrastructure-on-demand for AI workloads, many organizations may prefer on-premises solutions to improve cost management, security compliance, and performance consistency. The integrated AI stack makes on-premises deployments much easier tasks and accelerate time to value.
Learn more about how Teradata's AI Factory can help your organization with on-premises deployments.
[1] Petrie, K, Cloud, Prem, Hybrid, Oh My! Where AI employers host projects and why, Data Rail, April 3, 2025.
