Update: Updated images for comparison between ChatGPT 3.5 and vicuna-13B for better readability.
With the introduction of Large Language Models (LLM) for Generative Artificial Intelligence (GenAI), the world has become both enthralled and concerned about the potential of AI. The ability to carry on conversations, pass tests, develop research papers, and write software code are incredible feats of AI, but these are just the beginnings of what his GenAI can achieve in the years to come. It’s not too much. All this innovative functionality comes at a high cost in terms of processing performance and power consumption. So the possibilities for AI may be endless, but ultimately physics and cost may be the boundaries.
Tirias Research predicts that the combined cost of generative AI data center server infrastructure and operating costs will exceed $76 billion by 2028 on the current course, and that growth will drive the growth of search, content, and other technologies that incorporate GenAI. Creating business models for emerging services such as business automation and profitability becomes difficult. For reference, this cost is more than double the estimated annual operating costs of Amazon’s cloud service AWS, which currently accounts for one-third of the cloud infrastructure services market, according to estimates by Tirias Research. This forecast incorporates an aggressive 4x improvement in hardware computing performance, but despite the rapid pace of innovation in inference algorithms and their efficiency, this improvement is is outweighed by a 50-fold increase in Neural networks (NNs) designed to run at scale are even more highly optimized and continue to improve over time, increasing the capacity of each server. However, this improvement is offset by increased usage, more demanding use cases, and more sophisticated models with an order of magnitude more parameters. GenAI’s cost and scale require innovation in NN optimization, potentially pushing computational load out of the data center and onto client devices such as PCs and smartphones.
For background, today the majority of NN inference runs on servers accelerated by graphics or tensor processing units (GPUs or TPUs) designed to perform parallel computations of matrix computations. Each accelerator applies thousands of coefficient “parameters” (analogous to synapses) to each “node” (analogous to neurons). The network is arranged in layers, with each layer consisting of thousands of nodes and each node having thousands of connections to the nodes in the previous and next layers. In LLM, these nodes are ultimately mapped to tokens, that is, text language objects and symbols. It then uses the history of previously generated tokens (such as prompts and subsequent generated responses) to assign probabilities and selects one of the next most likely tokens.
The next wave of LLMs, such as GPT-4, are being trained on large data sets with the goal of creating neural networks estimated to have over a trillion parameters. Today, running a trained LLM often requires running a single model on multiple accelerators and multiple servers, increasing costs quickly. Memory capacity and powerful cloud-based GPU or TPU accelerators with massive memory designed to efficiently run algorithms, even on small models spanning tens to hundreds of billions of parameters Performance requirements can easily be exceeded.
To predict the operating costs of GenAI, Tirias Research applies a projected Total Cost of Operations (FTCO) model of complex data center workloads on various hardware configurations. The FTCO model incorporates technology advancements, changes in end-user demand, and changes in workloads such as media streaming, cloud gaming, and machine learning (ML). For GenAI, this means taking processing progress into account. This will continue to be driven by GPU accelerator technology for the foreseeable future. The dataset grows exponentially, which in turn increases the number of parameters in the trained NN model. Improved model optimization. And the insatiable demand for GenAI.
Respond to user requests first. GenAI is now being used to generate text, software code, and images, along with new applications such as video, sound, and 3D animation. In the future, these basic functions will power increasingly sophisticated GenAI applications such as video entertainment generation, metaverse creation, education, and even process generation for urban, industrial, and business applications. increase. Now, OpenAI’s ChatGPT monthly visitor numbers are fast approaching 2 billion, and his popular GenAI art community, Midjourney, has over 15 million users.
To forecast demand, Tirias Research analyzed GenAI’s three basic capabilities (text, image, and video) and categorized emerging markets into ad-driven consumers, paid subscription users, and automated tasks. For Text GenAI, the demand for word- and symbol-like tokens is projected to exceed 10 trillion by the end of 2023, with over 400 million monthly active his users concentrated in developed markets. The forecast predicts that by the end of 2028, the number of users will surpass his 6 billion, or about 90% of the smartphone market penetration, with annual tokens exceeding 100 billion, or a 100-fold increase. For image GenAI, with the advent of video, the increase in the number of images is projected to exceed 400-fold, a significant increase to over 10 trillion. Videos should use more sophisticated image generation tools and sophisticated prompts to create sequences of images that are thematically and visually relevant. loop.
Now let’s deal with the computational workload. As an unprecedented amount of academic and business knowledge pours into the field of machine learning (ML) and GenAI, GenAI models are becoming more efficient. The quality of GenAI images and tokens varies by segment and by factors such as resolution and model size, with paid usage allocated to higher quality outputs and a corresponding reduction in data center computing resource utilization. is also higher. Projected workloads combine demanding large models with smaller, more efficient and computationally optimized NNs. “The emergence of more efficient neural networks, trained by more sophisticated NNs, will be one of several drivers leading generative AI to more viable economics and reduced environmental impact. said Simon Solotko, senior analyst at Tirias Research and developer of the FTCO model. It enables you to quickly train small networks using large parameter networks and run more cost-effectively on distributed platforms such as PCs, smartphones, vehicles, and mobile XR. HuggingFace recently used Facebook’s LLaMA LLM framework, trained using ChatGPT user logs, to develop two new trained ChatGPT-like LLMs, the 30 billion parameter vicuna-30B and the 13 billion parameter vicuna -13B demonstrated. This clever technique has resulted in ChatGPT-like LLMs that can run on a single consumer device, and whose responses are no different from trained large-scale models. A highly optimized model, or a simpler and more specialized model, reduces the model size in the cloud and removes the workload entirely from the cloud to distribute GenAI applications to smartphones and PCs This is expected to reduce costs for large data centers. .
Tirias Research projects that data center power consumption will reach nearly 4,250 megawatts in 2028, and the total amortized capital and operating costs of servers will exceed $76 billion in today’s dollars, a year-on-year increase of 212 from 2023. double. This cost does not include the cost of the data center build structure, but does include labor, power, cooling, ancillary hardware, and server amortized costs over a three-year period. The FTCO model is based on a server benchmark using 10 Nvidia GPU accelerators with a peak power of just over 3000 watts and operating power at an average utilization of 50% just over 60% of peak. “Using a high-density 10 GPU server provided by data center innovator Krambu, Tirias Research will benchmark multiple open source generative AI models to derive computational demands for future higher parameter models. We can,” Sorotko continued. The forecast includes insights into his GPU and TPU accelerator roadmaps over the next five years, and with these roadmaps, what each server achieves for each use case like text, image, and video. Calculate your possible workload. Perhaps the greatest insight of his FTCO model is that equilibrium exists. Server throughput per token or image remains relatively stable year-over-year, even as workloads become more complex and server performance increases by about 4x.
As the demand for GenAI continues to grow exponentially, the slowdown of Moore’s Law makes breakthroughs in processing and chip design seem like a long gamble. Free lunch does not exist. Consumers will demand better his GenAI output, but that will hinder efficiency and performance gains. As consumer usage increases, costs will inevitably increase. Sorotko concludes: “We are just beginning to understand the data center economics of machine learning. By modeling the entire cycle of demand, processing and cost, we can see what ultimately drives workloads and economics to shift in the right direction. Moving computing to the edge and distributing it to clients such as PCs, smartphones and XR devices is an important path to reducing capital and operating costs.”
Five years ago, at the annual Hot Chip Semiconductor Technology Conference, companies began sounding the alarm about data center power consumption, predicting that global computing demand could exceed the world’s total electricity production within a decade. rice field. This was before the rapid adoption of GenAI, which could see an even faster pace of increase in computing demands. The processing challenges typified by the introduction of GenAI cannot be overcome by technology enhancements alone. It requires changes in how processing is performed, significant improvements in model optimization without significant loss of accuracy, and new business models to cover the continued costs of processing in the cloud. will be These points will be covered in Part 2. GenAI disrupts data centers: Move GenAI to the edge.
follow me twitter Or LinkedIn. check out my website.
The authors and members of the Tirias Research staff do not own any stock in the companies mentioned. Tirias Research tracks and consults companies across the electronics ecosystem, from semiconductors to systems and sensors to the cloud. Tirias Research consults to Nvidia and other semiconductor and technology companies that develop and deploy AI solutions.
