Researchers have increasingly focused on energy consumption in machine learning, recognizing it as a critical computing resource. Jae-Won Chung, Ruofan Wu, and Jeff J. Ma from the University of Michigan and the ML.ENERGY Initiative, along with Mosharaf Chowdhury, present a large-scale study examining inference time and energy usage across 46 generative AI models, 7 tasks, and 1,858 configurations on H100 and B200 GPUs. This research is important because it extends beyond simply measuring energy to diagnostics. why Differences occur, revealing orders of magnitude variation. For example, for large language models, energy usage can vary by as much as 25 times depending on the task, and video generation can consume more than 100 times more energy than image generation. The team proposes a framework that connects time and energy consumption to underlying metrics such as memory and utilization, providing a path to optimizing performance and throughput per watt in power-constrained data centers.
This research addresses a critical gap in understanding why certain AI configurations consume more energy than others, moving beyond simple measurements to detailed diagnostics and optimization. The research team achieved empirical findings that reveal orders of magnitude variation in energy usage, with large-scale language model (LLM) task types having a 25x energy difference, and video generation can consume more than 100 times more energy than image generation. These observations highlight the urgent need for a deeper understanding of the fundamental mechanisms controlling energy consumption in modern AI systems.
This study reveals a framework for inferring time and energy consumption and assumes that these metrics are determined by underlying factors such as memory and GPU utilization that are influenced by the algorithm, software, and hardware layers. Researchers investigated knobs and conducted controlled comparisons at both the model and system level to pinpoint the specific factors that influence energy consumption. Although counterintuitive, this study proves that lower precision does not necessarily mean faster performance or more energy efficiency, and that increasing the number of GPUs and, in some cases, increasing memory capacity can reduce total energy consumption. These unexpected findings provide important clues to the complex interplay of factors governing energy use.
The framework goes beyond simple real-time and energy measurements to encompass throughput per watt, an important metric for power-constrained data centers. Experiments show that energy consumption is driven by underlying factors such as memory availability, hardware utilization, and application constraints that are not directly observable from end metrics alone. After carefully analyzing 1,858 configurations, the researchers observed that the type of LLM task had a significant impact on energy usage, with problem-solving tasks consuming 25 times more energy per response than text conversation tasks due to their longer output token sequences. Additionally, this study details how energy consumption changes with different model architectures, LLM, multimodal LLM, and diffusion models, and reveals that task type has a significant impact on output length and thus energy usage. Specifically, the team compared Qwen 3 32B across tasks on a single B200 GPU and found that problem solving produced 11 times more output tokens and 23 times more energy per response than text conversation. This in-depth analysis provides the foundation for optimizing AI workloads and designing more energy-efficient AI infrastructure, paving the way for sustainable growth in the field.
Generative AI inference on H100 and B200 GPUs
Scientists have begun a large-scale measurement study of inference time and energy consumption across 46 generative AI models, including 7 tasks and 1,858 different configurations on both H100 and B200 GPUs. The research team carefully measured GPU energy and recognized that data center power consumption is overwhelming at 50-70%, so they leveraged the Zeus system for accurate energy tracking and used modern LLMs such as Qwen 3, DeepSeek R1, and GPT OSS in conjunction with current popular models. A production-grade serving stack was implemented with vLLM 0.11.1 for LLM and MLLM and xDiT 0.4.5 for the diffusion model, all running on NVIDIA H100 and B200 GPU nodes with NVSwitch support. This work pioneered a method for LLM to identify the steady state at which the batch size saturates and then calculate the energy per token by dividing the total steady state energy consumption by the number of tokens generated during that period.
In a spread model processed in batches, the energy per request is determined by dividing the total energy consumed by the batch size. The researchers systematically examined the batch size and number of GPUs for each model while setting BF16 precision by default, and included native FP8 weighted LLM and MLLM as separate models for comparative analysis. This rigorous approach allowed us to observe orders of magnitude variation, revealing that energy consumption can vary by a factor of 25 between LLM task types. The team further investigated the impact of input modality on energy usage, specifically comparing text, image, and video inputs for multimodal LLMs.
In our experiments, we used Qwen 3 VL 8B on a single B200 GPU and demonstrated that video generation can consume over 100 times more energy than image processing. Importantly, this study establishes a framework for linking energy consumption to underlying metrics such as memory and GPU utilization and understanding the underlying mechanisms governing time and energy. By analyzing KV cache utilization and batch size, the scientists demonstrated that longer output sequences tax memory capacity, hinder batch size growth, and increase energy per token. This detailed methodology allowed the researchers to quantify how a problem-solving task that generated 10 times more output tokens than a text conversation ended up consuming 23 times more energy per response, highlighting the significant impact of task type on overall energy consumption. This work goes beyond simple measurements to provide a framework that can be applied to inference about service capacity in power-constrained AI data centers, providing insights into resource allocation optimization.
The inference energy of generative AI varies greatly
Scientists conducted a large-scale measurement study of inference time and energy consumption across 46 generative AI models, including 7 tasks and 1,858 different configurations on H100 and B200 GPUs. Experiments revealed significant variation in energy usage, with task type having a 25x impact on energy consumption. Specifically, problem-solving tasks consumed 25 times more energy per response than text conversations. The data show that a longer output sequence for a problem-solving task limits the batch size and increases the energy per token, resulting in 23 times higher energy per response for this model compared to other models. Researchers have measured that video generation can require more than 100 times more energy than image generation, highlighting the significant energy demands of multimodal tasks.
The researchers noted that text and image input consumed 1.1 to 5.2 times more energy per token than text-only, and text and video input consumed 1.3 to 15.0 times more energy. Analysis of the Qwen 3 model demonstrated that even with sufficient GPU capacity, vision preprocessing on the CPU side becomes a bottleneck, limiting batch size and increasing energy per token. Measurements confirm that video input requires more CPU-side processing and generates more vision tokens, resulting in smaller batch sizes and higher energy per token. Tests have proven that the diffusion model exhibits an energy consumption pattern that is determined not only by the model size but also by factors such as the number of denoising steps, output resolution, and number of frames.
The results show that generating a single video can consume between 26 kJ and 1.16 MJ, which is 1 to 2 orders of magnitude more energy consuming than image generation. Specifically, CogVideoX 1.5 5B consumed more energy than Wan 2.1 14B due to its higher output resolution, while HunyuanVideo reached 1.16 MJ by generating 129 frames at 720p. Scientists also investigated the relationship between batch size and performance metrics and observed that increasing batch size generally increases throughput, but can also impact energy efficiency. DeepSeek R1 and Qwen 3 Coder 30B measurements show trends in energy per token, tokens per second, median ITL, and power as batch size changes, normalized to a percentage of the maximum value. These findings establish a framework for understanding memory and utilization metrics that have the potential to control time and energy consumption across algorithmic, software, and hardware layers, all the way down to throughput per watt in power-constrained data centers.
Inference time and energy for generative AI
Scientists conducted a large-scale measurement study of inference time and energy consumption across 46 generative AI models, examining seven tasks and 1,858 configurations on H100 and B200 GPUs. Their empirical findings reveal large variations in energy usage, with language model task types potentially having a 25% difference, and video generation can consume more than 100 times more energy than image generation. Differences in GPU utilization can cause energy consumption to vary by as much as 3 to 5 times. The researchers present a framework for understanding the mechanisms governing time and energy consumption, and hypothesize that these metrics are determined by underlying factors such as memory and utilization.
These potential factors are influenced by factors across algorithmic, software, and hardware layers, all the way down to throughput per watt, a key metric for power-constrained data centers. The authors acknowledge that there are limitations, such as the fixed batch size used in the analysis, which can inflate the energy numbers, and the generalizability of measurements taken with internal systems. This research proves that understanding inference energy consumption, not just measuring it, is critical to optimizing AI infrastructure. By mapping factors and their relationship to energy consumption, this framework goes beyond black-box observations and allows you to track how model, system, and application factors impact energy use. Future research could build on this framework to further refine energy optimization strategies in machine learning systems.
