Effectively measuring the performance of applications that leverage large-scale language models (LLMs) is critical to the adoption of AI technology in organizations. Legare Kerrison and Cedric Clyburn from the Red Hat team recently spoke at the Arc of AI 2026 Conference about practical ways to evaluate and optimize LLM inference. They discussed the resource requirements and cost implications of various workloads for AI applications such as search augmentation generation (RAG) and agent AI. Kerrison and Clyburn also talked about the importance of metrics such as requests per second (RPS), time to first token (TTFT), and ITL (intertoken latency) when evaluating applications.
The speakers began their presentation by highlighting 2023 as the year of LLM with Hugging Face and other models, 2024 as the year of RAG, 2025 as the year of model fine-tuning and AI agents, and predicted that 2026 will be the year of LLM evaluation. When it comes to AI adoption and LLM model evaluation and performance challenges, leaderboards can be helpful, but they tend to be generic. Some websites measure their models using criteria such as hard prompts, coding, math, and creative writing. These benchmarks do not reflect your specific business issues or data and should be used with limitations in mind. Software development teams need to understand the overall AI technology landscape and choose the best model and provider for their specific use case.
Speakers highlighted common challenges experienced in real-world projects implementing LLM. There, delivering a production-ready model required navigating a “triangle of trade-offs” between model quality (accuracy), responsiveness (latency), and overall cost. Optimizing any two of these factors affects the third factor. For example, focusing on high accuracy and low latency leads to increased implementation costs. Applications built for low cost and high accuracy typically experience high latency. Also, focusing too much on low cost and low latency will result in less accurate models. Clear measurements and evaluations can help you make informed decisions when choosing the right model, performance goals, and hardware infrastructure for your workloads.
To provide customers with the right solution, teams must move from simple model selection to actual application requirements and system priorities. Service level objectives (SLOs) with clearly defined key performance and quality metrics ensure that applications remain fast, useful, and reliable to end users, and enable structured comparisons across models and hardware to enable cost optimization. The requests per second (RPS) metric represents the number of inference requests that the system can process per second. You can use this to measure overall throughput and how well your serving stack scales under load. Time to first token (TTFT) is the time between sending a request and receiving the first generated token. Indicates the user-perceived delay. Also, Inter-Token Latency (ITL) is the time between each subsequent token after the first one. This highlights how fast the streaming output feels to the user and indicates the efficiency of the decoder.
They showed some examples of different SLOs for different workloads for different use cases and benchmark metrics. E-commerce chatbot solutions require quick, conversational responses. TTFT metrics for this use case are typically <200ms and ITL <50ms for 99% of requests (P99). RAG-based applications, on the other hand, require more accuracy and completeness than just speed and performance. RAG use cases tend to use more input tokens and fewer output tokens. The metrics for TTFT, ITL, and request latency are ≤ 300 ms, ≤ 100 ms (for streaming), and ≤ 3000 ms, respectively, for 99% of requests.
After determining application priorities, the team should focus on hardware requirements. The LLM inference phase has two stages called prefill, which is compute-bound, and a decode phase, which is memory-bound. Techniques such as structured generation, speculative decoding, prefix caching, and session caching help provide efficient LLM models. It is easier to load a prefill phase that uses the first token than a decode phase that relies on subsequent tokens. The speaker mentioned that if it makes sense to run LLM locally, it can be more efficient in certain use cases since it has the advantage of not accessing the cloud.
They defined the term model evaluation as the process of evaluating a model’s overall performance and suitability for its intended purpose across a variety of criteria. That is, how a particular model performs under a workload on particular hardware. Model benchmarking was defined as a standardized comparison of model performance against predefined datasets, tasks, and other models.
They talked about how their teams typically measure LLM for different types of workflow patterns, such as standard request flows where a token is generated for each new request. End-to-end request latency is a key metric for this pattern. On the other hand, in streaming request flows, LLM requests are not uniform and metrics such as TTFT and ITL must be formally tracked.
LLM performance metrics are influenced by factors such as model architecture and size, quantization (compressing the model by reducing weight precision), serving engine (Ollama, vLLM, TGI, Triton, etc.), hardware (GPU memory), batching and concurrency choices, etc.
Measuring LLM deployment is difficult because model inference performance evaluation is time-consuming and fragmented. Kerrison and Clyburn provided several examples of LLM workloads that teams need to plan for and ask questions for evaluation. For example, “Should I use Llama 3.1 8B or Llama 3.1 70B to create a customer service chatbot on NVIDIA H200?” or “How many servers do I need to keep the service running under maximum load?”
Benchmarking using open source toolkits such as GuideLLM. Achieve SLO-aware benchmarks for LLM deployments. Part of the vLLM project, GuideLLM works by simulating real-world traffic and measuring metrics such as throughput and latency. Its process flow includes steps such as model selection and customization, dataset selection with real or synthetic data, workload configuration, and benchmark test execution. If your model meets the required SLO objectives, you can deploy it to production on the vLLM engine.
Clyburn presented results from GuideLLM tests using datasets such as Hugging Face (ShareGPT), file-based, and in-memory datasets to simulate workloads such as synchronous (running a single request stream at a time) and concurrent (running a fixed number of synchronous streams in parallel). He shared benchmark statistics for P99 (99th percentile) and P90 (90th percentile) latency metrics for various workloads such as chat, RAG, summarization, and code generation.
In addition to LLM inference, evaluation of model accuracy should also be considered. LLM accuracy assessment use cases should include categories such as model accuracy, pipeline accuracy (for RAG and AI agents), and application accuracy. Open source assessment tools include:
The speaker concluded his talk by emphasizing the need for application teams to consider LLM optimization techniques such as quantization (model compression is more effective than niche optimization techniques). In one example, quantization using GPTQModifier reduced model size by 45%. Another technique is KV caching. This saves redundant calculations and speeds up decoding (but consumes more memory). They recommended the Hugging Face website with language models validated by Red Hat AI for additional learning on AI topics, and the deeplearning.ai website for training courses on AI in general.
