Dynamic Batching Explained: Improving LLM Inference Throughput and GPU Utilization

Dynamic batching is a server-side mechanism that merges multiple inference requests at runtime. This article explains how dynamic batching works, how it differs from Continuous Batching, and the deployment trade-offs in LLM inference throughput optimization.

What Is Dynamic Batching?

Dynamic Batching is an inference server-side mechanism: it merges multiple independent inference requests into a single batch for execution at runtime. Its optimization focus is request scheduling, batch formation, and resource usage, rather than directly changing model capabilities.

In online inference services, requests usually do not arrive neatly at the same time: some requests enter the queue first, while others arrive later; some finish generation quickly, while others continue running. What dynamic batching aims to solve is how to organize these requests into batches that are more suitable for execution within acceptable waiting times and service goals.

In some LLM inference contexts, Continuous Batching is also referred to as dynamic batching or iteration-level batching. Both emphasize runtime scheduling, but Continuous Batching more specifically highlights continuously adding new requests during the generation iteration process.

Key point: The core of dynamic batching is not simply fixing batch size at a larger value, but allowing the server side to dynamically organize inference batches based on request arrival patterns, resource status, and SLO (service-level objective).

Why Does LLM Inference Need Dynamic Batching?

Online LLM inference usually faces two goals at the same time: responding to individual user requests as quickly as possible, while also keeping compute resources such as GPUs highly utilized. Processing requests one by one is simple to schedule, but when concurrent requests continue to arrive, it is often difficult to fully utilize hardware resources.

Single-request Processing Can Easily Cause Resource Idling

If every request is executed individually, the server may frequently handle small-scale compute tasks. For high-concurrency question answering, chatbots, or text generation APIs, this approach can easily leave resources waiting between requests, limiting overall throughput.

The basic idea of batching is to merge multiple requests for execution so that one inference computation carries more effective work. Dynamic batching further decides “when to merge, how many to merge, and how to continue scheduling” at runtime, making it better suited to online services where request arrivals are uneven.

Inference Throughput Does Not Depend Only on the Model Itself

LLM inference is often affected by factors such as memory IO, and throughput bottlenecks do not necessarily come only from model parameter scale or compute power itself. Even if model weights remain unchanged, server-side scheduling methods, memory usage patterns, and batch organization strategies can also affect overall throughput performance.

Therefore, dynamic batching is often used for LLM inference throughput optimization: by merging requests at runtime and adjusting batches or block sizes, it balances resource constraints and SLO targets.

Common Goals Include Throughput, Resource Utilization, and Latency Metrics

Dynamic batching usually focuses on the following goals:

  • Improve effective throughput, completing more inference requests per unit time;
  • Improve GPU utilization and reduce resource idling caused by scattered requests;
  • Under specific implementations and workload conditions, improve latency metrics such as TBT that are related to the generation process;
  • Optimize inference service deployment efficiency, usually without modifying model weights.

Here, TBT can be understood as a generation-process latency metric used in documentation for specific inference frameworks. Its exact definition and calculation method should follow the corresponding framework documentation. It should be noted that dynamic batching does not inherently guarantee that all latency metrics will decrease. It usually requires a trade-off between “waiting a little longer to form a more suitable batch” and “executing as soon as possible to reduce waiting for a single request.”

Basic Workflow of Dynamic Batching

Dynamic batching can be understood as the server continuously performing scheduling around the request queue. A typical process is as follows.

1. Requests Enter the Server-side Queue

User requests first enter the inference service. Each request may contain inputs of different lengths and may correspond to different generation length requirements. The server records request arrival time, resource status, and the current scheduling window.

2. The Server Aggregates Requests That Can Be Merged

The scheduler observes requests in the queue at runtime and tries to merge multiple independent requests into the current batch. The key here is not to hard-code all batches in advance, but to dynamically form batches after requests actually arrive.

3. Form the Current Batch and Execute Inference

When the scheduler considers the conditions satisfied, it submits the selected requests as a batch for execution. These conditions may be related to current resources, waiting time, maximum batch size, business SLO targets, and other factors.

According to the vllm-ascend documentation, dynamic batching dynamically adjusts block size in each inference iteration of the chunked prefill strategy based on resources and SLO targets, to improve effective throughput and reduce TBT. Whether other frameworks adopt the same mechanism needs to be confirmed with their specific documentation and versions.

4. Continue Scheduling Based on Completion Status

During batch execution, different requests may complete at different times. For LLM generation tasks, some sequences may finish earlier, while others still need to continue generating.

A typical idea in continuous batching is: when a sequence in the batch completes, the scheduler can use the freed execution slot to insert a new sequence, allowing the inference process to continue carrying effective requests instead of waiting for the entire batch to finish before starting the next batch.

5. Continuously Balance Throughput and Latency

Dynamic batching is not a one-time action, but a continuous scheduling process. The server continuously receives new requests, forms batches, executes inference, releases resources, adds requests, and balances throughput, waiting time, and resource usage.

What Are the Differences Between Dynamic Batching, Static Batching, and Continuous Batching?

Static batching, dynamic batching, and continuous batching are all related to “batches,” but their scheduling granularity and applicable scenarios are different.

Comparison DimensionStatic BatchingDynamic BatchingContinuous Batching
When batches are formedMore inclined to be fixed in advance or formed according to fixed rulesMerges requests at runtime based on request arrivals and service statusContinuously schedules and adds requests during generation iterations
Request arrival patternMore suitable for relatively orderly requests or offline processingSuitable for uneven arrivals of online requestsSuitable for cases where different sequences complete at different times during LLM generation
Resource utilizationFixed batches may lead to waiting or empty slotsCan improve resource usage through runtime mergingCan continue adding requests after sequences complete, improving sustained utilization
Latency trade-offSimple rules, but weaker flexibilityNeeds to balance request waiting time and batching benefitsPlaces more emphasis on iteration-level scheduling, with higher scheduling complexity
Applicable scenariosOffline inference, fixed-scale tasksOnline inference services, text generation APIs, high-concurrency requestsLLM chat, question answering, and generation tasks with mixed long and short outputs

Static Batching Is More Oriented Toward Fixed Batches

Static batching usually executes according to a relatively fixed batch size or scheduling rule. It is simple to implement and suitable for scenarios where task scale is relatively stable, waiting time requirements are not high, or offline processing is used.

However, in online inference, request arrivals are often uneven. If the system waits too long to fill a fixed batch, it may affect individual request responses; if it does not wait, the batch may be too small, reducing resource utilization.

Dynamic Batching Emphasizes Runtime Merging and Adjustment

The focus of dynamic batching is runtime decision-making. Based on the current queue, resource status, and service goals, it merges multiple independent requests into batches for execution. Compared with fixed batches, it is better suited to inference services where requests continue to arrive and load fluctuates significantly.

Continuous Batching Emphasizes Iteration-level Scheduling

Continuous Batching more commonly appears in LLM generation tasks. Because the generation process usually advances by token or iteration, different requests may end at different times. Continuous batching uses iteration-level scheduling to allow new requests to enter the running batching process at appropriate times, thereby improving deployment throughput.

What Benefits Can Dynamic Batching Bring?

The benefits of dynamic batching mainly come from server-side scheduling and resource utilization optimization, rather than changes in the model’s capabilities themselves.

Improve Effective Throughput

Dynamic batching merges multiple individual requests into batches for execution, enabling one inference process to carry more requests. For high-concurrency LLM inference services, this helps increase the number of requests completed per unit time.

Continuous Batching is also often positioned as a method for improving LLM deployment throughput, especially in scenarios where generation lengths are inconsistent and requests continue to enter the system.

Improve GPU Utilization

When requests arrive in a scattered manner, executing them one by one can easily lead to insufficient GPU utilization. Dynamic batching helps reduce resource idling by merging requests at runtime and, in continuous batching scenarios, using freed sequence slots to add new requests.

What needs to be understood carefully is that improved GPU utilization does not mean every request returns faster. If the waiting time to form batches is too long, the perceived latency of a single request may increase. Therefore, scheduling strategies should be set according to business goals during deployment.

Usually Does Not Require Modifying Model Weights

Based on materials related to Continuous Batching, this type of optimization is closer to optimization at the level of inference service scheduling and memory usage, and does not require modifying model weights; it should generally not be understood as a transformation of the model’s capabilities themselves.

For businesses that have already completed model training or fine-tuning, dynamic batching can be one direction for deployment optimization, improving service efficiency without retraining the model.

May Improve Generation Latency Metrics in Specific Implementations

According to the vllm-ascend documentation, dynamic batching dynamically adjusts block size based on resources and SLO targets, to improve effective throughput and reduce TBT. This should be understood as: under specific implementations and workload conditions, scheduling strategies can improve time metrics related to the generation process.

However, different frameworks, models, hardware, and request distributions will produce different effects, and one should not directly assume that all latencies will decrease without actual stress testing.

What Boundaries Should You Pay Attention to When Using Dynamic Batching?

Dynamic batching is suitable for optimizing inference services, but it is not a universal solution. Before introducing it, it should be clear that it solves scheduling and resource utilization problems, not model semantic capability, inference correctness, or weight-related problems.

It Is Not a Transformation of Model Capability

Dynamic batching will not make the model “smarter,” nor will it change model weights. It mainly changes the way multiple requests are organized and executed on the server side.

If a business problem comes from insufficient model performance, unreasonable context design, or poor prompt quality, dynamic batching usually cannot directly solve these problems.

SLO Targets Affect Scheduling Strategies

Dynamic batching needs to make trade-offs around SLO targets. For example:

  • If the waiting time is too short, the batch may be too small, limiting throughput improvement;
  • If the waiting time is too long, the batch is more likely to become larger, but user-perceived latency may increase;
  • If the batch or block size is too large, resource usage may increase;
  • When resources are constrained, scheduling strategies need to avoid local throughput optimization affecting overall stability.

Key point: The key to dynamic batching is not pursuing the largest batch, but achieving better overall throughput and resource utilization within latency that the business can accept.

Fixed Configurations Should Not Be Applied Directly

The parameters, default values, version behaviors, and hardware adaptations of different frameworks may differ, so fixed configurations should not be directly applied in production environments. A more prudent approach is to conduct stress testing and evaluation around business workloads, latency targets, and resource usage:

  • Whether requests arrive continuously;
  • Whether input and output length distributions vary greatly;
  • Whether the business values throughput or response latency more;
  • Whether GPU utilization remains low over a long period;
  • Whether there are clear SLO targets;
  • Whether changes in batch size, waiting time, and resource usage can be observed through stress testing.

Which Application Scenarios Are Suitable for Dynamic Batching?

Dynamic batching is more suitable for online inference services, especially scenarios where requests continue to arrive, the execution cost of a single request is high, and there is a desire to improve overall throughput within acceptable latency.

User NeedSolutionPossible Effect
A high-concurrency question-answering service needs to handle more requestsMerge multiple question-answering requests at runtime on the server sideImprove effective throughput and reduce resource idling
Chatbot requests continue to enter the systemUse dynamic batching or continuous batching for iteration-level schedulingMake fuller use of GPU resources in generation tasks
Text generation API outputs vary greatly in lengthContinue scheduling new requests after some sequences completeReduce inefficiency caused by fixed batches waiting for all sequences to finish
A large model online inference service focuses on SLOAdjust batches based on resource status and service goals; specific implementations may also adjust block sizeBalance throughput, latency, and resource usage
An already deployed model needs improved inference efficiencyOptimize at the level of server-side scheduling and memory usageUsually improve deployment efficiency without modifying model weights

You can use the following checklist to determine whether dynamic batching is worth evaluating:

  • Whether there are stable or bursty concurrent inference requests;
  • Whether there is a desire to improve LLM inference throughput;
  • Whether insufficient GPU utilization or resource idling has been observed;
  • Whether server-side scheduling is needed under clear SLO targets;
  • Whether there is a desire to optimize deployment efficiency without modifying model weights;
  • Whether waiting time, batch size, and throughput changes can be validated through stress testing.

If the answer to most of the above questions is “yes,” dynamic batching is usually worth including in an LLM inference service optimization plan; if request volume is very low, latency is extremely sensitive, and there is almost no concurrency, the benefits of dynamic batching may be limited, and the waiting strategy may even need to be controlled carefully.