LLM Capacity Planning: Estimating QPS, Concurrency, VRAM, and GPU Count

LLM capacity planning is a method for evaluating QPS, concurrency, VRAM, and GPU count before launch. By combining parameter count, precision, sequence length, and KV cache VRAM estimation, it helps choose GPU specifications for inference deployment.

What Is LLM Capacity Planning?

LLM capacity planning is the process of evaluating business request volume, concurrency level, context length, model specifications, VRAM usage, and GPU count together before a model goes online or is scaled out. It is not about separately calculating how many GPUs are needed, but about first clarifying how requests enter the system, and then determining whether the inference service can run stably under constraints on VRAM, throughput, and response time.

For LLM inference deployment, the same QPS may correspond to completely different resource requirements. Short Q&A, long-document analysis, and multi-turn conversations have different token scales, and the VRAM pressure brought by KV cache also differs. Therefore, the key to capacity planning is to first unify the evaluation criteria, and then proceed to VRAM estimation and GPU specification selection.

Capacity planning can be understood as a resource budget before launch: the business side provides traffic and experience targets, while the engineering side converts these targets into a combination of model, VRAM, context, concurrency, and hardware specifications.

LLM Capacity Planning Must First Unify Evaluation Criteria

Before estimating the number of GPUs, you need to confirm the meanings of several commonly used metrics. Otherwise, different teams may all be discussing “capacity,” but they are actually focusing on different issues.

CriterionMeaningRole in capacity planning
QPSNumber of requests processed per unit timeDescribes request arrival intensity, but cannot alone represent compute pressure
ConcurrencyNumber of requests being processed at the same timeDirectly affects the context and KV cache occupied simultaneously
ThroughputNumber of completed requests or generated tokens per unit timeCloser to the system’s actual output capability
LatencyWaiting time from when a user initiates a request to when they receive a responseDetermines whether concurrency backlog is acceptable
Context lengthAssumptions about the scale of input and output tokensAffects VRAM usage, especially KV cache
Model specificationsParameter count, precision, deployment method, etc.Determines weight VRAM and baseline resource boundaries

Capacity planning cannot look only at the number of requests. For LLM inference, a single request may contain only dozens of tokens, or it may contain a long document, conversation history, and a long output. The VRAM required to deploy and fine-tune an LLM is affected by factors such as model parameter count, precision, and sequence length, so the request pattern needs to be described clearly during evaluation.

Key point: If only QPS is provided, without input length, output length, latency target, and concurrency assumptions, subsequent GPU count estimation can easily become distorted.

VRAM Requirements Are Jointly Determined by Parameter Count, Precision, and Sequence Length

VRAM estimation is the foundation of LLM capacity planning. When deploying and fine-tuning an LLM, VRAM requirements are usually affected by factors such as model parameter count, precision, and sequence length. The larger the parameter count, the higher the VRAM typically required by model weights; the precision setting affects the memory occupied by each parameter; and sequence length affects intermediate states and cache overhead during inference.

In the early planning stage, you can first use a VRAM estimation tool to quickly obtain resource boundaries, and then proceed to GPU specification filtering. The value of the tool is not to replace stress testing, but to help the team quickly rule out obviously unsuitable specifications and form initial capacity assumptions.

Checklist Before VRAM Estimation

Check itemQuestion to clarify
Model sizeWhich model is used, and roughly what is its parameter scale
Weight precisionWhat precision or quantization method is used for deployment
Maximum contextWhat is the maximum allowed input and output token scale per request
Concurrency assumptionHow many requests run simultaneously at peak time
Request patternIs it short Q&A, long-document analysis, multi-turn conversation, or a mixed scenario
Fine-tuning requirementIs it inference only, or are training or fine-tuning resources also needed
Runtime reserveWhether to reserve margin for frameworks, scheduling, temporary tensors, and monitoring

For production environments, it is not recommended to make VRAM estimation a single-point number. A more reliable approach is to provide multiple tiers, such as regular context, longer context, and peak concurrency, and then evaluate VRAM levels separately.

Focus on KV Cache VRAM During the Inference Stage

During LLM inference, one of the important sources of VRAM consumption comes from the KV cache. The KV cache is used to store attention-related caches for tokens that have already been processed, so that the model does not need to recompute the entire historical context when generating subsequent tokens.

The size of the KV cache is proportional to the number of tokens. This means that the longer the context and the more requests running simultaneously, the more obvious the VRAM pressure usually is. When doing capacity planning, you cannot estimate only model weight VRAM; you also need to include the KV cache caused by each request’s context length and the number of concurrent requests.

KV Cache Pressure from Different Request Patterns

ScenarioUser needSolutionImpact on VRAM
Short Q&AQuickly answer short questionsControl input and output length and pursue lower latencyKV cache pressure per request is small, but still accumulates under high concurrency
Multi-turn conversationRetain historical contextCarry part of the historical messages or summariesThe more conversation turns, the longer the context and the higher the KV cache pressure
Long-document analysisRead and summarize long textMore input tokens, and output may also be longerVRAM usage per request increases significantly, and concurrency headroom becomes smaller
Batch generationProcess many similar requests at the same timeImprove throughput through batchingNeed to balance throughput gains against VRAM usage

In capacity planning, maximum context length and typical context length should be recorded separately. Estimating based on maximum context is more conservative and results in higher resource requirements; estimating based on average length is closer to regular cost, but requires additional consideration of peak values and abnormally long requests.

Derive QPS, Concurrency, and Context Assumptions from Business Requests

Capacity planning should usually start from business requests, rather than directly from GPUs. The business side needs to provide peak requests, typical request length, acceptable response time, and burst traffic characteristics, and the engineering side then converts this information into inference resource requirements.

Inputs to Collect First

  • Peak QPS or peak request volume.
  • Average input token length and high-percentile input length.
  • Average output token length and high-percentile output length.
  • Acceptable first-token latency or full response time.
  • Whether there is obvious burst traffic, such as campaigns, batch processing tasks, or scheduled calls.
  • Whether requests require long context, multi-turn history, or external retrieved content.

Concurrency estimation should be based on realistic request length assumptions. If the maximum context length supported by the model is used to estimate request length, the result is usually more conservative, but it will also significantly increase VRAM and GPU resource requirements. If only average length is used for estimation, stress testing is needed to verify whether long-tail requests will cause queuing or insufficient VRAM.

Key point: QPS, concurrency, and latency are not independent of one another. The longer the request duration, the higher the concurrency that may be formed under the same QPS; the longer the context, the higher the VRAM occupied by a single concurrent request.

GPU Count Estimation Should Start from VRAM Constraints and Model Usage

GPU count estimation needs to first determine the relationship between single-GPU VRAM, GPU count, and model parameter VRAM usage. A common misconception is to look only at model parameter count while ignoring KV cache, runtime reserved space, and the additional VRAM consumption brought by concurrent requests.

A more reliable breakdown is to divide VRAM into three categories: model weight VRAM, KV cache VRAM, and runtime reserved space. Only when all these parts can fit under the target concurrency and context length do the GPU specifications and quantity have preliminary feasibility.

Confirm the Model and Precision

First clarify the model to be deployed, parameter scale, and weight precision. Parameter count and precision affect model weight VRAM, and are the basis for determining whether a single GPU can host the model.

Estimate Model Weight VRAM

Estimate weight VRAM based on model parameter count and precision. At this stage, you can use a VRAM estimation tool to obtain an initial range for filtering possible GPU specifications.

Estimate the KV Cache Brought by Context and Concurrency

Evaluate the possible VRAM usage brought by KV cache by combining input tokens, output tokens, maximum context, and target concurrency. Because KV cache size is proportional to the number of tokens, long context and high concurrency will simultaneously push up VRAM pressure.

Reserve Runtime Space

In addition to weights and KV cache, space also needs to be reserved for inference frameworks, scheduling, temporary tensors, communication, and monitoring. In production environments, VRAM should not be planned close to full utilization.

Select Single-GPU Specifications and Determine Whether Multiple GPUs Are Needed

If a single GPU’s VRAM cannot simultaneously accommodate model weights, KV cache, and reserved space, then larger-VRAM specifications, multi-GPU deployment, parallel strategies, or reduced context and concurrency assumptions need to be evaluated. If VRAM is sufficient, throughput and latency still need to be verified through stress testing.

Judgment itemMain issuePossible action
Weights do not fitA single GPU cannot host the model’s basic weightsChoose a larger-VRAM GPU, multi-GPU deployment, or a lower-precision solution
KV cache pressure is highLong context or concurrency causes insufficient VRAMLimit context, adjust concurrency, evaluate cache offloading
Throughput is insufficientVRAM is sufficient, but generation speed cannot meet business peak demandAdd replicas, optimize batching, or scale out GPUs
Latency is too highQueuing or generation time exceeds the targetReduce single-instance load, split traffic, or adjust the service strategy

When VRAM Is Insufficient, Cache Offloading and Memory Sharing Can Be Evaluated

When VRAM becomes a bottleneck, adding GPUs or choosing specifications with larger VRAM is not the only direction. CPU-GPU memory sharing can help access and process large models or datasets that exceed traditional GPU VRAM capacity limits, and KV cache offloading is also a typical application task in this type of approach.

For long-context and highly concurrent inference scenarios, the KV cache often occupies more VRAM as the number of tokens grows. Offloading part of the cache to the CPU side, or alleviating GPU VRAM pressure through CPU-GPU memory sharing, may allow the system to host larger models or longer contexts with limited GPU VRAM.

However, such solutions cannot be evaluated only by whether they can run. Cache offloading and memory sharing are affected by the hardware environment, data transfer, inference framework, latency targets, and throughput targets. The specific models, graphics cards, and concurrency results in public cases usually cannot be directly generalized into capacity conclusions for all businesses.

Key point: VRAM optimization solutions should be verified through stress testing. If the business has very high requirements for low latency, the additional overhead brought by cache offloading needs to be evaluated carefully.

Capacity Planning Results Need Continuous Calibration Through Stress Testing and Monitoring

Capacity planning provides an initial boundary before launch, not a one-time conclusion. After the real business goes online, request length, peak traffic, output token count, and user behavior may all change, so capacity levels need to be continuously calibrated through stress testing and monitoring.

Combinations That Stress Testing Should Cover

Stress test dimensionRecommended coverage
Context lengthShort context, typical context, long context close to the upper limit
Concurrency levelRegular concurrency, peak concurrency, burst concurrency
Request typeShort Q&A, multi-turn conversation, long-document analysis, batch generation
Output lengthShort output, typical output, longer output
Resource levelGPU VRAM usage, token generation speed, queuing status, response time

After stress testing, you need to observe where the bottleneck comes from. If VRAM reaches the limit first, adjust the context limit, concurrency strategy, model specifications, or cache solution; if VRAM is sufficient but throughput is insufficient, focus on generation speed, batching strategy, and number of replicas; if response time is unstable, queuing, scheduling, and long-tail requests need to be analyzed.

Post-Launch Capacity Review Checklist

  • Whether peak QPS exceeds the original assumptions.
  • Whether average input and output tokens have changed.
  • Whether the proportion of long-context requests has increased.
  • Whether GPU VRAM remains at a high level for a long time.
  • Whether queuing, timeouts, or long-tail response times occur.
  • Whether the current GPU count can still cover business growth.
  • Whether context limits, batching strategies, or scaling plans need to be adjusted.

The core of LLM capacity planning is not to pursue a perfectly accurate one-time estimate, but to establish an estimation process that is explainable, stress-testable, and reviewable. Only by first clarifying business requests, then breaking down VRAM and KV cache, and finally calibrating GPU count through stress testing can the inference service achieve a more reliable balance between cost and stability.