LLM Capacity Planning: Estimating QPS, Concurrency, VRAM, and GPU Count
LLM capacity planning is a method for evaluating QPS, concurrency, VRAM, and GPU count before launch. By combining parameter count, precision, sequence length, and KV cache VRAM estimation, it helps choose GPU specifications for inference deployment.
What Is LLM Capacity Planning?
LLM capacity planning is the process of evaluating business request volume, concurrency level, context length, model specifications, VRAM usage, and GPU count together before a model goes online or is scaled out. It is not about separately calculating how many GPUs are needed, but about first clarifying how requests enter the system, and then determining whether the inference service can run stably under constraints on VRAM, throughput, and response time.
For LLM inference deployment, the same QPS may correspond to completely different resource requirements. Short Q&A, long-document analysis, and multi-turn conversations have different token scales, and the VRAM pressure brought by KV cache also differs. Therefore, the key to capacity planning is to first unify the evaluation criteria, and then proceed to VRAM estimation and GPU specification selection.
Capacity planning can be understood as a resource budget before launch: the business side provides traffic and experience targets, while the engineering side converts these targets into a combination of model, VRAM, context, concurrency, and hardware specifications.
LLM Capacity Planning Must First Unify Evaluation Criteria
Before estimating the number of GPUs, you need to confirm the meanings of several commonly used metrics. Otherwise, different teams may all be discussing “capacity,” but they are actually focusing on different issues.
| Criterion | Meaning | Role in capacity planning |
|---|---|---|
| QPS | Number of requests processed per unit time | Describes request arrival intensity, but cannot alone represent compute pressure |
| Concurrency | Number of requests being processed at the same time | Directly affects the context and KV cache occupied simultaneously |
| Throughput | Number of completed requests or generated tokens per unit time | Closer to the system’s actual output capability |
| Latency | Waiting time from when a user initiates a request to when they receive a response | Determines whether concurrency backlog is acceptable |
| Context length | Assumptions about the scale of input and output tokens | Affects VRAM usage, especially KV cache |
| Model specifications | Parameter count, precision, deployment method, etc. | Determines weight VRAM and baseline resource boundaries |
Capacity planning cannot look only at the number of requests. For LLM inference, a single request may contain only dozens of tokens, or it may contain a long document, conversation history, and a long output. The VRAM required to deploy and fine-tune an LLM is affected by factors such as model parameter count, precision, and sequence length, so the request pattern needs to be described clearly during evaluation.
Key point: If only QPS is provided, without input length, output length, latency target, and concurrency assumptions, subsequent GPU count estimation can easily become distorted.
VRAM Requirements Are Jointly Determined by Parameter Count, Precision, and Sequence Length
VRAM estimation is the foundation of LLM capacity planning. When deploying and fine-tuning an LLM, VRAM requirements are usually affected by factors such as model parameter count, precision, and sequence length. The larger the parameter count, the higher the VRAM typically required by model weights; the precision setting affects the memory occupied by each parameter; and sequence length affects intermediate states and cache overhead during inference.
In the early planning stage, you can first use a VRAM estimation tool to quickly obtain resource boundaries, and then proceed to GPU specification filtering. The value of the tool is not to replace stress testing, but to help the team quickly rule out obviously unsuitable specifications and form initial capacity assumptions.
Checklist Before VRAM Estimation
| Check item | Question to clarify |
|---|---|
| Model size | Which model is used, and roughly what is its parameter scale |
| Weight precision | What precision or quantization method is used for deployment |
| Maximum context | What is the maximum allowed input and output token scale per request |
| Concurrency assumption | How many requests run simultaneously at peak time |
| Request pattern | Is it short Q&A, long-document analysis, multi-turn conversation, or a mixed scenario |
| Fine-tuning requirement | Is it inference only, or are training or fine-tuning resources also needed |
| Runtime reserve | Whether to reserve margin for frameworks, scheduling, temporary tensors, and monitoring |
For production environments, it is not recommended to make VRAM estimation a single-point number. A more reliable approach is to provide multiple tiers, such as regular context, longer context, and peak concurrency, and then evaluate VRAM levels separately.
Focus on KV Cache VRAM During the Inference Stage
During LLM inference, one of the important sources of VRAM consumption comes from the KV cache. The KV cache is used to store attention-related caches for tokens that have already been processed, so that the model does not need to recompute the entire historical context when generating subsequent tokens.
The size of the KV cache is proportional to the number of tokens. This means that the longer the context and the more requests running simultaneously, the more obvious the VRAM pressure usually is. When doing capacity planning, you cannot estimate only model weight VRAM; you also need to include the KV cache caused by each request’s context length and the number of concurrent requests.
KV Cache Pressure from Different Request Patterns
| Scenario | User need | Solution | Impact on VRAM |
|---|---|---|---|
| Short Q&A | Quickly answer short questions | Control input and output length and pursue lower latency | KV cache pressure per request is small, but still accumulates under high concurrency |
| Multi-turn conversation | Retain historical context | Carry part of the historical messages or summaries | The more conversation turns, the longer the context and the higher the KV cache pressure |
| Long-document analysis | Read and summarize long text | More input tokens, and output may also be longer | VRAM usage per request increases significantly, and concurrency headroom becomes smaller |
| Batch generation | Process many similar requests at the same time | Improve throughput through batching | Need to balance throughput gains against VRAM usage |
In capacity planning, maximum context length and typical context length should be recorded separately. Estimating based on maximum context is more conservative and results in higher resource requirements; estimating based on average length is closer to regular cost, but requires additional consideration of peak values and abnormally long requests.
Derive QPS, Concurrency, and Context Assumptions from Business Requests
Capacity planning should usually start from business requests, rather than directly from GPUs. The business side needs to provide peak requests, typical request length, acceptable response time, and burst traffic characteristics, and the engineering side then converts this information into inference resource requirements.
Inputs to Collect First
- Peak QPS or peak request volume.
- Average input token length and high-percentile input length.
- Average output token length and high-percentile output length.
- Acceptable first-token latency or full response time.
- Whether there is obvious burst traffic, such as campaigns, batch processing tasks, or scheduled calls.
- Whether requests require long context, multi-turn history, or external retrieved content.
Concurrency estimation should be based on realistic request length assumptions. If the maximum context length supported by the model is used to estimate request length, the result is usually more conservative, but it will also significantly increase VRAM and GPU resource requirements. If only average length is used for estimation, stress testing is needed to verify whether long-tail requests will cause queuing or insufficient VRAM.
Key point: QPS, concurrency, and latency are not independent of one another. The longer the request duration, the higher the concurrency that may be formed under the same QPS; the longer the context, the higher the VRAM occupied by a single concurrent request.
GPU Count Estimation Should Start from VRAM Constraints and Model Usage
GPU count estimation needs to first determine the relationship between single-GPU VRAM, GPU count, and model parameter VRAM usage. A common misconception is to look only at model parameter count while ignoring KV cache, runtime reserved space, and the additional VRAM consumption brought by concurrent requests.
A more reliable breakdown is to divide VRAM into three categories: model weight VRAM, KV cache VRAM, and runtime reserved space. Only when all these parts can fit under the target concurrency and context length do the GPU specifications and quantity have preliminary feasibility.
Recommended Estimation Steps
Confirm the Model and Precision
First clarify the model to be deployed, parameter scale, and weight precision. Parameter count and precision affect model weight VRAM, and are the basis for determining whether a single GPU can host the model.
Estimate Model Weight VRAM
Estimate weight VRAM based on model parameter count and precision. At this stage, you can use a VRAM estimation tool to obtain an initial range for filtering possible GPU specifications.
Estimate the KV Cache Brought by Context and Concurrency
Evaluate the possible VRAM usage brought by KV cache by combining input tokens, output tokens, maximum context, and target concurrency. Because KV cache size is proportional to the number of tokens, long context and high concurrency will simultaneously push up VRAM pressure.
Reserve Runtime Space
In addition to weights and KV cache, space also needs to be reserved for inference frameworks, scheduling, temporary tensors, communication, and monitoring. In production environments, VRAM should not be planned close to full utilization.
Select Single-GPU Specifications and Determine Whether Multiple GPUs Are Needed
If a single GPU’s VRAM cannot simultaneously accommodate model weights, KV cache, and reserved space, then larger-VRAM specifications, multi-GPU deployment, parallel strategies, or reduced context and concurrency assumptions need to be evaluated. If VRAM is sufficient, throughput and latency still need to be verified through stress testing.
| Judgment item | Main issue | Possible action |
|---|---|---|
| Weights do not fit | A single GPU cannot host the model’s basic weights | Choose a larger-VRAM GPU, multi-GPU deployment, or a lower-precision solution |
| KV cache pressure is high | Long context or concurrency causes insufficient VRAM | Limit context, adjust concurrency, evaluate cache offloading |
| Throughput is insufficient | VRAM is sufficient, but generation speed cannot meet business peak demand | Add replicas, optimize batching, or scale out GPUs |
| Latency is too high | Queuing or generation time exceeds the target | Reduce single-instance load, split traffic, or adjust the service strategy |
When VRAM Is Insufficient, Cache Offloading and Memory Sharing Can Be Evaluated
When VRAM becomes a bottleneck, adding GPUs or choosing specifications with larger VRAM is not the only direction. CPU-GPU memory sharing can help access and process large models or datasets that exceed traditional GPU VRAM capacity limits, and KV cache offloading is also a typical application task in this type of approach.
For long-context and highly concurrent inference scenarios, the KV cache often occupies more VRAM as the number of tokens grows. Offloading part of the cache to the CPU side, or alleviating GPU VRAM pressure through CPU-GPU memory sharing, may allow the system to host larger models or longer contexts with limited GPU VRAM.
However, such solutions cannot be evaluated only by whether they can run. Cache offloading and memory sharing are affected by the hardware environment, data transfer, inference framework, latency targets, and throughput targets. The specific models, graphics cards, and concurrency results in public cases usually cannot be directly generalized into capacity conclusions for all businesses.
Key point: VRAM optimization solutions should be verified through stress testing. If the business has very high requirements for low latency, the additional overhead brought by cache offloading needs to be evaluated carefully.
Capacity Planning Results Need Continuous Calibration Through Stress Testing and Monitoring
Capacity planning provides an initial boundary before launch, not a one-time conclusion. After the real business goes online, request length, peak traffic, output token count, and user behavior may all change, so capacity levels need to be continuously calibrated through stress testing and monitoring.
Combinations That Stress Testing Should Cover
| Stress test dimension | Recommended coverage |
|---|---|
| Context length | Short context, typical context, long context close to the upper limit |
| Concurrency level | Regular concurrency, peak concurrency, burst concurrency |
| Request type | Short Q&A, multi-turn conversation, long-document analysis, batch generation |
| Output length | Short output, typical output, longer output |
| Resource level | GPU VRAM usage, token generation speed, queuing status, response time |
After stress testing, you need to observe where the bottleneck comes from. If VRAM reaches the limit first, adjust the context limit, concurrency strategy, model specifications, or cache solution; if VRAM is sufficient but throughput is insufficient, focus on generation speed, batching strategy, and number of replicas; if response time is unstable, queuing, scheduling, and long-tail requests need to be analyzed.
Post-Launch Capacity Review Checklist
- Whether peak QPS exceeds the original assumptions.
- Whether average input and output tokens have changed.
- Whether the proportion of long-context requests has increased.
- Whether GPU VRAM remains at a high level for a long time.
- Whether queuing, timeouts, or long-tail response times occur.
- Whether the current GPU count can still cover business growth.
- Whether context limits, batching strategies, or scaling plans need to be adjusted.
The core of LLM capacity planning is not to pursue a perfectly accurate one-time estimate, but to establish an estimation process that is explainable, stress-testable, and reviewable. Only by first clarifying business requests, then breaking down VRAM and KV cache, and finally calibrating GPU count through stress testing can the inference service achieve a more reliable balance between cost and stability.