How much memory should I reserve for the KV cache?

Describe what stays in memory

During autoregressive generation, the keys and values of already processed tokens can be kept for subsequent steps. This cache belongs to the attention layers. Its volume therefore depends on the model and the retained history, not just the number of parameters. A conversation's context also includes instructions, previous messages and documents added by the application.

Your first decision is operational: how many sequences must stay active, up to what length? A queue of twenty requests of which two run concurrently does not necessarily represent twenty resident caches. Record the actual request admission and any extra sequences created by generation.

Prepare a sheet with the model revision, attention layers, KV heads, key and value dimension, their dtype and the cache strategy. Count tokens after the tokenizer and the chat template. A character limit does not describe this allocation.

Technical sources: Hugging Face — how per-layer caches work and their shape

Use KV heads, especially with GQA

The number of query heads Q and the number of KV heads can differ. In classic multi-head attention, they coincide. With MQA, a single KV head is shared; GQA groups several Q heads around one KV head. Read num_key_value_heads in the configuration when this field exists and verify what it means for the architecture.

For example, forty Q heads and eight KV heads form five Q heads per KV group. The stored-cache formula uses eight, not forty. This ratio does not describe all attention memory: operations or conversions can produce temporaries.

Do not simply change this number to reduce the memory requirement of an already-trained model. The attention scheme is part of its architecture. Two models with different head counts do not become equivalent variants through a capacity calculation; their quality must be evaluated separately.

Technical sources: Hugging Face — LlamaConfig fields and the distinction between MHA, MQA, GQA · PyTorch — Q/K/V dimensions and GQA attention constraints

State the formula and its units

In the uniform case, L is the number of layers, Hkv the number of KV heads, D their dimension, T the number of positions kept per sequence, B the number of sequences and q the number of bytes per value. The factor of two accounts for K and V. This approximation assumes keys and values of the same dimension and the same format, with no compression or prefix sharing.

Convert only the final result: one GiB is 1,073,741,824 bytes; one decimal GB is 1,000,000,000 bytes. Keep the bytes in your notes to avoid a rounding or unit change masking a difference.

For different lengths without padding, replace B × T with the sum of the positions actually stored. For heterogeneous layers, sum layer by layer. Dense storage with padding, block allocation or static reservation requires counting allocated slots, which may exceed the useful tokens.

Technical sources: Hugging Face — cache tensor dimensions · NIST — decimal units and binary prefixes

Worked example: six sequences, no GPU measurement

Let's take a fictional architecture of forty layers, eight KV heads and a dimension of 128. Assume a uniform cache at two bytes per value. Each sequence receives at most 3,072 input tokens and a reservation for 1,024 additional tokens, i.e. an upper bound of 4,096 positions. These are pedagogical assumptions, not the attested configuration of a model.

The computed cost per position and per sequence is 2 × 40 × 8 × 128 × 2 = 163,840 bytes. A sequence of 4,096 positions then represents 671,088,640 bytes, or 0.625 GiB. Six sequences give 4,026,531,840 bytes, i.e. 3.75 GiB for the cache alone.

Doubling the kept length doubles this line item in this formula. Replacing eight KV heads with forty multiplies it by five, all other assumptions unchanged. These proportions do not predict any speedup, quality loss or compatibility for a real model.

Adapt the budget to the cache strategy

A dynamic cache grows with the positions kept. A static cache reserves a maximum capacity: size that reservation, not just the short request observed at startup. For sliding-window attention, some layers may cap their history; full-attention layers require a separate calculation.

Cache quantization and its offloading to the CPU are other strategies, dependent on the model and the software. They change the storage, transfer or compute constraints. Quantizing the weights does not prove that the cache has the same format.

Note the cache class and its explicit parameters. Also check that slots are released after a request finishes or is cancelled. For a workload mixing short and long sequences, a uniform maximum reservation may weigh more than the sum of the useful contents alone.

Technical sources: Hugging Face — dynamic, static, quantized and offloaded caches

Check a representative workload step by step

Build three cases: usual input, expected long input and the maximum number of simultaneous requests allowed. Fix the model, tokenizer, template, generation limit and stop rule. Vary one dimension at a time; a shortened response or a truncated document changes the work performed.

Measure loading, initial input processing and generation separately. Synchronize the device around the timed phases, and keep the starting memory levels and the peaks. The memory method explains why allocated and reserved do not add up and why the difference between their maxima does not isolate the cache.

Your check must produce a tested bound and acceptable outputs: completed request IDs, errors, generated length and quality criterion. A run that holds up on a short input does not validate maximum concurrency. An out-of-memory error before completion is not a measurement of what a full run requires.

Technical sources: PyTorch — synchronizing work on the chosen device · PyTorch — scope of the memory counters

Rule out the errors that distort the choice

Do not turn the computed cache into total GPU capacity. Add an analysis of the weights, temporary activations, retained outputs and software. Nor should you automatically divide that volume by the number of cards: layer or head placement must actually be configured and verified on each device.

The IteraGPU Lab notebook is an instrumentation exercise on a small dense network without attention. It helps you read memory phases; its context parameter does not validate this KV computation. Use the workload sheet and the inference folder to prepare the test for your own model.

Compare the capabilities of the offerings only after distinguishing the arithmetic volume, the maximum actually observed and the cases not yet tested. The rental package organizes your working window; it guarantees no context length or generation rate.

• Confusing Q heads and KV heads: go back to the architecture configuration.
• Budgeting only the prompt: include generation and the actual reservation.
• Reading “4-bit weights” as “4-bit cache”: record both formats.
• Forgetting concurrency: count the sequences actually resident.
• Promising shared memory between cards: verify the actual placement.

Practical questions