Why does the memory peak exceed your estimate?

Define the workload before choosing the counter

"A 7B model" specifies neither the representation of the weights nor the work to be performed. Note its revision, the framework, the versions of the extensions, the format actually loaded and the operation: training, fine-tuning or generation. For text, note the input and output lengths; for vision, the resolution and the number of images. A multimodal model requires keeping both of these dimensions.

Prepare a typical input, a long but expected one and one close to your functional limit. Keep them during the comparisons. Check the shapes after tokenization, padding, grouping or resizing: the value written in the configuration does not prove the shape actually processed.

Also set what stays simultaneously in memory: one sequence, one microbatch, several requests or an evaluation run after training. Your question becomes verifiable: does this complete workload fit on each device used, including during its most demanding step?

• Trial identity: model or code, revision, input set and seed when relevant.
• Dimensions: batch, context, generated tokens, resolution or number of simultaneous requests.
• Environment: selected GPU, driver, Python, PyTorch, CUDA or HIP/ROCm backend, and allocator settings.
• Scope: loading, compute, transfer, evaluation, export; first pass or pass after warm-up.

Computing weights without mixing GB and GiB

One GB equals 1,000,000,000 bytes; one GiB equals 1,073,741,824 bytes. The PyTorch counters used here return bytes. Keep this raw value in the results file, then apply a single conversion to compare rows. A card's commercial label does not replace the capacity actually reported by the device.

For a dense set of seven billion parameters stored on two bytes each, the weights represent 14,000,000,000 bytes: 14 GB, or about 13.04 GiB. This operation includes no activations, no gradients, no KV cache, and no optimizer states. Adding a 4 GiB reserve gives about 17.04 GiB as a preparation assumption; this does not prove that a workload will fit in that envelope.

The theoretical four-bit division assumes uniform compact storage. A real quantized loading can add scales and other information, and keep certain modules in another precision. The weight storage format and the compute format must therefore appear separately in your record.

Technical sources: NIST — binary prefixes and GB/GiB comparison · Hugging Face — quantized formats and modules with bitsandbytes

In training, measure a full step

The weights coexist with other objects: gradients, optimizer states, activations needed for the backward pass, and temporary tensors. Their sizes depend on the loop, the precision, and the workload dimensions. A universal constant in bytes per parameter would notably mask the effect of the microbatch and the inputs.

Instrument the forward pass, the loss computation, the backpropagation, and the update. To observe the states actually created by your optimizer, do not stop at model loading. Also keep a measurement of the first full step: a successful warm-up may already have performed an initialization that you must be able to fund in memory at startup.

Add the evaluation and export your project needs. If the failure occurs during evaluation, reducing only the training batch does not fix that phase. An adaptation that trains few parameters can still keep a base model and large activations.

Technical sources: Hugging Face — memory categories during training

In inference, track context and concurrency

In an autoregressive generation with attention, the KV cache keeps states associated with tokens. For a uniform dense cache, its size depends on the layers, the KV heads, their dimension, the retained tokens, and the sequences present together. Use the model's KV heads, not automatically its query heads.

Arithmetic example: 32 layers, 8 KV heads, a dimension of 128, 8,192 tokens, two bytes per value, and one sequence give 1,073,741,824 bytes, or 1 GiB for K and V combined. Four identical sequences give 4 GiB for this single item. This calculation measures neither throughput nor full GPU occupancy.

Adapt the formula to the cache actually used. A sliding window does not necessarily retain the entire history; a static cache can preallocate its maximum capacity. Quantized and offloaded caches also change the problem. Measure the initial input processing and the generation separately, without automatically attributing their entire difference to the cache.

Technical sources: Hugging Face — cache strategies, static allocation, and windows

Allocated and reserved: two readings that do not add up

memory_allocated describes the bytes occupied by the tensors tracked by PyTorch on the device. memory_reserved describes the memory managed by its caching allocator, including the memory already used by those tensors. Adding the two double-counts part of the memory. Keep them in two separate columns.

Their max variants each record a peak since tracking began or its last reset. These are absolute peaks for the period, which include allocations already present at the start. The result does not automatically represent only the objects created by the phase.

A system-level reading may have a broader scope. Allocations made directly by a CUDA library, for example certain NCCL communications, are not all visible in the PyTorch allocator. A difference from a system tool therefore does not, on its own, establish a leak.

Technical sources: PyTorch — memory_allocated · PyTorch — memory_reserved · PyTorch — max_memory_allocated · PyTorch — allocations outside its allocator

Why the difference between the two peaks does not measure the cache

Consider only the two fictitious instants in the table. The allocated peak is 8 GiB and the reserved peak is 12 GiB. Their difference, 4 GiB, is not the difference observed at either of these two instants: that difference is 2 and 6 GiB respectively. Two maxima do not necessarily describe the same state.

To study their gap at a given moment, read allocated and reserved at the same checkpoint, after synchronization and without any new voluntary operation between the readings. You obtain a counter gap at that instant, not a measurement of the model's KV cache, nor a guarantee that all of this difference can satisfy the next allocation.

Also keep the allocator backend in the report. The PyTorch 2.14 documentation specifies that, with cudaMallocAsync, max_memory_reserved can combine the highest levels of two pools and provide an upper bound of the simultaneous peak. This reinforces the need to keep the name and scope of the counter.

Technical sources: PyTorch — definition and limit of max_memory_reserved

Delimit each phase before reading its peak

GPU operations can be queued before they complete. For a per-phase measurement, finish the previous work before resetting the peaks, then wait for the end of the phase before reading. torch.cuda.synchronize waits for the kernels of all streams of the selected device; this choice defines an explicit boundary for this protocol.

reset_peak_memory_stats resets peak tracking from the current state; it does not free the program's tensors. Read the starting levels first. At the end, keep both absolute peaks and both current levels. Do not present the subtraction of a starting level as the exact volume of all temporary tensors: earlier objects may also have been freed during the phase.

This instrumentation can alter the usual overlap of phases. Use it to locate the problem, then also verify the full loop with its actual scheduling. With multiple cards, repeat the readings for each device; a measurement on cuda:0 does not describe the other GPUs.

• 1. Give the phase a name and note its exact inputs.
• 2. Synchronize the device, then read the starting allocated and reserved.
• 3. Call reset_peak_memory_stats on that same device.
• 4. Run the defined phase while keeping the outputs needed for what follows.
• 5. Synchronize, read the peaks and the ending levels, then record success or error.
• 6. Keep the raw result, the dimensions and the configuration; do not fill any missing measurement with zero.

Technical sources: PyTorch — synchronizing a device · PyTorch — resetting peak statistics

Distinguish the first pass from passes after warm-up

A first run and a loop that has already been prepared do not answer the same question. Keep a record of loading and the first pass, then document the number of warm-up iterations before the repetitions. Do not discard an initialization failure on the grounds that later passes would have been lighter.

The IteraGPU script distinguishes model_load, inputs, cold_forward, warmup, and warm_forward. Its cold_forward is the first pass of the small model after device initialization. It does not measure the entire startup of a server, a driver, or a service. The warm_forward repetitions stay within the same process and benefit from its existing state.

For your model, start a new series in a new process when you change a condition likely to leave behind previous objects or allocations. Note the order of the runs and the warm-up policy. Running the same loop five times and launching five processes are not the same protocol.

Using the notebook and the IteraGPU Lab v1 script

Start with the README, then download the standalone notebook or the Python script. The estimate computation uses the standard library. Measurement requires PyTorch installed with a compatible GPU backend and an accessible device; it downloads neither a model nor a package. The notebook requires an environment capable of opening ipynb files.

The measurement exercise uses a small original dense network and synthetic inputs. The batch, context, and width options describe its tensors; context here is not the length of a real LLM with a KV cache. This material is meant to examine the measurement method and vary one dimension. It does not demonstrate a card's capacity for your research model.

Run the commands below from the folder containing the script. Start by reviewing the environment report. If PyTorch or the GPU is missing, measure must stop explicitly with exit code 2; no CPU result should be interpreted as a GPU measurement. The output JSON of a successful measurement is produced in your environment. Choose a new file name for each series: the script refuses to overwrite an existing result.

The first command recomputes the weights and the hypothetical 4 GiB reserve. For the following ones, use a device you are authorized to use and keep the dimensions modest at first. Record the version of PyTorch actually used: the technical references on this page notably describe version 2.14, without requiring that it be installed on your machine.

The script's behavior was checked on a small case: a local RTX 5070 not in the catalog, driver 610.62, Python 3.14.6, and PyTorch 2.11.0+cu128. The run used batch 1, context 16, width 64, float32, one warm-up, and two repetitions. It validates this execution path, without qualifying an LLM, training, or the GPUs offered for rental. The notebook is delivered without outputs and the comparison table without results; the guard for a missing PyTorch was also verified in a separate environment.

python mesure_memoire.py estimate --parameters 7000000000 --bits 16 --reserve-gib 4
python mesure_memoire.py environment --device cuda:0
python mesure_memoire.py measure --device cuda:0 --batch 2 --context 128 --width 1024 --dtype float32 --warmup 3 --repeats 5 --output mesures.json

• Computation and measurement notebook ↓

  Standalone notebook to open, review, and run in your environment.

• mesure_memoire.py script ↓

  Dependency-free computation, environment check, and explicit GPU measurement.

• Folder instructions ↓

  Prerequisites, commands, scope of the phases, and limits of interpretation.

• IteraGPU Lab v1 archive ↓

  The versioned resources collected, with their instructions and their license.

• Resource license ↓

  Conditions for reusing the provided files.

Interpreting a failure before changing cards

An incomplete run remains a useful observation. Keep the phase, the requested dimensions, the error message, and the last available values. A partial peak before saturation does not constitute the memory requirement of a full run. Reduce a single dimension to build a case that completes, then look for the boundary between success and failure.

empty_cache frees unused blocks from the allocator's cache, without freeing tensors that are still alive. It is not a universal fix for excessive load. Calling it between every repetition changes the conditions: document that choice instead of mixing those trials with the ones that keep the cache.

If the simple counters do not explain the situation, a memory trace can help identify allocations over time. Its scope remains limited to allocations visible to PyTorch. A low allocated peak therefore does not rule out an external allocation or another user of the card.

Technical sources: PyTorch — what empty_cache frees · PyTorch — traces and limits of memory visibility

Build a margin from comparable loads

Avoid a margin percentage presented as universal. The margin must cover identified variations: longer input, permitted batch, evaluation, export, library version or other use of the card. Test the expected edge cases, then record what remains out of scope. A successful run on a single small input does not validate the maximum load.

Change one variable at a time: batch 1 then 2 at constant context, or contexts 2,048 then 4,096 at constant batch. Keep the same content and the same preparation rules. A truncation that removes necessary information makes the task different, even if it reduces the peak.

If the weights dominate, study another format while controlling quality. If the activations dominate, microbatching or activation checkpointing may be options. The latter trades memory for recomputation: also measure the duration and verify the results. If the KV cache dominates, examine context, concurrency and cache strategy. The comparison case supplements this approach with a shared quality rule.

Technical sources: PyTorch — activation checkpointing and recomputation

From the trace to a configuration decision

Your expected output is a short record: weight estimate, maximum load tested, phases that succeeded or failed, four counters with units, environment and chosen option. Attach the raw result to this record. Separate what you calculated, what you observed and what you still assume.

Then compare the requirement with the capacity of each card, while retaining the software constraints. Multiple GPUs require distributing the work and the data; their presence does not automatically create a single pool of memory for the application. A failure on one card may persist despite free memory on another.

The PyTorch protocol uses the torch.cuda interface; a HIP/ROCm build of PyTorch reuses that name. Identify the backend actually installed before comparing two hardware families. Using the same Python function does not prove that the kernels, precisions or results are equivalent.

The sizing tool lets you revisit the initial assumption; the GPU records let you compare capacities. Then return to the same workload to verify the choice. The small network from the download remains an instrumentation exercise: only running your own workload, in its documented environment, can validate your own margin.

Technical sources: NVIDIA — distributing work across multiple GPUs · PyTorch — torch.cuda interface in HIP/ROCm builds