How much memory for your hypothesis?

Settle on a configuration for a defined workload

A list of GPUs, a recommendation for your model and an alternative to your current configuration all require the same starting sheet: model and revision, inference or fine-tuning, weight format, required length, concurrency or microbatch, and quality criterion. Keep these requirements in mind when you compare the offerings. A configuration that handles less context or a simplified task does not automatically meet the same need.

Classify each mandatory requirement: admissible if the available document confirms it within your scope; to confirm if it is missing; exclude if an established failure prevents meeting the load. A candidate is only retained when all these requirements are satisfied. Then break ties among admissible candidates by the total package cost, the useful margin and the documented schedule. A preference does not offset a disqualifying criterion.

Link each criterion to a document and a decision

The sales sheet attests to what is offered; your protocol establishes what works on the load. The environment requested at the time of ordering remains a preparation preference. A nominal memory capacity or a declared stock proves neither a software installation nor a duration of availability.

Keep the document with its version and scope. If a mandatory piece of information is not documented, write down precisely the condition to confirm. The table makes it possible to build a shortlist without turning these unknowns into guarantees.

Technical sources: Load sheet for inference · Measure the phases and counters · Set the quality criteria · Plan the cost of the experiment

Know what the result computes

The number of parameters corresponds to the values you represent in the dense computation. The format indicates their number of bits per value. The "Weights only" result converts this volume into GiB, then "Indicative envelope" adds your reserve. Choosing training or adaptation in the tool does not apply a hidden universal multiplier; it invites you to document the missing items.

This simplicity makes the assumption easier to review. It also requires knowing what remains outside the computation: quantization metadata, modules in another format, working memory, transient loading and uses outside the process. The suggested GPU sheet is a candidate to examine, not a guarantee that your model will work on it. The name and nominal memory of a card do not describe the rest of its server.

Read the units before comparing a card

A decimal GB equals one billion bytes; a GiB equals 2³⁰ bytes. The catalog displays a nominal capacity in GB. The tool applies the decimal convention; to validate a configuration, also note the capacity in bytes declared by the device. Your program's counters may be displayed in bytes or GiB. Convert quantities that share the same definition before comparing their size; do not mix nominal memory, free memory and reserved peak.

Example calculation: 24 billion bytes represent approximately 22.35 GiB. This result does not indicate the free amount of a 24 GB card. The system, the execution context and other allocations may count. Keep the unit and the scope in the name of each column of your test sheet.

Technical sources: NIST — binary and decimal prefixes

Worked example: seven billion parameters

For 7 billion parameters at 16 bits, the dense volume is 14 billion bytes, or about 13.04 GiB. With a chosen reserve of 6 GiB, the envelope becomes 19.04 GiB. The calculation does not prove that a 6 GiB reserve suits your model: that is precisely the assumption to check against the inputs and operations involved.

The table keeps the same reserve to show only the arithmetic effect of the weight format. In a real run, the other allocations may evolve differently. A four-bit representation often includes additional information and may leave some layers in another format. So do not read the last row as a guaranteed total peak.

Technical sources: Transformers 5.17 — bitsandbytes formats and bit limits

Building an explainable reserve

Break down the reserve instead of picking a percentage out of habit. In autoregressive inference, note the architecture, cache, retained tokens and simultaneous sequences. In training, note the learned parameters, gradients, optimizer, activations and buffers. Input length and microbatch are part of the record, even when the number of parameters does not change.

Some items do not reach their maximum at the same time. Adding up independent maximum margins can serve as a conservative envelope if it is declared as such, but it is not an observed peak. Record the estimated items, the measured ones and the ones still unknown. The KV cache guide details one of these calculations; the memory dossier places the counters back into the phases of the program.

Validate in stages without silently changing the task

Start with a short input and a small batch to spot preparation errors. Then move to a usual input, then to your actually necessary limit. If the program truncates the data, keeps fewer tokens or skips part of the corpus, the case that fits does not validate the stated workload. Note the dimensions actually executed.

Record the peak per phase and per GPU with its counter. The memory allocated to tensors and the memory reserved by the PyTorch allocator do not add up. A system reading may cover more than the instrumented process. If several batches are considered, document how they are distributed in software; two cards do not automatically form a single memory space.

Technical sources: PyTorch — CUDA memory management

Deciding after the first out-of-memory error

A failure at loading points to the weights, the format or a transient allocation. A failure at generation calls for examining context and concurrency. A failure during the backward pass may point to microbatch, activations or the computation strategy. This localization avoids randomly changing precision, model and data at the same time.

After each correction, check the quality and the scope covered. Reducing the required length may be unacceptable; changing quantization may alter outputs. If a different capacity is required, return to the catalog with a record that specifies the observed peak, the justified margin, the versions and the limit inputs. A margin without a reason is no more reliable than a result rounded to the nearest GB.

A short selection, under the same requirements

For the illustrative 7-billion-parameter inference in 16 bits with a 6 GiB reserve, the worked example gives 19.04 GiB. You can look at the 24 GB RTX 4090 or the 48 GB RTX 6000 Ada. Their listed offers for a one-card batch over 3 days are 47.14 USD and 55.71 USD respectively. These are two candidates with different capacities: context, concurrency, actually available memory, compatibility and quality remain to be checked. These prices establish no ranking of speed.

For an adaptation, reuse the same grid with the learned parameters and the additional phases: backward pass, first update, validation and export. The number of LoRA parameters is not enough to validate total memory. The fine-tuning walkthrough helps qualify the result; batch and accumulation serve to document a comparable update.

If you are looking for an alternative after an overrun, locate the failing phase and keep your output requirements. Compare one change at a time: capacity, cache, microbatch or precision. A different format must recover the minimum quality you set. If your need requires a managed application, a specific RAM amount or an attested interconnection, a GPU spec sheet alone leaves the decision to be confirmed. Two cards require verified software placement; their memories do not automatically constitute a single space.

Technical sources: RTX 4090 — batch, capacity and plans · RTX 6000 Ada — batch, capacity and plans · Quality after quantization · Preparing an adaptation · Defining batch and accumulation

Keep the computation together with what confirms it

Record the initial hypothesis, the line-item table, the procedure and the raw readings. Distinguish estimate, measured counter and commercial decision. The IteraGPU Lab notebook helps you understand this instrumentation on a small network; it does not replace a trial of your own model. The numerical examples on this page are calculations, with no GPU throughput or consumption claimed to have been observed.

A good sizing output fits on one sheet: model and revision, task, precision, length, microbatch or concurrency, memory per GPU and measurement conditions. Add what would invalidate the conclusion, for example a longer window or a different evaluation. You can then choose a plan matching the campaign, instead of redoing the entire estimate for each variant.

Practical questions