Define the useful result before looking for throughput
In interaction, measure the wait for the first token and for the complete response. For an offline corpus, measure the time needed to obtain all the expected outputs. In both cases, set the acceptance rule: accuracy on known answers, quality of a ranking or extraction of verified fields. A syntactically valid JSON can still contain a wrong answer.
Separate the queue wait, the processing and the full trip observed by your client when your tools allow it. A measurement internal to the engine does not have the same boundaries as that of the application. The vLLM metrics documentation notably distinguishes wait, first token and total duration: keep this distinction in your records, whatever engine is used.
Technical sources: vLLM — request and latency metrics
Turn your requests into selection criteria
Prepare short, usual and long inputs with stable identifiers. Keep the model, tokenizer, chat template and generation parameters. Count the tokens actually transmitted, including the history and the documents added by the application. Record separately the requested batch, the concurrency sent and the requests actually processed simultaneously: these are not necessarily the same numbers.
| Input to set | Measurement and unit | Consequence for the choice |
|---|---|---|
| Full prompt and output limit | Input and output tokens per request | Check the long cases and the responses cut off at the limit |
| Concurrent requests and arrival rate | Active, waiting and completed requests | Determine the sustainable workload with the required delay |
| Precision, quantization and cache | Memory peak per GPU, in bytes or GiB | Rule out settings that exceed memory on the intended workload |
| Quality rule and references | Accepted outputs / expected outputs; business metric | Compare only variants that meet the same criterion |
| Scope of the timer | First token, full response or corpus: seconds | Compare durations with the same boundaries |
| Timeline through to retrieved files | Total window, in hours or days | Then choose a 3, 7 or 30-day plan |
Measure memory with context and concurrency
For an autoregressive model that generates token by token, the KV cache stores attention states. Its size depends on the model and the tokens retained. A dynamic cache can grow during generation; a static cache reserves a maximum size. Some sliding-window layers limit this growth. So test the intended lengths and concurrency with the strategy actually used.
Weight quantization and cache quantization are two distinct choices. For example, bitsandbytes replaces certain linear layers with quantized versions; that does not describe all the allocations in your run. After a precision change, check memory and quality again instead of assuming the entire peak drops by the same proportion.
With PyTorch, record the peak of allocated tensors and the peak of memory reserved by the allocator separately. Do not add them together. State which GPU was measured and the unit: 1 GiB equals 2³⁰ bytes. These counters do not necessarily represent all device usage. The memory dossier details their limitations.
Technical sources: Hugging Face Transformers 5.17 — KV cache strategies · Hugging Face Transformers 5.17 — bitsandbytes quantization · PyTorch 2.14 — CUDA memory management and counters
A measurable trial on 300 documents
Example to carry out with your authorized corpus: extract date, amount and category from 300 documents. Review the expected values, specify how missing fields are handled and set the acceptance threshold before the trial. The number of documents describes the protocol; no time, score or throughput is assumed.
- Freeze the 300 IDs, the model revision, the prompts, the token limits and the parsing rule. Keep long documents identifiable in the report.
- Run a baseline with one request at a time. Separate loading, warm-up and measurement; keep the predictions and errors per document.
- Then increase a single parameter: batch for grouped processing, or concurrency for a request engine. Keep the same inputs and quality criteria.
- For each pass, record duration, memory peak, number of IDs retrieved without duplicates and accepted outputs. Repeat the measurements and keep the raw values with their dispersion.
- If a variant fails, log the reason: memory, truncation, format or quality. A reduced batch or a retry is a decision to document, not a failure to erase.
Technical sources: IteraGPU — detailed protocol for comparison at equivalent quality
Use IteraGPU Lab v1 resources within the right scope
The notebook and its companion script offer a weight calculation and a measurement on a small synthetic network. Their measurement code was run on a local RTX 5070 with PyTorch 2.11.0, on a small float32 configuration. This trial verifies that run case; it measures neither an LLM, nor a KV cache, nor the GPUs in the catalog on your requests.
Use the notebook to understand the counters, then measure your real workload with its environment. The quality protocol and the raw table help you prepare the comparison. The distributed notebook's outputs and the CSV result rows remain blank; the procedure installs no model or driver. Read the README prerequisites before running.
- IteraGPU Lab v1 memory notebook
Calculations and a small synthetic network to understand memory measurement.
- Quality protocol to complete
Fixed inputs, output acceptance and trial comparison.
- Blank raw table
One row per real pass, with parameters, measurements and verdict.
- Prerequisites and limits of IteraGPU Lab v1
Instructions and precise scope of the local trial already performed.
Moving from measurements to an offer
Your selection sheet should bring together compatible environment, memory per card, context, concurrency, achieved quality, and measured durations. Compare the configurations that meet these criteria, then the 3-, 7-, and 30-day plans against the full timeline: preparation, processing, evaluation, retries, and export. The benchmarks file calculator uses the entire plan and useful corpora that have actually been validated.
Keep this sheet and the code versions in your notebook. You choose your software and your processing; IteraGPU does not inspect the content of your files, prompts, or computations. An environment choice at checkout expresses your preparation need: it is not proof that your model has already been installed or tested.
Practical questions
Is 24 GB enough for my inference model?
A capacity of 24 GB is not enough to answer without knowing the model and the workload. Check weights, runtime allocations, context, and simultaneous requests together. The configuration must complete long cases with the required quality; a file size or a weights estimate alone does not demonstrate that.
What throughput should be compared for an offline corpus?
First compare the time needed to complete and evaluate the same corpus. If you publish a throughput, give the number of accepted useful outputs, the time used, and the errors. A throughput in tokens per second without response length or quality control is not enough to decide between two configurations.
Does an out-of-memory error require changing the GPU?
An out-of-memory error first requires identifying the setting and the input that caused it. You can examine batch, concurrency, length, or precision, while preserving the project goal. Any reduction that changes the documents processed or the expected responses requires a new quality check; more memory may be needed if those constraints must be preserved.
Does the provided notebook validate my inference application?
The provided notebook does not validate your application: it measures a small synthetic network and explains the memory counters. The documented local test covers only that case. Your application must be evaluated with its model, its inputs, its environment, and its own acceptance criteria before any conclusion about capacity or performance.