Choose a quantization based on its results, not its bits.

Describe the variant beyond its bit count

Name the method, its implementation, its version and the exact revision of the artifact. Two four-bit variants may use different representations, groups, metadata and kernels. Separate the storage format of the weights from the format of the compute operations. Also note the modules kept at another precision and any offloading to the CPU.

In bitsandbytes, quantized linear layers replace certain ordinary layers; the other modules follow their configured dtype. This behavior therefore does not on its own describe the process peak. Read the effective configuration after loading, then verify that the variant actually performs the expected processing rather than a different software fallback.

Technical sources: Hugging Face Transformers 5.17 — quantized layers and other dtypes

Separate calibration from evaluation

Some methods use examples to prepare the quantization. The GPTQ procedure documented in Transformers notably requires a calibration set and a tokenizer. This set takes part in producing the artifact: it does not constitute independent proof of quality. Other methods follow a different path; do not assume that the same calibration protocol applies to all formats.

Set aside the calibration examples in the authorized data to prepare the model, then record identifiers, provenance, lengths and the transformation applied. Keep validation for choosing the settings, and the final test for confirmation. If you choose several calibration sets after comparing their scores, that search is part of development and must be included in the report.

Technical sources: Hugging Face Transformers 5.17 — calibrating a GPTQ quantization · Build partitions according to their role

Compute a weight bound without selling it as a peak

Let's take a fictional model of three billion parameters, all stored at the same bit count. The raw volume is computed as parameters × bits / 8. At 16 bits, we get 6 billion bytes; at 8 bits, 3 billion; at 4 bits, 1.5 billion. These numbers are illustrative arithmetic, not the observed sizes of an artifact nor the requirements of a running model.

The raw difference between 16 and 4 bits is 4.5 GB, or about 4.191 GiB. It excludes scales, metadata, non-quantized modules, cache and temporaries. It allows an assumption about memory to be formulated and verified, without predicting a fourfold reduction in the peak. The memory guide explains how to separate the phases and interpret the counters.

Technical sources: NIST — binary and decimal prefixes · IteraGPU — memory estimates and peaks

Change the weights before changing the cache

Start with a baseline whose behavior you know. Then compare the weight variants with the same cache, the same lengths, the same batch or the same concurrency. If you also change these parameters, you are comparing complete configurations: say so and do not attribute the entire difference to the weight quantization.

The KV cache can itself be quantized when the model and the engine allow it. This is a separate choice. The Transformers documentation notably points out that a quantized cache can hurt latency for short contexts when enough memory remains. Test this second change in a separate series; more memory saved does not guarantee a better latency.

Technical sources: Hugging Face Transformers 5.17 — quantized KV cache and latency trade-offs

Example quality rule: a small drop may still be unacceptable

Here is an illustration of a decision, without running a model. A project evaluates 200 documents and defines, before the trials, a maximum loss of one percentage point against the reference. It also requires at least 90% success on 20 critical documents included among those 200. A document is accepted only if all its required fields are correct.

The fictitious values below make A admissible under both rules: 94% instead of 95%, and 18/20 on the critical group. B fails both. A cannot yet be declared the winner: neither peak memory nor duration is given. Moreover, the totals do not show which documents changed; examine the paired errors to detect any major new regressions.

Technical sources: Examine the errors rather than the total alone

Run a comparison whose gaps can be explained

First prepare the comparison contract, then the results files. The following protocol is to be carried out on your workload; the earlier figures do not replace it. If a variant does not load or lacks the required operators, keep that failure as compatibility information.

• Freeze the corpus, references, model, tokenizer, prompt and output rules; keep the long and difficult cases identifiable.
• Record the calibration, the exported artifact and the effective configuration. Check the devices used and any CPU/GPU transfers.
• Separate build, loading, warm-up and stabilized processing. Use the same measurement boundaries and keep the raw repetitions.
• Record the peak per device and per phase; keep allocated and reserved separate. Do not sum these counters and do not subtract their independent maxima.
• Evaluate the format, the content and the agreed subgroups, then match errors by identifier.
• Reload the chosen artifact in a new process and replay a defined check. A result obtained before export does not automatically validate the reload.

Technical sources: Measure memory correctly · Define repetitions and timing

Link the trade-off to the cost incurred

A memory saving can widen the configurations that are feasible, allow more concurrency or simply leave headroom. It does not automatically reduce the expense. If the period, the reserved batches and the accepted deliverables stay the same, the package cost stays the same, even if a run is faster.

Include in the schedule any calibration, quantization, evaluation, rejected attempts and reloading. Then compare the full 3-, 7- or 30-day packages between the options that meet your criteria. The ratio per useful corpus can only be computed with corpora that are actually completed and accepted; timing repetitions do not create new deliverables.

If a single rental is used to compare several variants, its amount is a shared campaign expense. Do not mentally charge the whole package to each variant and then add those amounts up as distinct actual expenses. To allocate an analytical share, state a convention; it does not change the total incurred.

Technical sources: IteraGPU — full package and cost of an experiment

Conclude with a configuration and its limits

The final decision names the artifact, the environment, the workload covered, the quality criterion met and the constraint improved. If the variants are close, keep that uncertainty and favor a choice you know how to reload and explain. The number of bits is not on its own an order of preference.

A conclusion drawn on a short corpus does not automatically extend to long contexts, to another language, or to more simultaneous requests. A quantization chosen for inference also does not define the trainable parameters of a fine-tuning. Keep these questions separate and confirm the chosen variant on the held-out test.

Practical questions