# Protocol: an accepted corpus before comparing cost

Version 1 — September 24, 2026. This sheet proposes a protocol to fill in; it contains no measured results or dataset. The accompanying notebook uses a small synthetic MLP to learn how to read memory. It does not run the business protocol below.

## 1. Define the useful work before the trials

Example of work: classify a corpus of **1,000 texts**. Build yourself an authorized set, representative of your use, with a unique identifier and a reference verified per text. Fix the revision of the corpus, the model, the tokenizer, the code and the seed. Keep the list of expected IDs and the individual outputs separately to allow auditing.

Define before any measurement: the classes, the output format, the main metric (for example macro-F1), its acceptance threshold and the maximum allowed degradation against a reference. Choose a tolerance relevant to your application; no universal value is provided here. The training, tuning and evaluation data must remain separate.

A corpus is accepted only if the 1,000 expected IDs are present exactly once, no extra response appears in the outputs, the format is valid and the pre-established quality rule is satisfied. Two files each containing 1,000 lines do not by themselves prove that the IDs are equal. Archive the set and duplicate check.

## 2. Change only the announced variant

To study the batch, prepare for example variants 1, 4 and 8. Keep the same corpus, input order, model, tokenizer, precision and context limit. If you then study precision, create a separate experiment and redo the quality check. Describe the truncation: shortening the texts changes the work performed.

Record the actual GPU, the number of GPUs actually used, the device, the driver, Python, PyTorch and the CUDA or ROCm runtime. Also record your code version, any generation options and any competition for the machine in `notes`. A commercial lot of two GPUs does not imply that the program uses both.

## 3. Measure comparable passes

State what the timer covers: processing alone or the full chain with reading, tokenization and writing. Separate loading, the first pass and the warmed-up passes. The memory script only measures its MLP and does not time a full classification chain.

Perform the announced warm-up, then five measured passes per variant in an alternating or pre-drawn order. Keep each raw duration. Reporting the median and the min–max range shows the dispersion; five observations do not justify a robust estimate of p95. Do not silently discard an error or a slow pass: keep its row and explain the incident. Rerun in a fresh process if you want to compare the first passes under similar conditions.

For PyTorch GPU measurements, synchronize the chosen device before the initial reading and after the operation. Record the `allocated` and `reserved` baselines, reset the peak statistics, then keep the absolute peaks of the phase. `allocated` is included in `reserved`: adding them would count part of the memory twice. The two maxima may be reached at different instants: their difference is not a measurement of the cache at a given instant. Allocations by other processes and those outside the PyTorch allocator are not covered.

## 4. Fill in resultats-bruts.csv

The distributed CSV contains only the headers. Write one row per pass, with a decimal point and seconds for times, bytes for memory and USD cents for the package. An empty cell means "not recorded"; zero genuinely means zero. Commas present in a note must be protected by the usual CSV rules.

| Fields | Meaning and input |
| --- | --- |
| `experiment_id`, `variant_id` | Stable identifiers for the experiment and the variant. |
| `corpus_revision`, `model_revision`, `tokenizer_revision`, `seed` | Immutable versions or fingerprints, and the declared seed. |
| `gpu_model`, `gpu_count`, `device`, `driver_version`, `python_version`, `torch_version`, `runtime_version` | Hardware actually used and observed environment; do not copy an offer's promise. |
| `precision`, `batch`, `context` | Configuration actually applied. |
| `phase`, `run_index`, `warmup_iterations` | Separate phase (`cold` or `warm`, for example), run number, number of warmups. Do not mix durations. |
| `expected_ids`, `observed_ids` | Expected and observed ID counts; detailed lists are kept with the outputs. |
| `ids_match`, `format_valid` | `true`/`false` after actual verification, including duplicates and extra responses. |
| `quality_metric`, `quality_threshold`, `quality_tolerance`, `quality_value` | Predefined name, threshold and tolerance, then the measured value. Specify the direction of the tolerance and the reference in `notes`. |
| `corpus_accepted` | `true` only if all validation conditions are met; otherwise `false`. |
| `elapsed_seconds` | Raw duration of the declared scope, never an expected value. |
| `baseline_allocated_bytes`, `baseline_reserved_bytes`, `peak_allocated_bytes`, `peak_reserved_bytes` | Separate readings for the measured device only. Leave blank if not measured. |
| `duration_days`, `lots`, `package_total_usd_minor` | Full package chosen, billed lots and total price in USD cents; a single expense must not be added five times. |
| `accepted_unique_corpora` | Number of distinct useful corpora accepted for the economic analysis; this field is not to be summed across repetitions. |
| `notes` | Scope, incidents, quality decisions, code revision and references for the retained artifacts. |

## 5. Calculate without inventing production

The price to compare is the full 3-, 7- or 30-day package multiplied by the lots. For B200, one lot contains two GPUs and the rate already covers that lot. `calcul_forfaits.py` applies this rule to the provided rates.

If the useful work chosen is a complete accepted corpus, `--accepted-results` must receive the number of distinct corpora actually validated over the period. The five repetitions of the same corpus serve to measure variability: they do not create five useful deliverables. Set the unit before the comparison and keep it for all variants. Without a measured and accepted quantity, leave this parameter out: only the package cost will be calculated. Do not automatically project a rate over 3, 7 or 30 days.

Keep the completed protocol, the individual outputs, the quality checks, the raw CSV and the economic calculation together. A faster but unaccepted configuration does not fulfill the same objective. A configuration that runs out of memory remains an observation of failure, not a time to be replaced with zero.