Buy a decision, not an endless series.

Define what will make it possible to close the campaign

"Train a model" defines neither the work to be purchased nor when it is finished. Prefer a goal such as comparing a control and two variants on a frozen corpus, then delivering their predictions, an error analysis, and a reloadable artifact. A negative result can be enough: showing that no variant meets the threshold avoids a longer, poorly targeted campaign.

Separate three categories before ordering: essential to conclude, useful if time allows, exploratory. The control, the data check, and a rerun verification generally belong to the first. Set a decision point after the pilot: continue, reduce an optional variant, or revisit the question. The available package must not become an obligation to fill every hour.

Build an end-to-end schedule

An extrapolation from an already warm training step sometimes overlooks loading, long inputs, validation, and output files. Distinguish the windows during which the machine is engaged from the human preparation time. Independent windows can overlap; two steps that use the same GPU do not become parallel just because they have two rows in a table.

Here is an illustrative schedule, with no measured GPU speed. It reserves 48 sequential hours from the start of preparation to the retrieval of results. Within a 72-hour window, it leaves 24 hours uncommitted. This subtraction only checks the schedule: neither setup nor a ten-hour variant are timelines promised by IteraGPU.

Compare packages without inventing hourly billing

The price of a lot covers the selected duration. An RTX 4090 from the catalog costs USD 47.14 for three days, USD 110 for seven days and USD 390 for thirty days. These amounts are IteraGPU rates; they do not measure the number of trained models. Two lots over three days represent USD 94.28. The B200 lot already includes two GPUs: its composition does not multiply its price a second time.

If the illustrative schedule grows by 30 hours, it reaches 78 hours and exceeds three days by six hours. Then consider the seven-day package, or a reduced protocol that still answers the question. Buying two lots can help with independent variants if your environment allows it; it does not automatically shorten a run and does not merge memories.

Technical sources: IteraGPU rates and rental units · Calculate packages and check the window

Prioritize variants by their usefulness to the decision

After the pilot, update your duration assumptions and ask what result would change the choice. A variant almost identical to the baseline can be less informative than a control that isolates a cause. Avoid launching all combinations of rank, precision, learning rate and length at once. The ablations guide shows how to set up a small, interpretable plan.

Write down the stopping rule before the trials: non-finite loss, quality below a threshold, essential input truncated or no usable recovery point. If the stop occurs, keep its identifier and its reason. An interrupted run does not become an accepted deliverable, but it remains an expense and sometimes decisive information. A new configuration after correction deserves a new identifier.

Plan for the cost of an interruption

Resuming training requires more than the weights alone. The PyTorch documentation distinguishes the model parameters from the optimizer state; progress must also be kept. Depending on the loop, add the necessary states of the scheduler, the scaler, the random seeds and the data traversal. Decide whether you want to resume training or simply rerun an inference.

Test an early checkpoint and then reload it in a new process. Count the write, transfer and restore time in the schedule. A checkpoint interval is not chosen solely to produce many files: match the tolerable loss of work with the observed cost of the operation. Also check where these files live and how you will retrieve them before the period ends.

Technical sources: PyTorch — checkpointing and recovery

Relate the budget to the useful result

At closing, report the package committed, the other expenses actually included and the number of deliverables that passed the announced checks. Any external expenses remain separate from the rental price; do not invent tax rates or transfer costs that were not provided. The cost of a campaign is not erased when its result is negative.

Conditional arithmetic example: if a $47.14 plan can deliver two complete, accepted, distinct corpora, its rental share is $23.57 per corpus. Rerunning the same corpus five times to measure its timing does not yield five useful corpora. With zero accepted corpora, the ratio is undefined; report the cost incurred and the reason for rejection, never a zero cost. For a research decision, describe the conclusion reached rather than inventing an artificial unit.

Close out with files you can reload

Prepare the output package during the campaign: manifest, parameters, raw measurements, predictions, stop reasons and reload instructions. Check the expected identifiers, empty files and paths that depend on a temporary directory. For an adapter, keep the exact revision of the base. A file that exists but cannot be reloaded is not a verified output.

Reconcile planned vs. actual without rewriting the initial plan: time reserved, time observed, explained deviations, decisions made. The IteraGPU ledger can keep the summary and the order reference; your raw files and backups remain yours to organize. The next campaign then starts with a better estimate and a shorter list of uncertainties.

Practical questions