Reduce the microbatch without losing track of the updates.

Separate microbatch, backward pass and update

The microbatch is the group of examples processed by one forward pass on one replica. The backward pass computes its contribution to the gradients. The optimizer update uses the available gradients to modify the parameters. With accumulation, several forward/backward passes precede that update; the parameters stay unchanged during the group.

In PyTorch, gradients accumulate in the tensors set aside for that purpose. Clearing the gradients after each microbatch would therefore cancel the intended accumulation. Conversely, forgetting to zero them between two groups would make examples from the previous update contribute.

Define your logging unit: microbatch number, optimizer update, examples or tokens seen. The word "step" alone is ambiguous. A loss curve cannot be compared properly if the axis represents eight times more examples in one of the experiments.

Technical sources: PyTorch — gradient accumulation and zeroing

Compute the effective batch without counting GPUs twice

Let m be the number of examples per microbatch and per replica, A the number of accumulated microbatches, D the number of data-parallelism replicas. If these sizes are constant and the examples are correctly distributed, the number of examples contributing to one global update is m × A × D.

The factor D does not necessarily designate all the cards in the machine. GPUs that share a single model through tensor or pipeline parallelism do not become that many data replicas. Write down the groups actually configured, not simply the commercial quantity of the batch.

Arithmetic example: two examples per microbatch, eight accumulations and two replicas give 32 examples per global update. Each replica processes sixteen examples in that group. The table compares counts; it does not rank their memory or speed.

Technical sources: PyTorch — effective batch and accumulation in mixed precision · PyTorch — behavior of DistributedDataParallel replicas

Normalize the loss according to the elements actually evaluated

For an average loss over microbatches containing the same number of relevant elements, dividing each contribution by A gives the group average. This rule assumes that the framework does not already perform this normalization. With a tool that handles accumulation, reread its contract before adding a manual division.

For a per-token average loss, different lengths change the denominator. You must relate the sum of the losses to the tokens actually supervised in the group, excluding padding and ignored positions. The average of the microbatch averages generally does not give the same objective.

Theoretical example: one microbatch has 512 supervised tokens with an average loss of 2; another has 1,536 with an average of 4. The weighted average is (512 × 2 + 1,536 × 4) ÷ 2,048 = 3.5. The unweighted average is 3 and overweights the small group. These values illustrate the calculation only.

Technical sources: Hugging Face Accelerate — accumulation with examples of variable sizes

Organize a complete accumulation group

First prepare the group boundaries and its denominator. For each microbatch, compute the output, the normalized loss and the backward pass without an intermediate update. Free the outputs you no longer need; keeping losses attached to their graph in a list can extend the lifetime of allocations.

After the last contribution, apply the operations intended for the full gradient, then the update. Then reset the gradients for the next group. If the pass ends with fewer than A microbatches, explicitly choose either to process that partial group with its true denominator or to discard it; note the examples involved.

With mixed precision using GradScaler, the scale factor remains constant during accumulation. The actual unscaling and any clipping occur after the contributions; the scaler update follows the step attempt. Checks for non-finite values can prevent the modification of the parameters.

The scheduler must follow the unit announced by your loop. If it is defined per optimizer update, calling it at each microbatch would change the schedule. Record step attempts and actually applied updates separately when your system can skip some.

Technical sources: PyTorch — autograd graphs and tensors retained for backward · PyTorch — accumulation, unscale, clipping and GradScaler

In multi-GPU, verify reduction and example distribution

DistributedDataParallel synchronizes gradients between replicas. In its usual behavior, the reduction averages them; a summed loss and a locally averaged loss therefore do not have the same scale. With different token counts per replica, the global denominator and this reduction must be considered together.

Verify the IDs actually processed: involuntarily duplicating the same examples on all cards does not increase the group's information by as much. To delay intermediate communications, no_sync can be used on the microbatches preceding the final synchronization; its context must also cover the forward pass.

Do not transpose this rule to all distributed systems. State sharding, pipeline, communication hooks and frameworks can change the effective operations. Start with the configuration supported by your tool, then check a complete group on each replica.

Technical sources: PyTorch — gradient reduction and scope of no_sync in DDP

Why the same effective batch does not guarantee the same experience

The equality m × A × D is a counting identity. Recovering a large-batch gradient requires, in particular, correctly weighted contributions, the same parameter state throughout the group, and operations compatible with this decomposition. Numerical proximity is verified with a suitable tolerance; it cannot be inferred from the product alone.

BatchNorm computes statistics from the inputs of its pass: several small microbatches do not present it with the same groups as one large batch. Random operations, the order of computations, and rounding can also vary. Do not promise bit-for-bit identical final weights.

A change in the global batch can also change the number of updates for the same number of examples seen. Fix the comparison axis and your quality rule in advance. Do not simultaneously change the learning rate, the scheduler, and the duration without documenting these new assumptions.

Technical sources: PyTorch — BatchNorm1d statistics · PyTorch — reproducibility limits

Control memory and decide what to do next

Instrument a group including the backward passes and the first update, then the subsequent groups. A successful forward pass does not validate the gradients or the states created by the optimizer. Counters must keep their scope per device. The memory dossier provides the method for reading baselines and peaks.

If a group fails, reduce the microbatch and recompute A to keep the target effective batch, when that choice remains relevant. This change guarantees neither a proportional division of the peak nor a better duration. If weights or states dominate, accumulation alone may be insufficient.

Before a campaign, check one update on a small controlled set: same examples, weighted loss, finite gradients, group boundary, and number of steps. Then evaluate quality with the chosen protocol. The small inference MLP in the downloadable dossier does not run this training recipe; it does not replace this check.

Your final record brings together m, A, D, the supervised tokens, precision, normalization, handling of the last group, and the observed peaks. Then return to the configuration choice and the experiment budget, separating preparation, trials, and results actually accepted.

• An abnormally small loss: look for a double division by the accumulation.
• A result that varies with the split: check ignored tokens and the average of averages.
• Accumulation with no effect: check zero_grad and optimizer.step.
• Growing memory: look for references kept between microbatches.
• Shifted schedule: distinguish backward passes from updates.

Technical sources: Hugging Face — memory checkpoints of a training run

Practical questions