Compare differences, not two best scores.

Define what varies and what constitutes a repetition

Start with the question: are you evaluating the effect of training initialization, data order, a train/test split, or stochastic generation? A seed controls a random generator; on its own, it does not identify all of these dimensions. Keep one row per run and record only the factors actually changed.

Re-running the deterministic evaluation of the same checkpoint on the same corpus ten times does not produce ten independent trainings. Likewise, a thousand predictions from a single model do not replace multiple trainings for estimating their variability. Choose the unit of repetition that matches the decision.

Before the series, fix the primary metric, its direction, and the minimal effect that would justify a change. In the following examples, a higher value is better and the illustrative useful threshold is 0.30 point on a metric displayed from 0 to 100. This threshold comes from the project decision, not from a universal statistical rule.

Pair configurations on common conditions

For a pair i, run reference A and candidate B on the same data split and with the planned common conditions. Then compute dᵢ = score(Bᵢ) − score(Aᵢ). Pairing relies on a correspondence defined before the results; it does not consist of matching scores that happen to look alike afterward.

Using the same list of seeds can help organize pairs, but two architectures may consume random numbers differently. Declare initialization, data, and generation seeds separately when your code distinguishes them. Also record the partitions or their identifiers: a shared integer does not prove that the data was traversed the same way.

If a pair contains a failure, keep both statuses and explain how it will be handled. Do not compare the mean of five successful candidates with that of six references without flagging the change in population.

Technical sources: NIST — pairing between matched observations

Keep the limits of the seed and determinism in mind

PyTorch states that fully reproducible results are not guaranteed across versions, platforms, or CPU and GPU runs, even with an identical seed. Its guide distinguishes control of generators from control of non-deterministic operations. Record the versions and options applied rather than summarizing the environment as "seed fixed".

A deterministic run is useful in particular for reproducing a behavior under given conditions. It does not demonstrate that the model withstands other initializations or data. Conversely, a difference between two identical reruns warrants diagnosis before being interpreted as the candidate's effect.

Keep the same determinism policy for both variants. If you change it to diagnose an error, identify that series separately. The experimental result must remain tied to the conditions under which it was obtained.

Technical sources: PyTorch 2.14 — reproducibility and randomness control

Worked example: five paired differences

Here are five fictitious pairs intended for the calculation, with no training run. The seeds are example identifiers. The reference and the candidate use the same comparison protocol here. Scores are expressed out of 100; differences are in points, not relative percentages.

The mean of the differences is (0.4 + 0.3 + 0.1 + 0.5 + 0.1) / 5 = 0.28 point. Their median is 0.30 point and their range goes from 0.10 to 0.50. The sample standard deviation of the differences is approximately 0.179 point, with a denominator of n − 1. All five signs are positive, but the spread remains large relative to the mean gain.

Reading an interval without giving it excessive scope

To illustrate a common calculation, suppose the differences are independent between pairs and come from an approximately normal distribution. The 95% Student interval for their mean uses four degrees of freedom here: 0.28 ± 2.776 × 0.080, i.e. approximately [0.058, 0.502] point. With only five pairs, the shape of the distribution is difficult to assess; the calculation does not make these assumptions true.

This interval is above zero in the example, but it overlaps the useful threshold set at 0.30 points. It therefore does not support the strict rule “the average gain exceeds 0.30 points.” A positive difference can still be too small or too uncertain to justify the change.

A 95% interval describes a coverage procedure under its assumptions, not a 95% probability attached to the parameter after the calculation. Here it covers only the variation represented by the pairs, not automatically another domain, another corpus or future hardware.

If you use SciPy, ttest_rel compares paired samples and provides a confidence interval. The order of the arrays sets the sign: for B − A, place B first. Keep the values and the method used, not just a p-value.

Technical sources: NIST — confidence interval for a mean · SciPy 1.18 — ttest_rel and interval of the differences

Preparing the decision and the next repetitions

The result of the example is “useful gain not established,” not “useless candidate.” Depending on the cost of the change, you can keep the baseline, continue collecting or test a more substantial modification. State this rule before you have the series in front of you. A lower bound above your useful threshold would support adoption more strongly, provided the rest of the protocol is valid.

Plan the repetitions based on the precision needed and the variability expected, with a reserve for failures. No number of seeds universally guarantees a conclusion. A pilot can help estimate the dispersion; keep its status exploratory and then fix the confirmation procedure.

Avoid looking after each pair and then stopping as soon as the result becomes favorable using an interval intended for a fixed sample size. Choose a sample size and an analysis in advance, or a suitable sequential method. Adding trials only to the disappointing candidate also changes the balance of the comparison.

Do not confuse training variability with corpus evaluation

A series of seeds on a fixed test set informs the variation of training runs on that set. It does not by itself measure the uncertainty related to the choice of test examples. If your question also concerns partitions or domains, plan a design that varies these dimensions without mixing them into a single repetition counter.

Keep a final evaluation separate from model choices. Selecting among many variants with the same test favors the options that look good on it by chance. Subgroups and complementary metrics should inform the decision, with their number and exploratory status visible.

The final output brings together the raw scores, the pairs, the differences, the dispersion, the interval method and the decision against the useful threshold. Attach the failures and the limits of generalization. You can then explain why the gap seems sufficient, insufficient or still undecidable.

Technical sources: scikit-learn — keeping the test set out of model choices

Practical questions