An ablation must isolate one decision.

Writing a hypothesis that can be contradicted

"Improve the model" does not specify the experiment. Instead write: "Class weighting improves quality on the rare classes, without exceeding the accepted regression on the frequent classes." Name a primary metric, the important subgroups and the useful-effect threshold. Also set the constraint that remains the priority: memory, latency, input coverage or model simplicity.

Distinguish adding from removing. Adding a component to a simple reference measures its contribution in that context. Removing it from a complete system measures what that system loses. The two questions can produce different answers when components interact. Your conclusion must therefore specify the reference and the state of the other factors.

The expected deliverable is a decision accompanied by a table of variants, not just a better score. Before you book the runs, write down what would make you keep, drop, or investigate each track further.

Technical sources: NIST — defining the objective of the experimental design

Building an interpretable control and variants

The control uses the reference procedure, with its versions, its data, and its checkpoint selection rule. It must be runnable in the current campaign. An old metric with no predictions and no complete protocol can serve as a benchmark, but it does not automatically replace this control.

For each factor, state exactly the two states being compared. "Augmentation enabled" is too vague: keep the transformation, the probability, and the data concerned. Describe the parameters that remain common: preprocessing outside the factor under study, splits, optimizer, training budget, and evaluation method. If the training steps stay fixed but the tokens processed change, note that consequence.

Keep the final test set out of variant choices. Learned transformations and tuning decisions must use the data intended for that purpose. An improvement obtained after tuning the model on test errors is no longer an independent evaluation.

Technical sources: scikit-learn — avoiding data leakage

Two factors require four situations

Suppose you are studying A, a class weighting, and B, a training-time augmentation rule. To see their interaction, examine the control, A alone, B alone, and A with B. This two-factor, two-level design contains four configurations. With k binary factors, the full design contains 2ᵏ configurations before repetitions: the number of runs grows quickly.

The following table is a made-up numerical example to explain the reasoning. Its scores come from no training. They represent fictitious macro-F1 averages on a scale of 0 to 100; in real work, the individual values and their variability remain indispensable.

Technical sources: NIST — full two-level factorial designs

Reading effects and their interaction

In this example, A adds 1.2 points without B, but only 0.7 point when B is already active. B adds 0.8 point without A and 0.3 point with A. The combined gain is 1.5 points, whereas the sum of the two isolated gains is 2.0 points. The difference of −0.5 point here describes a non-additive interaction.

This calculation establishes neither its robustness nor an explanation of the mechanism. It tells you which question to confirm: does adding B to the system containing A justify its complexity for only 0.3 point in this example? Also examine the planned subgroups. The same average can mask different gains and losses.

Compare these differences across paired repetitions when your protocol allows it. Do not pick the best seed for each variant. If the sign or magnitude changes sharply across runs, the decision remains uncertain.

Technical sources: NIST — combinations and interactions in a factorial design

Allocating budget to the decisions that matter

A run allowance is a planning tool, not a forecast of GPU speed. Example: you have an organizational budget of 24 full runs. You assign 12 runs to the four configurations with three seeds each, 10 to a confirmation of the control and one candidate with five new seeds, and 2 to justified repeats. The total is 24; nothing yet says how many hours they will take.

The first three seeds are used here to explore, not to certify a winner. Set the rule for choosing the candidate before observing this series. Confirmation uses the agreed protocol; new seeds reduce dependence on the initial selection but do not fix a test set already used to tune the model.

If the budget does not allow the necessary repetitions, reduce the factors or postpone a question. Prioritize changes that affect a real decision: removing an expensive component, resolving a failure, or deciding between two close options. Keep time for evaluation and artifacts before comparing pricing plans.

Prepare the order and stopping rules

Write the list of trials before launching them, with identifier, configuration, seed, and priority. Distribute the variants across the run order instead of finishing all controls and then all candidates. If a period, a dataset, or a machine constitutes a comparison block, document that block. A change of environment midway through the series must remain visible.

Prepare technical stopping reasons, such as repeated errors or memory overrun. Keep every attempt and its status. Do not silently replace a failure with a shorter run, nor a disappointing trial with a new seed until you get the expected score.

An early stop decided on the scores changes the analysis protocol. If you plan interim decisions, announce their rules and their exploratory scope. For a confirmatory conclusion, keep an analysis plan suited to those decisions.

Close the ablation with a verifiable conclusion

The final report states the hypothesis, the control, the factors, the repetitions, the gaps, and the failure cases. Link each value to the predictions and to the run that produced it. End with an explicit decision: component kept, component removed, or insufficient data to choose.

Avoid three shortcuts: attributing a change to A when the preprocessing also changed; adding up isolated gains without testing their combination; announcing a general rule from a single corpus. An ablation establishes an observation within a defined protocol. Its scope depends on the data, the model, and the variations actually studied.

Practical questions