Keep an evaluation that can still surprise you.

Define what each example represents

Start with a decision sentence: recognize the category of a new ticket, extract three fields from a document, or answer a question with its attachments. Describe the input available at the time of prediction, the expected output, and the out-of-scope cases. Information added after the ticket was resolved must not appear in an input meant to represent its arrival.

Then distinguish between a row and an independent unit. Five messages in a conversation share a context; ten pages of a file share their origin. If your goal concerns new files, keep each file in a single partition. A split by user, document, equipment, or period answers different questions: choose the one that resembles the future use, then record its justification in the manifest.

Technical sources: scikit-learn 1.9 — validation with groups and temporal dependencies

Assign a role to each set

The training set is used for model adjustments. The validation set guides development decisions: prompt, threshold, hyperparameters, or choice of a variant. The final test answers a question that has already been fixed. Consulting it to pick the best setting gradually turns that test into a development tool, even if no gradient is computed on it.

Also separate the data used to learn a transformation. A normalization, a feature selection, or an imputation fitted on the entire corpus can pass information to the model. Learn these transformations on the allowed partition, then apply the saved transformation to the other sets. A pipeline makes this control easier; it does not on its own fix incorrectly separated groups.

Technical sources: scikit-learn 1.9 — data leakage and transformations

Handle duplicates without erasing the hard cases

Keep the raw corpus, then build a working inventory with identifier, origin and group. A content fingerprint detects exact copies according to the chosen representation. Document that representation: removing all punctuation or lowercasing a text may merge entries whose difference matters for the task. Keep the mapping between the discarded copy and the retained example.

Near-duplicates require an extra review: a modified export, a reworded answer, a shared passage or a reissued document. High similarity is a signal to examine, not proof that two annotations are interchangeable. Group related variants before splitting the data. If two copies carry contradictory references, open an annotation question; do not automatically pick the one the model predicts best.

Worked example: 1,200 rows are not 1,200 independent observations

This example is entirely illustrative: no corpus or model was measured. Suppose 240 conversations, each exported over five rows. One row is an exact copy in each conversation; the other four are distinct. Removing these 240 copies leaves 960 examples, still organized into 240 groups. The following pedagogical choice reserves 70% of the groups for training, 15% for validation and 15% for testing.

This yields 168, 36 and 36 conversations, i.e. 672, 144 and 144 examples. The equality of proportions in rows and in groups comes here from the four examples per conversation. In a real corpus of variable sizes, it would not be automatic. These proportions are not a universal rule: they must leave enough groups and important cases in each set.

Technical sources: scikit-learn 1.9 — GroupShuffleSplit counts groups

Check classes, lengths and chronology

After the split, count the classes and the groups that carry them. A class present on many rows but in a single conversation does not offer many independent cases. Also examine lengths, languages actually covered, incomplete documents and categories that matter for the decision. A reassuring average can hide a split with no example of a critical case.

Stratification with groups seeks to preserve class proportions without scattering the groups. It does not guarantee a perfect balance when the groups are few or very uneven. If the task is to predict future observations, a chronological separation may be more relevant than a random shuffle. Also check that the variables were available at the prediction date.

Technical sources: scikit-learn 1.9 — StratifiedGroupKFold and its limits · scikit-learn 1.9 — time series data validation

Make the reference precise enough to judge an output

Write an annotation guideline with examples and edge cases. For extraction, specify the meaning of a missing field, the date format, the currency and the accepted equivalences. For classification, describe the boundaries between categories. An answer different from the reference is not always a model error: the reference may be ambiguous or incorrect.

Have a varied selection reviewed, especially disagreements and important cases, then record the arbitration. Keep the corrections in a new version of the dataset. If a correction changes the result of a comparison, recompute all the variants concerned on the same reference; do not fix only the row unfavorable to your preferred model.

Produce the evaluation package before the GPU campaign

The deliverable of this preparation is an identifiable set, accompanied by its rules. It makes it possible to reproduce the data selection without relying on a memory of the notebook. Preparing this package before renting avoids spending the compute period resolving differences in files or criteria.

• Set the identifiers, the groups, the splits and their justification; keep the script or the list that produces them.
• Check for intersections of identifiers, groups, and fingerprints between partitions. Any unexpected intersection must be explained or corrected.
• Export the counts per partition, class, and useful subgroup, as well as the list of exclusions and their reason.
• Lock in the baseline, normalization rules, primary metric, and thresholds before the comparison.
• Keep the revision, fingerprints, usage rights, and data location. A fingerprint provides identification, not anonymity.
• Restrict access to the final test until the planned decision; keep a log of consultations and protocol changes.

Know what the check demonstrates

Preparation is acceptable when the counts reconcile with the inventory, overlaps are under control, and every output can be judged by an explicit rule. It does not prove that the model will succeed, nor that all future uses are represented. A small set can describe a specific problem while remaining insufficient to draw conclusions about a rare class.

If you use the final test errors to improve the system, keep that test as a historical record and prepare a new independent confirmation. For a pretrained model whose original data is unknown, your partition cannot certify the absence of prior exposure. Document this limitation rather than calling the set completely untouched.

Practical questions