Turn a score into verifiable decisions.

Keep the outputs before summarizing them

An analysis starts with the join between inputs, references and predictions. Check that every expected identifier appears exactly once. Distinguish an incorrect answer from an unfinished request, a duplicate or an unreadable output. These problems do not have the same remedy and must not disappear when a mean is computed.

Keep the raw output alongside its normalized version, the model revision, the prompt, the settings and the reason for the verdict. A conversion that silently turns an ambiguous date into a value can create an artificial success. Start exploring on the validation set. If the final test set is used to invent the next fix, a new held-out confirmation will be needed.

Technical sources: scikit-learn 1.9 — keep the test set out of model choices

Read a confusion matrix with its counts

For a classification with one category per input, the matrix crosses the reference class with the predicted class. In the convention used here and in scikit-learn, rows carry the reference and columns the prediction. Keep the raw counts before normalizing: a percentage without the number of examples can exaggerate how solid a conclusion is.

The following example is fictitious and purely arithmetic. One hundred tickets are split between Billing, Access and Deletion. The diagonal contains 54 + 24 + 5 = 83 correct answers, or 83%. That total hides the Deletion class: only 5 of the 10 expected tickets are recognized. No model or GPU produced these figures.

Technical sources: scikit-learn 1.9 — definition of the confusion matrix

Link precision, recall and practical consequence

For Deletion, precision is 5/8 = 62.5%: among the tickets sent to this category, five are correct. Recall is 5/10 = 50%: half of the tickets in this category are found. F1 is 2 × 5 / (2 × 5 + 3 + 5), or about 55.56%. The three false positives and the five false negatives describe different problems.

The F1 scores of the Billing, Access and Deletion classes are 88.52%, 80% and 55.56% respectively. Their unweighted mean, the macro-F1, is about 74.69%. It complements the 83% of correct answers, without replacing the counts. If a class is absent from the references or predictions, some metrics may be undefined: state the convention used instead of hiding the case in a mean.

Technical sources: scikit-learn 1.9 — precision, recall, F1, averages and zero divisions

Build a short taxonomy tied to actions

An error category should help you decide what to examine. Start with a few categories, a definition, and a representative example. Add a primary label to count cases without double counting, then secondary labels if several phenomena coexist. Keep an "to examine" category rather than forcing an explanation.

This taxonomy is a working proposal, not an automatic diagnosis. A truncated document can also contain a reference ambiguity. The frequency of a label describes the cases reviewed; it does not yet demonstrate a cause. Keep the input passage that supports your interpretation and distinguish observed cause, hypothesis, and missing information.

For extraction, count fields and documents

A valid JSON format does not guarantee that the values are correct. Set the allowed normalizations: canonical date, decimal separator, spaces or category codes. Distinguish missing field, invented value, and allowed abstention. A value absent from the document must not be replaced by a guess to improve the fill rate.

Here is a second illustrative calculation, independent of the classification table. Fifty documents each have three expected fields, i.e., 150 values. Suppose 38 documents are entirely correct, six with two correct fields, and six with only one. That gives 38 × 3 + 6 × 2 + 6 × 1 = 132 correct values, i.e., 88%. Yet only 38/50 = 76% of documents are entirely acceptable if all three fields are required.

The 18 incorrect values affect twelve documents. Do not present them as eighteen failing documents. Depending on usage, the useful unit will be a verified field or a fully accepted document; define it before the comparison. Keep the per-field results to know whether the difficulty comes from dates, amounts, or categories.

Choose the fix before relaunching a campaign

Prioritize based on consequence and scope affected, not only on the number of rows. In the fictional matrix, the five Deletion errors may warrant review before the six Invoice errors if the project has defined that category as critical. This priority belongs to the project contract; the table does not allow inventing a business severity.

Formulate a testable hypothesis: "Long inputs lose the decisive passage during preparation." Choose a change that lets you examine it, then keep everything else constant. Simultaneously adding examples, changing the model, and increasing the context may improve the score, but it no longer lets you attribute the effect to a single fix.

• Review a few successes in addition to the errors, to verify that the criterion is applied consistently.
• Compare variants on the same identifiers and references; report any corpus change separately.
• Distinguish fixed errors, persistent errors, new errors, and unchanged cases.
• Also replay examples outside the targeted category to look for regressions.

Check the gain without erasing regressions

A paired calculation shows what a net score hides. Out of the hundred fictional tickets, imagine nine errors fixed but four former successes now wrong. The result goes from 83 to 83 + 9 − 4 = 88 correct answers. The gain is five points, with four regressions to examine. It does not mean that nine corrections were obtained without trade-offs.

The decision note keeps the before/after tables, the corrected cases, the new errors, the fix version and the criteria met. If a critical rule is still violated, a higher average is not enough to accept the variant. Cost and duration are then compared between the admissible options; speeding up a rejected result does not solve the quality problem.

Limit the conclusion to what was examined

A spectacular error is not necessarily representative. If you mainly re-read long entries or failures from a rare category, state that selection mode and do not present their frequencies as those of the whole corpus. Also keep the unadjudicated cases: they define an uncertainty in your evaluation.

Your analysis is usable when another reader can locate the entry, understand the verdict and verify the proposed correction. It proves neither the internal cause of a generated response nor a guaranteed future quality. Move on to the held-out final set after choosing the fix, then archive the limitations along with the conclusion.

Practical questions