AI One Lab / Study 3
Building trustworthy enterprise benchmarks and synthetic training data.
A practical method for designing enterprise decision benchmarks and auditable synthetic training data for task-specific models.
Overview
The quality of a decision model depends on the distinctions its data teaches and tests. This paper documents the benchmark behind Papers 1 and 2 and the separate synthetic corpus used to specialise Clef.
The benchmark contains 1,974 requests across 50 workflow categories. The new frozen corpus contains 6,497 examples split across training, validation and test. The paper explains how scenario families, difficult category boundaries, model routes and review records were handled.
What the paper covers
Define the decision before creating examples
Each example asks for the current primary workflow, with a shared policy for resolving ambiguous requests. The model sees the request and relevant context, while labels and audit notes remain separate.
Keep evaluation and training data distinct
The original benchmark was retained for evaluation only. Family exclusions and frozen packs were used to preserve the separation between the benchmark and the later training corpus.
Make the process inspectable
Recipes, generation runs and final training packs answer different questions. The paper records review, coverage, lineage and provider-reconciled production spend of $156.93.
How to read the findings
The reference labels were model-authored and automatically reviewed; they are not individually adjudicated human ground truth. The paper describes a repeatable construction and audit method, not a universal benchmark for every enterprise task.