"Who must approve before the account is created?"
It’s an innocuous question from a user, but the model receiving it now has to choose what type of intent the user really means. It has 50 predefined categories to choose from and in this test case Clef, the local model we were testing, chose “onboarding”. The expected answer was “approval routing”. Had a live system followed Clef's choice, someone asking for the next approval could have found themselves back in onboarding.
Of course, a person handling supplier operations could have explained the distinction in a few seconds. The eight-word question was asking who had authority to approve the account. Clef's answer identified the broader onboarding process and missed the more tightly scoped approval request.
At AI One Labs, we wanted to find out how well a small, locally hosted model could make these decisions. The job was kept deliberately narrow - given a request and its context, simply choose the current primary workflow from fifty defined categories. It covered familiar enterprise work we have seen in real implementations, including approvals, customer operations, document processing and software delivery.
We began with a broad comparison, testing eight configurations against the same 1,974 requests and fifty workflow categories. The test field included stock Clef and Jev, two generations of Luna with low and high reasoning through the Responses API, and Luna's native Decisions API in two request formats. That gave us a baseline view across both reasoning models and dedicated decision interfaces.
Vanilla Clef got 1,830 requests right, or 92.71%, and returned valid output every time. Its result made it a useful starting point for local routing, with accuracy close to Jev and GPT-6 Luna's low-reasoning Responses configuration. The strongest hosted frontier result reached 96.10%, getting another 67 requests right. We had a capable open-weight starting point and a clear measure of what ‘better’ looked like.
Our job was to make that useful local model great at its particular task. Training on 4,998 examples took it straight to 96.56% on the historical benchmark, into the performance range of the strongest hosted configuration we tested. A separate sealed test showed the same direction of improvement.
Before Clef could make its first decision, we had to fix the serving configuration. Later, a peculiar failure in the hosted comparison sent us back to the way we presented the choices.
A local model worth taking seriously
There are many decisions like the supplier example inside an enterprise. A request about a refund might ask for the policy, an investigation, or the refund itself. A conversation about a software incident may have moved from diagnosis to recovery. The vocabulary stays similar, user questions are occasionally somewhat obtuse, while the work required changes.
Our benchmark contained 1,974 synthetic requests. We had 496 scenarios, each represented in three intended difficulty variants, plus 486 cases designed around deliberately awkward category boundaries. Some depended on a previous exchange. Others contained a prominent word that pointed towards the wrong workflow, or quietly changed the scope of the request.
The median prompt and context together contained 75 words. These were compact decisions, close to the sort of follow-up a person writes when they assume the system remembers the conversation. They demanded care with meaning even though the input was short.
Other examples exposed questions the organisation itself had to settle. One request said, "COI cleared. Take the next step." Its context described a procedure whose next action was creating an ERP vendor record. Both Clef and Luna high chose “system-of-record update” as their decision. The reference favoured “standard operating procedure execution”. Both readings were plausible. The policy had to establish which should take precedence and establish it correctly.
For the vendor-record example, the policy needed to give either the enclosing procedure or its immediate action priority. That choice determines what a correct answer means. It also gives the people labelling examples a rule they can apply consistently. If the owners of two workflows would route the same request differently, they need to settle that before asking a model to learn their decision.
We used Clef Flash, an open-weight model with roughly nine billion parameters, running on a standard RTX Nvidia system. Before measuring its quality, we had to get the serving configuration right. Clef's native decision pass needs the complete rendered request to fit the effective physical batch. Our requests included the policy and all fifty category definitions, which brought them to roughly two thousand tokens despite the short user messages.
The initial default 512-token physical batch was too small. Increasing the context window alone left that constraint in place. A larger batch passed the smoke test, and the full baseline used a conservative 16,384-token capacity with one prediction slot. After that, all 1,974 requests produced valid decisions.
Stock Clef classified 1,830 correctly, or 92.71%. Median response time was 221 milliseconds, with 95% of requests finishing within 306 milliseconds. The strongest hosted result, GPT-6 Luna through the Responses API with high reasoning, was 96.10%. It had a median of a much larger 2.912 seconds in its recorded environment. The serving paths and concurrency differed, so those timings describe the systems we ran.
A little over three percentage points separated our local model from the strongest hosted result. There were 67 additional correct decisions to find. The local system already had the low latency and usable output format we wanted. That made its mistakes worth studying in a little more depth.
The result that sent us back to the request
One row in the comparison looked a little strange. GPT-6 Luna through the native Decisions API had scored 76.70%, far below its high-reasoning Responses result. In that run, the choices were defined as numeric category IDs, presented in their canonical order – much like you might get from a back office system table.
Of its 456 valid but incorrect answers, 332 selected exactly the next category ID in sequence from that list. That is the sort of pattern that makes you inspect your own code. We checked the raw returned values, the mapping into our categories and the maxima of the returned probability distributions. They agreed. The audit found no local off-by-one scoring error to explain the pattern – this was an unusual, and somewhat unexpected feature of the model itself.
We tried a different representation of the same task. The choice values became descriptive category-name slugs. We also shuffled each complete value-description pair using a frozen plan derived independently of the answers. The user request, context, classification policy and category descriptions stayed the same.
The new run scored 95.39%. It corrected 397 previous mistakes and introduced 28 brand new ones. Of the 332 adjacent-category errors, 327 disappeared.
Both request formats passed API validation, yet their accuracy differed by almost nineteen percentage points. The change made us inspect the choice representation as closely as the model configuration.
A smaller diagnostic separated some of the changes. It used 24 deliberately selected cases, including twelve adjacent-ID errors and six correct controls. Shuffling numeric choices helped, and descriptive values helped. Keeping all fifty choices while changing their presentation recovered most of the selected failures. The experiment gave us a direction to investigate, with label binding and position sensitivity among the plausible explanations.
On those selected cases, the original format scored six correct answers. Descriptive values alone raised that to nineteen. Shuffling the numeric choices produced twenty-one, as did combining descriptive values with the shuffle. The combined format and numeric-shuffled format returned the same predictions. For that diagnostic, adding meaningful labels to shuffled choices brought no further improvement.
We also examined whether fifty choices were simply too many for the models to deal with. In sweeps that varied the candidate count, the two-choice and fifty-choice endpoints had identical case-level correctness. Those reduced sets used the known reference to retain candidates, so they served as diagnostic probes. They gave us little reason to blame the number fifty for the original pattern.
The full run changed two factors together, and the diagnostic set was chosen to examine known failures. We wanted repeated permutations and independent examples before turning either formatting choice into a universal rule. For this task, however, the revised request was a substantial improvement.
It also changed how one could read a decision-model comparison. It’s necessary to see the question, the choice values and their order, alongside the model and endpoint. An enterprise can spend considerable effort comparing models while leaving this part of the system almost unexamined. Our most dramatic early gain came from examining it.
Teaching the specialist its fifty workflows
We still had the local model's quality gap to work on. Many of its mistakes concerned distinctions that could be expressed in the task policy and demonstrated with examples. The supplier request was one. The model needed to follow the current action through the context, even when another category shared most of the vocabulary.
We built a new synthetic corpus and froze three splits. There were 4,998 training examples, 500 validation examples for checkpoint selection, and 999 sealed examples for a final test. Every category was represented. The earlier benchmark and related source examples were excluded from the training corpus.
The distinction between a reference label and a teacher's opinion mattered here. Clef trained on one hard category label per example. The input contained the public prompt and context, the fixed policy and the fifty choices. Author explanations and teacher probability estimates stayed outside the training input. The deployed task would give the model the same kinds of evidence it saw while learning.
We also wanted to preserve Clef's native decision architecture. A conventional language model generates an answer as text. Clef's decision interface scores the permitted choices and returns a probability distribution. Our training updated that choice-scoring behaviour through its original decision head.
The approximately nine-billion-parameter backbone was held in a four-bit representation to reduce memory requirements. Low-rank adapters added trainable updates to the model, and the original decision head was trained alongside them. Together, the adapters and head accounted for about 165 million trainable parameters.
The loss function was cross-entropy. In practical terms, a correct category with very little assigned probability incurs a large penalty. Training moves the parameters towards assigning it more. The objective fit our task directly, because each example had one primary workflow.
We ran three epochs on a single GPU. The full interval was 4.07 hours, including initialisation, validation and checkpoint handling. An interrupted save required a restart from update 625 and the replay of 75 updates. The retained training history recorded the recovery, and a fresh reload checked the saved adapters and decision-head tensors.
All three epochs classified 499 of the 500 validation examples correctly. Accuracy alone therefore could not choose between them. Our selection rule used lower negative log-likelihood as the next criterion, rewarding better probability assigned to the reference answers. Epoch three won that tie-break.
With a validation set of 500, a single answer changes accuracy by 0.2 percentage points. That is fairly coarse once the score is near the ceiling. The probability-based tie-break gave us a predeclared way to choose, while the two final evaluations answered what the chosen model could do.
The gain was larger than the leaderboard made it look
For the training comparison, we put stock and tuned Clef through the same native evaluator, with the same numerical format and choice mapping. This matched stock baseline scored 1,827 out of 1,974. Its small difference from the earlier 1,830-result Q8 serving run reflects a separate evaluation configuration.
The trained model scored 1,906. Looking at the individual requests, it corrected 86 stock errors and introduced seven. Another 61 requests remained wrong under both models. Eleven of those changed from one wrong category to another, which is why counting changed labels would have exaggerated the improvement.
Reducing 147 errors to 68 is a 53.7% reduction. Expressed as accuracy, it is a move from 92.55% to 96.56%, about four percentage points. I find the error counts easier to reason about when thinking about operations. They make both the improvement and the new mistakes visible.
The paired 95% interval for the accuracy gain was +2.99 to +5.04 percentage points. The calculation resampled whole scenario families, keeping the three related variants together. That accounts for the fact that three versions of one scenario tell us less than three entirely independent scenarios.
Against Luna high, the difference was much smaller. Clef's 1,906 correct answers put it nine ahead of Luna's 1,897. The interval for that difference ran from -0.36 to +1.29 percentage points, leaving room for either model to lead. Frontier-range performance is how we would generally describe the result on this task.
The two models also differed in where they struggled. On the deliberately difficult boundary cases, Luna high classified 455 correctly and tuned Clef 449. On the scenario variants, Clef did better. A team whose traffic consists mostly of terse, ambiguous follow-ups could care a great deal about that breakdown.
On the sealed 999-example test, stock Clef got 984 right and the selected model got 998. It corrected all fifteen stock mistakes and introduced one. This test had been kept out of checkpoint selection, but it came from the same generation recipe and teacher process as the training data. Its 99.90% result describes performance within that distribution.
The probabilities needed another look
The local model had one operational advantage throughout these comparisons. Every request produced a valid native decision. In contrast to this, Luna high had 21 outputs that entirely failed the benchmark's output requirements, which included the structure and probabilities of its ranked answer.
Our primary score counted every attempted request. A correctly named category inside an unusable response still left the caller without a valid decision. Looking only at the 1,953 requests where both models returned valid output changed the ordering. Luna scored 97.13% and Clef 96.52%.
Ideally, one should keep both views in an enterprise evaluation. The full-cohort score counts how often the application received a valid, correct decision. The jointly valid comparison helps separate classification quality from output handling. A caller that repairs or retries an invalid response creates another system to evaluate, with its own latency and cost.
Then there was confidence. Clef returns a full distribution over the fifty choices, which makes it possible to inspect more than its winning label. Its Brier score, which measures probability error across all fifty categories, improved from 0.123 to 0.062. Expected calibration error compares reported confidence with observed accuracy in groups of predictions. That improved too, from 0.088 to 0.029.
Yet the number of incorrect decisions with at least 95% confidence rose from five to 43. Forty-three of the trained model's 68 remaining mistakes were now highly confident.
Those observations can coexist. The model can assign better probabilities across the whole dataset while concentrating confidence on a smaller set of wrong decisions. For someone designing automatic acceptance, those particular mistakes deserve attention.
So what do we take from this? In short, coverage belongs beside accuracy. A probability threshold of 0.80 on the original stock serving result would have accepted 1,532 requests. Of those, 1,507 were correct and 25 were wrong. That is 98.37% accuracy among the accepted 77.61% of the cohort, with 442 requests left for another route.
What did the improvement cost?
The primary data-generation run cost $156.93. That covered all 6,497 exported training, validation and test examples, including failed, repaired and unused candidate work. The process also produced probability targets that the native hard-label training did not use.
Under the study's depreciation and electricity assumptions, the 4.07-hour training interval added about $2.80. Together, those items came to $159.73. Engineering, earlier pilots, final evaluation and ongoing serving operations sit outside that subtotal.
Someone still has to define the task, inspect the data, connect the native head correctly and verify the exported checkpoint. A small direct compute bill gives a team room to experiment, with that engineering work included in the development budget.
A good approach is to treat the first successful run as a reusable investment in the decision. Once the taxonomy, data checks and evaluation process exist, the team has a way to examine a revised model or a changed policy. The model will need attention as the work changes, and the evaluation set becomes part of maintaining it.
Serving raises a different economic question. The original stock configuration delivered about 3.975 decisions per second while active. Using that measured rate, a $15,000 installation, a three-year write-down and the study's operating assumptions, fully allocated cost was about $0.694 per thousand decisions at 10% utilisation. At 50% utilisation it would be about $0.144.
The hardware cost continued through the quiet hours. Buying a dedicated machine for a small queue therefore produced different economics from keeping an existing shared machine busy. The modelled allocation included maintenance, operator time and idle electricity.
Ten per cent utilisation sounds modest until it is translated into volume. Under the original stock throughput assumption, it represents about 12.54 million decisions a year. The fifty per cent case represents about 62.68 million.
There is a different calculation for the cost of specialisation itself. Spread the $159.73 data-and-training subtotal over one million decisions and it contributes about $0.160 per thousand. Over ten million, that falls to about $0.016 per thousand, before serving and maintenance. For a stable, repeated task, the up-front experiment can become a small component of the eventual unit cost.
The tuned model still needs a deployed serving profile. Its batched evaluation measured forward computation, while the earlier latency numbers came from an actual stock endpoint. Before applying the same cost calculation, a firm would need to measure the tuned endpoint under the expected concurrency and latency target.
Even so, an enterprise can make a local-deployment decision for reasons that go beyond the per-call price. Requests can contain internal policy, customer details or commercially sensitive context. Keeping that material within an approved environment, keeping all context sovereign to the organisation and within its technology perimeter, pinning a model version and controlling the serving path can easily outweigh expenditure. The financial comparison should include those requirements explicitly.
A practical enterprise capability
Open-weight models gave us a useful local starting point. Specialisation through careful training made it substantially better at the enterprise's job. A model reached the strongest hosted configuration's performance range on our fifty-workflow task, while keeping its native decision interface.
That combination fits an enterprise architecture well. The specialist selects a workflow and returns its probabilities. The application applies the organisation's access rules and controls how the selected work proceeds. For the supplier example, approval routing leads into the company's approval process, where the relevant policy and authorisation still govern the action.
The organisation also owns the definition of the task. It can encode the distinctions its teams use, version the model alongside those definitions and keep the business context within its chosen environment. Improvements become part of a capability it operates, with their effect visible in the decisions the model gets right.
Enterprises should be comfortable treating open-weight specialists as a serious part of their AI architecture. We have shown how a capable stock model can be improved for a defined business task at a modest experimental scale. The result establishes a practical route to a high-quality local decision layer, shaped around a company's workflows and operated under its own control.
This essay draws on the first two AI One Labs studies, Are open-source decision models useful for enterprises?, dated 2 November 2026, and Can an open-weight decision model match frontier performance for an enterprise?, dated 6 November 2026. Figures were redrawn from their reported methods and results. Model names, configurations and costs refer to the recorded experiments. The examples and reference labels were synthetic, and the task was workflow classification.