AI One Lab / Study 1
Are open-source decision models useful for enterprises?
A technical evaluation of a locally hosted, open-weight decision model against hosted decision and reasoning models on enterprise workflow classification.
Overview
Enterprises may be able to route routine requests with a locally hosted decision model, but accuracy, speed, operating cost and the consequence of a wrong route all matter. This study compares stock Clef with hosted models on 1,974 synthetic requests across 50 enterprise workflow categories.
The task was classification: identify which workflow a request belongs to. It did not test whether a model could safely execute that workflow.
What the paper covers
Local classification is viable for a bounded task
Clef Flash correctly classified 92.71% of requests, returned a valid result for every request and had a median response time of 221 ms in the tested configuration.
The strongest hosted result was more accurate
GPT-6 Luna with high reasoning reached 96.10% top-1 accuracy, with a median response time of 2.912 seconds. A hosted Decisions configuration reached 95.39% at 286 ms median.
Economics depend on utilisation
The paper models local serving cost at about $0.694 per 1,000 decisions at 10% utilisation and $0.144 at 50%, before treating data locality and the cost of errors as decision factors.
How to read the findings
The benchmark is synthetic and nearly balanced across categories. Results from a real deployment will depend on its request mix, error-handling policy and hardware utilisation. The paper includes the full method, category results and cost assumptions.