Smaller Models for Better Document Understanding

At Cyberhaven, we replaced our previous LLM-based approach to document understanding with a small model that makes 83% fewer decision errors at less than 1% of the inference cost. We achieved this by breaking complex tasks into simpler decisions, using uncertainty as a training signal, and heavily optimizing our inference stack. The model produces calibrated decision probabilities while preserving multimodal understanding and generative capabilities.

Decision models such as TypeSafe AI’s Jev have been attracting significant attention for their low inference costs and calibrated probabilities that reflect uncertainty. This interest highlights an important distinction: large language models are capable generalists, but that does not make them the best fit for every application. Processing terabytes of data with frontier LLMs can become prohibitively expensive.

At Cyberhaven, we process immense amounts of data and have been using an in-house trained document-understanding model for several months. Our model is orders of magnitude smaller than frontier LLMs, delivers competitive quality, and produces calibrated probabilities. It also retains multimodal understanding and generative capabilities, allowing us to reuse document representations between decision tasks and tasks that require generation.

Our approach addresses three connected challenges:

1. Cost: making large-scale document processing affordable
2. Quality: reducing decision errors while maintaining high recall
3. Calibration: producing probabilities that reflect uncertainty.

We explain how task decomposition, shared document processing, and uncertainty-aware training contribute to these goals, then evaluate the results along each dimension.

Simplifying the Problem

Large language models (LLMs) handle a wide range of tasks well. They let us express complex requirements in a prompt and get a working solution quickly. But making something work and making it scalable are different challenges. For our document-processing workloads, LLM inference was prohibitively expensive. Small language models offered a path to 100–1,000× lower costs, but preserving quality required more than switching models. Replacing an LLM like Gemini 2.5 Flash with an SLM like Gemma 4 E2B under the same task formulation increases decision errors by more than 100% and introduces output-schema failures in more than 1% of documents.

Closing that gap meant returning to the fundamentals: matching the problem formulation to the model’s capacity. A frontier LLM can consider multiple definitions, reason across all criteria, and assemble a complex structured response in one pass. We needed to reduce the demands placed on each inference.

We therefore decomposed the task into individual binary decisions, each evaluating one criterion. With this formulation, we can assign decision scores directly from the probabilities assigned to the answer tokens, eliminating the need to generate and parse a complex response schema.

Simplifying the task eliminates approximately 80% of the additional decision errors introduced by switching from Gemini to the two billion parameter model. Our in-house SLM reflects a further quality improvement, as reflected in the figure below:

Decomposition would be expensive if every decision required processing the full document again. We therefore organize prompts and inference to maximize reuse of cached document computation. The first request processes the document; subsequent requests reuse the shared context rather than recomputing it. Self-hosting gives us control over cache management and scheduling, making this reuse practical at scale.

Handling Ambiguity

Some document-analysis tasks admit more than one reasonable answer. Different human annotators or frontier models can disagree because a definition allows different interpretations, the document omits information needed to decide, or they lack relevant internal knowledge. A useful model should reflect these sources of uncertainty in its confidence: less certainty when the available evidence or knowledge is insufficient, and greater certainty when the evidence strongly supports a decision.

Language models work by assigning probabilities to each possible token continuation. When these probabilities are accessible, we can derive a decision score from the probabilities assigned to the tokens representing the decision s.a. True or False. However, these scores are not automatically interpretable as calibrated decision probabilities.

Research across several model families shows that instruction tuning can improve task accuracy while worsening calibration (Sanz-Guerrero et al., Huang et al.). Training a model to follow instructions and reproduce reference answers does not necessarily teach it to express uncertainty in ambiguous cases. Its probabilities can become more decisive without a corresponding increase in how reliably its answers are supported.

The figure below illustrates the difference. A well-calibrated model follows the diagonal: decisions assigned a probability of 80%, for example, should have positive outcomes approximately 80% of the time. Gemma’s probabilities diverge substantially from that relationship, while our model’s probabilities track the observed outcomes more closely, in fact, for this task our in-house model is significantly better calibrated than Jev, a decision model.

Post-hoc calibration methods, such as temperature scaling, adjust a model’s scores after training. Their effectiveness depends on how well the calibration dataset represents the deployment data, and they can easily overfit. Our approach diverges from these methods by incorporating uncertainty into the training signal itself. During fine-tuning, the model learns both how to answer the task and how strongly the evidence supports that answer, encouraging less decisive probabilities in ambiguous cases.

Frontier Performance at a Fraction of The Cost

We benchmarked frontier LLMs, small language models, and decision models on selected document-understanding tasks using our internal Golden Dataset. The evaluation reflects how we use these models: LLMs receive the full task in a single prompt, while SLMs receive simpler, decomposed decision-like prompts. The results therefore compare complete task-solving approaches, rather than models under identical prompts.

Claude models used adaptive thinking at their default effort levels: medium for Opus 5.5 and high for the other evaluated variants. Gemini 3.8 Flash used default dynamic thinking. SLMs, decision models, and the Gemini 2.5 Flash baseline ran without generated reasoning traces.

Cyberhaven SLM delivers 83% fewer decision errors at more than 100× lower inference cost than our initial Gemini 2.5 Flash approach. It also produces 9.6% fewer decision errors than Claude Opus 4.6 at more than 1,000× lower inference cost, while approaching the quality of the strongest frontier models tested.

Our main quality metric is F0.5, which combines precision and recall while giving greater weight to precision. In our setting, false positives can trigger unnecessary investigations and interrupt legitimate work. We therefore prioritize precision while maintaining high recall.

Cyberhaven SLM achieves a F0.5 score of 97.8%, up from 88.1% for our previous Gemini 2.5 Flash approach. It outperforms Claude Opus 4.6 (97.47%) and Claude Sonnet 4.6 (96.73%), as well as the other evaluated SLMs and decision models: Gemma 4 26B A4B (91.41%), Kev 4B (90.69%), and Jev 1.13 (88.17%).

The strongest results come from Claude Sonnet 5.5, Claude Opus 5.5, and Gemini 3.8 Flash, with nearly perfect scores of 99.7–99.8%. Cyberhaven SLM comes within two percentage points of these models at more than 1,000× lower inference cost, while recording the lowest reported inference cost among the evaluated configurations.

Conclusion

Making document understanding practical at scale required us to address cost, quality, and uncertainty together. We decomposed complex tasks into simpler decisions, incorporated uncertainty into model tuning, and optimized inference to reuse document computation across tasks.

The resulting Cyberhaven SLM delivers:

1. Lower cost: more than 100× lower inference cost than our previous approach and more than 1,000× lower cost than the strongest frontier models tested.
2. Higher quality: 83% fewer decision errors than our previous approach, with an F0.5 score that surpasses Claude Opus 4.6.
3. Better calibration: probabilities that more closely match observed outcomes than those of the evaluated decision models, including Jev 1.13 and Kev 4B.

These improvements preserve the model’s multimodal understanding and generative capabilities. For our workloads, matching the task formulation, training signal, and serving setup to a small model made near-frontier document understanding practical at production scale.

Learn more about Cyberhaven Labs.

Note: No customer data was used to train or fine-tune the model.