The task and what "good" means
An assistant that answers in fluent, correct Slovenian and, in the middle of a conversation, decides whether to call a tool, which one and with which arguments. "Good" was agreed before any model was run:
- Slovenian knowledge and reasoning: the share of correct answers on seven Slovenian tasks.
- Tool use: four separate scores, because they fail differently: should it call at all, the right tool, the right arguments, and staying silent when no tool is needed.
In a customer evaluation this step uses fifty of your real examples and the people who know the work. The definition of "good" is written down and signed off before measuring, so the result cannot be argued afterwards.
The frozen test sets
- Slovenian-LLM-Eval (
cjvt/slovenian-llm-eval): 500 examples per task, 7 tasks, 3,500 in total, zero-shot, the same code for every model. - Next-step tool calling: 2,471 held-out conversation steps the model never saw in training.
Frozen means the sets are versioned and never change. Every later model, retrain or update is scored on the same examples, which is how a regression is caught before it reaches users.
Results: Slovenian knowledge and reasoning
| Task | Untuned base | Tuned | Δ |
|---|---|---|---|
| arc_easy | 0.968 | 0.982 | +0.014 |
| arc_challenge | 0.942 | 0.950 | +0.008 |
| hellaswag | 0.686 | 0.752 | +0.066 |
| piqa | 0.852 | 0.862 | +0.010 |
| openbookqa | 0.876 | 0.920 | +0.044 |
| winogrande | 0.708 | 0.748 | +0.040 |
| boolq | 0.878 | 0.886 | +0.008 |
| Average | 0.844 | 0.871 | +0.027 |
The evaluation itself produces the first column. The second shows what training later changed, so you can see both halves of the cycle on one page.
Results: tool use
| Score | Untuned base | Tuned | Δ |
|---|---|---|---|
| Decides correctly whether to call a tool | 0.893 | 0.922 | +0.029 |
| Picks the right tool | 0.792 | 0.801 | +0.009 |
| Fills in the right arguments | 0.628 | 0.651 | +0.023 |
| Stays silent when no tool is needed | 0.847 | 0.956 | +0.109 |
The biggest gain is the least visible one: knowing when not to call a tool. An untuned model calls a tool in roughly one of seven cases where it should just answer. In production that means wasted calls, and in a system with approvals, a person approving things nobody asked for.
What goes wrong, with examples
Scores hide the type of error, so every report lists the failures that matter, grouped by cause:
This part decides what to do next: knowledge gaps call for text or targeted data, format errors for more examples, and some errors are simply not worth fixing.
Recommendation
- Train: yes, for tool use. The gap in "stay silent when no tool is needed" is large and it closes with training.
- Slovenian: continued pre-training, not longer fine-tuning. A second pass over the same fine-tuning data bought nothing: tool scores moved by less than a point and Slovenian dropped slightly. Slovenian text is what moved the knowledge scores.
- Keep checking facts. Knowledge errors remain. For answers that must be exact, pair the model with retrieval from your documents.
- Deploy locally as GGUF on one workstation for a small team; the same weights serve a department through vLLM.
What a customer report adds
- Your fifty examples and your definition of "good", signed off before measuring.
- The same test set scored on several candidates, including a large model through an API, so you see what training has to beat.
- A cost comparison: API fees at your volume against your own model.
- Where the data stayed: your examples are anonymised and not used for anything else.