Running a private LLM on your own server: models, setup and real costs

A company calls and asks: can we run our own language model, on our own server, so that contracts, patient letters or tickets never leave the building? And what does that really cost?

Short answer: yes, and for most narrow tasks a small model is enough. A 4B to 12B model trained on your examples handles extraction, classification, support answers and tool calling. You need a larger one (27B to 35B) when questions are open-ended, documents are long and messy, or the model has to combine several sources. The machine is rarely the expensive part. The data, the evaluation and keeping the model current are.

Which model size do you need?

Start from the task, not from the model.

Model size	Where it runs	Format	Typical tasks
4B	a laptop or an ordinary workstation	GGUF Q4–Q8	routing incoming mail, classification, extraction into fixed fields, offline field work
8B–12B	a workstation or a small server	GGUF or vLLM	support answers in your terminology, summaries, tool calling with a few tools
27B–35B	a workstation or a server	GGUF or vLLM	long documents, broader questions, agents with many tools, image input

Our own Sokol 4B runs on a laptop, reads images and calls tools; the model card lists the GGUF builds. Our strongest, Sraka 27B, scores 0.871 on Slovenian-LLM-Eval against 0.844 for its base, but it is experimental and non-commercial. For your work we train a model on your data. The public models show what the recipe does, not what you will run.

Q8 stays closer to the original weights; Q4 is smaller and faster and loses a little. How much it loses on your task is something we measure on your frozen set, not something we guess.

Why there is no memory or speed table here

Because it would mislead you. Memory use and answers per second depend on context length, on how many people use the model at once and on quantization. A figure from someone else's benchmark says little about your load. During the trial we measure it on your task, and then write down what hardware you need, if any. Sometimes the machine you already have is enough for a small model.

vLLM or llama.cpp/GGUF: when to use which

• GGUF with llama.cpp or LM Studio. One file, simple install, runs on a laptop or a single workstation. Good for a handful of users, offline work and pilots.
• vLLM. Built for servers: many people at once, batched requests, an OpenAI-compatible API your existing tools can call. The standard for a department or a customer-facing service.

A rough rule: a few simultaneous users, GGUF. A whole department or your customers, vLLM. We deliver both from the same weights, so moving from one to the other later does not mean training again.

Air-gapped: what changes when the server has no network

Some customers want a machine that never touches a network. We can train on your own isolated hardware, deliver on a drive, and the model runs on a network with no outside connection. Three things then need a plan.

Updates. There is no pip install. Everything arrives on a drive: weights, runtime, dependencies and checksums. Agree on a rhythm, for example monthly or quarterly. Before each update, run the frozen evaluation set on the new version and only roll it out if nothing got worse.

Backups of the weights. Weights are files. Keep every version with its hash, next to the version of the dataset and the evaluation set it was measured on. Two copies, one of them elsewhere. Rolling back then means swapping a file.

Logging. Decide before go-live what you record: model version, time, user, input and output, or only their hashes when the content is sensitive. Logs are personal data too, so set who can read them and how long you keep them.

GDPR and the AI Act: what on-premise solves and what it does not

What it solves. Data is not transferred to a third country and there is no foreign sub-processor your data protection officer has to justify. For health data, banking secrecy or trade secrets, "the data never left" can be checked rather than promised.

What it does not solve.

• GDPR still applies in full: legal basis, purpose, minimisation, retention, access rights, and an impact assessment where the risk is high. Personal data you train on can end up in the weights, so we anonymise or pseudonymise before training.
• The AI Act does not care where the server stands. Your role and the risk class depend on what the system is used for. AI literacy (Article 4) and disclosure for chatbots (Article 50) apply the same way. If you put the system into service under your own name, you can be its provider; we are the supplier. More in our note on what companies must do by 2 December 2026.
• On-premise is not a legal requirement. For ordinary personal data, an EU-hosted provider under a data-processing agreement is lawful.

This is not legal advice. For your case, ask your DPO or a lawyer.

The cascade: when data leaves the house anyway

Most requests go to your small local model. When it is unsure, the format is unknown or the question is outside its domain, the request can go to a large frontier model through an API. That is the moment data leaves the house, so decide it on paper first:

• which kinds of data may never go to the fallback;
• whether to pseudonymise before sending and restore names afterwards;
• that every fallback call is logged and counted.

The share of fallback calls is useful in itself: it shows where the small model needs more examples. We pass frontier API calls through at cost. If no data may leave at all, there is no fallback and the case goes to a person.

When an EU cloud is the better choice

Your own server is not always the right answer. An EU-hosted service is usually better when:

• volume is low. At a few hundred calls a month an API is often cheaper than running a machine;
• nobody in the company wants to maintain a server;
• the load comes in spikes;
• the data is ordinary personal data and a processing agreement is enough;
• you need a large general model for broad questions, not a narrow task.

We also host your trained model in the EU, from €600 a month. The weights stay yours, and you can move them in-house later.

What it costs

Prices from our public price list, in EUR, excluding VAT:

What	Price
One-day evaluation on your documents	€1,900, credited against a pilot
One-week trial under NDA	billed as Pilot S (€3,900), credited
Pilot M, when the data has to be built	€12,000, 4–5 weeks
Full program	from €20,000
Monthly retainer (retraining, re-evaluation)	from €1,200
On-site day (install, air-gapped hand-over)	€1,500 + travel

The hardware you buy yourself, from whoever you like. We do not resell it.

When not to call us. If a few people need a general assistant for writing e-mails, a business subscription to a large model is simpler. If you have no repeated task with examples, start by collecting fifty of them. Then we can measure something.

A one-week trial under NDA

You choose fifty anonymised examples and one task. Within a week you get a model running on your server, numbers before and after training, and a written recommendation. If the result is not good, it stops there.

Frequently asked questions

Can a private LLM run with no internet at all? Yes. Everything we deliver works with the network cable unplugged. Updates arrive on a drive.

How large a model does a company need? For narrow tasks, a 4B to 12B model trained on your examples. 27B to 35B for open-ended questions, long documents or agents with many tools.

Does GDPR require an on-premise model? No. An EU provider under a processing agreement is lawful for ordinary personal data. On-premise matters most for health data, banking secrecy and trade secrets.

GGUF or vLLM? GGUF for a few users, laptops and offline work. vLLM for servers with many simultaneous users.

What does it cost to start? A one-day evaluation is €1,900. A one-week trial is billed as Pilot S (€3,900). Both are credited against what follows.