Technology · Model serving
Ollama
Local runtime · Model library · OpenAI-compatible API
Ollama runs open-weight models on ordinary hardware with a single command. Its value is not raw throughput but access: a model on a laptop or a modest server, behind a standard API, with nothing leaving the machine. That makes two things practical — genuinely private inference, and development against a real model without a meter running.
How TrivialWorks uses it.
Two ways. During development we point work at a local model so engineers can iterate on prompts and retrieval without an API bill or a rate limit shaping the design. And for clients whose data cannot leave the building at all — air-gapped estates, on-premise deployments, sensitive internal tooling — Ollama is often the shortest honest path to a working system. For production concurrency we move to vLLM; Ollama is not built to serve a queue.
Decision guide
Should your project use Ollama?
Practical selection guidance — the conversation we would have with you before writing a line of code.
When it’s the right choice
- Air-gapped or on-premise environments where no external API call is permissible
- Development and evaluation against real models without per-token cost distorting how the team experiments
- Internal tools with modest, bursty usage where a single machine comfortably covers demand
- Proof-of-concept work that must demonstrate feasibility on the client's own hardware before budget is committed
When it isn’t
- Production workloads with real concurrency — it is not designed for high-throughput batched serving, and it will not pretend otherwise under load
- Applications needing frontier-model quality, which local open weights on commodity hardware do not reach
- Teams without anyone to maintain the machine: a local model is infrastructure, and it acquires an owner whether or not one is nominated
Consider vLLM — The same privacy requirement, but with production concurrency — vLLM is built for throughput, Ollama is not.
Consider Commercial model APIs — Data residency is not a constraint — you get better models with no hardware to own or patch.
Best use cases
Where Ollama makes practical sense.
Air-gapped deployment
Assistants and document tooling running entirely inside a closed network, where an outbound API call is not an option.
Zero-cost development loop
Prompt and retrieval iteration against a real model locally, so experimentation is shaped by the problem rather than the invoice.
On-premise internal tooling
Summarisation, drafting and classification for internal teams, served from a machine the client already owns.
Feasibility proofs on client hardware
Demonstrating that a workload runs acceptably on infrastructure they have, before anyone commits to buying more.
Proof
Where it shows up in our work.
No published case study names this stack yet — most of our work ships under NDA, and we would rather show you nothing than invent something. The work ledger shows how we handle that honestly.
Related services
Services that commonly use it.
Questions
Ollama, asked straight.
Can Ollama serve a production application?
For low, bursty internal load, yes. For concurrent production traffic, no — it is not built for batched high-throughput serving and degrades under load. When a private deployment needs real concurrency we use vLLM, and we say so before the pilot rather than after it.
Does anything leave our network?
No. Inference runs on your hardware and prompts never traverse an external boundary. Models are downloaded once from a public registry, which can be done ahead of time and moved in manually for genuinely air-gapped estates.
Which models can we run, and how good are they?
The open-weight families, sized to your hardware. For retrieval-grounded answering, classification and extraction they are frequently good enough; for the hardest reasoning they trail the frontier. We test against your evaluation set rather than asking you to take a view on it.
What hardware do we need?
Less than most people expect for smaller models — a well-specified workstation handles useful work, with GPU memory the binding constraint as models grow. We size it against the specific models and load you need, and it is often the cheapest way to prove the concept before committing to serving infrastructure.
Thinking about Ollama?
Send the requirement and you get back a functional specification — screens, data model, stack and an estimate — at no charge. If Ollama is the wrong choice for it, that will be in there too.