Technology · Model serving
vLLM
PagedAttention · Batching · OpenAI-compatible API
vLLM is an inference server for open-weight models, built to keep expensive GPUs busy. Through continuous batching and efficient memory handling it serves many concurrent requests on hardware that a naive setup would exhaust after a handful — which is what makes self-hosting a commercial proposition rather than a demonstration.
How TrivialWorks uses it.
We reach for vLLM when data residency forbids a commercial API, or when volume is high and steady enough that owning inference is genuinely cheaper than renting it. It exposes an OpenAI-compatible endpoint, so an application written against a commercial provider can move behind it with little change. We size the hardware against measured load before recommending it — the decision is arithmetic, and the arithmetic frequently favours the API.
Decision guide
Should your project use vLLM?
Practical selection guidance — the conversation we would have with you before writing a line of code.
When it’s the right choice
- Regulated or sovereign workloads where content cannot leave your infrastructure at all
- High, sustained request volume where per-token API pricing has overtaken the cost of owning GPUs
- Fine-tuned or domain-specific open-weight models that no commercial provider hosts for you
- Predictable latency requirements that a shared multi-tenant API cannot contractually guarantee
When it isn’t
- Low or spiky volume — idle GPUs bill continuously while an API bills per token, and the break-even is further away than most teams assume
- Teams with no appetite for operating GPU infrastructure: capacity planning, driver versions and model updates become your responsibility permanently
- Products that need frontier-model quality, which open weights do not yet match on the hardest reasoning tasks
Consider Commercial model APIs — Volume is moderate or uneven — you pay only for what you use and somebody else runs the hardware at three in the morning.
Consider Ollama — You want local models for development or light internal use rather than production concurrency.
Best use cases
Where vLLM makes practical sense.
Sovereign and regulated deployment
Models served inside your own network, so documents and queries never cross a boundary your compliance obligations forbid.
High-volume inference economics
Steady, heavy workloads moved off per-token pricing once measured demand shows ownership costs less than renting.
Serving fine-tuned models
Domain-adapted open-weight models put into production behind a standard API surface, with versioning and rollback.
Migration without rewriting
An OpenAI-compatible endpoint dropped in behind an existing application, so the switch is configuration rather than a project.
Proof
Where it shows up in our work.
No published case study names this stack yet — most of our work ships under NDA, and we would rather show you nothing than invent something. The work ledger shows how we handle that honestly.
Related services
Services that commonly use it.
Questions
vLLM, asked straight.
Is self-hosting actually cheaper than using an API?
Sometimes, and later than people expect. A GPU bills whether or not it is serving anything, so self-hosting wins on sustained high volume and loses on spiky or modest load. We model it against your measured traffic before recommending either — and we have talked clients out of self-hosting more often than into it.
What hardware does it need?
It depends on the model's parameter count, the precision you serve at and your concurrency target — the ceiling is usually GPU memory rather than compute. We size it from a load profile, then validate against real traffic, because the first estimate is always the optimistic one.
Are open-weight models good enough for our use case?
Often, and it is testable rather than a matter of opinion. For retrieval-grounded answering, classification and extraction, current open models are frequently sufficient. For the hardest reasoning, frontier models still lead. We run your evaluation set against both and let the numbers decide.
Can we use both self-hosted and commercial models?
Yes, and it is a sound default — sensitive workloads served locally, everything else routed to whichever commercial model is best value that quarter. A routing layer in front of both makes that a policy decision rather than an architectural one.
Thinking about vLLM?
Send the requirement and you get back a functional specification — screens, data model, stack and an estimate — at no charge. If vLLM is the wrong choice for it, that will be in there too.