Single-stream chat and coding
One long session at a time. Watch time to first token and decode speed.
Local deployment
Choose an inference engine for a local model
An inference engine decides how a model fits in GPU memory, how requests are batched, how long a context can be, and how stable tool calls and structured output are. The same weights can change speed and quality limits when moved to another engine.
What this covers
The comparison is about the engine itself: scheduling, quantization support, context and KV management, speculative decoding, API shape, and the class of machine it fits. A public score from one engine is not a quality result for another.
Concurrent requests
Several requests at once. Watch continuous batching and prefix cache.
Long material
Where the KV lives, and whether experts or cold weights can sit in system memory.
Integration
Whether it offers an OpenAI-compatible API, and whether tool calls and JSON stay stable.
How to choose
Fix the hardware and the weight format first. The engine follows from those two constraints.
Look at the GPU and its memory before the engine
On a consumer card, prefer engines that can run NVFP4, FP8, or GGUF. On a data-center card, a compiled stack such as TensorRT-LLM becomes reasonable.
One user and several users are different problems
A single stream wants low latency and a long context. Many streams want continuous batching, prefix reuse, and fair scheduling.
The quantization format narrows the candidates
GGUF goes to a llama.cpp-class runtime. Official NVFP4 / FP8 exports more often go to SGLang, vLLM, or a purpose-built engine.
Run only one service that fills the GPU
Two engines loaded on the same card usually only contend for memory. They do not go faster.
By use case
The same kind of task does not have to be tied to one engine. The three groups below are ways of using a model, not a fixed deployment on any particular machine.
SGLang
Serving framework · prefix cache and structured output
Built for high-throughput serving. RadixAttention keeps KV for identical prefixes in a radix tree, so repeated system prompts, multi-turn chats, and batch evaluations can skip part of prefill. Conservatively quantized official exports can be compared for quality here. Community weights with a wider quantized surface buy a longer context, and should not be turned around and used as the default for exact work.
NInfer
Purpose-built runtime · single-stream decode speed
A runtime tuned for single-request generation. A common recipe keeps sensitive layers at higher precision, quantizes the remaining MLPs to NVFP4, and uses MTP speculative decoding to raise decode speed. It fits one person waiting on one long reply. It does not replace a multi-request serving framework.
Strata
Purpose-built runtime · mixture-of-experts and long material
Used to run mixture-of-experts models. Active experts can sit in system memory, leaving GPU memory for attention and KV. The cost is a slower first token on long prompts, and more disagreement with higher-precision weights. It serves a larger class of model. It is not a slow substitute for a small dense model.
DSpark is a draft accelerator attached to SGLang, not a fourth engine. It changes generation speed, not model quality. Tool calls and strict JSON often do not take that path.
Common engines
SGLang
A Python serving framework with an OpenAI-compatible API by default. Its strengths are prefix cache, continuous batching, and constrained decoding. For local deployment, it fits a coding assistant or evaluation service that repeats one system prompt, has occasional concurrency, and needs structured output. Official low-bit exports are often checked for quality here first.
vLLM
The most widely used high-throughput serving framework. PagedAttention manages KV in pages, continuous batching is mature, and it covers many models and quantization backends. It is usually the default candidate for several people sharing one machine, an OpenAI-compatible gateway, or a rotation of Hugging Face weights. Compared with SGLang, the choice is more about model coverage and operational familiarity than about being faster on every consumer card.
TensorRT-LLM
NVIDIA's compiled engine. Weights are compiled into an engine and then run on the matching GPU architecture. Peak throughput and the FP8 / NVFP4 kernels are often the best. The cost is a long compile path: a new card or version means compiling again, which is a poor fit for day-to-day iteration on a consumer card. Consider it when the card is a data-center GPU, the model is fixed, and batch throughput is the goal.
llama.cpp
A C/C++ runtime whose main format is GGUF. It runs on CPUs, integrated graphics, Apple silicon, and NVIDIA consumer cards. Quantization steps are fine-grained, and distribution can be a single file. It fits personal trials, models without an official NVFP4 export, and cases where layers must spill to memory because GPU memory is short. High concurrency and prefix cache are not its main ground.
Ollama
A local application wrapped around a llama.cpp-class runtime. One command pulls a model, picks a quantization, and exposes a local API. It fits people who will not edit launch flags. Once the quantized surface, context limit, or a quality comparison against another service must be locked, go back to the underlying runtime instead of staying on its defaults.
LM Studio / other desktop shells
A graphical interface for loading GGUF or a compatible format. It is useful for checking whether a model can hold a conversation. It is a poor reproducible service baseline: context, GPU offload, and sampling parameters can change with the interface version.
NInfer and Strata
These two are purpose-built runtimes for particular weights and hardware paths, not another general-purpose server. NInfer focuses on single-stream decode speed. Strata focuses on MoE capacity. A new model should not be dropped in by default. First confirm that its quantization recipe and layer types are supported.
Comparison
| Engine | Main format | Better for | Not a substitute for |
|---|---|---|---|
| SGLang | HF / NVFP4 / FP8 | Quality baseline, repeated prefixes, structured output | A GGUF model with no matching export |
| NInfer | Mixed NVFP4 + FP8 | Single-stream coding, decode speed | A concurrent gateway |
| Strata | Quantized MoE weights | Long material, edits across many files | A general accelerator for a 27B dense model |
| vLLM | HF and several quantizations | Shared use, many model families | The only speed answer on a consumer card |
| TensorRT-LLM | A compiled engine | Peak throughput after the model is fixed | A personal setup that changes weights daily |
| llama.cpp | GGUF | Single-machine trials, CPU or integrated graphics, fine quantization | High-concurrency production serving |
| Ollama | Mostly GGUF | A quick local trial | A reproducible quality comparison |
Recommendations
| Situation | First choice | Why |
|---|---|---|
| Exact code, protocol implementation, financial analysis | SGLang + conservatively quantized weights | A narrower quantized surface, suitable as a quality reference |
| One person developing, decode speed first | NInfer | Mixed precision plus MTP, aimed at single-stream generation |
| A very long repository, or edits across many files | A MoE runtime such as Strata | Trade memory capacity for context instead of forcing a dense-model window to its maximum |
| Material too long for conservative quantization | SGLang + weights with a wider quantized surface | Extra numeric error in exchange for window length. Not the default for exact work. |
| Several people sharing one machine | vLLM or SGLang | Continuous batching and an OpenAI-compatible API fit better than a desktop shell |
| No NVIDIA card, or a conversation is enough for now | llama.cpp / Ollama | Broad GGUF coverage and a low setup cost |
| The model is fixed and peak batch throughput is required | TensorRT-LLM | Compile once, and compile again when the architecture changes |
Before comparing speed on a new model, fix four things: weight version, quantized surface, context limit, and sampling parameters. Leave any one out and the engine comparison cannot be reproduced.
Limits
A stated context length is not a recommendation. A longer window raises prefill time and can slow decoding. A task that fits a shorter context does not need the maximum window.
Speculative decoding and draft models only speed up generation. Tool calls, strict JSON, and some constrained decoding can bypass that path, and speed falls back to the base model.
Putting layers or experts in system memory buys capacity, but time to first token and bandwidth become the new limits.
Public leaderboard scores cannot be subtracted across engines or quantizations. Order of use should come from the same tasks and the same prompts, not from scores of different origins.
A desktop shell can change its default context and offload. For a service meant to stay in use, write the launch flags into a fixed configuration instead of remembering them in the interface.