inferno.rs blog

Act 3: Recap

chat-server is a working server now. It speaks OpenAI's HTTP and SSE protocols, so any OpenAI client connects to it unmodified. It admits concurrent conversations through a background scheduler, stores KV memory in fixed-size paged blocks so appends never reallocate, reuses cached prompt prefixes across requests so a shared system prompt is prefilled once, and fuses concurrent decode steps into one batched forward pass.

What you shipped in Act 3

  • A chat pipeline (III.1) that loads the model's trained prompt format (its Jinja chat template) from GGUF metadata, renders a message list through minijinja, and drives one chat turn end to end, streaming text deltas through a callback. Smoke-tested with the interactive chat-repl binary.
  • An OpenAI-compatible HTTP API (III.2) built on axum and tokio: GET /health, GET /v1/models, and non-streaming POST /v1/chat/completions, with request and response JSON shaped exactly like OpenAI's. The chat-server binary.
  • SSE streaming (III.3): stream: true returns a live sequence of chat.completion.chunk events terminated by [DONE], so a client watches the reply appear token by token.
  • A paged KV cache (III.4) replacing the contiguous, reallocating cache from II.2: a pool of fixed-size blocks, a per-layer page table, O(1) append with no per-step reallocation or copy (the pool is per-request here; a production engine shares one pool across all requests). Selectable at runtime with --kv basic|paged.
  • A radix prefix cache (III.5): a tree keyed on token ids storing KV snapshots at terminal nodes, with LRU eviction and RAII pins that keep an entry resident while a request reads it. Shared prompt prefixes are prefilled once and reused.
  • A decode scheduler (III.6): a background worker thread owning the model, tokenizer, backend, and prefix cache, holding a fixed set of slots, admitting jobs and interleaving their decode steps. The HTTP handlers submit jobs and await results over channels.
  • Batched decode (III.7): when two or more slots decode together, their forward passes fuse into one: projections and the MLP become single matmuls with a real batch dimension, attention stays a per-slot loop. A GuideLLM A/B benchmark measures the throughput gain.

Together these turn the engine from a fast single-user CLI into a multi-tenant server you can point an off-the-shelf chat client at.

The whole journey

Three acts, one consistent arc:

  • Act 1 made it work. From a GGUF file on disk to a model-generate CLI: a binary parser, a BPE tokenizer, a Tensor type, a Backend trait with a scalar CPU implementation, the 28-layer Qwen3 forward pass, and the greedy autoregressive loop. Slow (about a second per token, and worsening as the sequence grew) but every number understood.
  • Act 2 made it fast. A benchmark harness to measure before optimizing, a KV cache to kill quadratic decode, then three faster backends behind the same Backend trait (SIMD, multithreaded, Metal GPU) and Q8_0 quantized weights to quarter the memory footprint. The same model, an order of magnitude quicker.
  • Act 3 made it serve. Chat templates, an OpenAI HTTP and SSE API, a paged KV cache, a radix prefix cache, a decode scheduler, and batched decode. The same fast engine, now answering many users at once.

Every act reacted to the one before it. Act 2's KV cache only mattered because you had watched Act 1's forward pass crawl. Act 3's paged cache only mattered because you had built Act 2's contiguous one and seen it recopy the whole history on every appended token. Nothing was abstract.

What's still missing

This is a small-but-real inference engine, not a clone of vLLM or SGLang. Several things production engines do, this one does not:

  • More quantization formats. II.6 fused Q8_0 dequantization into the matmul, but that's one format; a production engine also supports Q4 and FP8, with far more heavily tuned kernels. For anything larger than 0.6B, that is table stakes.
  • Speculative decoding. A small draft model proposes several tokens, the target model verifies them in one pass, often 2-3× the decode throughput. Not built.
  • Chunked prefill. A new request's prefill currently runs synchronously inside admit(), on the worker thread; while it runs, every active conversation's decode stalls. Splitting prefill into chunks so it interleaves with other requests' decode steps is the next scheduler refinement.
  • Multi-GPU. The engine runs on one device. A 70B-class model needs tensor parallelism across several.
  • Structured output and tool calls. Constrained decoding (JSON mode, grammars) and function-calling APIs: all unbuilt.
  • Production robustness. Graceful shutdown under load, backpressure (slowing or rejecting clients when the server is saturated), admission control, per-tenant quotas, multi-model hosting. Everything a real API needs and a tutorial codebase skips.

What you learned

More than the specific code: you learned the shape of the problem. Every production inference engine in 2026 is a variation on the primitives you built here: a tensor type, a backend abstraction, a KV cache, a paged allocator, a prefix cache, a scheduler, a batched forward pass. Open the vLLM, SGLang, or TGI source and it is no longer foreign: it's a matter of mapping their names onto yours.

You started with a bag of numbers in a binary file. You finished with a server that streams tokens to concurrent users over an OpenAI-compatible API, and you wrote every line between.

Continue to Where to from here.