Inference Backend: Serving Engine Selection

InferenceBackend specifies PyTorch, vLLM, TGI, llama.cpp, and ONNX Runtime. Backend selection, batching, KV cache, and speculative decoding are covered.

InferenceBackend selects and configures the execution engine for model serving. This page specifies the supported backends, the selection decision tree, latency and throughput profiles, and optimization features including batching, KV cache management, and speculative decoding.

Supported Backends

Backend	Execution Mode	Best For	GPU Required
PyTorch eager	Python, dynamic	Research, prototyping, custom ops	Optional
vLLM	PagedAttention, continuous batching	High-throughput GPU serving	Yes
TGI (Text Generation Inference)	Rust + Python, flash attention	Production Hugging Face models	Yes
llama.cpp	C/C++, quantized	CPU, edge, mobile	No
ONNX Runtime	Graph optimization, cross-platform	Standardized deployment, interoperability	Optional

Backend Selection Decision Tree

The selection logic balances hardware, latency requirements, throughput targets, and model format.

Latency and Throughput Profiles

The table below shows placeholder targets for each backend. Values are engineering assumptions pending benchmarking.

Backend	TTFT Target	Throughput Target	Memory per 7B
PyTorch eager	EXP-TBD-001	EXP-TBD-002	EXP-TBD-003
vLLM	EXP-TBD-004	EXP-TBD-005	EXP-TBD-006
TGI	EXP-TBD-007	EXP-TBD-008	EXP-TBD-009
llama.cpp	EXP-TBD-010	EXP-TBD-011	EXP-TBD-012
ONNX Runtime	EXP-TBD-013	EXP-TBD-014	EXP-TBD-015

All latency and throughput values are marked EXP-TBD and will be populated after benchmark runs on reference hardware.

Batching Strategies

Strategy	Backend Support	Description
Static batching	All	Fixed batch size, simple but inefficient for variable load
Dynamic batching	vLLM, TGI, ONNX	Batches requests arriving within a time window
Continuous batching	vLLM, TGI	Adds new requests to running batches as others complete

Continuous batching is preferred for high-throughput serving. InferenceBackend configures the batching mode via backend-specific parameters passed from DeploymentManager.

KV Cache and Prefix Cache

The KV cache stores key and value tensors from prior tokens to avoid recomputation during autoregressive generation. InferenceBackend manages cache allocation, eviction, and reuse.

• PagedAttention (vLLM): Allocates KV cache in fixed-size blocks, enabling memory sharing and reducing fragmentation.
• Prefix cache: Reuses KV cache for shared prompt prefixes across multiple requests. Enabled by default in vLLM and TGI when prompts share a common system message or instruction prefix.

Speculative Decoding

Speculative decoding accelerates generation by drafting future tokens with a smaller, faster model and verifying them in parallel with the target model. InferenceBackend supports speculative decoding when a draft model is registered in ModelRegistry and the target backend implements verification.

• Draft model must be compatible in tokenizer vocabulary and architecture.
• Verification accepts or rejects draft tokens in blocks.
• Rejected tokens trigger standard autoregressive fallback.

Integration

InferenceBackend is configured by DeploymentManager during environment setup. It reads model artifacts from ArtifactStore and reports metrics to Observability. Backend health is monitored by GPU Health and Monitoring.