> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference Backend: Serving Engine Selection

> InferenceBackend specifies PyTorch, vLLM, TGI, llama.cpp, and ONNX Runtime. Backend selection, batching, KV cache, and speculative decoding are covered.

InferenceBackend selects and configures the execution engine for model serving. This page specifies the supported backends, the selection decision tree, latency and throughput profiles, and optimization features including batching, KV cache management, and speculative decoding.

## Supported Backends

| Backend                         | Execution Mode                      | Best For                                  | GPU Required |
| ------------------------------- | ----------------------------------- | ----------------------------------------- | ------------ |
| PyTorch eager                   | Python, dynamic                     | Research, prototyping, custom ops         | Optional     |
| vLLM                            | PagedAttention, continuous batching | High-throughput GPU serving               | Yes          |
| TGI (Text Generation Inference) | Rust + Python, flash attention      | Production Hugging Face models            | Yes          |
| llama.cpp                       | C/C++, quantized                    | CPU, edge, mobile                         | No           |
| ONNX Runtime                    | Graph optimization, cross-platform  | Standardized deployment, interoperability | Optional     |

## Backend Selection Decision Tree

The selection logic balances hardware, latency requirements, throughput targets, and model format.

```mermaid theme={null}
flowchart TD
    START([START]) --> INPUT[INPUT: Model Format, Hardware, Latency Target, Throughput Target]
    INPUT --> GPU{GPU Available?}
    GPU -->|NO| CPU_PATH{Latency < 100ms?}
    GPU -->|YES| GPU_PATH{Throughput > 1000 tok/sec?}
    CPU_PATH -->|YES| ONNX[ONNX Runtime]
    CPU_PATH -->|NO| LLAMA[llama.cpp]
    GPU_PATH -->|YES| VLLM[vLLM]
    GPU_PATH -->|NO| TGI_PATH{Hugging Face native?}
    TGI_PATH -->|YES| TGI[TGI]
    TGI_PATH -->|NO| PYTORCH[PyTorch eager]
    ONNX --> OUTPUT[OUTPUT: Selected Backend + Config]
    LLAMA --> OUTPUT
    VLLM --> OUTPUT
    TGI --> OUTPUT
    PYTORCH --> OUTPUT
    OUTPUT --> END([END])
```

## Latency and Throughput Profiles

The table below shows placeholder targets for each backend. Values are engineering assumptions pending benchmarking.

| Backend       | TTFT Target | Throughput Target | Memory per 7B |
| ------------- | ----------- | ----------------- | ------------- |
| PyTorch eager | EXP-TBD-001 | EXP-TBD-002       | EXP-TBD-003   |
| vLLM          | EXP-TBD-004 | EXP-TBD-005       | EXP-TBD-006   |
| TGI           | EXP-TBD-007 | EXP-TBD-008       | EXP-TBD-009   |
| llama.cpp     | EXP-TBD-010 | EXP-TBD-011       | EXP-TBD-012   |
| ONNX Runtime  | EXP-TBD-013 | EXP-TBD-014       | EXP-TBD-015   |

<Note>
  All latency and throughput values are marked EXP-TBD and will be populated after benchmark runs on reference hardware.
</Note>

## Batching Strategies

| Strategy            | Backend Support | Description                                                |
| ------------------- | --------------- | ---------------------------------------------------------- |
| Static batching     | All             | Fixed batch size, simple but inefficient for variable load |
| Dynamic batching    | vLLM, TGI, ONNX | Batches requests arriving within a time window             |
| Continuous batching | vLLM, TGI       | Adds new requests to running batches as others complete    |

Continuous batching is preferred for high-throughput serving. InferenceBackend configures the batching mode via backend-specific parameters passed from [DeploymentManager](/deployment/deployment-specification).

## KV Cache and Prefix Cache

The KV cache stores key and value tensors from prior tokens to avoid recomputation during autoregressive generation. InferenceBackend manages cache allocation, eviction, and reuse.

* **PagedAttention** (vLLM): Allocates KV cache in fixed-size blocks, enabling memory sharing and reducing fragmentation.
* **Prefix cache**: Reuses KV cache for shared prompt prefixes across multiple requests. Enabled by default in vLLM and TGI when prompts share a common system message or instruction prefix.

## Speculative Decoding

Speculative decoding accelerates generation by drafting future tokens with a smaller, faster model and verifying them in parallel with the target model. InferenceBackend supports speculative decoding when a draft model is registered in [ModelRegistry](/tracking/model-registry) and the target backend implements verification.

* Draft model must be compatible in tokenizer vocabulary and architecture.
* Verification accepts or rejects draft tokens in blocks.
* Rejected tokens trigger standard autoregressive fallback.

## Integration

InferenceBackend is configured by [DeploymentManager](/deployment/deployment-specification) during environment setup. It reads model artifacts from [ArtifactStore](/tracking/artifact-registry) and reports metrics to [Observability](/architecture/observability). Backend health is monitored by [GPU Health](/operations/gpu-health) and [Monitoring](/operations/monitoring).