Quantization Engine: Precision Reduction for Inference

QuantizationEngine reduces model weights to FP16, BF16, FP8, INT8, or INT4. This page specifies the decision tree, flowchart, and quality gates.

QuantizationEngine compresses model weights and activations to lower precision formats for faster inference and smaller memory footprints. This page specifies the supported methods, the selection decision tree, and the validation flow that prevents quality regression.

Supported Methods

Method	Precision	Use Case	Notes
FP16	16-bit float	General GPU inference	Widely supported, minimal range loss
BF16	16-bit bfloat	Ampere+ GPUs	Wider dynamic range than FP16
FP8	8-bit float	Hopper+ GPUs, throughput	NVIDIA transformer engine support
INT8	8-bit integer	Edge, CPU, latency-sensitive	Requires calibration; static or dynamic scaling
INT4	4-bit integer	Maximum compression, memory-bound	Group-wise or block-wise; GPTQ/AWQ style (Engineering assumption)

FP8 and INT4 support depend on hardware availability and backend implementation. The engine falls back to the nearest supported precision if the target backend cannot execute the chosen format.

Decision Tree

The engine selects a quantization method based on target hardware, latency requirements, memory constraints, and quality tolerance. The decision tree below shows the selection logic.

Quantization Flowchart

Every quantization job follows the same pipeline. The flowchart below shows the full lifecycle from model input to acceptance or rejection.

Quality Gates

Pre-quant and post-quant benchmarks run on the same dataset split to ensure comparability. The engine computes perplexity, accuracy, and task-specific metrics. A regression gate compares post-quant results against pre-quant baselines and against the thresholds defined in OQ-006.

Gate	Metric	Action on Failure
Integrity Check	SHA-256 hash match, layer count, tensor shape	Reject and retry with same config
Perplexity Regression	Perplexity delta > OQ-006 threshold	Reject and retry with higher precision
Accuracy Regression	Accuracy delta > OQ-006 threshold	Reject and retry with higher precision

Threshold values in OQ-006 are placeholders pending experimental calibration. Do not deploy quantized models in production until thresholds are finalized.

Retry Policy

On rejection, the engine retries with the next higher precision in the hierarchy: INT4 → INT8 → FP8 → BF16 → FP16. After three retries, the job fails and the model remains in its original precision. The retry count and final state are logged to ExperimentTracker.

Integration

QuantizationEngine is invoked by DeploymentManager before staging and by Offline Deployment when building secure transfer packages. It reads from ArtifactStore and writes quantized artifacts back to the same store with a new version tag.