Performance Validation: Latency and Throughput Gates

PerformanceValidation measures latency, throughput, memory, tokens per second, and TTFT. Statistical protocol, regression gate, and flowchart specified.

PerformanceValidation measures the runtime behavior of a deployed model. This page specifies the metrics, the measurement protocol, the statistical criteria for acceptance, and the regression gate that prevents degraded models from reaching production.

Metrics

Metric	Definition	Unit
Latency	End-to-end request duration	milliseconds
Throughput	Total tokens generated per second	tokens/sec
Memory	Peak GPU or RAM usage during inference	GiB
Tokens/sec	Generation speed per request	tokens/sec
TTFT	Time to first token	milliseconds

All metrics are collected under controlled load conditions to ensure comparability across runs.

Measurement Flowchart

The validation pipeline runs benchmarks, collects samples, computes statistics, and applies the regression gate.

Statistical Protocol

PerformanceValidation uses a bootstrap confidence interval protocol to reduce noise and false positives.

1. Minimum sample size: 100 requests per load profile. (Engineering assumption)
2. Warmup: 10 requests executed and discarded before measurement.
3. Bootstrap: 10,000 resamples with replacement to compute a 95% CI for the mean of each metric.
4. Comparison: The new model passes if the upper bound of its 95% CI is below the baseline mean plus the regression threshold.

Metric	Regression Threshold	Direction
Latency	+10%	Higher is worse
Throughput	-10%	Lower is worse
Memory	+10%	Higher is worse
Tokens/sec	-10%	Lower is worse
TTFT	+10%	Higher is worse

Threshold values are engineering assumptions. Final values will be calibrated in OQ-006 after reference benchmark runs.

Regression Gate

The regression gate compares the new candidate against the current production baseline. If any metric exceeds its regression threshold, the candidate fails and cannot proceed to canary or production.

• Failure is recorded in ExperimentTracker with full sample distributions.
• The candidate status is set to REGRESSION.
• DeploymentManager blocks promotion.

Load Profiles

Profile	Concurrent Requests	Request Length	Use Case
Single	1	512 tokens	Latency baseline, TTFT
Moderate	8	1024 tokens	Throughput, memory pressure
Heavy	32	2048 tokens	Saturation, stability

Every candidate is tested against all three profiles. Results are stored in ArtifactStore and linked to the model lineage.

Integration

PerformanceValidation is invoked by DeploymentManager during staging and by Offline Deployment during acceptance testing. It reads models from ArtifactStore and reports results to Observability.