EMEP Performance Testing and Metrics

Latency, throughput, memory, tokens per second, and TTFT performance tests for EMEP with cross-links to deployment validation.

Performance testing measures how EMEP components behave under load. Tests cover inference latency, throughput, memory consumption, tokens per second, and time to first token (TTFT). Results are compared against thresholds defined in Performance Validation.

Test Categories

Latency

Measure end-to-end latency for:

• Model registration and validation.
• Compatibility analysis.
• Merge execution.
• Benchmark evaluation.
• Candidate promotion.

Latency is reported as p50, p95, and p99.

Throughput

Measure jobs completed per minute for:

• Concurrent merge jobs.
• Concurrent benchmark runs.
• Concurrent evolution generations.

Throughput tests use a fixed GPU pool and measure saturation behavior.

Memory

Measure peak memory usage for:

• Model loading (full precision and quantized).
• Merge execution (intermediate tensors).
• Benchmark inference (batch size sweep).

Memory is reported in GiB and as a fraction of available GPU memory.

Tokens Per Second

Measure generation throughput for InferenceBackend across batch sizes and sequence lengths. Report prefill and decode phases separately.

Time to First Token (TTFT)

Measure latency from request submission to first generated token. TTFT is critical for interactive use cases.

Test Matrix

Metric	Batch Size	Sequence Length	Dtype	Backend
Latency	1	512	bf16	vLLM
Throughput	1, 4, 8, 16	512	bf16	vLLM
Memory	1, 4, 8	512, 2048	bf16, fp8	vLLM
Tokens/sec	1, 4, 8	512, 2048	bf16	vLLM
TTFT	1	512, 2048	bf16	vLLM

Load Testing

Load tests simulate multiple concurrent users:

1. Ramp up requests over 60 seconds.
2. Hold peak load for 300 seconds.
3. Ramp down over 60 seconds.
4. Measure error rate, latency percentiles, and resource utilization.

Baselines and Thresholds

Performance baselines are established on a reference hardware configuration. Deviations greater than 10% relative trigger investigation. Thresholds are documented in Performance Validation.

Integration Points

• GPU Orchestration: performance tests validate GPU scheduling and memory limits. See GPU Orchestration.
• Inference Backend: tests exercise InferenceBackend under load. See Inference.
• Deployment: performance results feed into deployment decisions. See Performance Validation.
• CI: performance tests run weekly and on release candidates.
• Monitoring: metrics are exported to the observability stack. See Monitoring.