EMEP Evaluation Testing and Benchmark Harness

Benchmark harness tests, deterministic seeds, and cross-backend validation for EMEP EvaluationEngine and BenchmarkEngine.

Evaluation testing validates that EMEP produces consistent, comparable benchmark scores across runs, backends, and model variants. The benchmark harness is the core artifact under test.

Benchmark Harness Tests

The harness test suite verifies:

• Correct prompt formatting for each benchmark suite.
• Accurate metric computation (perplexity, accuracy, F1, BLEU, etc.).
• Timeout handling for long-running evaluations.
• Graceful degradation when a backend returns an error.

Deterministic Seeds

Every benchmark run uses a fixed random seed for:

• Dataset shuffling and sampling.
• Backend initialization (where applicable).
• Dropout and sampling during generation.

The seed is logged in ExperimentTracker. Re-running the same benchmark with the same seed must produce identical scores within the tolerance defined in Numerical Testing.

Cross-Backend Validation

Critical benchmarks are run on multiple InferenceBackend implementations to detect backend-specific drift:

Backend	Primary Use	Cross-Backend Target
vLLM	Production benchmarks	Run subset on Hugging Face Transformers
Hugging Face Transformers	Reference implementation	Run subset on vLLM

Scores must agree within 1% relative across backends. Disagreements trigger an investigation into implementation differences.

Benchmark Splits

Tests use the same split definitions as production:

• Optimization Set: used for model selection and evolution. Available to EvolutionEngine.
• Validation Set: used for final candidate ranking. Available to EvaluationEngine.
• Hidden Test Set: never used in testing or development. Reserved for independent audit.

Hidden Test Set is never used by EvolutionEngine, EvaluationEngine, or any test suite.

Regression Detection

Benchmark regression tests compare scores against a historical baseline. A regression is defined as a drop greater than 2% relative on any metric. Regressions block promotion.

Test Scenarios

1. Single-model benchmark: run a known model on a single benchmark suite. Verify score matches baseline.
2. Multi-suite benchmark: run a candidate on all configured suites. Verify aggregation logic.
3. Timeout benchmark: configure a very short timeout. Verify graceful abort and partial results.
4. Backend failure benchmark: simulate a backend crash mid-evaluation. Verify retry and fallback.

Integration Points

• Benchmark Catalog: test suites reference benchmark definitions in Benchmark Catalog.
• Benchmark Specification: test parameters align with Benchmark Specification.
• CI: evaluation tests run nightly and on every evaluation-related change.
• ExperimentTracker: each test run is logged as an experiment with full seed and backend metadata.