EMEP Model Merge Tests and Golden Outputs

Tensor correctness tests per MergeStrategy in EMEP with golden outputs and tolerance policies for SLERP, TIES, DARE, Task Arithmetic, and Franken-Merge.

Model merge tests verify that every MergeStrategy produces mathematically correct and deterministic outputs. Each strategy has a golden reference test and a defined tolerance policy.

Test Structure

For each strategy, the test suite:

1. Loads a fixed pair of small reference models.
2. Executes the merge with fixed random seeds and fixed parameters.
3. Compares output tensors against a precomputed golden reference.
4. Reports PASS if all tensors are within tolerance, FAIL otherwise.

Strategies and Tolerances

Strategy	Golden Reference	Tolerance	Notes
SLERP (Shoemake 1985)	Precomputed fp32	1e-5 relative	Spherical interpolation of weights
TIES (Yadav et al. 2023)	Precomputed fp32	1e-5 relative	Trim, elect, sign consensus
DARE (Yu et al. 2023)	Precomputed fp32	1e-5 relative	Drop and rescale with fixed seed
Task Arithmetic (Ilharco et al. 2022)	Precomputed fp32	1e-5 relative	Linear combination with task vectors
Franken-Merge	Precomputed fp32	1e-5 relative	Layer-wise stitching

Tolerance Policy

Tolerance is relative to the maximum absolute value of the reference tensor:

|output - reference| / max(|reference|) < tolerance

For fp16 and bf16 merges, tolerance is relaxed to 1e-3 relative to account for reduced precision. fp8 and int8 merges are tested separately with wider tolerances defined in Numerical Testing.

Golden Output Generation

Golden references are generated once using fp32 arithmetic on a fixed CPU backend. The generation script is versioned in ArtifactStore. Regeneration requires a manual trigger and a code review. Golden references are stored as safetensors files with SHA-256 checksums.

Edge Cases

The test suite covers these edge cases for every strategy:

• Merging identical models (output must equal input).
• Merging models with zero delta (output must equal input).
• Single-layer models.
• Models with tied embeddings.
• Models with different vocabularies (must fail compatibility before merge).
• Empty parameter sets (must error gracefully).

Regression Detection

Any change to TensorEngine, MergeEngine, or a MergeStrategy implementation must pass the full merge test suite. A failure blocks the merge request. The CI job uploads diff tensors to ArtifactStore for inspection.

Integration Points

• TensorEngine: merge tests exercise the full tensor operation stack.
• Numerical Testing: tolerance thresholds are shared with the numerical test suite.
• CI: merge tests run on every commit and nightly.
• ArtifactStore: stores golden references, test outputs, and diff artifacts.