Phase 2: Basic Merge Engine with Linear, SLERP, and Task Arithmetic

Implement the foundational merge engine for EMEP. Phase 2 delivers TensorEngine, MergeEngine, and three core strategies: Linear Merge, SLERP (Shoemake 1985), and Task Arithmetic (Ilharco et al. 2022).

Phase 2 introduces the first executable merge capabilities in EMEP. You build the TensorEngine for low-level tensor operations, the MergeEngine for orchestrating weight combination, and three foundational merge strategies that cover the simplest to the most widely used merging techniques in the literature. This phase establishes the plug-in architecture that later strategies will extend.

Purpose

The purpose of Phase 2 is to make model merging operational. You need a system that can take two or more registered models, verify compatibility through the Phase 1 infrastructure, and produce a merged checkpoint. The three strategies implemented here (Linear Merge, SLERP, and Task Arithmetic) represent the baseline techniques that more advanced methods build upon. You also design the MergeStrategy plug-in system so that Phases 3 and 5 can add new algorithms without modifying core engine code.

Entry Criteria

• Phase 1 exit criteria satisfied: ModelRegistry, ModelLoader, and ModelCompatibilityAnalyzer are operational
• At least two baseline model checkpoints registered in the ModelRegistry for merge testing
• TensorEngine compute backend selected (e.g., PyTorch, JAX, or custom CUDA kernels)
• Merge algorithm literature reviewed and algorithm specifications documented

Exit Criteria

• TensorEngine executes element-wise, matrix, and interpolation operations correctly and efficiently
• MergeEngine orchestrates a full merge pipeline: load models, check compatibility, apply strategy, save result
• Linear Merge produces a weighted average of model weights with configurable coefficients
• SLERP (Shoemake 1985) performs spherical linear interpolation between model parameter vectors
• Task Arithmetic (Ilharco et al. 2022) applies task-specific vectors with configurable scaling
• MergeStrategy plug-in system allows registration of new strategies at runtime
• All strategies produce deterministic outputs for identical inputs
• Unit and integration tests pass for every strategy

Primary Components

Phase 2 implements and extends the following canonical components:

• TensorEngine: the low-level tensor operation layer. It provides element-wise arithmetic, matrix operations, memory-efficient intermediates, and device placement. All merge strategies depend on TensorEngine for numerical execution.
• MergeEngine: the orchestrator for combining model weights. It coordinates ModelLoader, ModelCompatibilityAnalyzer, MergeStrategy selection, and ArtifactStore persistence. It is the primary user-facing interface for merging.
• MergeStrategy: the plug-in abstraction for merge algorithms. Each strategy implements a consistent interface: accept a list of loaded models and parameters, return merged weights. Phase 2 provides the first three implementations.

Supporting components from Phase 1:

• ModelRegistry: supplies model metadata and lineage for merge inputs
• ModelLoader: loads checkpoints into memory for TensorEngine processing
• ModelCompatibilityAnalyzer: validates that input models can be merged before execution begins
• ArtifactStore: persists merged checkpoints and provenance logs

Deliverables

Deliverable	Path / Reference	Description
TensorEngine	TensorEngine	Low-level tensor operations with device and memory management
MergeEngine	MergeEngine	Merge orchestration pipeline
Linear Merge	MergeStrategy	Weighted average of model parameters
SLERP	MergeStrategy	Spherical linear interpolation (Shoemake 1985)
Task Arithmetic	MergeStrategy	Task vector arithmetic with scaling (Ilharco et al. 2022)
MergeStrategy Plug-in System	MergeStrategy	Runtime registration and discovery for merge algorithms

Dependencies

Phase 2 depends directly on Phase 1:

• Phase 1: Model Registry + Compatibility: provides ModelRegistry, ModelLoader, and ModelCompatibilityAnalyzer, which MergeEngine uses to acquire and validate inputs

Key Tasks

Track Phase 2 work in the project task registry:

• Project Tasks

Typical Phase 2 tasks include:

1. Implement TensorEngine core operations (add, multiply, interpolate, normalize)
2. Build MergeEngine pipeline: load, validate, merge, save
3. Implement Linear Merge with per-layer and global coefficient support
4. Implement SLERP with quaternion and vector variants for parameter spaces (Shoemake 1985)
5. Implement Task Arithmetic with support for task vector extraction and scaling (Ilharco et al. 2022)
6. Design and implement MergeStrategy interface with registration and discovery
7. Write unit tests for each strategy against synthetic and real checkpoints
8. Write integration tests for full MergeEngine workflows
9. Document strategy parameters, constraints, and expected outputs

Risks

• Numerical instability in SLERP: Near-antipodal parameter vectors can produce unstable interpolation. Mitigate by adding epsilon guards and fallback to Linear Merge in degenerate cases.
• Memory exhaustion during merge: Large models may exceed available RAM during intermediate tensor creation. Mitigate by implementing chunked computation in TensorEngine.
• Task vector contamination: Task Arithmetic assumes clean task vectors; mixed training data may produce ineffective vectors. Mitigate by documenting assumptions and adding diagnostic logging.

See the full risk register for tracked items and mitigations:

• Risk Register

Quality Gate

Phase 2 is complete when the following checklist is fully satisfied:

• TensorEngine passes numerical accuracy tests against reference implementations
• MergeEngine executes a full merge pipeline end-to-end with no manual intervention
• Linear Merge output is deterministic and matches hand-calculated expectations for toy models
• SLERP output preserves vector norms and interpolates smoothly between endpoints
• Task Arithmetic reproduces published results on a known benchmark pair (Ilharco et al. 2022)
• MergeStrategy plug-in system loads and unloads strategies without restarting MergeEngine
• All strategies run within established memory bounds for the largest registered model
• Integration tests cover compatibility failure paths and graceful error handling

Roadmap Position

The diagram below shows Phase 2 as the first merge implementation phase, building on Phase 1 and feeding into advanced strategies in Phase 3.