> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Merge Strategies: Selection, Parameters, and Limitations

> Overview of EMEP merge strategies: Linear Merge, SLERP, TIES, DARE, DARE+TIES, Task Arithmetic, Franken-Merge, and Passthrough. Includes decision tree for strategy selection and per-strategy specs.

Merge strategies are the algorithms that combine two or more models into one candidate. This page defines each strategy, its inputs, parameters, preconditions, failure modes, and known limitations. A decision tree at the end guides strategy selection.

## Strategy Selection Decision Tree

```mermaid theme={null}
flowchart TD
    START([START]) --> INPUT[Input: source models, compatibility level, goal]
    INPUT --> COUNT{Model count?}
    COUNT -->|1| PASSTHROUGH[Passthrough]
    COUNT -->|2| TWO[Two-model path]
    COUNT -->|3+| MULTI[Multi-model path]
    TWO --> SAME{Same task, different seeds?}
    SAME -->|yes| LINEAR[Linear Merge]
    SAME -->|no| TASK_GOAL{Goal: add skill or remove bias?}
    TASK_GOAL -->|add skill| SLERP[SLERP]
    TASK_GOAL -->|remove bias| TASK_ARITH[Task Arithmetic]
    TASK_GOAL -->|both| TIES[TIES]
    MULTI --> SIMILAR{Models similar in size and task?}
    SIMILAR -->|yes| DARE[DARE]
    SIMILAR -->|no| STRUCT{Structural merge needed?}
    STRUCT -->|yes| FRANKEN[Franken-Merge]
    STRUCT -->|no| DARE_TIES[DARE+TIES]
    PASSTHROUGH --> END([END])
    LINEAR --> END
    SLERP --> END
    TASK_ARITH --> END
    TIES --> END
    DARE --> END
    DARE_TIES --> END
    FRANKEN --> END
```

## Linear Merge

**Purpose:** Weighted average of two or more models trained on the same task with different seeds or data splits.

**Inputs:** Two or more COMPATIBLE models.

**Parameters:**

| Parameter | Type         | Default | Range      |
| --------- | ------------ | ------- | ---------- |
| weights   | List\[float] | equal   | sum to 1.0 |

**Preconditions:** All models COMPATIBLE. Same architecture, same task.

**Math:** See [Merge Math](/math/merge-math).

**Failure modes:** Weights do not sum to 1.0; dtype mismatch.

**Known limitations:** Assumes models are in the same basin. Models from different tasks may cancel skills.

## SLERP

**Purpose:** Spherical linear interpolation between two models. Preserves weight vector direction better than linear interpolation.

**Inputs:** Two COMPATIBLE models.

**Parameters:**

| Parameter | Type  | Default | Range       |
| --------- | ----- | ------- | ----------- |
| t         | float | 0.5     | \[0.0, 1.0] |

**Preconditions:** Models COMPATIBLE. Same parameter count.

**Math:** See [SLERP Math](/math/slerp-math). Reference: Shoemake 1985.

**Failure modes:** Theta near 0 (fallback to lerp); theta near pi (ambiguous shortest path).

**Known limitations:** Only defined for two models. Extension to N models requires pairwise chaining or geodesic averaging.

## TIES

**Purpose:** Trim, elect sign, and merge. Reduces interference by keeping only the most significant parameter changes and resolving sign conflicts.

**Inputs:** Two or more COMPATIBLE models, or task vectors.

**Parameters:**

| Parameter       | Type  | Default    | Range             |
| --------------- | ----- | ---------- | ----------------- |
| density         | float | 0.6        | (0.0, 1.0]        |
| sign\_consensus | str   | "majority" | "majority", "all" |

**Preconditions:** Models COMPATIBLE or task vectors precomputed.

**Math:** See [TIES Math](/math/ties-math). Reference: Yadav et al. 2023.

**Failure modes:** Density too low (all parameters trimmed); sign consensus unreachable.

**Known limitations:** Requires task vectors for best effect. Direct model merge may retain pretraining noise.

## DARE

**Purpose:** Drop and rescale. Randomly sparsifies task vectors, then rescales survivors to preserve expected magnitude.

**Inputs:** Two or more COMPATIBLE models, or task vectors.

**Parameters:**

| Parameter | Type  | Default | Range       |
| --------- | ----- | ------- | ----------- |
| p         | float | 0.5     | \[0.0, 1.0) |

**Preconditions:** Models COMPATIBLE or task vectors precomputed.

**Math:** See [DARE Math](/math/dare-math). Reference: Yu et al. 2023.

**Failure modes:** p too high (all parameters dropped); p = 0 (no effect, degenerate).

**Known limitations:** Random sparsification is non-deterministic without fixed seed. Best with multiple task vectors.

## DARE+TIES

**Purpose:** Combine DARE sparsification with TIES sign election for robust multi-model merging.

**Inputs:** Three or more COMPATIBLE models, or task vectors.

**Parameters:**

| Parameter       | Type  | Default    | Range             |
| --------------- | ----- | ---------- | ----------------- |
| p               | float | 0.5        | \[0.0, 1.0)       |
| density         | float | 0.6        | (0.0, 1.0]        |
| sign\_consensus | str   | "majority" | "majority", "all" |

**Preconditions:** Models COMPATIBLE. Task vectors recommended.

**Math:** See [DARE Math](/math/dare-math) and [TIES Math](/math/ties-math). Reference: Yu et al. 2023.

**Failure modes:** Combines DARE and TIES failure modes.

**Known limitations:** More hyperparameters to tune. Best for 3+ models.

## Task Arithmetic

**Purpose:** Add or subtract skills by combining task vectors (fine-tuned minus pre-trained).

**Inputs:** Pre-trained base model, one or more fine-tuned models.

**Parameters:**

| Parameter | Type         | Default | Range    |
| --------- | ------------ | ------- | -------- |
| alphas    | List\[float] | \[1.0]  | any real |

**Preconditions:** Base model and fine-tuned models COMPATIBLE. Task vectors computed as theta\_ft - theta\_base.

**Math:** See [Task Arithmetic Math](/math/task-arithmetic). Reference: Ilharco et al. 2022.

**Failure modes:** Alpha too large (catastrophic forgetting); base model missing.

**Known limitations:** Requires a shared pre-trained base. Cannot merge two independent fine-tunes without a common ancestor.

## Franken-Merge / Structural Merge

**Purpose:** Combine layers or blocks from different models to create a new architecture.

**Inputs:** Two or more models, possibly cross-family.

**Parameters:**

| Parameter  | Type                 | Default  | Description                              |
| ---------- | -------------------- | -------- | ---------------------------------------- |
| layer\_map | Dict\[int, str]      | required | Which source provides each layer         |
| adapters   | List\[AdapterConfig] | \[]      | Adapter layers between mismatched blocks |

**Preconditions:** Models CONDITIONALLY\_COMPATIBLE with explicit adapters, or INCOMPATIBLE with franken-merge override.

**Failure modes:** Layer map references non-existent layer; adapter shape mismatch; activation distribution mismatch at boundary.

**Known limitations:** Requires manual layer map. No automatic search for optimal layer combinations in the initial core.

## Passthrough

**Purpose:** Return the single input model unchanged. Used for baseline evaluation or when only one model is available.

**Inputs:** One model.

**Parameters:** None.

**Preconditions:** Any model.

**Failure modes:** None.

**Known limitations:** Not a merge. Included for pipeline completeness.

## Cross-Links

* [Merge Engine](/merge/merge-engine) for the pipeline that uses these strategies.
* [Tensor Operations](/merge/tensor-operations) for the low-level ops strategies call.
* [Merge Validation](/merge/merge-validation) for post-merge checks.
* [Merge Math](/math/merge-math) for shared notation.