Strategy Selection Decision Tree
Linear Merge
Purpose: Weighted average of two or more models trained on the same task with different seeds or data splits. Inputs: Two or more COMPATIBLE models. Parameters:
Preconditions: All models COMPATIBLE. Same architecture, same task.
Math: See Merge Math.
Failure modes: Weights do not sum to 1.0; dtype mismatch.
Known limitations: Assumes models are in the same basin. Models from different tasks may cancel skills.
SLERP
Purpose: Spherical linear interpolation between two models. Preserves weight vector direction better than linear interpolation. Inputs: Two COMPATIBLE models. Parameters:
Preconditions: Models COMPATIBLE. Same parameter count.
Math: See SLERP Math. Reference: Shoemake 1985.
Failure modes: Theta near 0 (fallback to lerp); theta near pi (ambiguous shortest path).
Known limitations: Only defined for two models. Extension to N models requires pairwise chaining or geodesic averaging.
TIES
Purpose: Trim, elect sign, and merge. Reduces interference by keeping only the most significant parameter changes and resolving sign conflicts. Inputs: Two or more COMPATIBLE models, or task vectors. Parameters:
Preconditions: Models COMPATIBLE or task vectors precomputed.
Math: See TIES Math. Reference: Yadav et al. 2023.
Failure modes: Density too low (all parameters trimmed); sign consensus unreachable.
Known limitations: Requires task vectors for best effect. Direct model merge may retain pretraining noise.
DARE
Purpose: Drop and rescale. Randomly sparsifies task vectors, then rescales survivors to preserve expected magnitude. Inputs: Two or more COMPATIBLE models, or task vectors. Parameters:
Preconditions: Models COMPATIBLE or task vectors precomputed.
Math: See DARE Math. Reference: Yu et al. 2023.
Failure modes: p too high (all parameters dropped); p = 0 (no effect, degenerate).
Known limitations: Random sparsification is non-deterministic without fixed seed. Best with multiple task vectors.
DARE+TIES
Purpose: Combine DARE sparsification with TIES sign election for robust multi-model merging. Inputs: Three or more COMPATIBLE models, or task vectors. Parameters:
Preconditions: Models COMPATIBLE. Task vectors recommended.
Math: See DARE Math and TIES Math. Reference: Yu et al. 2023.
Failure modes: Combines DARE and TIES failure modes.
Known limitations: More hyperparameters to tune. Best for 3+ models.
Task Arithmetic
Purpose: Add or subtract skills by combining task vectors (fine-tuned minus pre-trained). Inputs: Pre-trained base model, one or more fine-tuned models. Parameters:
Preconditions: Base model and fine-tuned models COMPATIBLE. Task vectors computed as theta_ft - theta_base.
Math: See Task Arithmetic Math. Reference: Ilharco et al. 2022.
Failure modes: Alpha too large (catastrophic forgetting); base model missing.
Known limitations: Requires a shared pre-trained base. Cannot merge two independent fine-tunes without a common ancestor.
Franken-Merge / Structural Merge
Purpose: Combine layers or blocks from different models to create a new architecture. Inputs: Two or more models, possibly cross-family. Parameters:
Preconditions: Models CONDITIONALLY_COMPATIBLE with explicit adapters, or INCOMPATIBLE with franken-merge override.
Failure modes: Layer map references non-existent layer; adapter shape mismatch; activation distribution mismatch at boundary.
Known limitations: Requires manual layer map. No automatic search for optimal layer combinations in the initial core.
Passthrough
Purpose: Return the single input model unchanged. Used for baseline evaluation or when only one model is available. Inputs: One model. Parameters: None. Preconditions: Any model. Failure modes: None. Known limitations: Not a merge. Included for pipeline completeness.Cross-Links
- Merge Engine for the pipeline that uses these strategies.
- Tensor Operations for the low-level ops strategies call.
- Merge Validation for post-merge checks.
- Merge Math for shared notation.