Genome Encoding for Model Merge Candidates

Specification for the genome encoding used by EMEP to represent merge candidate configurations, including structure, fields, and JSON example.

The genome is the complete genetic representation of a merge candidate in EMEP. It encodes every decision the MergeEngine needs to materialize a candidate model from parent weights. This page defines the genome structure, field semantics, and serialization format.

Genome Structure Diagram

Field Definitions

Parent IDs

Ordered list of parent model identifiers from the ModelRegistry. The first parent is the base model. All parents must be in REGISTERED or VALIDATED state. The genome references models by their canonical ID, not by path or version hash.

Merge Method

The strategy applied to combine parent weights. Supported values include:

• SLERP (Shoemake 1985)
• TIES (Yadav et al. 2023)
• DARE (Yu et al. 2023)
• Task Arithmetic (Ilharco et al. 2022)
• Franken-Merge

Alpha Coefficients

Global interpolation weights for each parent. For two-parent merges, a single scalar alpha determines the blend ratio. For multi-parent merges, alpha is a vector summing to 1.0. Values outside [0, 1] are clipped during materialization.

Per-Layer Coefficients

Layer-specific alpha overrides. When present, these take precedence over global alpha for the specified layer indices. This enables fine-grained control, for example applying TIES at early layers and SLERP at late layers.

Density

Sparsity parameter for methods like TIES and DARE. Density controls the fraction of parameters retained during merge. A density of 1.0 means no sparsity. Values below 0.5 are experimental and may produce unstable candidates.

Task Vectors

When using Task Arithmetic (Ilharco et al. 2022), the genome encodes which task vectors to apply and their scaling coefficients. Task vectors are derived from fine-tuned parent models and represent direction of adaptation.

Layer Routing

For Franken-Merge configurations, layer routing specifies which parent contributes each layer or block. Routing is expressed as a map from layer index to parent index. Gaps or overlaps are validated by the MergeEngine before materialization.

Structural Parameters

Optional architectural overrides: hidden size adjustments, attention head count, or intermediate dimension changes. These are only valid when all parents share the same base architecture family. Structural parameters trigger a compatibility check against the TensorEngine.

Quantization Option

Post-merge quantization setting: none, int8, int4, or fp16. Quantization is applied after merge materialization, not during. This field is only active when the QuantizationEngine is enabled for the experiment.

Genome JSON Example

{
  "genome_id": "genome_7a3f9e2",
  "version": "1.0",
  "parents": [
    "model_llama_7b_base_v3",
    "model_llama_7b_instruct_v2",
    "model_llama_7b_code_v1"
  ],
  "merge_method": "TIES",
  "alpha": [0.5, 0.3, 0.2],
  "per_layer_alpha": {
    "0-15": [0.6, 0.2, 0.2],
    "16-31": [0.4, 0.4, 0.2]
  },
  "density": 0.7,
  "task_vectors": [
    {
      "parent_index": 1,
      "scaling": 0.8
    },
    {
      "parent_index": 2,
      "scaling": 1.0
    }
  ],
  "layer_routing": {},
  "structural_parameters": {
    "hidden_size": null,
    "num_attention_heads": null,
    "intermediate_size": null
  },
  "quantization": "none",
  "metadata": {
    "created_by": "EvolutionEngine",
    "generation": 12,
    "lineage": ["genome_4b1c8d0", "genome_9e2f5a1"]
  }
}

Validation Rules

1. Parent IDs must resolve to existing, compatible models
2. Alpha coefficients must sum to 1.0 (within epsilon 1e-6)
3. Per-layer alpha keys must be valid layer index ranges
4. Density must be in (0, 1]
5. Task vectors require parent models with fine-tuned deltas
6. Layer routing must cover all layers without overlap
7. Structural parameters require architecture-family match
8. Quantization option must be supported by the QuantizationEngine

Encoding Rationale

The genome uses a flat JSON structure for portability and inspectability. Every field is optional except parents and merge_method. Defaults are applied by the CandidateGenerator when fields are absent. This design supports both manual specification and automated evolution without schema changes.

Genomes are immutable once created. Mutations and crossovers produce new genome instances with updated IDs. The ExperimentTracker records the full lineage chain for every genome.