EMEP Research Reference Cards

Annotated bibliography of 18 papers underpinning EMEP merge strategies, evaluation, evolution, and PEFT methods. Each card links to a component and experiment ID.

EMEP is built on established research in model merging, evolutionary optimization, parameter-efficient fine-tuning, and benchmark integrity. This page provides structured reference cards for every foundational paper.

Model Merging

Model Soups

• Authors: Wortsman et al.
• Year: 2022
• Research problem: How to combine multiple fine-tuned models without additional training
• Method: Weight-space averaging of models fine-tuned with different hyperparameters; interpolation coefficients optimized on a validation set
• Main finding: Averaging weights of models fine-tuned in the same soup outperforms individual models and reduces hyperparameter sensitivity; works because weights in the same basin can be linearly combined
• Limitations: Requires models fine-tuned from the same initialization; does not address task interference
• Relevance to EMEP: Justifies MergeEngine weight averaging as a baseline strategy
• Component affected: MergeEngine
• Experiment: EXP-TBD-001

Task Arithmetic

• Authors: Ilharco et al.
• Year: 2022
• Research problem: How to edit pre-trained models by adding or removing capabilities via arithmetic in weight space
• Method: Compute task vectors (fine-tuned minus pre-trained weights), then add or subtract scaled task vectors to the pre-trained base; scaling coefficients tuned on validation data
• Main finding: Task vectors encode semantic capabilities; arithmetic operations correspond to capability addition and removal; enables multi-task composition without joint training
• Limitations: Task interference causes degradation when combining many tasks; scaling is task-dependent
• Relevance to EMEP: Foundational merge strategy implemented in TensorEngine
• Component affected: MergeEngine, math/task-arithmetic
• Experiment: EXP-TBD-002

TIES-Merging

• Authors: Yadav et al.
• Year: 2023
• Research problem: How to reduce interference when merging multiple task-specific models
• Method: Three-step process: (1) Trim low-magnitude task vector elements, (2) Elect the majority sign for each parameter across task vectors, (3) Merge only parameters where signs agree, disjoint otherwise
• Main finding: Significant reduction in interference compared to naive averaging; preserves more individual task performance when merging many models
• Limitations: Sign election can discard valid updates; trimming threshold is hyperparameter-sensitive
• Relevance to EMEP: Primary advanced merge strategy for multi-model composition
• Component affected: MergeEngine, math/ties-math
• Experiment: EXP-TBD-003

DARE

• Authors: Yu et al.
• Year: 2023
• Research problem: How to merge task vectors while controlling capacity and reducing noise
• Method: Drop And REscale: randomly drop elements from task vectors with probability p, then rescale surviving elements by 1/(1-p); merge dropped vectors with base model
• Main finding: DARE outperforms magnitude-based trimming at high sparsity; random dropping preserves task performance better than low-magnitude removal
• Limitations: Optimal drop rate varies by task and model scale; requires calibration
• Relevance to EMEP: Sparsification strategy for noise reduction in merge pipelines
• Component affected: MergeEngine, math/dare-math
• Experiment: EXP-TBD-004

Evolutionary Model Merge

• Authors: Akiba et al.
• Year: 2024
• Research problem: How to automatically discover optimal merge recipes across diverse models
• Method: Evolutionary search over merge layer configurations, model combinations, and merge ratios; fitness evaluated on downstream tasks; uses NSGA-II for multi-objective optimization
• Main finding: Evolutionary search discovers merge combinations that outperform manual design; layer-wise merge ratios are more important than global ratios
• Limitations: Computationally expensive; requires large evaluation budget; search space grows combinatorially
• Relevance to EMEP: Direct justification for EvolutionEngine and genome encoding of merge recipes
• Component affected: EvolutionEngine, CandidateGenerator
• Experiment: EXP-TBD-005

Model Merging Survey

• Authors: Yang et al.
• Year: 2024
• Research problem: Systematic categorization of model merging techniques and their trade-offs
• Method: Taxonomy of merging methods by operation type (averaging, stitching, branching), application domain, and evaluation protocol
• Main finding: No single method dominates; choice depends on model similarity, task relatedness, and evaluation budget
• Limitations: Survey does not propose new methods; evaluation protocols vary across papers
• Relevance to EMEP: Informs MergeStrategy catalog and strategy selection heuristics
• Component affected: MergeStrategy
• Experiment: EXP-TBD-006

Parameter-Efficient Fine-Tuning

LoRA

• Authors: Hu et al.
• Year: 2021
• Research problem: How to fine-tune large models without updating all parameters
• Method: Low-Rank Adaptation: inject trainable rank-decomposition matrices into each layer; freeze pre-trained weights
• Main finding: LoRA achieves comparable accuracy to full fine-tuning with orders of magnitude fewer trainable parameters; rank can be tuned per task
• Limitations: Optimal rank is task-dependent; does not reduce inference memory without merging
• Relevance to EMEP: Optional fine-tuning path for producing task vectors before merge
• Component affected: optional-finetuning, lora
• Experiment: EXP-TBD-007

QLoRA

• Authors: Dettmers et al.
• Year: 2023
• Research problem: How to fine-tune 65B parameter models on a single 48GB GPU
• Method: 4-bit NormalFloat quantization of base model weights with double quantization; paged optimizers; LoRA adapters for training
• Main finding: QLoRA matches 16-bit full fine-tuning and LoRA on downstream tasks; enables large model fine-tuning on consumer hardware
• Limitations: 4-bit quantization introduces small accuracy degradation; inference requires dequantization or adapter merging
• Relevance to EMEP: Enables optional fine-tuning in resource-constrained environments
• Component affected: qlora
• Experiment: EXP-TBD-008

PEFT Survey

• Authors: Han et al.
• Year: 2024
• Research problem: Comprehensive review of parameter-efficient fine-tuning methods
• Method: Categorization by adaptation location, parameter type, and computational cost; empirical comparison across model scales
• Main finding: Adapter-based methods (LoRA, adapters, prefix tuning) offer the best accuracy-efficiency trade-off; prompt tuning underperforms on small datasets
• Limitations: Benchmark coverage is English-centric; long-context tasks underrepresented
• Relevance to EMEP: Guides optional fine-tuning strategy selection in DatasetRegistry
• Component affected: peft
• Experiment: EXP-TBD-009

Catastrophic Forgetting

Catastrophic Forgetting (McCloskey & Cohen)

• Authors: McCloskey & Cohen
• Year: 1989
• Research problem: Why neural networks lose previously learned information when trained on new tasks
• Method: Controlled experiments on sequential learning in feedforward networks
• Main finding: New learning interferes with old representations; problem is fundamental to gradient-based optimization in shared parameter spaces
• Limitations: Studied on small networks; modern LLM scale was not accessible
• Relevance to EMEP: Motivates regression-evaluation and catastrophic forgetting controls
• Component affected: catastrophic-forgetting-controls
• Experiment: EXP-TBD-010

Elastic Weight Consolidation (Kirkpatrick)

• Authors: Kirkpatrick et al.
• Year: 2017
• Research problem: How to protect important parameters during sequential learning
• Method: Compute Fisher information for each parameter; add penalty for changing parameters critical to previous tasks
• Main finding: EWC significantly reduces forgetting in sequential task learning; importance estimation is computationally expensive
• Limitations: Fisher information is approximate; does not scale cleanly to hundreds of tasks
• Relevance to EMEP: Candidate approach for merge-time forgetting mitigation
• Component affected: catastrophic-forgetting-controls
• Experiment: EXP-TBD-011

Evolutionary Optimization

NSGA-II

• Authors: Deb et al.
• Year: 2002
• Research problem: Efficient multi-objective optimization with Pareto-based selection
• Method: Non-dominated sorting with crowding distance; elitism via combined parent-offspring population; binary tournament selection
• Main finding: NSGA-II achieves diverse Pareto fronts with low computational cost; dominates predecessor NSGA on standard benchmarks
• Limitations: Performance degrades beyond 3-4 objectives; crowding distance becomes less discriminative
• Relevance to EMEP: Default multi-objective optimizer in EvolutionEngine
• Component affected: multi-objective-optimization
• Experiment: EXP-TBD-012

NSGA-III

• Authors: Deb & Jain
• Year: 2014
• Research problem: Scaling multi-objective optimization to many objectives (5+)
• Method: Reference point-based selection replaces crowding distance; uniform distribution of reference points ensures diversity
• Main finding: NSGA-III outperforms NSGA-II on many-objective problems; reference point specification requires domain knowledge
• Limitations: Reference point design is non-trivial; performance on 2-3 objectives is comparable to NSGA-II
• Relevance to EMEP: Alternative optimizer for experiments with more than four fitness objectives
• Component affected: multi-objective-optimization
• Experiment: EXP-TBD-013

CMA-ES

• Authors: Hansen
• Year: 2001
• Research problem: Derivative-free optimization for continuous, non-convex, ill-conditioned problems
• Method: Covariance Matrix Adaptation Evolution Strategy: adapts mutation distribution via covariance matrix update; uses rank-based selection
• Main finding: CMA-ES is state-of-the-art for black-box continuous optimization; sample efficiency is high for moderate dimensions
• Limitations: Computational cost scales with the square of dimension; not designed for discrete or combinatorial spaces
• Relevance to EMEP: Potential optimizer for continuous hyperparameter search (merge ratios, scaling coefficients)
• Component affected: EvolutionEngine
• Experiment: EXP-TBD-014

Interpolation

SLERP

• Authors: Shoemake
• Year: 1985
• Research problem: Smooth interpolation between orientations in 3D graphics
• Method: Spherical linear interpolation: interpolate along the great circle path on a unit sphere; preserves constant angular velocity
• Main finding: SLERP avoids the acceleration artifacts of linear interpolation; optimal for quaternion and rotation interpolation
• Limitations: Requires normalized inputs; undefined for antipodal vectors
• Relevance to EMEP: Applied to weight vector interpolation in MergeEngine to preserve angular relationships between model checkpoints
• Component affected: math/slerp-math
• Experiment: EXP-TBD-015

Evaluation

LM Evaluation Harness

• Authors: Gao et al.
• Year: 2023
• Research problem: Standardized evaluation framework for few-shot and zero-shot LLM benchmarking
• Method: Unified API for task loading, prompt formatting, metric computation, and result aggregation; supports hundreds of tasks
• Main finding: Standardized evaluation reduces implementation variance; reproducible few-shot settings are critical for fair comparison
• Limitations: Task coverage is English-centric; custom tasks require boilerplate; contamination detection is manual
• Relevance to EMEP: EvaluationEngine may integrate Harness tasks but maintains independent tracking and signing
• Component affected: EvaluationEngine
• Experiment: EXP-TBD-016

HELM

• Authors: Liang et al.
• Year: 2022
• Research problem: Holistic evaluation of language models across scenarios, metrics, and domains
• Method: Taxonomy of scenarios (QA, summarization, toxicity, etc.) with standardized metrics; multi-metric reporting instead of single leaderboard score
• Main finding: No model dominates all scenarios; trade-offs between accuracy, fairness, robustness, and efficiency are universal
• Limitations: Evaluation is expensive; scenario coverage still incomplete; some metrics are noisy
• Relevance to EMEP: Informs multi-objective fitness design and benchmark catalog structure
• Component affected: benchmark-specification
• Experiment: EXP-TBD-017

Benchmark Contamination

• Authors: Sainz et al.
• Year: 2023
• Research problem: Detecting and measuring test data leakage into pre-training corpora
• Method: N-gram overlap analysis, template-based detection, and dynamic benchmark construction to identify contaminated examples
• Main finding: Contamination is widespread in popular benchmarks; even small overlap inflates scores significantly
• Limitations: Detection methods have false positives; dynamic benchmarks are expensive to construct
• Relevance to EMEP: Justifies strict benchmark-integrity and Hidden Test Set isolation
• Component affected: benchmark-integrity
• Experiment: EXP-TBD-018