Merge Math: Notation, Shapes, and Precision

Foundational merge math notation for EMEP. Defines scalars, vectors, matrices, tensors, LLM weight shapes, parameter deltas, task vectors, norms, and precision effects. Base of all merge math.

Merge math is the shared notation and shape conventions that every strategy in EMEP uses. This page defines the symbols, weight tensor shapes, and precision rules that appear in all per-strategy math pages.

Notation

Symbol	Meaning
theta	Model parameters, a set of tensors indexed by key
theta_pre	Pre-trained base model parameters
theta_ft	Fine-tuned model parameters
tau	Task vector: tau = theta_ft - theta_pre
lambda	Scalar interpolation weight
t	SLERP interpolation parameter in [0, 1]
p	Drop probability in DARE, in [0, 1)
d	Density parameter in TIES, in (0, 1]
epsilon	Small constant, typically 1e-8
		.		_2	L2 norm
		.		_inf	Infinity norm (max absolute value)
circle_dot	Element-wise (Hadamard) product
R	Real numbers
R^d	d-dimensional real vector space

Weight Tensor Shapes for LLMs

For a transformer with hidden size H, vocab size V, intermediate size I, attention heads A, and KV heads K:

Parameter	Shape	Description
embed_tokens.weight	(V, H)	Token embeddings
layers.l.self_attn.q_proj.weight	(H, H)	Query projection
layers.l.self_attn.k_proj.weight	(H, K * H / A)	Key projection (GQA)
layers.l.self_attn.v_proj.weight	(H, K * H / A)	Value projection (GQA)
layers.l.self_attn.o_proj.weight	(H, H)	Output projection
layers.l.mlp.up_proj.weight	(H, I)	MLP up projection
layers.l.mlp.gate_proj.weight	(H, I)	MLP gate projection (SwiGLU)
layers.l.mlp.down_proj.weight	(I, H)	MLP down projection
layers.l.input_layernorm.weight	(H,)	Pre-attention layernorm
layers.l.post_attention_layernorm.weight	(H,)	Post-attention layernorm
norm.weight	(H,)	Final layernorm
lm_head.weight	(V, H)	Output projection (often tied to embed)

Parameter Deltas and Task Vectors

A parameter delta is the difference between two parameter sets:

delta = theta_B - theta_A

A task vector is a parameter delta between a fine-tuned model and its pre-trained base:

tau = theta_ft - theta_pre

Task vectors are central to Task Arithmetic (Ilharco et al. 2022), TIES (Yadav et al. 2023), and DARE (Yu et al. 2023).

Norms

The L2 norm of a flattened parameter tensor:

||theta||_2 = sqrt(sum(theta_i^2))

The infinity norm:

||theta||_inf = max_i |theta_i|

Norms are used for normalization, envelope checks, and magnitude-based trimming.

Precision Effects

Precision	Range	Notes
fp32	~1e-45 to ~3.4e38	Safe for all intermediate compute
bf16	~1e-38 to ~3.4e38	Same range as fp32, less precision
fp16	~5.96e-8 to ~65504	Risk of overflow in large norms
fp8	Narrow	Experimental, limited support
int8	-128 to 127	Quantized, requires dequant for merge
int4	0 to 15	Highly quantized, dequant mandatory

All merge operations should accumulate in fp32 and cast to the target dtype at the end. This is an engineering assumption: fp16 accumulation causes measurable drift in merged models.

Cross-Links

• SLERP Math for spherical interpolation formulas.
• TIES Math for trim, elect, and disjoint merge.
• DARE Math for sparsification and rescaling.
• Task Arithmetic for task vector combinations.
• Tensor Math for broadcasting and elementwise rules.