> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Merge Math: Notation, Shapes, and Precision

> Foundational merge math notation for EMEP. Defines scalars, vectors, matrices, tensors, LLM weight shapes, parameter deltas, task vectors, norms, and precision effects. Base of all merge math.

Merge math is the shared notation and shape conventions that every strategy in EMEP uses. This page defines the symbols, weight tensor shapes, and precision rules that appear in all per-strategy math pages.

## Notation

| Symbol      | Meaning                                           |   |   |       |                                    |
| ----------- | ------------------------------------------------- | - | - | ----- | ---------------------------------- |
| theta       | Model parameters, a set of tensors indexed by key |   |   |       |                                    |
| theta\_pre  | Pre-trained base model parameters                 |   |   |       |                                    |
| theta\_ft   | Fine-tuned model parameters                       |   |   |       |                                    |
| tau         | Task vector: tau = theta\_ft - theta\_pre         |   |   |       |                                    |
| lambda      | Scalar interpolation weight                       |   |   |       |                                    |
| t           | SLERP interpolation parameter in \[0, 1]          |   |   |       |                                    |
| p           | Drop probability in DARE, in \[0, 1)              |   |   |       |                                    |
| d           | Density parameter in TIES, in (0, 1]              |   |   |       |                                    |
| epsilon     | Small constant, typically 1e-8                    |   |   |       |                                    |
|             |                                                   | . |   | \_2   | L2 norm                            |
|             |                                                   | . |   | \_inf | Infinity norm (max absolute value) |
| circle\_dot | Element-wise (Hadamard) product                   |   |   |       |                                    |
| R           | Real numbers                                      |   |   |       |                                    |
| R^d         | d-dimensional real vector space                   |   |   |       |                                    |

## Weight Tensor Shapes for LLMs

For a transformer with hidden size H, vocab size V, intermediate size I, attention heads A, and KV heads K:

| Parameter                                  | Shape           | Description                             |
| ------------------------------------------ | --------------- | --------------------------------------- |
| embed\_tokens.weight                       | (V, H)          | Token embeddings                        |
| layers.l.self\_attn.q\_proj.weight         | (H, H)          | Query projection                        |
| layers.l.self\_attn.k\_proj.weight         | (H, K \* H / A) | Key projection (GQA)                    |
| layers.l.self\_attn.v\_proj.weight         | (H, K \* H / A) | Value projection (GQA)                  |
| layers.l.self\_attn.o\_proj.weight         | (H, H)          | Output projection                       |
| layers.l.mlp.up\_proj.weight               | (H, I)          | MLP up projection                       |
| layers.l.mlp.gate\_proj.weight             | (H, I)          | MLP gate projection (SwiGLU)            |
| layers.l.mlp.down\_proj.weight             | (I, H)          | MLP down projection                     |
| layers.l.input\_layernorm.weight           | (H,)            | Pre-attention layernorm                 |
| layers.l.post\_attention\_layernorm.weight | (H,)            | Post-attention layernorm                |
| norm.weight                                | (H,)            | Final layernorm                         |
| lm\_head.weight                            | (V, H)          | Output projection (often tied to embed) |

## Parameter Deltas and Task Vectors

A parameter delta is the difference between two parameter sets:

```text theme={null}
delta = theta_B - theta_A
```

A task vector is a parameter delta between a fine-tuned model and its pre-trained base:

```text theme={null}
tau = theta_ft - theta_pre
```

Task vectors are central to Task Arithmetic (Ilharco et al. 2022), TIES (Yadav et al. 2023), and DARE (Yu et al. 2023).

## Norms

The L2 norm of a flattened parameter tensor:

```text theme={null}
||theta||_2 = sqrt(sum(theta_i^2))
```

The infinity norm:

```text theme={null}
||theta||_inf = max_i |theta_i|
```

Norms are used for normalization, envelope checks, and magnitude-based trimming.

## Precision Effects

| Precision | Range                | Notes                                 |
| --------- | -------------------- | ------------------------------------- |
| fp32      | \~1e-45 to \~3.4e38  | Safe for all intermediate compute     |
| bf16      | \~1e-38 to \~3.4e38  | Same range as fp32, less precision    |
| fp16      | \~5.96e-8 to \~65504 | Risk of overflow in large norms       |
| fp8       | Narrow               | Experimental, limited support         |
| int8      | -128 to 127          | Quantized, requires dequant for merge |
| int4      | 0 to 15              | Highly quantized, dequant mandatory   |

All merge operations should accumulate in fp32 and cast to the target dtype at the end. This is an engineering assumption: fp16 accumulation causes measurable drift in merged models.

## Cross-Links

* [SLERP Math](/math/slerp-math) for spherical interpolation formulas.
* [TIES Math](/math/ties-math) for trim, elect, and disjoint merge.
* [DARE Math](/math/dare-math) for sparsification and rescaling.
* [Task Arithmetic](/math/task-arithmetic) for task vector combinations.
* [Tensor Math](/math/tensor-math) for broadcasting and elementwise rules.