> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Tensor Operations: Low-Level Merge Primitives

> Specification of every low-level tensor operation used by EMEP merge strategies. Covers input/output shapes, dtype rules, broadcasting, and numerical stability.

TensorEngine exposes a fixed set of low-level operations that all merge strategies use. This page defines each operation, its shape contracts, dtype rules, and numerical stability notes.

## Generic Tensor Op Flow

Every operation follows this pipeline.

```mermaid theme={null}
flowchart TD
    START([START]) --> INPUT[Input A + Input B]
    INPUT --> SHAPE[Shape Check]
    SHAPE -->|mismatch| BROADCAST{Broadcastable?}
    BROADCAST -->|no| FAIL[FAIL: shape incompatible]
    BROADCAST -->|yes| BROADCAST_OP[Apply Broadcasting]
    SHAPE -->|match| DTYPE[Dtype Check]
    BROADCAST_OP --> DTYPE
    DTYPE -->|incompatible| CAST[Cast to higher precision]
    DTYPE -->|compatible| OP[Execute Operation]
    CAST --> OP
    OP --> NUMERICAL[Numerical Validation]
    NUMERICAL -->|NaN/Inf| FAIL_NUM[FAIL: numerical instability]
    NUMERICAL -->|pass| OUTPUT[Output Tensor]
    OUTPUT --> END([END])
    FAIL --> END
    FAIL_NUM --> END
```

## Operation Catalog

### Element-wise Add / Sub

| Property            | Value                                                              |
| ------------------- | ------------------------------------------------------------------ |
| Inputs              | A, B: tensors of broadcastable shapes                              |
| Output              | Same broadcasted shape                                             |
| Dtype rule          | Promote to higher precision: fp16 + fp32 → fp32                    |
| Numerical stability | Watch for catastrophic cancellation in fp16. Use fp32 accumulator. |

### Scalar Mul

| Property            | Value                                                   |
| ------------------- | ------------------------------------------------------- |
| Inputs              | A: tensor; s: scalar                                    |
| Output              | Same shape as A                                         |
| Dtype rule          | Scalar cast to A.dtype before multiply                  |
| Numerical stability | Overflow risk in fp16 for s > 65504. Clamp or use fp32. |

### Weighted Average

| Property            | Value                                        |
| ------------------- | -------------------------------------------- |
| Inputs              | A, B: tensors; w: scalar weight in \[0, 1]   |
| Output              | Same broadcasted shape                       |
| Dtype rule          | Compute in fp32, cast output to target dtype |
| Numerical stability | w \* A + (1 - w) \* B. Accumulate in fp32.   |

### Normalization

| Property            | Value                                               |
| ------------------- | --------------------------------------------------- |
| Inputs              | A: tensor; norm type: L1, L2, or inf                |
| Output              | Same shape as A                                     |
| Dtype rule          | Compute norm in fp32 regardless of input dtype      |
| Numerical stability | Guard against division by zero with epsilon = 1e-8. |

### Linear Interpolation (Lerp)

| Property            | Value                                      |
| ------------------- | ------------------------------------------ |
| Inputs              | A, B: tensors; t: scalar in \[0, 1]        |
| Output              | Same broadcasted shape                     |
| Dtype rule          | Compute in fp32, cast to target            |
| Numerical stability | Straightforward. No special guards needed. |

### Spherical Interpolation (SLERP)

| Property            | Value                                             |            |                                              |
| ------------------- | ------------------------------------------------- | ---------- | -------------------------------------------- |
| Inputs              | A, B: 1-D parameter vectors; t: scalar in \[0, 1] |            |                                              |
| Output              | 1-D vector, same length                           |            |                                              |
| Dtype rule          | Normalize in fp64 for precision, cast to target   |            |                                              |
| Numerical stability | Fallback to lerp when theta \< 1e-6 or            | theta - pi | \< 1e-6. See [SLERP Math](/math/slerp-math). |

### Sparsification

| Property            | Value                                                               |
| ------------------- | ------------------------------------------------------------------- |
| Inputs              | A: tensor; p: drop probability in \[0, 1)                           |
| Output              | Same shape as A, with (1 - p) fraction of values kept               |
| Dtype rule          | Mask in bool, apply to A in original dtype                          |
| Numerical stability | Rescale survivors by 1 / (1 - p). See [DARE Math](/math/dare-math). |

### Sign Masking

| Property            | Value                                                 |
| ------------------- | ----------------------------------------------------- |
| Inputs              | A: tensor; sign: +1, -1, or 0                         |
| Output              | Same shape as A, zeroed where sign mismatch           |
| Dtype rule          | Sign computed in fp32, mask applied in original dtype |
| Numerical stability | No special guards.                                    |

### Rescaling

| Property            | Value                                         |
| ------------------- | --------------------------------------------- |
| Inputs              | A: tensor; scale: scalar                      |
| Output              | Same shape as A                               |
| Dtype rule          | Scale cast to A.dtype                         |
| Numerical stability | Overflow risk in fp16. Use fp32 intermediate. |

### Tensor Concat / Slice

| Property            | Value                                  |
| ------------------- | -------------------------------------- |
| Inputs              | A, B: tensors; dim: int                |
| Output              | Concatenated shape along dim           |
| Dtype rule          | Both tensors must have identical dtype |
| Numerical stability | N/A                                    |

### Layer Selection

| Property            | Value                                          |
| ------------------- | ---------------------------------------------- |
| Inputs              | model: state\_dict; layer\_indices: List\[int] |
| Output              | Subset state\_dict with only selected layers   |
| Dtype rule          | Preserved from input                           |
| Numerical stability | N/A                                            |

## Cross-Links

* [Merge Engine](/merge/merge-engine) for the pipeline that calls these ops.
* [Merge Strategies](/merge/merge-strategies) for which strategies use which ops.
* [Merge Math](/math/merge-math) for shared notation.
* [SLERP Math](/math/slerp-math), [TIES Math](/math/ties-math), [DARE Math](/math/dare-math) for strategy-specific formulas.