Tensor Operations: Low-Level Merge Primitives

Specification of every low-level tensor operation used by EMEP merge strategies. Covers input/output shapes, dtype rules, broadcasting, and numerical stability.

TensorEngine exposes a fixed set of low-level operations that all merge strategies use. This page defines each operation, its shape contracts, dtype rules, and numerical stability notes.

Generic Tensor Op Flow

Every operation follows this pipeline.

Operation Catalog

Element-wise Add / Sub

Property	Value
Inputs	A, B: tensors of broadcastable shapes
Output	Same broadcasted shape
Dtype rule	Promote to higher precision: fp16 + fp32 → fp32
Numerical stability	Watch for catastrophic cancellation in fp16. Use fp32 accumulator.

Scalar Mul

Property	Value
Inputs	A: tensor; s: scalar
Output	Same shape as A
Dtype rule	Scalar cast to A.dtype before multiply
Numerical stability	Overflow risk in fp16 for s > 65504. Clamp or use fp32.

Weighted Average

Property	Value
Inputs	A, B: tensors; w: scalar weight in [0, 1]
Output	Same broadcasted shape
Dtype rule	Compute in fp32, cast output to target dtype
Numerical stability	w * A + (1 - w) * B. Accumulate in fp32.

Normalization

Property	Value
Inputs	A: tensor; norm type: L1, L2, or inf
Output	Same shape as A
Dtype rule	Compute norm in fp32 regardless of input dtype
Numerical stability	Guard against division by zero with epsilon = 1e-8.

Linear Interpolation (Lerp)

Property	Value
Inputs	A, B: tensors; t: scalar in [0, 1]
Output	Same broadcasted shape
Dtype rule	Compute in fp32, cast to target
Numerical stability	Straightforward. No special guards needed.

Spherical Interpolation (SLERP)

Property	Value
Inputs	A, B: 1-D parameter vectors; t: scalar in [0, 1]
Output	1-D vector, same length
Dtype rule	Normalize in fp64 for precision, cast to target
Numerical stability	Fallback to lerp when theta < 1e-6 or	theta - pi	< 1e-6. See SLERP Math.

Sparsification

Property	Value
Inputs	A: tensor; p: drop probability in [0, 1)
Output	Same shape as A, with (1 - p) fraction of values kept
Dtype rule	Mask in bool, apply to A in original dtype
Numerical stability	Rescale survivors by 1 / (1 - p). See DARE Math.

Sign Masking

Property	Value
Inputs	A: tensor; sign: +1, -1, or 0
Output	Same shape as A, zeroed where sign mismatch
Dtype rule	Sign computed in fp32, mask applied in original dtype
Numerical stability	No special guards.

Rescaling

Property	Value
Inputs	A: tensor; scale: scalar
Output	Same shape as A
Dtype rule	Scale cast to A.dtype
Numerical stability	Overflow risk in fp16. Use fp32 intermediate.

Tensor Concat / Slice

Property	Value
Inputs	A, B: tensors; dim: int
Output	Concatenated shape along dim
Dtype rule	Both tensors must have identical dtype
Numerical stability	N/A

Layer Selection

Property	Value
Inputs	model: state_dict; layer_indices: List[int]
Output	Subset state_dict with only selected layers
Dtype rule	Preserved from input
Numerical stability	N/A

Cross-Links

• Merge Engine for the pipeline that calls these ops.
• Merge Strategies for which strategies use which ops.
• Merge Math for shared notation.
• SLERP Math, TIES Math, DARE Math for strategy-specific formulas.