> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Tensor Math: Broadcasting, Operations, and Precision

> Tensor math specification for EMEP. Defines broadcasting rules, elementwise operations, reductions, matmul shape rules for transformer weights, and precision ranges for fp32, bf16, fp16, fp8, int8, and int4.

Tensor math is the foundation of all operations in TensorEngine. This page defines broadcasting rules, elementwise operations, reductions, matrix multiplication shape contracts, and precision ranges.

## Broadcasting Rules

Two tensors are broadcastable if, for each dimension from trailing to leading, the dimensions are equal or one of them is 1. A missing dimension is treated as 1.

| Shape A | Shape B   | Broadcasted Shape |
| ------- | --------- | ----------------- |
| (H, I)  | (H, 1)    | (H, I)            |
| (H, I)  | (1, I)    | (H, I)            |
| (H, I)  | (I,)      | (H, I)            |
| (V, H)  | (H,)      | (V, H)            |
| (H,)    | ()        | (H,)              |
| (H, I)  | (H, I, K) | Not broadcastable |

Broadcasting is applied before elementwise operations. The TensorEngine validates broadcastability before executing any op.

## Elementwise Operations

For elementwise add, sub, mul, div:

```text theme={null}
C = A op B

shape(C) = broadcast(shape(A), shape(B))
dtype(C) = promote(dtype(A), dtype(B))
```

| dtype A | dtype B | Promoted dtype          |
| ------- | ------- | ----------------------- |
| fp32    | any     | fp32                    |
| bf16    | fp16    | fp32                    |
| bf16    | bf16    | bf16                    |
| fp16    | fp16    | fp16                    |
| int8    | int8    | int16                   |
| int4    | any     | fp32 (dequantize first) |

## Reductions

| Operation | Input     | Output | Notes                  |   |           |
| --------- | --------- | ------ | ---------------------- | - | --------- |
| sum       | (H, I)    | scalar | Accumulate in fp32     |   |           |
| mean      | (H, I)    | scalar | Accumulate in fp32     |   |           |
| max       | (H, I)    | scalar | Direct on input dtype  |   |           |
| argmax    | (H, I)    | scalar | Returns index          |   |           |
| L2 norm   | flattened | scalar | sqrt(sum(x^2)) in fp32 |   |           |
| L1 norm   | flattened | scalar | sum(                   | x | ) in fp32 |

## Matmul Shape Rules for Transformer Weights

For a linear layer with weight W and input x:

```text theme={null}
x: (batch, seq_len, H)
W: (H, I)
output: (batch, seq_len, I)
```

For attention Q/K/V/O projections:

```text theme={null}
x: (batch, seq_len, H)
W_q: (H, H)
W_k: (H, K * H / A)
W_v: (H, K * H / A)
W_o: (H, H)
```

For MLP SwiGLU:

```text theme={null}
x: (batch, seq_len, H)
W_up: (H, I)
W_gate: (H, I)
W_down: (I, H)
```

## Precision and Numerical Range

| Precision | Bits | Exponent | Mantissa | Approx Range      | Notes                                  |
| --------- | ---- | -------- | -------- | ----------------- | -------------------------------------- |
| fp32      | 32   | 8        | 23       | 1.4e-45 to 3.4e38 | Safe for all compute                   |
| bf16      | 16   | 8        | 7        | 1.2e-38 to 3.4e38 | Same range as fp32, less precision     |
| fp16      | 16   | 5        | 10       | 5.96e-8 to 65504  | Overflow risk in large norms           |
| fp8       | 8    | varies   | varies   | Narrow            | Experimental, limited support          |
| int8      | 8    | N/A      | N/A      | -128 to 127       | Quantized, requires scale              |
| int4      | 4    | N/A      | N/A      | 0 to 15           | Highly quantized, dequantize mandatory |

## Numerical Stability Rules

1. All elementwise accumulation uses fp32 intermediates.
2. Norms are computed in fp32 regardless of input dtype.
3. SLERP dot products use fp64 for precision.
4. int4 and int8 weights are dequantized to fp32 before merge ops.
5. Results are cast to the target dtype only at the final step.

## Cross-Links

* [Merge Math](/math/merge-math) for shared notation and weight shapes.
* [Tensor Operations](/merge/tensor-operations) for the TensorEngine operation catalog.
* [SLERP Math](/math/slerp-math) for spherical interpolation formulas.
* [TIES Math](/math/ties-math) for trim and disjoint merge.
* [DARE Math](/math/dare-math) for sparsification and rescaling.