> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# DARE Math: Drop and Rescale for Task Vectors

> DARE mathematical specification for EMEP. Defines random sparsification with drop probability p, rescaling by 1/(1-p), and application per task vector. Reference: Yu et al. 2023.

DARE (Drop And REscale) randomly sparsifies task vectors and rescales the remaining values to preserve expected magnitude. This page defines the math used by the TensorEngine.

## Formula

Given a task vector tau and drop probability p in \[0, 1):

```text theme={null}
M ~ Bernoulli(1 - p)   // mask: 1 with probability (1 - p), 0 with probability p

tau_dare = tau circle_dot M / (1 - p)
```

| Symbol      | Meaning                                          |
| ----------- | ------------------------------------------------ |
| tau         | Task vector: tau = theta\_ft - theta\_pre        |
| p           | Drop probability: fraction of parameters to drop |
| M           | Random binary mask, same shape as tau            |
| circle\_dot | Element-wise (Hadamard) product                  |
| / (1 - p)   | Rescale survivors by 1 / (1 - p)                 |

## Expected Value

The expected value of tau\_dare equals the original tau:

```text theme={null}
E[tau_dare_j] = E[ tau_j * M_j / (1 - p) ]
              = tau_j * (1 - p) / (1 - p)
              = tau_j
```

This is the key property: DARE preserves the expected task vector while reducing density.

## Parameters

| Parameter        | Symbol | Range       | Default |
| ---------------- | ------ | ----------- | ------- |
| drop probability | p      | \[0.0, 1.0) | 0.5     |
| random seed      | -      | int         | 42      |

p = 0 is a no-op (no parameters dropped). p approaching 1 drops almost all parameters and amplifies noise from rescaling.

## Application Per Task Vector

DARE is applied independently to each task vector before merging:

```text theme={null}
for each task vector tau_i:
    tau_i_dare = DARE(tau_i, p)

theta_merged = theta_pre + sum_i tau_i_dare
```

## Combining with TIES

DARE and TIES can be combined: first trim by magnitude (TIES step 1), then apply DARE sparsification to the trimmed vectors, then elect sign and disjoint merge.

```text theme={null}
tau_i_trimmed = Trim(tau_i, d)
tau_i_dare = DARE(tau_i_trimmed, p)
tau_merged = DisjointMerge(tau_i_dare)
theta_merged = theta_pre + tau_merged
```

See [TIES Math](/math/ties-math) for trim and disjoint merge definitions.

## Failure Modes

| Condition                        | Outcome                                   |
| -------------------------------- | ----------------------------------------- |
| p = 1.0                          | Division by zero; INVALID                 |
| All parameters dropped by chance | tau\_dare = 0; merge has no effect        |
| p too high with few task vectors | High variance in merged model; QUARANTINE |

## Numerical Stability

Rescaling by 1 / (1 - p) amplifies any remaining noise. For p > 0.9, the amplification factor exceeds 10 and may cause overflow in fp16. Compute in fp32 and cast to target dtype after rescaling.

## Cross-Links

* [Merge Math](/math/merge-math) for shared notation and task vector definitions.
* [Merge Strategies](/merge/merge-strategies) for when DARE is selected.
* [TIES Math](/math/ties-math) for combining DARE with TIES.
* [Tensor Operations](/merge/tensor-operations) for the TensorEngine implementation.