DARE Math: Drop and Rescale for Task Vectors

DARE mathematical specification for EMEP. Defines random sparsification with drop probability p, rescaling by 1/(1-p), and application per task vector. Reference: Yu et al. 2023.

DARE (Drop And REscale) randomly sparsifies task vectors and rescales the remaining values to preserve expected magnitude. This page defines the math used by the TensorEngine.

Formula

Given a task vector tau and drop probability p in [0, 1):

M ~ Bernoulli(1 - p)   // mask: 1 with probability (1 - p), 0 with probability p

tau_dare = tau circle_dot M / (1 - p)

Symbol	Meaning
tau	Task vector: tau = theta_ft - theta_pre
p	Drop probability: fraction of parameters to drop
M	Random binary mask, same shape as tau
circle_dot	Element-wise (Hadamard) product
/ (1 - p)	Rescale survivors by 1 / (1 - p)

Expected Value

The expected value of tau_dare equals the original tau:

E[tau_dare_j] = E[ tau_j * M_j / (1 - p) ]
              = tau_j * (1 - p) / (1 - p)
              = tau_j

This is the key property: DARE preserves the expected task vector while reducing density.

Parameters

Parameter	Symbol	Range	Default
drop probability	p	[0.0, 1.0)	0.5
random seed	-	int	42

p = 0 is a no-op (no parameters dropped). p approaching 1 drops almost all parameters and amplifies noise from rescaling.

Application Per Task Vector

DARE is applied independently to each task vector before merging:

for each task vector tau_i:
    tau_i_dare = DARE(tau_i, p)

theta_merged = theta_pre + sum_i tau_i_dare

Combining with TIES

DARE and TIES can be combined: first trim by magnitude (TIES step 1), then apply DARE sparsification to the trimmed vectors, then elect sign and disjoint merge.

tau_i_trimmed = Trim(tau_i, d)
tau_i_dare = DARE(tau_i_trimmed, p)
tau_merged = DisjointMerge(tau_i_dare)
theta_merged = theta_pre + tau_merged

See TIES Math for trim and disjoint merge definitions.

Failure Modes

Condition	Outcome
p = 1.0	Division by zero; INVALID
All parameters dropped by chance	tau_dare = 0; merge has no effect
p too high with few task vectors	High variance in merged model; QUARANTINE

Numerical Stability

Rescaling by 1 / (1 - p) amplifies any remaining noise. For p > 0.9, the amplification factor exceeds 10 and may cause overflow in fp16. Compute in fp32 and cast to target dtype after rescaling.

Cross-Links

• Merge Math for shared notation and task vector definitions.
• Merge Strategies for when DARE is selected.
• TIES Math for combining DARE with TIES.
• Tensor Operations for the TensorEngine implementation.