TIES Math: Trim, Elect Sign, and Disjoint Merge

TIES mathematical specification for EMEP. Defines the three-step trim-by-magnitude, elect-sign, and disjoint-merge formulas with density parameter and rescaling. Reference: Yadav et al. 2023.

TIES (Trim, Elect Sign, and Merge) reduces interference in multi-model merging by keeping only the most significant parameter changes and resolving sign conflicts. This page defines the three-step math used by the TensorEngine.

Overview

TIES operates on task vectors tau_i = theta_ft_i - theta_pre. The three steps are:

1. Trim: Keep only the top-d fraction of parameters by magnitude in each task vector.
2. Elect Sign: Resolve sign conflicts across task vectors by majority vote.
3. Disjoint Merge: Merge only parameters that agree with the elected sign.

Step 1: Trim by Magnitude

Given a task vector tau and density parameter d in (0, 1]:

M_d(tau) = mask where |tau_j| >= threshold_d

threshold_d = d-th percentile of |tau|

tau_trimmed = tau circle_dot M_d(tau)

Symbol	Meaning
tau	Task vector
d	Density parameter, fraction of parameters to keep
M_d(tau)	Binary mask: 1 if	tau_j	is in top d, else 0
circle_dot	Element-wise (Hadamard) product

Parameters with magnitude below the d-th percentile are zeroed out.

Step 2: Elect Sign

Given N trimmed task vectors tau_trimmed_i, elect a sign for each parameter position j:

s_j = sign( sum_i tau_trimmed_i_j )

Symbol	Meaning
s_j	Elected sign for position j: +1, -1, or 0
sign(x)	+1 if x > 0, -1 if x < 0, 0 if x = 0

If the sum is exactly zero, the elected sign is 0 and the parameter is excluded from merging.

Step 3: Disjoint Merge

Merge only parameters where the elected sign matches the task vector’s sign:

tau_merged_j = sum_i [ tau_trimmed_i_j if sign(tau_trimmed_i_j) == s_j else 0 ]

The final merged model is:

theta_merged = theta_pre + tau_merged

Rescaling

After disjoint merge, the magnitude of tau_merged may differ from the average magnitude of the source task vectors. Rescaling is optional and controlled by a config flag:

tau_rescaled = tau_merged * ( mean_i ||tau_trimmed_i||_2 / ||tau_merged||_2 )

If ||tau_merged||_2 is zero (all parameters excluded), the merge fails and produces INVALID.

Parameters

Parameter	Symbol	Range	Default
density	d	(0, 1]	0.6
sign consensus	-	”majority” or “all"	"majority”
rescale	-	bool	false

Failure Modes

Condition	Outcome
d = 0	All parameters trimmed; INVALID
All signs conflict at every position	tau_merged = 0; INVALID
Rescaling with zero norm	INVALID

Cross-Links

• Merge Math for shared notation and task vector definitions.
• Merge Strategies for when TIES is selected.
• DARE Math for combining TIES with DARE sparsification.
• Tensor Operations for the TensorEngine implementation.