Task Arithmetic: Task Vectors and Linear Combinations

Task Arithmetic mathematical specification for EMEP. Defines task vectors as theta_ft minus theta_pre, linear combinations, skill addition, forgetting/negation, and combination rules. Reference: Ilharco et al. 2022.

Task Arithmetic treats fine-tuning as a vector in parameter space. This page defines task vectors, their linear combinations, and the rules for adding skills and removing biases. Reference: Ilharco et al. 2022.

Task Vector Definition

Given a pre-trained base model theta_pre and a fine-tuned model theta_ft:

tau = theta_ft - theta_pre

tau is a task vector: a set of parameter deltas that encode the skill or behavior learned during fine-tuning.

Linear Combinations

Multiple task vectors can be combined with scalar coefficients:

theta_merged = theta_pre + sum_i (alpha_i * tau_i)

Symbol	Meaning
theta_pre	Pre-trained base model parameters
tau_i	Task vector i
alpha_i	Scalar coefficient for task vector i
theta_merged	Resulting merged model

Adding Skills

To add a skill, set the corresponding alpha to a positive value:

theta_with_skill = theta_pre + 1.0 * tau_skill

Multiple skills can be added simultaneously:

theta_multi_skill = theta_pre + 0.5 * tau_math + 0.5 * tau_code

Forgetting and Negation

To remove a bias or unwanted behavior, negate the task vector:

theta_without_bias = theta_pre + (-1.0) * tau_bias

This is the basis of forgetting controls in EMEP. See Catastrophic Forgetting Controls for extended discussion. Reference: McCloskey & Cohen 1989; Kirkpatrick et al. 2017.

Combination Rules

Operation	Formula	Effect
Add skill	theta_pre + alpha * tau_skill	Enhances skill
Remove bias	theta_pre - alpha * tau_bias	Suppresses bias
Merge skills	theta_pre + sum_i alpha_i * tau_i	Combines multiple skills
Average models	theta_pre + average(tau_i)	Model Soups (Wortsman et al. 2022)

Preconditions

All task vectors must be derived from the same theta_pre. Merging task vectors from different base models is undefined and produces INVALID.

Condition	Outcome
Different base models	INVALID
Missing theta_pre	INVALID
alpha too large	Catastrophic forgetting; QUARANTINE

Failure Modes

Condition	Outcome
Base model missing	Cannot compute task vector; FAIL
alpha_i sum much greater than 1	Parameter drift; QUARANTINE
alpha_i sum much less than 0	Inverted skills; QUARANTINE

Cross-Links

• Merge Math for shared notation and weight shapes.
• Merge Strategies for when Task Arithmetic is selected.
• Catastrophic Forgetting Controls for forgetting mitigation.
• Tensor Operations for the TensorEngine implementation.