> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# Task Arithmetic: Task Vectors and Linear Combinations

> Task Arithmetic mathematical specification for EMEP. Defines task vectors as theta_ft minus theta_pre, linear combinations, skill addition, forgetting/negation, and combination rules. Reference: Ilharco et al. 2022.

Task Arithmetic treats fine-tuning as a vector in parameter space. This page defines task vectors, their linear combinations, and the rules for adding skills and removing biases. Reference: Ilharco et al. 2022.

## Task Vector Definition

Given a pre-trained base model theta\_pre and a fine-tuned model theta\_ft:

```text theme={null}
tau = theta_ft - theta_pre
```

tau is a task vector: a set of parameter deltas that encode the skill or behavior learned during fine-tuning.

## Linear Combinations

Multiple task vectors can be combined with scalar coefficients:

```text theme={null}
theta_merged = theta_pre + sum_i (alpha_i * tau_i)
```

| Symbol        | Meaning                              |
| ------------- | ------------------------------------ |
| theta\_pre    | Pre-trained base model parameters    |
| tau\_i        | Task vector i                        |
| alpha\_i      | Scalar coefficient for task vector i |
| theta\_merged | Resulting merged model               |

## Adding Skills

To add a skill, set the corresponding alpha to a positive value:

```text theme={null}
theta_with_skill = theta_pre + 1.0 * tau_skill
```

Multiple skills can be added simultaneously:

```text theme={null}
theta_multi_skill = theta_pre + 0.5 * tau_math + 0.5 * tau_code
```

## Forgetting and Negation

To remove a bias or unwanted behavior, negate the task vector:

```text theme={null}
theta_without_bias = theta_pre + (-1.0) * tau_bias
```

This is the basis of forgetting controls in EMEP. See [Catastrophic Forgetting Controls](/finetuning/catastrophic-forgetting-controls) for extended discussion. Reference: McCloskey & Cohen 1989; Kirkpatrick et al. 2017.

## Combination Rules

| Operation      | Formula                                | Effect                             |
| -------------- | -------------------------------------- | ---------------------------------- |
| Add skill      | theta\_pre + alpha \* tau\_skill       | Enhances skill                     |
| Remove bias    | theta\_pre - alpha \* tau\_bias        | Suppresses bias                    |
| Merge skills   | theta\_pre + sum\_i alpha\_i \* tau\_i | Combines multiple skills           |
| Average models | theta\_pre + average(tau\_i)           | Model Soups (Wortsman et al. 2022) |

## Preconditions

All task vectors must be derived from the same theta\_pre. Merging task vectors from different base models is undefined and produces INVALID.

| Condition             | Outcome                             |
| --------------------- | ----------------------------------- |
| Different base models | INVALID                             |
| Missing theta\_pre    | INVALID                             |
| alpha too large       | Catastrophic forgetting; QUARANTINE |

## Failure Modes

| Condition                        | Outcome                          |
| -------------------------------- | -------------------------------- |
| Base model missing               | Cannot compute task vector; FAIL |
| alpha\_i sum much greater than 1 | Parameter drift; QUARANTINE      |
| alpha\_i sum much less than 0    | Inverted skills; QUARANTINE      |

## Cross-Links

* [Merge Math](/math/merge-math) for shared notation and weight shapes.
* [Merge Strategies](/merge/merge-strategies) for when Task Arithmetic is selected.
* [Catastrophic Forgetting Controls](/finetuning/catastrophic-forgetting-controls) for forgetting mitigation.
* [Tensor Operations](/merge/tensor-operations) for the TensorEngine implementation.