> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# PEFT Overview for EMEP Fine-Tuning

> Parameter-Efficient Fine-Tuning methods in EMEP including LoRA, prefix-tuning, prompt-tuning, and trade-offs versus full fine-tuning.

Parameter-Efficient Fine-Tuning (PEFT) reduces trainable parameters by updating small adapter modules instead of full weights. EMEP uses PEFT to produce specialist inputs for MergeEngine without the cost of full fine-tuning. The PEFT survey (Han et al. 2024) catalogs the adapter families EMEP supports.

## Adapter Families

EMEP implements three adapter families. Each adds parameters to a frozen base model.

### LoRA

LoRA (Hu et al. 2021) decomposes weight updates into low-rank matrices A and B. For a weight matrix W, the update is delta W = A B^T with rank r. Only A and B are trained. LoRA is the default PEFT method in EMEP because it merges cleanly into base weights and has low memory overhead.

See [LoRA](/finetuning/lora) for math, hyperparameters, and the training flowchart.

### Prefix-Tuning

Prefix-tuning prepends trainable prefix vectors to keys and values in attention layers. The base model remains frozen. Prefix-tuning works well for classification and generation tasks but is less flexible for merging than LoRA because prefix vectors do not compose with base weights in the same way.

### Prompt-Tuning

Prompt-tuning trains soft prompt embeddings at the input layer. It is the most parameter-efficient of the three but also the most task-specific. Prompt-tuned models may require task-specific prompts at inference time, which complicates deployment in InferenceBackend.

## Trade-Offs

| Method        | Trainable Params       | Mergeability           | Forgetting Risk | Inference Cost               |
| ------------- | ---------------------- | ---------------------- | --------------- | ---------------------------- |
| Full FT       | 100%                   | N/A (replaces weights) | High            | Same as base                 |
| LoRA          | less than 1% (typical) | High (mergeable)       | Low (partial)   | Same as base after merge     |
| Prefix-tuning | about 0.1%             | Low                    | Low             | Same as base + prefix length |
| Prompt-tuning | about 0.01%            | Very low               | Very low        | Same as base + prompt length |

## When to Prefer PEFT Over Full FT

Prefer PEFT when:

* The base model is large and GPU memory is constrained.
* You need multiple specialist variants of the same base model.
* The target task is close to the base model's pretraining distribution.
* You intend to merge the fine-tuned model with others in MergeEngine.

Use full fine-tuning only when adapters fail to reach target accuracy or when the task distribution diverges significantly from pretraining.

## Integration with MergeEngine

LoRA adapters can be merged into base weights before entering MergeEngine, or kept as separate adapter artifacts. MergeEngine treats a merged LoRA model as a standard model with VALIDATED status. Prefix-tuned and prompt-tuned models require conversion or are treated as separate model variants with limited merge compatibility.

## Unsupported Claim

Prefix-tuning and prompt-tuning are not guaranteed to be compatible with all MergeStrategy implementations. Compatibility is determined at runtime by ModelCompatibilityAnalyzer.