> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# LoRA Specification for EMEP Fine-Tuning

> LoRA (Hu et al. 2021) math, hyperparameters, merging into base weights, and the EMEP training flowchart.

LoRA (Hu et al. 2021) is the default parameter-efficient fine-tuning method in EMEP. It decomposes weight updates into low-rank matrices, trains only those matrices, and merges them back into base weights for inference and merging.

## Low-Rank Decomposition

For a pretrained weight matrix W, LoRA represents the update as:

```text theme={null}
W' = W + delta W = W + A B^T
```

* A has shape (d, r)
* B has shape (k, r)
* r is much less than min(d, k)

Only A and B are updated during training. W remains frozen. The rank r controls the capacity of the adapter.

## Hyperparameters

| Parameter       | Default                 | Range                       | Description                                  |
| --------------- | ----------------------- | --------------------------- | -------------------------------------------- |
| r               | 8                       | 1-64                        | Rank of the decomposition                    |
| lora\_alpha     | 16                      | 1-128                       | Scaling factor; effective scale is alpha / r |
| lora\_dropout   | 0.0                     | 0.0-0.5                     | Dropout applied to A and B                   |
| target\_modules | \["q\_proj", "v\_proj"] | list                        | Which weight matrices to adapt               |
| bias            | "none"                  | "none", "all", "lora\_only" | Whether to train biases                      |

The scaling factor is applied as: `h = W x + (alpha / r) * A B^T x`.

## Merging LoRA into Base Weights

After training, the adapter is merged for inference and compatibility analysis:

```text theme={null}
W_merged = W + (alpha / r) * A * B^T
```

MergeEngine treats W\_merged as a standard weight matrix. The merge is deterministic and reversible if the original W and adapter are preserved in ArtifactStore.

## LoRA Training Flowchart

```mermaid theme={null}
flowchart TD
    START(["START"]) --> INPUT["INPUT: Base Model + Dataset"]
    INPUT --> INIT["Initialize A (Gaussian), B (Zero)"]
    INIT --> TRAIN["Training Loop"]
    TRAIN --> FORWARD["Forward: h = Wx + (alpha/r) AB^T x"]
    FORWARD --> LOSS["Compute Loss"]
    LOSS --> BACKWARD["Backward: grad A, grad B only"]
    BACKWARD --> CHECK["Epoch / Step Check"]
    CHECK -->|"Continue"| TRAIN
    CHECK -->|"Done"| SAVE["Save Adapter (A, B, config)"]
    SAVE --> MERGE["Optional: Merge into W"]
    MERGE --> REGISTER["Register to ModelRegistry"]
    REGISTER --> END(["END"])
```

## Integration Points

* **ModelLoader**: loads base model and freezes all weights except LoRA parameters.
* **DatasetRegistry**: provides training and validation splits.
* **ExperimentTracker**: logs hyperparameters, loss curves, and checkpoint paths.
* **ArtifactStore**: stores adapter checkpoints and merged weights.
* **ModelRegistry**: receives the final model after VALIDATED status.

## Engineering Assumption

LoRA adapters for attention layers (q\_proj, v\_proj) generalize across transformer architectures. EMEP assumes this holds for all supported models, but compatibility is verified by ModelCompatibilityAnalyzer before merge.