LoRA Specification for EMEP Fine-Tuning

LoRA (Hu et al. 2021) math, hyperparameters, merging into base weights, and the EMEP training flowchart.

LoRA (Hu et al. 2021) is the default parameter-efficient fine-tuning method in EMEP. It decomposes weight updates into low-rank matrices, trains only those matrices, and merges them back into base weights for inference and merging.

Low-Rank Decomposition

For a pretrained weight matrix W, LoRA represents the update as:

W' = W + delta W = W + A B^T

• A has shape (d, r)
• B has shape (k, r)
• r is much less than min(d, k)

Only A and B are updated during training. W remains frozen. The rank r controls the capacity of the adapter.

Hyperparameters

Parameter	Default	Range	Description
r	8	1-64	Rank of the decomposition
lora_alpha	16	1-128	Scaling factor; effective scale is alpha / r
lora_dropout	0.0	0.0-0.5	Dropout applied to A and B
target_modules	[“q_proj”, “v_proj”]	list	Which weight matrices to adapt
bias	”none"	"none”, “all”, “lora_only”	Whether to train biases

The scaling factor is applied as: h = W x + (alpha / r) * A B^T x.

Merging LoRA into Base Weights

After training, the adapter is merged for inference and compatibility analysis:

W_merged = W + (alpha / r) * A * B^T

MergeEngine treats W_merged as a standard weight matrix. The merge is deterministic and reversible if the original W and adapter are preserved in ArtifactStore.

LoRA Training Flowchart

Integration Points

• ModelLoader: loads base model and freezes all weights except LoRA parameters.
• DatasetRegistry: provides training and validation splits.
• ExperimentTracker: logs hyperparameters, loss curves, and checkpoint paths.
• ArtifactStore: stores adapter checkpoints and merged weights.
• ModelRegistry: receives the final model after VALIDATED status.

Engineering Assumption

LoRA adapters for attention layers (q_proj, v_proj) generalize across transformer architectures. EMEP assumes this holds for all supported models, but compatibility is verified by ModelCompatibilityAnalyzer before merge.