> ## Documentation Index
> Fetch the complete documentation index at: https://doc.blueapi.ir/llms.txt
> Use this file to discover all available pages before exploring further.

# QLoRA Specification for EMEP Fine-Tuning

> QLoRA (Dettmers et al. 2023) with NF4 quantization and LoRA adapters for memory-efficient fine-tuning in EMEP.

QLoRA (Dettmers et al. 2023) combines 4-bit Normal Float (NF4) quantization with LoRA adapters to reduce memory during fine-tuning. EMEP uses QLoRA when GPU memory is insufficient for full-precision LoRA or when training very large models on limited hardware.

## NF4 Quantization

NF4 is a information-theoretically optimal 4-bit data type for normally distributed weights. The quantization steps are:

1. Quantize pretrained weights W to 4-bit NF4.
2. Store quantized weights in a blockwise format with 64-element blocks.
3. Dequantize on-the-fly during the forward pass to 16-bit or 32-bit compute.
4. Apply LoRA to the dequantized weights.

The base model weights remain frozen and quantized. Only the LoRA adapters (A and B) are trained in full precision.

## Memory Savings

QLoRA reduces memory compared to full-precision LoRA by quantizing the base model weights. The exact savings depend on model size, sequence length, batch size, and optimizer state. Memory for adapter parameters and gradients remains full precision.

## QLoRA Training Flowchart

```mermaid theme={null}
flowchart TD
    START(["START"]) --> INPUT["INPUT: Base Model + Dataset"]
    INPUT --> QUANTIZE["Quantize W to NF4 (blockwise)"]
    QUANTIZE --> INIT["Initialize LoRA A, B"]
    INIT --> TRAIN["Training Loop"]
    TRAIN --> DEQUANT["Dequantize W to bf16/fp16"]
    DEQUANT --> FORWARD["Forward: h = W_dequant x + (alpha/r) AB^T x"]
    FORWARD --> LOSS["Compute Loss"]
    LOSS --> BACKWARD["Backward: grad A, grad B only"]
    BACKWARD --> CHECK["Epoch / Step Check"]
    CHECK -->|"Continue"| TRAIN
    CHECK -->|"Done"| SAVE["Save Adapter + NF4 Base Reference"]
    SAVE --> MERGE["Optional: Dequantize and Merge"]
    MERGE --> REGISTER["Register to ModelRegistry"]
    REGISTER --> END(["END"])
```

## Hyperparameters

QLoRA inherits all LoRA hyperparameters and adds quantization-specific settings:

| Parameter                     | Default  | Description                            |
| ----------------------------- | -------- | -------------------------------------- |
| bnb\_4bit\_quant\_type        | "nf4"    | Quantization type (nf4 or fp4)         |
| bnb\_4bit\_use\_double\_quant | true     | Nested quantization for memory savings |
| bnb\_4bit\_compute\_dtype     | bfloat16 | Compute dtype for dequantized weights  |

## Integration Points

* **QuantizationEngine**: handles NF4 quantization and dequantization.
* **ModelLoader**: loads quantized base model and attaches LoRA adapters.
* **GPU Orchestration**: allocates adapters and optimizer state in full precision, base weights in NF4.
* **ArtifactStore**: stores quantized base reference and adapter checkpoints separately.

## Unsupported Claim

QLoRA does not guarantee identical convergence to full-precision LoRA. The quantization error is small for normally distributed weights, but outlier features may degrade performance. Regression evaluation is required before promotion.