QLoRA Specification for EMEP Fine-Tuning

QLoRA (Dettmers et al. 2023) with NF4 quantization and LoRA adapters for memory-efficient fine-tuning in EMEP.

QLoRA (Dettmers et al. 2023) combines 4-bit Normal Float (NF4) quantization with LoRA adapters to reduce memory during fine-tuning. EMEP uses QLoRA when GPU memory is insufficient for full-precision LoRA or when training very large models on limited hardware.

NF4 Quantization

NF4 is a information-theoretically optimal 4-bit data type for normally distributed weights. The quantization steps are:

1. Quantize pretrained weights W to 4-bit NF4.
2. Store quantized weights in a blockwise format with 64-element blocks.
3. Dequantize on-the-fly during the forward pass to 16-bit or 32-bit compute.
4. Apply LoRA to the dequantized weights.

The base model weights remain frozen and quantized. Only the LoRA adapters (A and B) are trained in full precision.

Memory Savings

QLoRA reduces memory compared to full-precision LoRA by quantizing the base model weights. The exact savings depend on model size, sequence length, batch size, and optimizer state. Memory for adapter parameters and gradients remains full precision.

QLoRA Training Flowchart

Hyperparameters

QLoRA inherits all LoRA hyperparameters and adds quantization-specific settings:

Parameter	Default	Description
bnb_4bit_quant_type	”nf4”	Quantization type (nf4 or fp4)
bnb_4bit_use_double_quant	true	Nested quantization for memory savings
bnb_4bit_compute_dtype	bfloat16	Compute dtype for dequantized weights

Integration Points

• QuantizationEngine: handles NF4 quantization and dequantization.
• ModelLoader: loads quantized base model and attaches LoRA adapters.
• GPU Orchestration: allocates adapters and optimizer state in full precision, base weights in NF4.
• ArtifactStore: stores quantized base reference and adapter checkpoints separately.

Unsupported Claim

QLoRA does not guarantee identical convergence to full-precision LoRA. The quantization error is small for normally distributed weights, but outlier features may degrade performance. Regression evaluation is required before promotion.