All Projects
Fine-tuning
QLoRA
LLM
HuggingFace
AI
LLM Fine-tuning Pipeline
QLoRA fine-tuning of Phi-2 on custom instruction dataset with streaming inference
Full fine-tuning of large language models requires 80 GB+ VRAM and significant compute cost. Applied QLoRA to fine-tune Phi-2 (2.7B) on a custom instruction dataset using a free GPU (Colab T4) and served it with streaming inference.
January 1, 2025
10 viewsTech Stack
QLoRAHuggingFacePEFTTRLBitsAndBytesFastAPIPhi-2
Overview
Applied QLoRA — 4-bit NF4 quantisation via BitsAndBytes reduced VRAM from ~12 GB to ~6 GB; trained only LoRA adapter matrices (r=16, alpha=32) using TRL SFTTrainer with completion-only loss masking, cosine LR schedule, and gradient accumulation on a Google Colab T4. Fine-tuned Phi-2 (2.7B) adapter evaluated with ROUGE-1/2/L, corpus BLEU, and perplexity; served via FastAPI with token-by-token SSE streaming; adapter pushed to HuggingFace Hub.
The Problem
Full fine-tuning of large language models requires 80 GB+ VRAM and significant compute cost, making domain adaptation inaccessible on standard hardware.
The Solution
QLoRA with 4-bit NF4 quantisation reducing VRAM from ~12 GB to ~6 GB, LoRA adapter matrices, TRL SFTTrainer, evaluated with ROUGE/BLEU/perplexity, served via FastAPI SSE.