ML & Model Training

LoRA Doctor

Fine-tuned a language model on medical Q&A using LoRA — training 0.23% of the parameters of a full fine-tune.

View code

0.23%

of params trained (LoRA)

294K

adapter params vs 125M

r=8

LoRA rank on q/v projections

PyTorchPEFTHuggingFaceLoRATransformers

The problem

Full fine-tuning is expensive and out of reach on consumer hardware. The goal: adapt a base model to a specialised domain (medical Q&A) cheaply, and prove the efficiency gain with hard numbers.

The approach

Applied LoRA (Low-Rank Adaptation) to facebook/opt-125m: freeze the base weights and train small rank-8 adapter matrices on the attention q/v projections only. Trained on the medalpaca medical-QA dataset for 3 epochs on an Apple M1 (MPS) — no GPU cluster. The result learns domain terminology and response patterns while touching a tiny fraction of the weights.

Architecture

Base model: facebook/opt-125m   (weights frozen)
        │
        ▼
LoRA adapters (r=8, α=32) on q_proj, v_proj
        │
Full fine-tune:  125,534,208 trainable params
LoRA fine-tune:      294,912 trainable params   ← 0.23%
        │
        ▼
Train 3 epochs · medalpaca medical_meadow_medqa · Apple M1 (MPS)

Key decisions & trade-offs

▸Target only q_proj / v_proj — the highest-leverage attention projections for adaptation.
▸Rank-8 adapters: enough capacity for the domain, small enough to train on a laptop.
▸Quantified everything — 'it works' means nothing without the parameter and cost numbers.

Results / impact

✓Adapted a base model to medical Q&A while training just 0.23% of the parameters (294K vs 125M).
✓Ran end-to-end on an Apple M1 laptop — no GPU cluster required.
✓Learned domain terminology and response patterns with the base model fully frozen.

Roadmap

▸Host the adapter on a free Hugging Face Space for live inference.
▸Benchmark before/after on a held-out medical-QA set.

← All projects