WHAT IS LoRA AND WHY IT PREVAILED
Full fine-tuning of an LLM means retraining all model weights. For 7B parameters in fp16 you need to keep in memory the weights themselves, gradients, and Adam optimizer states. By a rough estimate that’s 14 GB for the weights plus another 14 GB for gradients plus double that for Adam moments — totaling around 80 GB. That’s an A100 with 80 GB, and even that is tight. Expensive, slow, and out of reach for most teams.
LoRA (Low-Rank Adaptation) solves the problem differently. The idea from the Microsoft 2021 paper is that weight changes during fine‑tuning have low effective rank. So instead of moving the entire weight matrix W of size d×d, we can freeze it and add two small matrices alongside: A of size d×r and B of size r×d, where r (rank) is a small number, usually 8–64. We train only A and B. The final forward pass is h = Wx + BAx, where BA is the learned adjustment.
The math is simple, but the effect is huge. For the same 7B model, trainable parameters drop from 7 billion to a few million. Memory for the optimizer is negligible because Adam stores moments only for trainable weights. This is PEFT — Parameter‑Efficient Fine‑Tuning, an umbrella term for a whole family of methods where LoRA is the most popular representative.
WHY IT’S CHEAP: NUMBERS
The main savings come not only from training memory but also from inference and storage. One base checkpoint plus dozens of adapters for different tasks instead of dozens of full model copies.
- Memory. QLoRA (LoRA on top of a 4‑bit quantized base) enables tuning a 7–13B model on a consumer card with 16–24 GB VRAM. A 70B model fits into 48 GB, which used to sound like science fiction.
- Artifact size. An adapter for a 7B model with rank 16 weighs roughly 30–80 MB versus 14 GB for a full checkpoint. It can be committed to a repository, run through CI, and shared with the team.
- Time. On a dataset of a few thousand examples, tuning a 7B model takes 1–3 hours on a single modern GPU. Renting such a card costs just a few dollars per run.
- Task switching. You swap the adapter in milliseconds without reloading the base model. One server can serve many «specializations».
QLoRA, DoRA AND OTHER VARIANTS — WHAT TO CHOOSE
Over the years LoRA has accumulated modifications. Fresh releases and tool forks are easy to track in the REDDYX catalog, but the baseline choice in 2026 is as follows:
| Method | Essence | VRAM (7B) | When to use |
|---|---|---|---|
| Full FT | All weights | ~80 GB+ | Lots of data, have a cluster, need maximum quality |
| LoRA | Adapters on fp16 base | ~24–40 GB | Have one card of A100/4090 level, speed matters |
| QLoRA | Adapters on 4‑bit base | ~10–16 GB | Consumer GPU, minimal budget |
| DoRA | Separates magnitude and direction | ~like LoRA | Want to be closer to full FT at the same rank |
According to community benchmarks, QLoRA loses minimal quality compared to regular LoRA while using far less memory — making it a sensible default for most applied tasks. DoRA often gives a small boost at low ranks, but adds overhead; take it if LoRA hits a quality ceiling and you don’t want to increase the rank.
WORKING EXAMPLE: QLoRA ON YOUR DATA
Below is a minimal but complete script using the transformers + peft + trl + bitsandbytes stack. The data format is JSONL, where each line has a text field with a ready-made prompt in the model's chat template.
from datasets import load_dataset
from transformers import (AutoModelForCausalLM, AutoTokenizer,
BitsAndBytesConfig)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
import torch
model_id = "meta-llama/Llama-3.1-8B-Instruct"
# 4-битная загрузка базы — это и есть QLoRA
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tok = AutoTokenizer.from_pretrained(model_id)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_id, quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model)
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
bias="none", task_type="CAUSAL_LM",
# вешаем адаптеры на все линейные слои внимания и MLP
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora)
model.print_trainable_parameters() # ~0.5% параметров обучаемы
ds = load_dataset("json", data_files="train.jsonl", split="train")
cfg = SFTConfig(
output_dir="out-adapter",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # эффективный батч = 16
learning_rate=2e-4,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
bf16=True,
logging_steps=10,
save_strategy="epoch",
dataset_text_field="text",
max_seq_length=2048,
)
trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds)
trainer.train()
trainer.save_model("out-adapter") # сохранится ТОЛЬКО адаптер, ~50 MB
After training, the adapter loads onto the base model in a second:
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
model = PeftModel.from_pretrained(base, "out-adapter")
# при желании слить в один вес для продакшена:
merged = model.merge_and_unload()
merged.save_pretrained("llama-my-task")
HYPERPARAMETERS: WHAT ACTUALLY MATTERS
Rank r and alpha
Rank is the adapter dimension. Start with r=16. For simple stylistic tasks, 8 is enough; for complex domain knowledge, consider 32–64. Above 64, returns usually diminish. For alpha, there's a hard practical rule: keep lora_alpha equal to 2×r or set normalization via rslora — then changing the rank won't break the effective learning rate.
target_modules
Early guides attached LoRA only to q_proj and v_proj. 2026 practice: attach adapters to all linear layers, including MLP (gate/up/down). This noticeably boosts quality with a penny increase in parameters.
Learning rate
For LoRA, the working range is 1e-4 … 3e-4 — an order of magnitude higher than full fine-tuning, because there are few trainable weights and it's not a pity to move them sharply. Cosine schedule with short warmup is a reliable default.
DATA DECIDES MORE THAN METHOD
The main mistake of beginners is tweaking hyperparameters on a bad dataset. LoRA cannot pull useful signal from garbage data. Priorities are exactly as follows:
- Quality matters more than quantity. 500–2000 cleaned, diverse examples often beat 50,000 noisy ones. Duplicates, contradictory answers, and a malformed format are a straight path to degradation.
- Unified chat template. Training prompts must exactly match how the model will be called in production. If the template diverges, you lose quality for no reason.
- Prompt masking. Compute loss only on the assistant's answer, not on the whole text. Otherwise the model wastes capacity memorizing your instructions instead of generating answers.
- Honest holdout. Keep a held-out set and evaluate on it, not just on train loss.
WHEN LoRA IS NOT NEEDED
Fine-tuning is not a silver bullet. Before tuning, honestly ask: can the task be solved cheaper?
- You need fresh facts and knowledge — take RAG, not fine-tuning. LoRA learns form and behavior, not memorization of facts that become outdated.
- The task is formulated in the prompt — first extract the maximum from few-shot and system prompt. Often that's enough.
- Very little data (dozens of examples) — the risk of overfitting outweighs the benefit.
LoRA shines when you need a stable style, specific output format, domain jargon, strict adherence to an answer protocol, or compressing a large system prompt into weights to save tokens and latency.
Frequently Asked Questions
How does LoRA differ from full fine-tuning?
With full fine-tuning, all model weights are updated, requiring tens of gigabytes of memory and powerful hardware. LoRA freezes the base weights and trains small low-rank adapters — usually 0.1–2% of parameters. The quality result on narrow tasks is comparable, but memory and cost are needed far less.
How much data is needed for LoRA?
For stylistic and formatting tasks, 500–2000 quality examples are enough. For complex domain skills — from several thousand. Data cleanliness and diversity matter more than volume: 1,000 cleaned examples usually beat 50 000 noisy ones.
What is QLoRA and when to use it?
QLoRA is LoRA on top of a base model loaded in 4-bit quantization. It reduces VRAM consumption several-fold with almost no loss in quality, allowing fine-tuning of 7–13B models on a consumer graphics card with 16–24 GB of memory. This is a sensible default when budget is limited.
Can the adapter be merged back into the model?
Yes. The merge_and_unload() method from the PEFT library embeds the learned BA correction directly into the weights and returns a regular model without inference overhead. Convenient for production, but then you lose the ability to quickly switch adapters.
LoRA and PEFT in 2026 are the default, not an exotic skill for applied developers. Tools around them are updated literally every week: new quantizers, adapter formats, wrappers for serving. To not miss fresh releases — subscribe to the Telegram channel REDDYX AI — new repositories every 30-60 minutes.