LoRA & QLoRA Recipes

This skill assumes the routing decision already happened — finetuning-method-selection should have already pointed here because the data shape is demonstrations (SFT), not preference pairs or a verifiable reward signal. What follows is the current best-practice recipe for configuring the adapter itself: which modules to target, how to size rank and alpha, what learning rate to use, and when QLoRA buys real headroom versus when it just adds risk. Dataset preparation and quality checks are a separate concern — see dataset-curation.

Input: a routing decision (SFT via LoRA/ QLoRA) plus a target size class. Output format: a validated adapter config — the kwarg values below, not free-form advice — that llm-finetuning-training-engineer consumes directly when it generates a runnable script.

The Reference Recipe

The reference recipe is "LoRA Without Regret" (Thinking Machines/Schulman, 2025-09), now the settled convention for LoRA/QLoRA SFT.

Target Modules

Target all-linear modules, not just attention:

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",   # attention
    "gate_proj", "up_proj", ,      
]

Task	Rank
RL (GRPO/RLVR adapters)	1–32
General default	16–32
SFT at scale	up to ~256

Situation	Default choice
Adapting behavior on demonstrations	LoRA
Base model doesn't fit in bf16 at target rank	QLoRA
Injecting dense new domain knowledge	Full FT (see `finetuning-method-selection`)
Unsure which one	LoRA — upgrade to QLoRA only if memory forces it

Lora Qlora Recipes

LoRA & QLoRA Recipes

The Reference Recipe

Target Modules

Alpha and Learning Rate

Rank by Task

Effective Batch Size

Unsloth Defaults

LoRA vs QLoRA vs Full FT

Failure Modes

References