Preference Optimization

This skill assumes finetuning-method-selection already routed here because the data shape is preference pairs or unpaired thumbs-up/down feedback, not demonstrations (that's lora-qlora-recipes) or a verifiable reward signal (that's grpo-rlvr-training). What follows is method selection among the DPO family, the evidence for how much that selection actually matters, the production training pattern, and how to build the pairs in the first place.

Input: a routing decision (preference optimization) plus preference pairs or unpaired feedback, usually from an SFT checkpoint. Output format: a validated method choice plus a config — the kwarg values in references/method-configs.md, not free-form advice — that llm-finetuning-training-engineer consumes directly.

Method Selection

Data shape	Method	Key parameters
Preference pairs, default case	DPO	β=0.1, LR 5e-7–1e-6, 1–2 epochs
Memory-bound or no SFT checkpoint	ORPO	reference-free, fused SFT+preference in one loss
Unpaired thumbs-up/down	KTO	binary label per example, no pairing needed

Preference Optimization

Preference Optimization

Method Selection

Worked Examples

The Low-Leverage Truth

Production Pattern: Iterative On-Policy DPO

Pair Construction

References