Trace to Training Data

Add skill

Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.

Permissions

Files

SKILL.md

trace-to-training-data

Trace To Training Data

This skill assumes eval-harness-first already graded the traces being converted here — goldens, graders, and runs/<run-id>/results.json all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream.

Input: graded traces — eval/goldens.jsonl plus runs/<run-id>/results.json, each row carrying a task_id, a verdict from the grader, and a reward when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier):

2 files · 14 KB6 KB

Version history

Trace to Training Data

Trace To Training Data

The Principle

SFT From Traces

Preference Pairs From Traces

Hygiene

Related Skills