Spark Training Gotchas

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

When to Use This Skill

A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
A run OOMs while nvidia-smi still shows headroom.
Throughput degrades partway through a run that started fine.
Before any multi-hour or multi-epoch job on GB10.
Wiring two Sparks together, before picking a parallelism strategy.

#	Symptom	Fix
G1	undefined symbol / segfault	cu130 wheel or container
G2	flash-attn wrong backend used	skip pip build; monkeypatch on NGC
G3	OOM despite headroom	drop page cache
G4	throughput drop / reboot	expect ~100W sustained cap
G5	memory-bound step slow	budget 180–192 GB/s
G6	cache evicted mid-run	one GPU server at a time
G7	NVFP4 slower than FP8	stay FP8 unless `sm_121a`
G8	playbook fails outright	check upstream issues
G9	env breaks after install	use a container
G10	2-Spark TP hangs	DDP/FSDP only, never TP

Spark Training Gotchas

Spark Training Gotchas

When to Use This Skill

Common Issues Quick Reference

The Ten Gotchas

G1: CUDA 12/13 ABI Mismatch

G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect

G3: UMA OOM Below 128GB

G4: Thermal Throttling

G5: Bandwidth Ceiling

G6: Global UMA Resource Contention

G7: NVFP4 Slower Than FP8 on SM121

G8: Stale Official Playbooks

G9: Container-First, Not Bare Pip

G10: Dual-Spark Is DDP/FSDP Only

Fast Triage