Spark Memory Thermal Ops

Add skill

Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.

Permissions

Files

SKILL.md

spark-memory-thermal-ops

Spark Memory & Thermal Ops

DGX Spark's GB10 chip has one 128GB unified memory (UMA) pool shared by CPU and GPU, and a sustained power ceiling well below its rated figure. Both break discrete-GPU assumptions: headroom isn't what nvidia-smi reports, and a run that starts fast will slow down mid-job with nothing misconfigured. This skill covers planning memory headroom, working an actual OOM, and watching thermals across a long job. For launch-time failure modes (ABI mismatches, flash-attn, playbook breakage), see spark-training-gotchas — this skill assumes the job starts.

Common Issues Quick Reference

Situation	Do this

3 files · 13 KB8 KB

Version history

Planning headroom before launch	Budget against `free -g`, not `nvidia-smi` — see UMA Memory Model
Job OOMs on unified memory	Work the OOM Ladder in order: flush, then batch/pack, then method downgrade
Throughput drops mid-run	Check the power/temp log before assuming a config bug — see Thermal Monitoring
Trainer + inference server both wanted	Run one at a time — see Concurrent Workloads

Spark Memory Thermal Ops

Spark Memory & Thermal Ops

Common Issues Quick Reference

When to Use This Skill

UMA Memory Model

Planning Sequence

Example: Sizing a 70B QLoRA Run

The OOM Ladder

Thermal Monitoring

Concurrent Workloads