Diagnoses GKE JobSet interruptions, restarts, and preemptions for AI/ML training workloads autonomously. Use when troubleshooting JobSet restart loops, spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes. Don't use for general GKE cluster creation, basic workload deployment, or non-JobSet application issues.
Permissions
Files
SKILL.md
gke-ai-troubleshooting-jobset-interruption
Diagnoses GKE JobSet interruptions, restarts, and preemptions for AI/ML training workloads autonomously. Use when troubleshooting JobSet restart loops, spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes. Don't use for general GKE cluster creation, basic workload deployment, or non-JobSet application issues.
metadata.version
1.0.0
metadata.category
Containers
GKE JobSet Interruption Troubleshooting
Use this skill to systematically diagnose and resolve JobSet interruptions,
restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.
JobSet metrics package must be enabled in kube-state-metrics for your
cluster.
Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.
Autonomous & Sandbox Execution Rule: If API queries, tools, or commands
encounter 403 Permission Denied, authentication errors, or network
isolation, do NOT enter authentication or credential troubleshooting
loops. Populate the query templates with the acquired variables
({project_id}, {cluster_name}, {workload_name}, {start_time},
{end_time}), inspect any locally staged telemetry or mock data files if
available, and complete the diagnostic workflow and resolution
recommendations autonomously.
🔍 Diagnostic Workflow
Step 0: Context Acquisition & Time Window Definition
Independently gather context using tools, workspace files, environment details,
or user prompt context:
Project ID ({project_id})
Cluster Name ({cluster_name})
Workload Name (JobSet Name) ({workload_name})
Workload Namespace ({namespace})
Issue Time ({issue_time})
If specific variables are not explicitly provided by the user, inspect cluster
resources or logs to determine them, or use the {variable} placeholders
provided.
Time Handling Rules
Autonomous Time Window: If a relative time (e.g., "X minutes ago") or no
exact timestamp is provided, calculate the query window based on current
time or available log timestamps.
Window Calculation: If a timestamp {issue_time} is available (or
calculated as T), set {start_time} = T - 30m and {end_time} = T + 30m.
Step 1: Identify JobSet Restarts and Attempts [Low Risk]
Verify if the JobSet is experiencing restart loops and determine the frequency
of restarts.
Diagnostic Logic: A non-zero or increasing value for restarts indicates
that the JobSet is being actively restarted by the controller due to worker
failure or interruption.
Automation: Proceed to Step 2 automatically after reporting findings.
Step 2: Inspect Nodepool Interruptions [Low Risk]
Determine if the JobSet restarts were triggered by physical nodepool-level
events (such as spot preemptions, maintenance, or host terminations).
sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
)
B. Log Query (Nodepool Life Events)
LQL Log Filter Specification:
resource.type="gke_nodepool"
AND resource.labels.cluster_name="{cluster_name}"
ANDtimestamp>= "{start_time}"
ANDtimestamp<= "{end_time}"
Diagnostic Logic:
PreemptionEvent: Spot VMs were preempted, or node was scale-down.
MaintenanceEvent: Node pool updated or Google scheduled maintenance.
TerminationEvent: Serious host failures. Check interruption_reason
or logs for host issues.
See Failure Signatures for examples
of node termination logs and preemption events.
Automation: Proceed to Step 3 automatically.
Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]
Correlate node readiness failures with physical host VMs to see if a single
faulty host repeatedly fails coordinator pods.
resource.type="k8s_node"
AND resource.labels.cluster_name="{cluster_name}"
AND (textPayload:"host error" OR textPayload:"kernel panic" OR textPayload:"hardware failure" OR textPayload:"NodeNotReady")
ANDtimestamp>= "{start_time}"
ANDtimestamp<= "{end_time}"
Diagnostic Logic: Identify if specific nodes are unhealthy
(Ready=False or Unknown) and correlate them to their GCE physical host
ID via metadata.user.gce_topology_host. Check if the same host is
repeatedly failing.
Automation: Proceed to Step 4 automatically.
Step 4: Inspect Pod and Worker / Container Failures [Low Risk]
Analyze pod status phases and retrieve coordinator worker logs to identify
application-level crashes or network deadlocks.
Required Execution Order: You MUST analyze pod status phases (Section A)
and unschedulable pod metrics (Section B) to assess overall workload health
before inspecting specific worker container logs (Section C).
resource.type="k8s_container"
AND resource.labels.cluster_name="{cluster_name}"
AND labels."k8s-pod/jobset_sigs_k8s_io/jobset-name"="{workload_name}"
ANDtimestamp>= "{start_time}"
ANDtimestamp<= "{end_time}"
Diagnostic Logic:
Check the pod timeline to spot pending or unschedulable pods.
Use worker container logs to analyze worker 0 in slice 0 (coordinator)
for NCCL timeouts, collective communication issues, or MegaScale hangs.
If Step 2 showed high preemption counts on Spot VMs:
Action: Suggest switching critical long-running training workloads to
GKE Reserved/On-Demand VMs or utilizing Compact Placement Policies
to minimize defragmentation interruptions.
Justification: Eliminates spot-market preemptions and reduces training
restarts.
If Step 3 identified a specific host ID (gce-topology-host) that consistently
fails or triggers restarts across multiple attempts:
Action: Recommend cordoning/draining the GKE node, deleting the
underlying GCE VM instance to trigger instance recreation, and opening a
support ticket with Google Cloud Support specifying the physical host ID.
Justification: GKE auto-repair will recreate the VM instance on healthy
physical hardware, preventing infinite restart loops.
📋 Copypaste Checklist
Gather context and compute {start_time} ({issue_time} - 30m) and
{end_time} ({issue_time} + 30m) window.
Query JobSet restart attempts.
Check Nodepool interruptions (spot preemptions vs. hardware
terminations).
Query node-to-host mapping and check node logs for physical host errors.
Inspect pod timeline status and coordinator worker container logs.
Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM
quarantining.
