Troubleshoot GKE TPU VMs stuck in Compute Engine repair (REPAIRING)

This skill checks the VM status, the host events, and the GKE repair status, and routes recovery through GKE instead of direct Compute Engine changes.

Prerequisites

• Tools:
  • For installing and initializing gcloud (which covers authentication and project selection), follow Before you begin.
  • For installing kubectl and required plugins, follow Install kubectl and Install required plugins.
• Cloud Billing & Project Configuration: Ensure the project has an active Cloud Billing account. Follow Before you begin to initialize gcloud and enable the Google Kubernetes Engine API (container.googleapis.com), ensure the Compute Engine API (compute.googleapis.com) and Cloud Logging API (logging.googleapis.com) are enabled, and follow Store cluster information for kubectl to configure kubectl cluster credentials.
• Required IAM Roles:
  • Kubernetes Engine Viewer (roles/container.viewer)
  • Compute Viewer (roles/compute.viewer)
  • Logs Viewer (roles/logging.viewer)
  • For the user who enables node auto-repair in Step 5, item 1: Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• Documentation:
  • View information about MIGs and managed instances (Sections: "List MIGs", "Check the status of managed instances")
  • Compute Engine instance lifecycle (Sections: "REPAIRING state")
  • Troubleshooting VM shutdowns and reboots (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")
  • Configure cluster access for kubectl (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
  • Add and manage node pools (Sections: "View node pools in a Standard cluster", "Deploy a Pod to a specific node pool")
  • Troubleshoot NodeNotReady state (Sections: "Check the node's status and conditions", "Confirm node preemption")
  • About TPUs in GKE (Sections: "Multi-host TPU slice node pools")
  • Use dynamic slicing with a custom scheduler (Sections: "Requirements", "Verify the status of the nodes and the partitions", "Monitor the status of the Slice custom resource", "Hardware maintenance and failures")
  • About GKE dynamic slicing (Sections: "Key characteristics of dynamic slicing configurations")
  • Deploy TPUs in All Capacity mode in GKE (Sections: "Manage node failures", "Node failure and recovery")
  • Report and repair faulty TPU hosts in All Capacity mode (Sections: "Understand the faulty host report process", "Review report faulty host operations")
  • Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Repair criteria", "Node repair process", "Verify node auto-repair is enabled for a Standard node pool", "Enable auto-repair for an existing Standard node pool", "Get information about recent automated repair events", "Node auto repair in TPU slice nodes")
  • Deploy TPU workloads in GKE Standard (Sections: "Before you begin", "Configure auto repair for TPU slice nodes")
  • Get support (Sections: "Before you contact support", "Recommendations for efficient support")
  • REST Resource: projects.locations.clusters.nodePools (Sections: "Resource: NodePool")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

Diagnostic workflow

Step 0: Collect context [Low Risk]

Collect the target parameters:

• {project_id}: Google Cloud project ID
• {cluster_name}: GKE cluster name
• {location}: Cluster region or zone
• {cluster_mode}: Cluster mode (Autopilot or Standard)
• {tpu_version}: TPU version (such as Ironwood tpu7x, TPU v6e, v5p, v5e, or v4) and whether the node pool is a single-host or multi-host TPU slice node pool
• {nodepool_name}: TPU node pool name
• {group_name} and {zone}: Name and zone of the managed instance group (MIG) that backs the node pool:
  • In a Standard cluster, follow View node pools in a Standard cluster to describe {nodepool_name} and read its instanceGroupUrls field (see Resource: NodePool).
  • In an Autopilot cluster (or if {nodepool_name} isn't known), follow List MIGs to list the managed instance groups in {project_id} and find {group_name} and {zone}.
• {instance_name}: Affected TPU VM name, if known. If the user doesn't know {instance_name}, Step 1 finds the instances in REPAIRING.
• {issue_time}: Incident timestamp in RFC3339 UTC, if known

Step 1: Check the VM status in the managed instance group [Low Risk]

To list the instances in {group_name} with their status (STATUS / instanceStatus) and current action (ACTION / currentAction), follow the instructions in Check the status of managed instances.

• Analysis:
  • STATUS (instanceStatus) is REPAIRING: Compute Engine is repairing the VM because it encountered an internal error or its host server is unavailable due to maintenance. The VM can't be used during the repair. See REPAIRING state. Record {instance_name} and proceed to Step 2.
  • No instance is in REPAIRING (every instance reports RUNNING with ACTION NONE, or an instance reports another status or action such as ACTION RECREATING, which means "The instance is being replaced" per Check the status of managed instances): Check each node's status and conditions following Check the node's status and conditions and hand off to the matching sibling skill:
    • If a node's Ready condition is Unknown (NodeStatusUnknown), hand off to gke-ai-troubleshooting-node-unresponsive-timeout.
    • If a node isn't Ready for another reason, hand off to gke-node-notready.
    • If every node is Ready (per Repair criteria, "A node reporting a Ready status is considered healthy") but the workload's Pods are still Pending, hand off to gke-workload-troubleshooting.

Step 2: Check Compute Engine system events [Low Risk]

To list the system events for {instance_name} around {issue_time}, follow the instructions in the section Querying Cloud Audit Logs. Compare the method field with the table in the Reviewing Cloud Audit Logs section of the same document:

• compute.instances.hostError, compute.instances.terminateOnHostMaintenance, or compute.instances.automaticRestart: Record the matching system event log entries and timestamps, and continue to Steps 3 and 4.
• compute.instances.preempted: Compute Engine preempted a Spot VM or preemptible VM. If the affected workload is a JobSet, hand off to gke-ai-troubleshooting-jobset-interruption. Otherwise, follow Confirm node preemption.
• No method from the table, or another method: Per the note in Reviewing Cloud Audit Logs, if a VM rebooted and none of the listed methods appears in the query results, the reboot likely happened because the VM's operating system initiated it. If the VM is still in REPAIRING in Step 1 (or a method such as compute.instances.reportHostAsFaulty appears), continue to Steps 3 and 4.

Step 3: Check the node status and slice impact [Low Risk]

Follow Check the node's status and conditions to check the status and conditions of the nodes in the cluster and match {instance_name} using the node's ProviderID field. To filter nodes to {nodepool_name}, use the cloud.google.com/gke-nodepool node label documented in Deploy a Pod to a specific node pool.

• If the node for {instance_name} reports Ready (per Repair criteria, "A node reporting a Ready status is considered healthy") and the VM is no longer in REPAIRING, proceed to Step 5, item 4.
• Otherwise, determine how the unhealthy node affects the node pool based on the node pool configuration:
  • Single-host TPU slice node pool: Per Configure auto repair for TPU slice nodes, "In a single-host TPU slice node pool, only the unhealthy TPU node is auto-repaired."
  • Dynamic slicing node pool: To check whether {nodepool_name} uses dynamic slicing, see Requirements: dynamic slicing requires a Standard cluster, Ironwood (tpu7x), and All Capacity mode reservations. For a Standard cluster with Ironwood (tpu7x), check whether the node has cloud.google.com/gke-accelerator-topology-mode: PROVISION_ONLY as shown in step 2 of Verify the status of the nodes and the partitions (per Key characteristics of dynamic slicing configurations, "dynamic slicing uses incremental provisioning"). Per Hardware maintenance and failures, "If you trigger customer-initiated maintenance or if a hardware failover occurs, only the targeted nodes are affected rather than the entire node pool." For an active dynamic sub-slice, the Slice custom resource transitions to FAILED and the scheduler attempts to re-create the slice on other available healthy nodes. Check the Slice status following Monitor the status of the Slice custom resource; if the Slice custom resource is in FAILED or another non-active state, hand off Slice troubleshooting to gke-ai-troubleshooting-tpu-dynamic-slices-monitoring alongside checking the VM repair in Step 4.
  • TPU All Capacity mode node pool: Per Manage node failures and Node failure and recovery, in TPU All Capacity mode, if a VM fails due to issues like hardware failure, Google Cloud "attempts to recover the VM on the same host (in-place repair)", failed nodes have a status of NotReady, and the node pool's status doesn't change to ERROR.
  • Multi-host TPU slice node pool (without dynamic slicing or All Capacity mode): Per the section Multi-host TPU slice node pools, the VMs in a multi-host TPU slice node pool are treated as a single atomic unit. When a node in the slice requires repairing, GKE shuts down all VMs in the TPU slice, forcing eviction of all the Kubernetes Pods in the workload, and the Pods can be scheduled again only after all VMs in the slice are up and running.

Step 4: Check node auto-repair status and faulty host reports [Low Risk]

Find out whether GKE or Compute Engine is repairing the node:

1. Check whether node auto-repair is enabled:
   • Autopilot clusters: Autopilot always repairs nodes, and you can't turn this off (see Settings for Autopilot and Standard).
   • Standard clusters: Follow Verify node auto-repair is enabled for a Standard node pool to check whether autoRepair is enabled on {nodepool_name}.
2. Check the GKE repair operation history: Per Configure auto repair for TPU slice nodes, check the repair status, including any failure reason, in the operation history described in Get information about recent automated repair events:
   • If a repair operation failed because of insufficient quota, tell the user to contact their Google Cloud account representative to increase the quota.
   • If autoRepair is enabled and no repair operation has appeared yet, check how long the node has been unhealthy (from {issue_time} or the Ready condition's LastTransitionTime in Step 3). Per Repair criteria, GKE initiates a repair when a node reports a NotReady status, or doesn't report any status at all, over a given time threshold of approximately 10 minutes. If the node has reported NotReady (or no status) for less than approximately 10 minutes, check the operation history again after that threshold has passed. If it has been longer than approximately 10 minutes and there is still no repair operation (and no All Capacity mode faulty host report in item 3), escalate in Step 5, item 3.
3. For TPU All Capacity mode node pools, check for a faulty host report: Per Understand the faulty host report process, after a user reports a faulty host, the TPU VM is stopped, its state changes to REPAIRING, and "The host repair process can take between 3 and 14 days, or longer." Before escalating a VM in REPAIRING in an All Capacity mode reservation, follow Review report faulty host operations to check for a compute.instances.reportHostAsFaulty operation on {instance_name}.

Step 5: Resolution [High Risk]

Guardrails:

• Never change GKE-managed instance groups or VMs directly through Compute Engine, for example with gcloud compute instances delete or gcloud compute instance-groups managed commands that change the group. Manage the node pool through GKE.
• Never run kubectl delete node on nodes in the node pool. Per Settings for Autopilot and Standard, if you use kubectl to delete an unhealthy node when auto-repair is disabled, the underlying VM might become orphaned from the cluster, and you would continue to be billed for it.

1. Enable node auto-repair (Standard clusters only):
   • Autopilot clusters always auto-repair nodes, so skip this step.
   • Before the user enables auto-repair on a multi-host TPU slice node pool that doesn't use dynamic slicing or All Capacity mode, tell them that GKE re-creates the entire node pool when a node in it needs repair (see item 2 of this step).
   • If autoRepair is disabled on {nodepool_name}, link the user (who needs the Kubernetes Engine Cluster Admin role, roles/container.clusterAdmin) to Enable auto-repair for an existing Standard node pool and Configure auto repair for TPU slice nodes so they can apply the change themselves.
2. Let GKE or All Capacity mode repair the node or slice:
   • Single-host TPU slice node pool: Per Configure auto repair for TPU slice nodes, "In a single-host TPU slice node pool, only the unhealthy TPU node is auto-repaired." Per Node repair process, for a single node repair, GKE drains the node and waits one hour for the drain to complete; if the drain doesn't complete, GKE shuts the node down and creates a new node.
   • Multi-host TPU slice node pool (without dynamic slicing or All Capacity mode): Per Node auto repair in TPU slice nodes and Configure auto repair for TPU slice nodes, if a TPU slice node in a multi-host TPU slice node pool is unhealthy and requires auto repair, the entire node pool is re-created.
   • Dynamic slicing node pool: Per Hardware maintenance and failures, only the targeted nodes are affected rather than the entire node pool, and for dynamic sub-slicing the scheduler attempts to re-create failed slices on other available healthy nodes. If Slice custom resources fail to form or remain FAILED, hand off to gke-ai-troubleshooting-tpu-dynamic-slices-monitoring.
   • TPU All Capacity mode with a faulty host report: If Step 4, item 3 found a compute.instances.reportHostAsFaulty operation for {instance_name} (tracked via Review report faulty host operations), do not escalate the VM as stuck. Tell the user that per Understand the faulty host report process, the VM stops during the repair and restarts on the same host after the repair is complete, and "The host repair process can take between 3 and 14 days, or longer."
3. Contact Cloud Customer Care if the repair doesn't start or complete:
   • If the VM stays in REPAIRING (and isn't waiting on a 3-to-14-day All Capacity mode faulty host repair), if the node has been NotReady (or reported no status) for longer than the ~10-minute threshold in Repair criteria with autoRepair enabled and no repair operation in Step 4, or if the operation history shows a failed repair that isn't caused by quota, direct the user to Before you contact support and Recommendations for efficient support to contact Cloud Customer Care. In addition to the details listed in Recommendations for efficient support, include the node pool name, instance name, the system event log entries from Step 2, and the repair operation name from Step 4.
4. Verify recovery:
   • Repeat Step 1 and Step 3. When recovery is complete, every instance in {group_name} reports RUNNING with action NONE, and every node in {nodepool_name} is Ready.
   • If the instances report RUNNING with action NONE (or another status or action) and a node still isn't Ready, follow the handoffs in Step 1 (gke-ai-troubleshooting-node-unresponsive-timeout or gke-node-notready) or escalate to Cloud Customer Care in item 3 of this step.