Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.
Permissions
Files
Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.
This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.
Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided {variable} placeholders:
{project_id}: The GCP Project ID.Version history
{cluster_name}{location}: The GKE Cluster Location (region or zone).{node_name}: (Optional) The name of the specific GKE node.{node_pool_name}: (Optional) The name of the GKE node pool.Before analyzing runtime metrics, verify that the workload is configured to export them. This ensures the cluster and container environment are set up for automated metric scraping and visibility into accelerator health.
containerPort: 8431 exposed on the TPU container (required for Prometheus metric scraping).0.4.14 or later if using JAX (earlier versions do not export runtime metrics).1.27.4-gke.900 or later (required for TPU runtime metric support).If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources k8s_node and k8s_container):
kubernetes.io/container/accelerator/duty_cycle: Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip.kubernetes.io/container/accelerator/memory_used: Amount of accelerator memory allocated in bytes.kubernetes.io/container/accelerator/memory_total: Total accelerator memory in bytes.kubernetes.io/node/accelerator/duty_cyclekubernetes.io/node/accelerator/memory_usedkubernetes.io/node/accelerator/memory_totalQuery the status condition of GKE nodes (GKE version 1.32.1-gke.1357001 or later).
kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", node_name="{node_name}", condition="Ready", status="True"}
kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition!="Ready", status="True"}
kubernetes_io:node_status_condition{monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready", status="False"}
avg by (condition,status)(avg_over_time(kubernetes_io:node_status_condition{monitored_resource="k8s_node"}[5m]))
Query the status of multi-host TPU node pools.
kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}", node_pool_name="{node_pool_name}", status="Running"}
count by (status)(count_over_time(kubernetes_io:node_pool_status{monitored_resource="k8s_node_pool"}[5m]))
Provisioning, Running, Error, Reconciling, Stopping.Query if all nodes in a multi-host TPU node pool are available.
avg by (node_pool_name)(avg_over_time(kubernetes_io:node_pool_multi_host_available{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[5m]))
1 (True, all nodes available) or 0 (False, some nodes unavailable).Query the count of interruptions for GKE nodes.
sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node"}[5m]))
TerminationEvent, MaintenanceEvent, PreemptionEvent.
Interruption Reasons: HostError, Eviction, AutoRepair.sum by (interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", interruption_reason="HW/SW Maintenance"}[5m]))
sum by (node_pool_name,interruption_type,interruption_reason)(sum_over_time(kubernetes_io:node_pool_interruption_count{monitored_resource="k8s_node_pool", interruption_reason="HW/SW Maintenance", node_pool_name="{node_pool_name}"}[5m]))
Calculate Mean Time to Recovery (MTTR) and Mean Time Between Interruptions (MTBI) over the last 7 days.
sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_sum{monitored_resource="k8s_node_pool", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_pool_accelerator_times_to_recover_count{monitored_resource="k8s_node_pool",cluster_name="{cluster_name}"}[7d]))
sum(count_over_time(kubernetes_io:node_memory_total_bytes{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d])) / sum(sum_over_time(kubernetes_io:node_interruption_count{monitored_resource="k8s_node", node_name=~"gke-tpu.*|gk3-tpu.*", cluster_name="{cluster_name}"}[7d]))
For GKE version 1.28.1-gke.1066000 or later, monitor TPU host performance.
kubernetes.io/container/accelerator/tensorcore_utilization: Current percentage of the TensorCore that is utilized.kubernetes.io/container/accelerator/memory_bandwidth_utilization: Current percentage of the accelerator memory bandwidth that is being used.kubernetes.io/node/accelerator/tensorcore_utilizationkubernetes.io/node/accelerator/memory_bandwidth_utilizationIn these kits
More from @google
Works with
Claude, Codex, Cursor & more