Gke AI Troubleshooting Tpu Metrics Monitoring

Add skill

Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.

Permissions

Files

SKILL.md

gke-ai-troubleshooting-tpu-metrics-monitoring

metadata.category: CloudObservabilityAndMonitoring

GKE TPU Metrics Monitoring Guide

This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure.

Step 0: Mandatory Context

Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided {variable} placeholders:

{project_id}: The GCP Project ID.
{cluster_name}: The GKE Cluster Name.
{location}: The GKE Cluster Location (region or zone).
{node_name}: (Optional) The name of the specific GKE node.

3 files · 9 KB7 KB

Version history

Gke AI Troubleshooting Tpu Metrics Monitoring

GKE TPU Metrics Monitoring Guide

Step 0: Mandatory Context

Diagnostic Steps

Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto]

Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto]

Step 3: Check Node Status Condition [Low Risk] [Auto]

Step 4: Check Node Pool Status [Low Risk] [Auto]

Step 5: Check Node Pool Availability [Low Risk] [Auto]

Step 6: Analyze Node Interruptions [Low Risk] [Auto]

Step 7: Calculate Recovery and Interruption Metrics [Low Risk] [Auto]

Step 8: Monitor TPU Host Metrics [Low Risk] [Auto]