Lightlines
By google

gke-ai-troubleshooting-tpu-vbar-oom

google3.4k installs

Find why your GKE TPU nodes crash during resets.

What it does

Diagnoses TPU v6e node crashes and metrics corruption in GKE and recommends ways to prevent recurrence.

When to use it

Use for vbar_control_agent crashes, serial-console memory cgroup OOMs, metrics checksum corruption, or custom metrics collection conflicts on GKE TPU v6e nodes, not general container OOMs or routine node lifecycle operations.

How to use it

Give it a TPU crash or metrics failure to investigate; it reports diagnostic findings and recommendations, or supplies log filters when live diagnostics are not applicable.

Uses

Latest review

What you need · 2

Live diagnostics require Cloud Logging to be enabled for the Google Cloud project.

Live diagnostics require access to the target project and cluster through gcloud or an equivalent tool; the troubleshooting scope is TPU v6e nodes.


Access · 2

This skill

Cloud Logging

Read

Reads node and container logs

GKE

Read

Inspects your cluster workloads