Find out why your GKE training jobs restart.
What it does
Diagnoses the causes of GKE JobSet interruptions and restarts in large-scale AI/ML training workloads.
When to use it
Use for JobSet restart loops, spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes, not general cluster creation or non-JobSet issues.
How to use it
Give it a JobSet interruption problem; it gathers context, investigates telemetry, and returns diagnostic findings and resolution recommendations.
Uses
CM
CL
GK
Latest review
What you need · 4
The investigation concerns an existing GKE cluster hosting JobSet AI/ML workloads.
The cluster must have the JobSet metrics package enabled in kube-state-metrics.
Access · 4
This skill
Google Cloud APIs
Write
Sends skill usage attribution
Cloud Monitoring
Read
Queries JobSet restart metrics