gke-ai-troubleshooting-jobset-interruption

Find why AI training JobSets keep restarting.

AI infrastructure troubleshooting

What it does

Diagnoses JobSet interruptions, restarts, and preemptions in large-scale GKE AI/ML workloads.

When to use it

Use for JobSet restart loops, Spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes.

Give it a GKE JobSet interruption problem; it returns diagnostic findings and resolution recommendations.

This skill

GKE cluster resources

Read

Inspects GKE cluster resources

Cloud Monitoring metrics

Read

Queries Cloud Monitoring metrics

Cloud Logging logs

Read

Reads Cloud Logging logs

Requires a GKE cluster hosting the JobSet workload to be diagnosed.

The JobSet metrics package must be enabled in kube-state-metrics for the cluster.

Cloud Logging must be enabled for the Google Cloud Project.

Cloud Monitoring must be enabled for the Google Cloud Project.