GPU Failure Modes: What Breaks and How to Debug It

GPU incidents get expensive when every layer is debugged at once. A pending pod, a driver failure, a video-memory (VRAM) exhaustion, and a stalled model load can all look like “the GPU is broken” from the client.

I start by locating the first broken boundary. That cuts the search space before anyone changes drivers or restarts a node.

TL;DR

• Classify the failure as scheduling, device/runtime, memory, loading, or request path.
• Read pod events and previous-container logs before restarting anything.
• Separate system-memory OOMs from VRAM OOMs.
• Use DCGM metrics for NVIDIA and AMD-aware metrics for AMD.
• Treat Loading as a lifecycle with substages, not a sufficient diagnosis.
• Alert on user symptoms first; use saturation and hardware signals to explain them.

The decision tree

My first pass is read-only:

# Placement and restarts
kubectl get pods -A -o wide

# Scheduler and kubelet evidence
kubectl describe pod -n <namespace> <pod>

# Current and previous container output
kubectl logs -n <namespace> <pod> -c <container> --tail=200
kubectl logs -n <namespace> <pod> -c <container> --previous --tail=200

# Node condition, taints, and advertised GPU resources
kubectl describe node <node>

Do not delete the pod until you have captured events, termination state, and previous logs. A clean restart can erase the best evidence.

1. Scheduling failures

A pod in Pending has not reached the model runtime. Start with the Events section from kubectl describe pod.

Event pattern	Likely boundary	Check next
Insufficient nvidia.com/gpu or an AMD GPU resource	Capacity or a stale device-plugin advertisement	Node capacity/allocatable, device-plugin pod
untolerated taint	Workload contract	Node taints and pod tolerations
didn't match Pod's node affinity	Label or architecture mismatch	Node labels and affinity
exceeded quota	Namespace policy	kubectl describe resourcequota
node(s) were unschedulable	Cordoned or unhealthy node	Node condition and maintenance state

List all advertised vendor resources without assuming one resource name:

kubectl get nodes -o json | jq -r '
  .items[] |
  [.metadata.name,
   (.status.allocatable | to_entries |
    map(select(.key | test("^(nvidia\\.com|amd\\.com)/gpu"))) |
    map("\(.key)=\(.value)") | join(",")),
   ((.spec.taints // []) | map("\(.key)=\(.value):\(.effect)") | join(","))] |
  @tsv'

In the current FlexInfer API, models declare spec.gpu.vendor and spec.gpu.count; the controller injects the extended resource. The CRD rejects GPU entries placed directly in spec.resources, which prevents two competing sources of truth.

2. Device and runtime failures

The driver stack has several boundaries:

For NVIDIA, capture:

nvidia-smi
nvidia-smi -q
journalctl -k --since '-30 min' | rg -i 'nvrm|xid|pcie'

For AMD, capture:

ls -l /dev/kfd /dev/dri
rocminfo | sed -n '1,120p'
rocm-smi --showmeminfo vram --showtemp --showpower
journalctl -k --since '-30 min' | rg -i 'amdgpu|kfd|pcie'

Then run the equivalent check inside the failing container. Host success plus container failure points at the device plugin, runtime class, permissions, or device selection. Host failure means the model image is not the first repair target.

Architecture matters. A container can include a valid runtime and still lack kernels for the installed card. I record the tested GPU architecture and immutable image together; “ROCm image” or “CUDA image” is too broad.

3. System memory and VRAM are different incidents

System-memory exhaustion commonly leaves OOMKilled and exit code 137 in container state:

kubectl get pod -n <namespace> <pod> -o json | jq '
  .status.containerStatuses[] |
  {name, restartCount, lastState}'

VRAM exhaustion usually appears as a CUDA, HIP, PyTorch, vLLM, or llama.cpp allocation error. The pod may remain Running while the backend subprocess has failed.

The sizing model is:

serving VRAM ~=
  model weights
  + quantization metadata
  + KV cache
  + activation/workspace buffers
  + graph/compiler allocations
  + driver/runtime overhead

Weight size alone is not a fit test. KV-cache size depends on layers, key/value heads, head dimension, token capacity, concurrency, and element width. Grouped-query attention and hybrid architectures make a generic “B parameters equals X GiB” rule even less reliable.

Fix the actual term that is too large:

• reduce context or total token capacity,
• reduce concurrency or batch size,
• choose a validated quantization,
• lower the runtime's GPU memory target,
• remove an invisible co-tenant,
• leave explicit headroom for compilation and temporary buffers.

Do not assume that calling torch.cuda.empty_cache() in a pod shutdown hook repairs isolation. Process exit releases the context; persistent leaks or fragmentation need runtime evidence.

4. Loading is not one state

Model startup crosses image pull, initialization, weight loading, kernel or graph compilation, and health checks. A single Loading phase hides which step stopped advancing.

FlexInfer now reports status.loadingSubstage, a short message, and status.loadingProgressAt. The current substages include ImagePulling, Initializing, LoadingWeights, Compiling, HealthCheckPending, and Preempted.

kubectl -n flexinfer-system get models.ai.flexinfer \
  -o custom-columns=\
NAME:.metadata.name,\
PHASE:.status.phase,\
SUBSTAGE:.status.loadingSubstage,\
PROGRESS:.status.loadingProgressAt,\
MESSAGE:.status.message

That distinction matters. A slow image pull, a storage stall during shard reads, and a compiler hang require different owners and repairs.

The proxy also increments flexinfer_proxy_stalled_load_total{model,substage} when progress stops beyond its threshold. A counter increase is evidence of a stall; a model merely spending several minutes in LoadingWeights is not automatically one.

5. Shared-GPU contention and preemption

Two models can each be healthy and still produce an unhealthy service if they repeatedly evict one another.

I watch:

sum by (group) (rate(flexinfer_sharedgroup_preemptions_total[15m]))

histogram_quantile(0.95,
  sum by (le, group) (
    rate(flexinfer_model_swap_duration_seconds_bucket[15m])
  )
)

max by (group, model, state) (flexinfer_sharedgroup_state)

The fix is usually policy, not a faster restart: adjust priority, increase swap cooldown, remove an overlapping route, or give the workload a dedicated lane. Current FlexInfer models expose spec.gpu.swapCooldown, and forcePromotion is an operator override rather than a normal steady-state setting.

6. Investigate the request path after the backend

Low GPU utilization plus high client latency does not prove a network problem. First compare these timestamps:

• ingress to proxy,
• proxy queue wait,
• cold-start or swap wait,
• upstream connect,
• time to first token,
• completion time.

Only then test the suspect hop. A temporary iperf3 or direct service request can help, but create diagnostic pods under the same network policy and clean them up afterward.

I avoid jumping to hostNetwork, kernel sysctls, or overlay changes. Those widen the blast radius and can hide an application queue or storage stall.

7. Build a vendor-aware operating view

The current platform uses separate collectors for the hardware layer:

• DCGM Exporter is scheduled only on NVIDIA nodes.
• An AMD sysfs exporter provides amdgpu_* and normalized gpu_* series.
• The FlexInfer agent publishes normalized flexinfer_gpu_* metrics across the fleet.

The service layer adds:

# Request errors by model
sum by (model) (rate(flexinfer_proxy_requests_total{status="error"}[5m]))

# Cold-start queue wait
histogram_quantile(0.95,
  sum by (le, model) (
    rate(flexinfer_proxy_queue_wait_duration_seconds_bucket[15m])
  )
)

# Backend process failures
sum by (model, backend) (
  rate(flexinfer_runtime_backend_crashes_total[15m])
)

# Per-card compute and VRAM pressure
max by (node, gpu, vendor) (flexinfer_gpu_compute_utilization_percent)
max by (node, gpu, vendor) (flexinfer_gpu_vram_utilization_percent)

Alert on latency, errors, and unavailable capacity first. Temperature, VRAM, queue depth, and preemption explain those symptoms. A low-utilization alert belongs in cost review, not the incident pager.

Results and limitations

This layered workflow has been most useful for three real classes of failure in my cluster: AMD device visibility, model-load storage stalls, and shared-GPU swap behavior. It produces evidence before a repair and leaves enough context for a post-incident review.

It is not a vendor hardware diagnostic manual. Exact reset procedures, hardware error meanings, and safe temperature limits depend on the card and vendor guidance. Managed Kubernetes also limits host-level access, so the cloud provider may own part of the investigation.

Takeaways

1. Preserve events, termination state, and previous logs.
2. Separate scheduling, device visibility, memory, loading, and request-path failures.
3. Size serving memory as a budget, not as model weights alone.
4. Make loading progress and shared-GPU swaps observable.
5. Use vendor-specific hardware metrics and normalized service metrics.
6. Page on user symptoms; use GPU signals for diagnosis and capacity review.

Related reading:

• SLOs for Inference
• GPU Cost Baseline
• AI Infra Readiness Audit checklist