Running LLMs on Radeon GPUs with ROCm

November 20, 20255 min readUpdated 10 days ago

Labrocmamdllminference

Running LLMs on Radeon GPUs with ROCm — hero illustration

I run LLM inference in my homelab because I want:

predictable cost for always-on endpoints,
low latency on my LAN,
and the ability to test “production-ish” operational work (deploys, rollbacks, telemetry) without renting GPUs.

RX 7900 XTX cards remain the main interactive inference hardware in my cluster. The useful question is not “does ROCm start?” It is “which exact card, runtime image, backend, and model shape stay healthy under the intended workload?”

TL;DR

Treat ROCm like part of the platform: pin versions and don’t casually drift.
Validate the stack bottom-up: device nodes -> host tooling -> container runtime -> model server.
Keep architecture-specific defaults in one profile instead of copying environment variables into every model.
llama.cpp is still my broadest compatibility path; vLLM is faster or more capable on selected, certified workloads.
Split workloads by lane, not just by model size. Predictable contention matters more than theoretical peak density.

What I am running now

The fleet has changed enough that node-to-model lists age badly. The durable shape is:

gfx1100 / 24 GiB: demand-driven vLLM and llama.cpp text lanes, plus media workloads; a GamingSession resource can temporarily repurpose one card.
gfx906 / 16 GiB: a llama.cpp embedding and reranking plane. Its GPUProfile allows multiple small models to remain active when the combined VRAM estimate fits.
sm_52 / 6 GiB: an older NVIDIA card dedicated to a constrained SD 1.5 image-generation lane.

FlexInfer owns model lifecycle, routing labels, shared-GPU policy, and architecture profiles. The active model set still changes during qualification work, so deploy/models/kustomization.yaml is the source of truth for what reconciles. A YAML file merely existing under deploy/models/ does not mean it is active.

MLC-LLM was useful during the original RX 7900 XTX bring-up, but I retired the active MLC serving manifests and caches on 2026-08-19. I keep the backend implementation and historical write-up; it is no longer the current serving path.

Bottom-Up Checklist (Host -> Container -> Model)

When ROCm is “broken,” it’s usually one of these:

The host doesn’t have the right device nodes or kernel modules.
The container can’t see the device nodes (permissions / runtime mismatch).
The model server runs, but VRAM behavior under load causes OOMs, fragmentation, or tail latency.

This is the checklist I run before I blame the model.

1) Host sanity: device nodes and basic tooling

On AMD ROCm nodes, I want to see:

/dev/kfd (ROCm kernel driver interface)
/dev/dri/* (DRM devices)

And I want at least one host-level tool to confirm the GPU is visible:

ls -la /dev/kfd /dev/dri
rocminfo | head -n 50
rocm-smi || true

If rocminfo fails on the host, nothing above it is going to be stable.

2) Container sanity: can a pod see the GPU?

Before running a model server, I run a tiny “does the container see the device?” check. The goal is to catch runtime issues (missing device mounts, wrong security context) early.

If you’re on Kubernetes, make sure the pod has access to /dev/kfd + /dev/dri and that your device plugin/runtime setup is consistent. A lot of “ROCm is flaky” reports are really “my containers don’t reliably get the device nodes.”

3) Put RDNA3 defaults in a profile

For RDNA3 (gfx1100), I keep tested defaults in the gfx1100 GPUProfile instead of letting every manifest guess:

PYTORCH_ROCM_ARCH=gfx1100 (PyTorch/ROCm builds, when applicable)
HSA_OVERRIDE_GFX_VERSION=11.0.0 (workaround for tooling/runtime mismatches on consumer RDNA3)
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
HIP_FORCE_DEV_KERNARG=1
TORCH_BLAS_PREFER_HIPBLASLT=1

The profile also declares the usable device index. That matters on a node where the AMD device plugin sees both an integrated GPU and the discrete card. Model manifests should not hard-code HIP_VISIBLE_DEVICES=0 or 1 and hope that enumeration stays identical.

These values describe my pinned runtime image and hardware. They are not a generic ROCm tuning recipe.

4) vLLM on gfx1100: certify the image and workload together

The earlier version of this post recommended forcing the vLLM V0 engine and disabling several kernels. That guidance is stale for the current gfx1100 profile. The pinned vLLM line now targets the v1 engine and uses eager execution as the safe default. Experimental features remain explicitly marked in the profile.

I also bind narrow capability claims to evidence. For example, the repository contains an image-and-artifact certificate for one Qwen3.5 mixture-of-experts workload with native MTP-1 speculative decoding. That certificate records the image digest, artifact digest, context limit, GPU-memory target, evidence path, and validation date. It does not prove that speculative decoding is a free win for every model.

The rule I keep: promote a combination, not a brand name. “vLLM works on Radeon” is too broad to operate.

A practical starting point on a 24 GiB card

For a new llama.cpp lane, I start conservatively:

choose one exact GGUF and record its size and hash,
begin with an 8,192-token context unless the product requires more,
use one request slot while measuring the memory budget,
increase GPU offload, context, batch, and concurrency one variable at a time,
retain at least 2 GiB of observed headroom for runtime and temporary allocations.

The 2 GiB value is my current gfx1100 profile margin, not a universal safe amount. I verify it under the intended prompt length and concurrency.

Failure Modes I Actually Hit

These are the ones that cost me time:

Version drift: a kernel update or ROCm update that “sort of works” and then fails under load. Fix: pin versions, upgrade intentionally, write it down.
VRAM lies: a model fits until you add concurrency and a real context window. Fix: budget KV cache and set explicit limits (model len, parallelism).
Container visibility: the pod is “Running” but doesn’t have usable /dev/kfd access. Fix: validate the device nodes in a trivial pod before deploying the model server.

Results and limitations

The profile-first approach has made Radeon changes easier to review and has reduced per-model environment drift. The same cluster now exposes normalized per-card compute and VRAM metrics alongside model loading, backend crash, and shared-group state.

The limits are still real:

Backend support is architecture-specific. gfx1100 results do not transfer automatically to gfx906.
Consumer-card support often depends on a pinned image or local patch.
A model that loads successfully can still fail at longer context or higher concurrency.
My measurements describe a small homelab fleet, not a broad hardware benchmark.
Host kernel and driver updates remain platform changes even when Kubernetes manifests are unchanged.

If you want the deeper war story version (dual nodes, MLC-LLM compilation/JIT, and the real “what broke first” timeline), see: Deploying MLC-LLM on Dual RX 7900 XTX GPUs.

Takeaways

Validate host, container, and backend as separate boundaries.
Pin the architecture profile and runtime image.
Treat a backend/model/image combination as the unit of support.
Budget weights, KV cache, workspaces, and driver overhead together.
Measure warm, cold, and shared-GPU behavior separately.
Keep active manifests distinct from historical recipes.

Related surfaces:

8 min read

amdrocm

Getting Gemma 4 Running on a Radeon 7900 XTX (with and without TurboQuant)

Field notes from serving Gemma 4 E4B on Radeon: the stable TRITON path, an experimental TurboQuant long-context lane, and the GPTQ work still in progress.

13 min read

rocmamd

Deploying MLC-LLM on Dual RX 7900 XTX GPUs: Debugging VRAM, KV Cache, and K8s GPU Scheduling

What actually broke when I deployed MLC-LLM across two RX 7900 XTX nodes, and the fixes that made it stable: quantization, KV cache sizing, and Kubernetes GPU hygiene.

6 min read