Operating a Kubernetes LLM Platform on Consumer GPUs

The first version of this platform was four model servers and a router. It worked, but it put too much operational knowledge in manifests and in my head. Which model fit which card? What could be unloaded? Would a rollout interrupt an active generation?

The current system moves those decisions into FlexInfer, a Kubernetes operator for model placement, caching, routing, and lifecycle. Applications still see a familiar API. The machinery behind that API can change without every consumer learning the cluster topology.

TL;DR

• A stable, OpenAI-compatible boundary matters more than preserving one serving backend or model lineup.
• GPU memory is a scheduling constraint, not a hopeful container limit. Model artifacts, runtime compatibility, key-value (KV) cache, and concurrency all belong in the placement decision.
• Model promotion needs workload evidence. This platform uses benchmark and coherence checks instead of treating “the pod became Ready” as a quality verdict.
• Safe rollouts require connection-aware draining. A readiness check alone does not protect a generation already in flight.
• The figures below are tied to dated repository and deployment evidence. I removed the old latency, uptime, cost, and “four models” claims because they no longer had a reproducible measurement window.

Context and evidence

This is a self-hosted platform running on a mixed-GPU K3s cluster. FlexInfer owns the model custom resources, scheduling, serverless proxy, cache lifecycle, and runtime status. Flux owns delivery of the control plane and model declarations.

I refreshed this study against these sources on August 30, 2026:

• services/flexinfer/README.md for the implemented control-plane and backend contracts;
• services/flexinfer/ROADMAP.md for the deployed-state attestation dated August 1, 2026 and the July rollout-under-load proof;
• services/flexinfer/deploy/ for the model and runtime resources; and
• platform/gitops/k3s/ai/flexinfer/ plus the Flux application definitions for cluster wiring.

This evidence establishes implementation and a dated operating snapshot. It is not a public availability service-level objective (SLO), a cloud cost comparison, or a timeless claim about which models are running today.

The challenge

“Run a model on a GPU” is a demo. “Keep an inference endpoint useful while hardware, models, and demand change” is an operating problem.

The failure modes compound:

• A model can fit at startup and still run out of memory when KV cache and concurrent requests grow.
• A runtime image can support one AMD architecture and fail on another.
• A healthy Kubernetes pod can return incoherent output at long context.
• A rollout can pass readiness while terminating an in-flight request.
• A manually chosen placement can become wrong as soon as another workload claims the card.

The platform needed to keep a stable client contract while making those conditions explicit and observable.

Constraints

Constraint	Consequence
Consumer AMD GPUs	ROCm, kernel, image, and GPU architecture have to be treated as one compatibility set.
Mixed nodes and workloads	Placement needs hardware labels, capacity, priority, and current utilization.
Finite local storage	Model artifacts need caching, deduplication, and deliberate promotion.
Interactive requests	Scale-to-zero and rollouts must account for queueing, warm-up, and active connections.
GitOps ownership	Runtime changes need a declarative source of truth and a rollback path.
Model quality varies by workload	Throughput alone cannot decide whether a candidate should serve production traffic.

Decisions and implementation

Make the model a reconciled resource

FlexInfer’s Model custom resource describes source, backend, GPU requirements, sharing policy, and lifecycle intent. The controller turns that intent into cache and serving resources, then reports status back on the resource.

That changes the operator question from “which Deployment do I patch?” to “what model state do I want?” It also gives scheduling, scale-to-zero, and cleanup one owner.

Keep the application boundary boring

The serverless proxy exposes an OpenAI-compatible interface. Applications do not need to know whether the selected runtime is vLLM, MLC-LLM, llama.cpp, or another supported backend.

Compatibility here means request shape and routing contract, not behavioral equivalence with a hosted OpenAI model. Model capabilities still need to be declared and tested.

Treat AMD compatibility as part of the release

The repository documents separate backend guidance for RDNA3 (gfx1100) and older NVIDIA hardware. Node labels describe vendor, architecture, and memory. Backend images and model configurations are selected against those facts.

This avoids a common failure pattern: scheduling a GPU workload based only on the presence of amd.com/gpu: 1, then discovering that the image or kernel path does not support the actual card.

Promote with evidence, not just readiness

The benchmark gauntlet records throughput and chat-aware coherence verdicts. Candidate experiments run in isolation and release hardware without mutating the Flux-owned production models. Failed experiments remain failed instead of quietly becoming production defaults.

Quantization is handled the same way. A smaller artifact is useful only if it fits the workload and passes its quality gate. I no longer describe a blanket “4× reduction with minimal quality loss”; both the compression ratio and the loss depend on the model, format, runtime, and evaluation set.

Drain before replacing

The proxy tracks active connections and uses readiness-first graceful draining during routine rollouts. New traffic moves away from the old pod before Kubernetes removes it.

That is a stronger invariant than “the replacement pod is Ready.” It protects work already accepted by the old process.

Outcomes and measurement context

A dated production snapshot

The FlexInfer roadmap’s August 1, 2026 refresh records two Ready Qwen3.5 9B RP parent models across the gfx1100 nodes. It also records two 35B workhorse models in Idle, not “four models served simultaneously.” That distinction matters: declared, cached, Ready, and actively serving are different states.

A rollout-under-load proof

On July 18, 2026, a controlled test sent a 2,048-token request through a proxy rollout. The request survived a 38.2-second drain, and all 60 service probes returned HTTP 200 during the test window. Those numbers describe one recorded rollout certificate, not a general latency or uptime SLO.

A repeatable promotion loop

The repository now contains recurring benchmark, backfill, and isolated model-experiment paths. The operational gain is not a universal tokens-per-second figure. It is the ability to attach a verdict to a specific artifact, runtime, hardware profile, and workload before promotion.

Limits and open work

• The deployed-state claim is a snapshot last attested on August 1, 2026. Live status can change after publication.
• Smart routing exists, but the project still calls out load and performance validation as ongoing work.
• Consumer AMD hardware keeps cost bounded for this environment, but this study does not establish a lower total cost than a managed service.
• Multi-tenancy and further gaming-mode hardening remain roadmap work.
• A compatibility API does not make local models interchangeable. Tool use, context behavior, and output quality still need model-specific evaluation.

Takeaways

1. Stabilize the client contract, not the topology. Models and runtimes will change more often than application integrations should.
2. Schedule the real memory budget. Weights are only part of VRAM demand; KV cache and concurrency are first-class inputs.
3. Separate readiness from quality. A responding server still needs workload-specific evaluation.
4. Make promotion reversible. GitOps, isolated candidates, and recorded verdicts turn experiments into controlled changes.
5. Test the handoff. Connection draining is a user-facing reliability feature, even in a small cluster.

The platform is documented under FlexInfer. The public playground focuses on the resource model rather than exposing the private cluster.