Standing Up a GPU-Ready Private AI Platform (Harvester + K3s + Flux + GitLab)

Cloud capacity is a strong default for experiments and bursts. I built a private GPU platform because my steady-state constraints were different:

• I want predictable cost for always-on services.
• I want data locality (and the option to keep sensitive data off the public internet).
• I want a workflow that’s boring: Git push → CI → Flux sync → running workloads.

This is a case study, not a reference architecture for every team. I use Harvester for virtualization, K3s for the workload cluster, and GitLab plus Flux for delivery. I do not use Fleet in this path; Flux owns reconciliation.

TL;DR

• Harvester provides the virtual-machine and Longhorn storage substrate; K3s runs the application workloads.
• GitLab CI builds artifacts; Flux reconciles the FlexInfer deployment from its repository.
• GPUProfile resources now hold architecture-specific runtime images, memory budgets, device selection, and backend support.
• The platform is mixed-vendor. AMD gfx1100, AMD gfx906, and NVIDIA sm_52 have intentionally different capability matrices.
• GitOps makes desired state and rollback reviewable. It does not remove host-driver or hardware operations.

The important evolution since the original build is that GPU behavior moved out of one-off model manifests and into platform contracts. That reduced configuration drift more than any individual runtime tweak.

Context: what “done” means

I’m not trying to recreate a hyperscaler. “Done” for this platform looks like:

• A clean path to deploy and roll back GPU workloads (Helm/Kustomize + Flux).
• A repeatable way to deploy FlexInfer changes without snowflake cluster edits.
• Guardrails: secrets management, resource isolation, observability, and upgrades that don’t require heroics.

Layer 0: Harvester as the substrate

Harvester gives me a useful boundary between the physical hosts and workload-cluster virtual machines:

• VMs for cluster nodes: workload cluster nodes are just VMs with predictable sizing.
• Storage: Longhorn as the default makes PVC behavior consistent (and debuggable).
• Networking: I keep the path observable and change it only after measurements identify a bottleneck.

The operational win is repeatable VM and storage management. GPU passthrough, firmware, and failed physical devices still require host-level procedures.

Layer 1: K3s for the workload cluster

I’m using K3s for the workload cluster because it’s lightweight, well-understood, and easy to repair. The tradeoff is that you have to be disciplined about what you add:

• Prefer “one controller per concern” (Flux, cert-manager, ingress, observability) instead of kitchen-sink bundles.
• Keep the cluster API surface small: fewer CRDs, fewer moving parts.

Layer 2: GitLab + Flux GitOps (no Fleet)

The contract I want is simple: the cluster is a projection of Git.

At a high level:

1. services/* repos build and test artifacts in GitLab CI.
2. Images publish to a registry with immutable tags.
3. A separate GitOps repo declares desired state (HelmRelease/Kustomization).
4. Flux reconciles that state into the cluster.

A minimal Flux Kustomization looks like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: flexinfer-platform
  namespace: flux-system
spec:
  interval: 10m
  path: ./deploy
  prune: true
  sourceRef:
    kind: GitRepository
    name: platform-gitops

The key operational move is to make “how things ship” legible:

• CI owns tests and artifact builds.
• GitOps owns rollout policy and drift correction.
• Emergency edits are bounded and recorded, then either moved into Git or allowed to reconcile away.

GPU enablement: make it boring

GPU support fails in predictable ways: driver mismatch, runtime mismatch, or scheduling mismatch. I aim for boring invariants:

• Drivers/runtime: pick a known-good AMD driver + ROCm combo and don’t drift casually.
• Device plugin / operator: deploy via GitOps, pin versions, and treat upgrades like real changes.
• Scheduling: label/taint GPU nodes and make workloads declare intent.

On AMD nodes, my “sanity check” is intentionally unglamorous: confirm /dev/kfd + /dev/dri exist, validate the host with rocminfo/rocm-smi, then validate the container runtime can see the device nodes (before I even look at model code).

The patterns I rely on are:

• Taint GPU nodes and require a toleration.
• Use node selectors or affinity for “GPU-capable” pools.
• Request GPUs explicitly through Model.spec.gpu; the controller turns that into the vendor resource request.

If the platform is healthy, a GPU workload should fail fast and obviously when it’s misconfigured.

One AMD footgun: KFD wants all GPUs bound

On my RX 7900 XTX node, the integrated GPU is a trap: if it isn’t bound to the amdgpu driver, /dev/kfd breaks and everything above it becomes “mysteriously flaky.”

The fix is unglamorous but effective:

• keep the iGPU bound,
• use the AMD device plugin in mixed resource-naming mode,
• and keep the usable device index in the gfx1100 GPUProfile, where the controller can inject it consistently.

That’s the kind of thing I mean by “make it boring”: encode the footguns into the platform so model deploys don’t have to rediscover them.

Another footgun: gfx1100 needs a tested profile

On consumer RDNA3 (gfx1100), I have hit hangs and backend-specific failures when runtime defaults drifted. I now keep the tested defaults in one GPUProfile, including:

• HSA_OVERRIDE_GFX_VERSION=11.0.0
• PYTORCH_ROCM_ARCH=gfx1100
• TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
• HIP_FORCE_DEV_KERNARG=1
• TORCH_BLAS_PREFER_HIPBLASLT=1

These are specific to my validated image and card. They are not universal ROCm recommendations. The current gfx1100 vLLM profile targets the v1 engine with eager execution as its safe default; the earlier V0 guidance in this post is no longer current.

Versioning: What I Pin (and Why)

The biggest operational lesson from GPU homelab work is that “latest” is not a plan.

I treat these as part of the platform contract and pin them:

• Harvester (virtualization substrate)
• K3s (workload cluster distribution)
• Flux + Helm controller versions
• Container images that depend on GPU runtimes
• ROCm + kernel/driver combo on the AMD nodes

When I do upgrade, I do it like a production change: one variable at a time, with an easy rollback and a short “known-good” note in Git.

What changed after the initial build

The repository now has three concrete controls that were missing from the original version of this case study:

• Architecture profiles: deploy/gpuprofiles/ records supported backends, memory limits, immutable images, device indices, and tested feature certificates.
• Declarative node mode: a GamingSession custom resource can drain inference from a selected GPU node, switch it to a gaming runtime, and restore inference on deletion.
• Fleet observability: GitOps-managed Grafana dashboards cover per-card compute and VRAM, runtime model state, backend crashes, health failures, and shared-group state.

I also learned a storage lesson the hard way. A replicated Longhorn volume is useful for durability, but one cross-node serving-path read stalled a model load for 8 minutes and 47 seconds. I now separate durable artifact storage from local serving caches.

Results and limitations

This stack gives me a reviewable path from source change to reconciled workload and a central place for GPU compatibility decisions. It has also made mixed-vendor scheduling and model swaps observable.

The limits are important:

• It is a small private cluster, not evidence of hyperscale behavior.
• Host firmware, drivers, passthrough, and physical recovery remain outside normal pod reconciliation.
• Consumer GPU support is validated per architecture, backend, image, and model shape.
• Local caches improve startup but need rebuild, disk-pressure, and recovery policies.
• The current Git repository contains some older status documents; I treat executable manifests and tests as the stronger source.

Takeaways

1. Keep Flux as the desired-state loop, but document break-glass operations.
2. Encode GPU compatibility once in architecture profiles.
3. Make GPU ownership visible to the scheduler.
4. Separate durable artifact storage from fast local serving storage.
5. Measure model lifecycle and shared-GPU behavior, not just node readiness.