Hybrid/On-Prem GPU: The Boring GitOps Path

Buying GPUs is the easy part. The hard part is making drivers, model artifacts, scheduling, rollouts, and recovery behave like one system.

I use GitOps to make those contracts explicit. It does not turn a small on-prem cluster into a managed cloud service. It does make drift visible and routine changes repeatable.

TL;DR

• Model the total cost before buying hardware; there is no universal savings percentage or break-even month.
• Keep desired state in Git, but keep emergency procedures available and documented.
• Pin the host/runtime compatibility matrix and immutable serving images.
• Make GPU ownership schedulable. A process using a device without a resource request is an invisible noisy neighbor.
• Use fast local storage for the serving path and durable shared storage for artifact distribution and recovery.
• Treat cloud burst as a separate capacity policy, not a magic overflow switch.

Context: the platform I am describing

My reference system is a small private platform with Harvester underneath a K3s workload cluster. GitLab CI builds artifacts, a registry stores images, and Flux reconciles the application and model manifests.

The GPU fleet is mixed: AMD gfx1100 and gfx906 nodes plus an older NVIDIA sm_52 card. That mix is useful because it forces the platform to describe capability instead of assuming every GPU can run every backend.

Decide with workload math, not a percentage

On-prem capacity is worth evaluating when the workload is steady, data locality matters, and the team can own hardware and Kubernetes operations. Cloud capacity remains a better fit for short experiments, sharp bursts, or hardware requirements that change faster than the asset lifecycle.

I compare both options with the same boundary:

• throughput and latency target,
• required availability and recovery time,
• average and peak demand,
• hardware, financing, power, cooling, space, support, and spares,
• storage, network egress, and backup costs,
• operator time and planned upgrade work,
• residual value and the risk of workload or model drift.

The result should be a range with sensitivity analysis. A point estimate that excludes operator time from one side or assumes perfect utilization on the other is not a decision model.

The four contracts

1. Hardware and runtime compatibility

The real unit of support is not “AMD” or “NVIDIA.” It is a tested combination of GPU architecture, kernel driver, runtime libraries, backend, model format, and container image.

In FlexInfer I encode that combination in GPUProfile resources. The current profiles declare architecture-specific backend support, immutable runtime images where available, memory budgets, device indices, and environment defaults. A gfx1100 profile and a gfx906 profile intentionally disagree about vLLM capabilities; hiding that difference would create a false portability promise.

For AMD nodes, I validate bottom-up:

ls -l /dev/kfd /dev/dri
rocminfo | sed -n '1,80p'
rocm-smi --showmeminfo vram

For NVIDIA nodes, I use the same sequence with nvidia-smi and a container built for the card's compute capability.

Pinning matters at two layers:

• Git pins the device-plugin, controller, and runtime configuration.
• Workload manifests pin images by digest or by a reviewed tag plus digest field.

2. Scheduling and ownership

Kubernetes can schedule an extended GPU resource. It cannot infer that an untracked host process or privileged container is using the same device.

I require four declarations:

1. GPU vendor and count.
2. Compatible node labels or profile.
3. GPU-node tolerations.
4. A sharing policy when more than one model can use the card.

With the FlexInfer v1alpha2 API, the application-facing shape is compact:

apiVersion: ai.flexinfer/v1alpha2
kind: Model
metadata:
  name: qwen3-8b
  namespace: flexinfer-system
spec:
  backend: llamacpp
  source: HF://Qwen/Qwen3-8B-GGUF
  gpu:
    vendor: amd
    count: 1
    shared: interactive-text
    priority: 200
  serverless:
    minReplicas: 0
    idleTimeout: 15m
  nodeSelector:
    flexinfer.ai/gpu.arch: gfx1100

The controller turns that intent into the vendor resource request and the rest of the pod contract. Raw Deployments can work too, but then every author must get the same details right.

Shared GPUs need observability. I track the active model, queue state, preemption count, and swap-to-ready latency. If interactive traffic frequently waits for model swaps, the lane needs a different policy or dedicated capacity.

3. Artifact and storage lifecycle

Model storage has two different jobs:

• Distribution and recovery: durable storage or an object/OCI registry.
• Serving: predictable local reads with enough throughput for model loading and memory mapping.

Those jobs do not have to use the same volume. In my cluster, a multi-replica distributed volume was acceptable for durability but caused an 8-minute, 47-second cross-node model-load stall on the serving path. The repair was to use a local, single-replica cache for serving and retain a durable source for rebuilding it.

The practical pattern is:

registry or shared artifact store
          |
          v
node-local NVMe cache -> model runtime -> request path

Track cache readiness separately from pod readiness. A pod that starts before its model artifact is local has not solved the cold-start problem; it has moved it.

4. Delivery and rollback

My delivery contract is:

service repository -> CI -> immutable image/artifact
                                  |
                                  v
GitOps repository -> Flux -> K3s desired state

A minimal Flux source and reconciliation pair looks like this:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: inference-platform
  namespace: flux-system
spec:
  interval: 5m
  url: ssh://[email protected]/platform/inference.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: inference-models
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: inference-platform
  path: ./deploy
  prune: true
  wait: false

Rollback must cover more than a Deployment image. I keep the runtime image, model artifact, model configuration, and GPU profile independently traceable. Otherwise a Git revert can restore YAML while leaving an incompatible artifact or driver in place.

GitOps also needs a break-glass rule. During an incident I may make a bounded imperative change to stop damage, record it, then either encode it in Git or let Flux revert it. “No kubectl ever” is not a safety property.

Vendor-specific telemetry

I do not use a DCGM dashboard as a generic GPU dashboard. The current platform runs DCGM Exporter only on NVIDIA nodes and a separate sysfs-based exporter on AMD nodes. FlexInfer's agent provides normalized per-card compute, VRAM, and temperature metrics for fleet views.

The minimum operating view includes:

• per-card compute utilization,
• used and total VRAM,
• temperature and power when the hardware exposes them,
• model phase and loading substage,
• backend crashes and health-check failures,
• queue depth, cold-start time, and shared-group swap time.

A node being Ready does not prove the GPU or model runtime is healthy.

Cloud burst is an application policy

Hybrid overflow requires more than a second cluster. The router needs to know:

• which models and artifacts exist in each location,
• how data and credentials may cross the boundary,
• whether the remote capacity is warm,
• how much latency and egress the path adds,
• what happens when the link or cloud capacity is unavailable.

I prefer to route an explicit workload class to remote capacity. Blindly overflowing every request after a local queue threshold can violate data-locality rules or turn a local saturation event into an expensive retry storm.

Results and limitations

This design has made routine model and platform changes reviewable in my small cluster. Architecture-specific profiles also stopped individual model manifests from accumulating contradictory runtime knobs.

It does not remove these limits:

• host driver and firmware changes still sit outside normal pod reconciliation,
• local caches improve loading but need rebuild and disk-pressure policies,
• consumer GPUs have narrower tested backend matrices,
• a small fleet has limited failure-domain diversity,
• GitOps can reproduce a bad declaration just as reliably as a good one.

Takeaways

1. Compare total cost under the same workload and reliability assumptions.
2. Treat GPU architecture plus runtime as a tested contract.
3. Make every GPU user visible to the scheduler.
4. Separate durable artifact storage from the fast serving cache.
5. Make rollback cover images, artifacts, configuration, and host compatibility.
6. Route hybrid overflow deliberately.

Related reading:

• GPU Cost Baseline
• Standing Up a GPU-Ready Private AI Platform
• AI Infra Readiness Audit checklist