GPU Cost Baseline: What to Measure, What Lies

The invoice is not a GPU cost baseline. It is one input.

I want a baseline to explain which capacity was paid for, what work used it, and how much of that work produced a successful result. If those three views cannot be joined, a precise-looking cost-per-request number is usually fiction.

TL;DR

• Measure allocation, compute utilization, VRAM pressure, and request outcomes separately.
• Use vendor-specific collectors. NVIDIA Data Center GPU Manager (DCGM) metrics do not describe AMD cards.
• Define the cost basis before comparing cloud and owned hardware.
• Attribute cost to successful work, while keeping retries, cold starts, and idle headroom visible.
• Treat thresholds as workload-specific. Low utilization can be waste, deliberate latency headroom, or both.

Context: what the baseline must answer

For each workload and reporting window, I want four answers:

1. What capacity was available and billable?
2. How much of that capacity was allocated?
3. How much GPU compute and VRAM did the workload use?
4. How many useful outcomes did it produce?

“Useful outcome” depends on the product. It might be a successful completion, generated image, transcribed minute, or finished batch item. Request count alone is weak when retries, different context lengths, or failed outputs consume materially different GPU time.

The visibility stack

The layers have different jobs:

Layer	What it answers	What it misses
Billing or asset ledger	What capacity cost	Whether the capacity did useful work
Kubernetes	Which workload reserved a GPU	Whether the GPU was busy
GPU telemetry	Compute, VRAM, temperature, and power	Which request created the load unless labels are joined
Application/proxy	Requests, tokens, latency, errors, retries	The full cost of idle or unavailable capacity

1. Inventory paid capacity

For cloud capacity, export line-item billing data rather than copying a console total. For owned hardware, choose and document an amortization period, then include power, cooling, space, spares, support, and operator time.

In Kubernetes, start with advertised resources:

kubectl get nodes -o json | jq -r '
  .items[] |
  [.metadata.name,
   (.status.allocatable | to_entries |
    map(select(.key | test("^(nvidia\\.com|amd\\.com)/gpu"))) |
    map("\(.key)=\(.value)") | join(","))] |
  @tsv'

Then find pods that reserve those resources:

kubectl get pods -A -o json | jq -r '
  .items[] as $pod |
  $pod.spec.containers[] |
  (.resources.limits // {}) as $limits |
  ($limits | to_entries |
   map(select(.key | test("^(nvidia\\.com|amd\\.com)/gpu")))) as $gpu |
  select($gpu | length > 0) |
  [$pod.metadata.namespace, $pod.metadata.name, $pod.spec.nodeName,
   ($gpu | map("\(.key)=\(.value)") | join(","))] |
  @tsv'

This reports allocation, not utilization. kubectl top reports CPU and system memory; it does not provide GPU usage.

2. Measure the correct GPU signal

Compute utilization and VRAM utilization are different. A model can occupy most VRAM while doing almost no compute. That state may be intentional for a latency-sensitive warm endpoint, but it is still paid idle capacity.

For NVIDIA, DCGM Exporter is the usual Prometheus source. Typical series include DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, and DCGM_FI_DEV_FB_TOTAL.

For AMD, use an AMD-aware source such as sysfs, ROCm tooling, or an exporter built on those interfaces. In my current GitOps configuration, the NVIDIA collector and AMD collector are separate. The AMD path exports amdgpu_gpu_busy_percent and VRAM byte gauges; the FlexInfer node agent also exposes normalized flexinfer_gpu_compute_utilization_percent and flexinfer_gpu_vram_utilization_percent series.

Useful queries on that normalized surface include:

# Per-card compute utilization
max by (node, gpu, vendor) (
  flexinfer_gpu_compute_utilization_percent
)

# Per-card VRAM utilization
max by (node, gpu, vendor) (
  flexinfer_gpu_vram_utilization_percent
)

# Fraction of the last 24 hours spent below 5% compute utilization
avg_over_time((
  max by (node, gpu) (flexinfer_gpu_compute_utilization_percent) < bool 5
)[24h:5m])

The last query returns a fraction, not GPU-hours. Multiply each card's fraction by the 24-hour window, then sum across cards to estimate idle GPU-hours.

3. Join cost to useful work

At minimum, keep these counters by model or workload:

• successful and failed requests,
• retries and queue rejections,
• prompt and completion tokens for language models,
• request latency and cold-start latency,
• model loads, unloads, and shared-GPU swaps.

The basic equations are deliberately boring:

capacity_cost = hourly_capacity_cost × billable_hours

cost_per_success = total_capacity_cost / successful_outcomes

cost_per_1k_output_tokens =
  total_capacity_cost / (successful_output_tokens / 1,000)

waste_cost_estimate =
  total_capacity_cost × (idle_fraction + failed_work_fraction)

Do not add idle_fraction and failed_work_fraction if their time windows overlap. The last equation is only valid when the categories are mutually exclusive.

A synthetic example

This example is arithmetic, not a market price quote:

workload: inference-api
window_hours: 720

capacity:
  gpu_count: 4
  cost_per_gpu_hour_usd: 22.93

outcomes:
  successful_requests: 12000000
  failed_requests: 240000
  successful_output_tokens: 1800000000

observations:
  compute_idle_fraction: 0.25
  note: 'Idle is based on per-card compute below 5% in 5-minute samples.'

That produces about $66,038 in capacity cost, $0.0055 per successful request, and $0.0367 per 1,000 successful output tokens. Those values are only comparable across periods if the workload mix and accounting basis stay consistent.

4. Explain headroom instead of hiding it

Average utilization is not a grade. I do not label 70% “good” without looking at the latency objective, burst shape, model-load time, and failure policy.

Classify idle capacity into at least three buckets:

• Intentional headroom: capacity held for an explicit latency or availability target.
• Operational overhead: model loading, compilation, health checks, and rolling changes.
• Unexplained idle: paid capacity without an owner or stated service objective.

Only the last bucket is an obvious optimization target. Intentional headroom can still be too expensive, but that is a product tradeoff rather than a monitoring defect.

5. Compare procurement strategies with one model

Cloud on-demand, commitments, spot capacity, and owned hardware must share the same boundary:

• same workload and throughput target,
• same availability and recovery assumptions,
• same networking and storage costs,
• same operator-time treatment,
• same hardware lifetime and residual-value assumptions.

I do not use generic “50% cheaper” claims. Provider prices, commitment terms, utilization, financing, and support costs move too much. I calculate a range and show the variables that can reverse the decision.

Results and limitations

A useful baseline produces a monthly reconciliation table, per-workload unit costs, idle GPU-hours, and a short list of attribution gaps. It should also make a repeated run cheap.

The main limitations are predictable:

• GPU metrics may lack pod labels, especially on custom AMD collectors.
• Streaming clients may omit token-usage details.
• Shared GPUs make time-based allocation approximate.
• Quality failures need an application-specific evaluation signal; HTTP 200 is not enough.
• Owned-hardware comparisons are sensitive to amortization and operator-time assumptions.

When one of those gaps applies, I publish a range or mark the result directional.

Takeaways

1. Start with a declared cost boundary and reporting window.
2. Keep allocation, compute utilization, and VRAM pressure separate.
3. Use vendor-specific collection, then normalize with recording rules.
4. Attribute cost to successful work and expose retries, cold starts, and idle headroom.
5. Optimize the unexplained portion first.

Related reading:

• SLOs for Inference
• GPU Failure Modes
• AI Infra Readiness Audit checklist