SLOs for Inference: Latency, Errors, Saturation

An inference endpoint can be up while users wait through a cold start, receive an error after a model swap, or get a syntactically valid answer that fails the task.

I define service-level objectives (SLOs) around those outcomes. GPU utilization belongs on the same dashboard, but it is usually a cause or capacity signal, not the user-facing objective.

TL;DR

• Define good events before choosing a percentage.
• Split warm-request latency from cold-start and shared-GPU swap latency.
• Track queue rejection and queue wait explicitly.
• Keep model-quality evaluation separate from transport availability.
• Use request-based error budgets for request SLOs; “minutes of downtime” only fits time-based availability.
• Page on budget burn or user symptoms. Use GPU saturation to diagnose them.

Context: the service boundary

This post assumes an OpenAI-compatible proxy in front of one or more model runtimes. The proxy owns routing, cold-start queueing, and request accounting. A controller owns model lifecycle and shared-GPU policy.

That boundary gives me three distinct paths:

1. Warm request: a ready model serves immediately.
2. Cold activation: the request waits while a model becomes ready.
3. Shared-GPU swap: one model yields the card and another loads.

Combining those paths into one latency histogram can be useful for the product SLO, but it is not enough for diagnosis or capacity planning.

SLI, SLO, and SLA

• A service-level indicator (SLI) is a measured ratio or distribution.
• An SLO is the target for that indicator over a window.
• A service-level agreement (SLA) is an external commitment with consequences.

I do not start with “99.9%.” I start by defining what counts as a valid event and which traffic is in scope.

The core indicators

1. Request success

For the current FlexInfer proxy, flexinfer_proxy_requests_total{model,status} records success and error outcomes.

sum(rate(flexinfer_proxy_requests_total{status="success"}[5m]))
/
clamp_min(sum(rate(flexinfer_proxy_requests_total[5m])), 1e-9)

Decide which caller errors belong in the denominator. Invalid input is often excluded from a service-availability SLO, but a proxy-generated rejection caused by full internal queues usually belongs in it.

Queue rejection deserves its own rate too:

sum by (model) (rate(flexinfer_proxy_queue_rejected_total[5m]))

2. End-to-end request latency

The product SLO should cover the time the proxy spends processing the request, including activation wait when applicable:

histogram_quantile(0.95,
  sum by (le, model) (
    rate(flexinfer_proxy_request_duration_seconds_bucket[15m])
  )
)

Percentiles are useful for dashboards, but a threshold SLO is easier to budget as good and bad events. For a five-second threshold:

sum(rate(flexinfer_proxy_request_duration_seconds_bucket{le="5"}[30d]))
/
clamp_min(sum(rate(flexinfer_proxy_request_duration_seconds_count[30d])), 1e-9)

Choose the threshold from user research or observed abandonment, not from a generic inference table.

The proxy does not currently expose a dedicated time-to-first-token histogram. Streaming request duration is not a substitute. If first-token latency is the user contract, instrument it at the proxy or client before declaring that SLO measurable.

3. Activation and queue wait

Cold-start queue time is visible separately:

histogram_quantile(0.95,
  sum by (le, model) (
    rate(flexinfer_proxy_queue_wait_duration_seconds_bucket[15m])
  )
)

histogram_quantile(0.95,
  sum by (le, model, result) (
    rate(flexinfer_proxy_activation_duration_seconds_bucket[1h])
  )
)

These are diagnostic SLIs and may also be product SLOs for serverless models. Keep warm and cold traffic distinguishable; otherwise a tiny cold-request population can disappear inside a healthy aggregate.

4. Shared-GPU swap behavior

For a shared group, I watch both frequency and cost:

sum by (group) (rate(flexinfer_sharedgroup_preemptions_total[15m]))

histogram_quantile(0.95,
  sum by (le, group) (
    rate(flexinfer_model_swap_duration_seconds_bucket[1h])
  )
)

A high preemption rate with stable swap latency is different from a single storage-backed swap that stalls for minutes. Both can violate the request SLO, but the repairs are different.

5. Saturation and traffic

Saturation explains risk. It is not automatically a failure.

# GPU compute, kept separate from VRAM occupancy
max by (node, gpu, vendor) (
  flexinfer_gpu_compute_utilization_percent
)

max by (node, gpu, vendor) (
  flexinfer_gpu_vram_utilization_percent
)

# Work waiting or executing at the proxy
sum by (model) (flexinfer_proxy_queue_depth)
sum by (model) (flexinfer_proxy_active_connections)

# Current model throughput gauge
sum by (model) (flexinfer_tokens_per_second)

I alert on saturation only when it predicts a user symptom or an imminent hard limit. A warm model occupying most VRAM can be healthy. A GPU at high compute utilization can also be healthy while latency stays inside its objective.

6. Model quality

HTTP success does not measure answer quality. Quality needs a separate indicator tied to the task:

• schema-valid output,
• retrieval or ranking quality,
• evaluation-suite pass rate,
• grounded-answer rate,
• human escalation or correction rate.

Keep the evaluation set, sampling method, grader version, and model version with the result. An automated judge without those fields produces a number, not an auditable SLI.

A concrete SLO document

This is an illustrative shape. The targets are examples, not recommendations:

service: inference-proxy
owner: ml-platform
window: 30d

traffic_scope:
  include: authenticated completion requests
  exclude: rejected invalid input

slos:
  - name: request-success
    good_event: proxy status is success
    target: 99.5%

  - name: request-under-5s
    good_event: end-to-end proxy duration <= 5s
    target: 99.0%

  - name: cold-activation-under-120s
    good_event: activation result is success and duration <= 120s
    target: 95.0%
    scope: models with minReplicas=0

  - name: evaluated-quality
    good_event: pinned evaluation case passes the versioned rubric
    target: 97.0%
    evaluation_window: weekly

The targets differ because the user contracts differ. A scale-to-zero lab model may have a looser cold-start objective than a warm interactive route.

Error budgets: count bad events correctly

For a request-success SLO of 99.5%, the request error budget is 0.5% of eligible requests during the window.

allowed_bad_requests = eligible_requests × (1 - target)

The familiar “43.2 minutes per month at 99.9%” calculation applies to a time-based availability indicator over a 30-day month. It does not describe a request-based SLO when traffic is uneven.

Quality budgets also need their own denominator. Do not combine an offline evaluation failure with a 5xx response into one percentage unless they are measured on the same eligible population.

Alert on burn, then diagnose

A burn-rate alert compares the observed bad-event ratio with the allowed ratio:

(
  1 -
  sum(rate(flexinfer_proxy_requests_total{status="success"}[15m]))
  /
  clamp_min(sum(rate(flexinfer_proxy_requests_total[15m])), 1e-9)
)
/
0.005

For a 99.5% SLO, a value of 1 consumes budget at exactly the sustainable rate. A value of 6 burns it six times faster. In a real alert, pair a short window with a longer window so a brief spike and a slow leak do not page identically.

When the budget burns, I pivot to:

• queue depth and queue rejection,
• activation failures and retry wait,
• stalled-load counters and loading progress,
• shared-group preemption and swap time,
• backend crashes and health-check failures,
• GPU compute, VRAM, temperature, and node health.

Results and limitations

The current FlexInfer metrics support request success, total request duration, queueing, activation, model lifecycle, and shared-GPU swap indicators. They also expose prompt and completion token histograms for responses whose upstream usage data is available.

Coverage still has limits:

• streaming token histograms require a terminal usage chunk,
• time to first token is not yet a first-class proxy metric,
• model quality remains workload-specific,
• low-volume models need longer windows or event counts beside percentiles,
• a histogram cannot recover detail finer than its configured buckets.

I write those limitations into the SLO document. An unmeasurable objective is a roadmap item, not a green checkmark.

Takeaways

1. Define the service boundary, traffic scope, and good event first.
2. Separate warm latency, cold activation, queue wait, and swap time.
3. Budget request failures with requests, not downtime minutes.
4. Keep quality evaluation versioned and independent from transport success.
5. Page on budget burn; diagnose with lifecycle and GPU signals.
6. Publish telemetry gaps beside the objectives they limit.

Related reading:

• GPU Cost Baseline
• GPU Failure Modes
• AI Infra Readiness Audit checklist