AI Infra Readiness Audit: What I Check (and What You Get)

Most AI infrastructure conversations go straight to tooling. I start with evidence. The recurring problems are usually less exotic:

• GPU spend climbs without a clear baseline.
• Production reliability is shaky (and telemetry doesn’t answer the first two questions you ask).
• Deploys are stressful because rollback paths aren’t real.

Those symptoms do not prove that you need a new platform. They prove that you need a baseline strong enough to separate capacity, software, and operating-process failures.

TL;DR

• Start with a fixed question and a bounded evidence window.
• Build cost and reliability baselines before proposing a redesign.
• Separate verified facts, estimates, and missing telemetry.
• Turn the findings into a sequenced roadmap with owners and acceptance checks.

If you want the packaged version of this, start here: AI Infra Readiness Audit.

Context: what I am auditing

This checklist is for a serving platform with GPU-backed inference, a deployment control plane, and enough request volume to make tail latency and cost visible. It also works for a smaller cluster, but some signals will be directional rather than statistically strong.

I ask for a representative traffic window, billing or amortized hardware data, deployment history, and read-only access to workload and telemetry configuration. I do not treat a single dashboard screenshot as a baseline.

The checklist

1) Workloads and success metrics

I want to know:

• Which workloads matter to revenue, customers, or internal dependencies?
• What does success mean for each one: time to first token, end-to-end latency, errors, throughput, quality, or cost?
• Which constraints are verified: compute, video memory (VRAM), system memory, storage, network, queueing, or model behavior?

2) GPU cost baseline (and what moves it)

You can’t optimize what you can’t explain.

Baseline questions:

• What did GPU capacity cost over the chosen window?
• Which cost basis applies: cloud bill, lease, or amortized hardware plus power, cooling, and operator time?
• What are the unit economics: cost per successful request, output token, image, or job?
• How much paid capacity was idle, retrying failed work, loading models, or reserved as deliberate headroom?

Output:

• A reproducible cost model with its source period, equations, and assumptions.

3) Reliability and operational visibility

If service-level indicators (SLIs) and objectives (SLOs) do not exist yet, I start with raw signals:

• Error rates (by endpoint/model)
• Latency percentiles
• Saturation (GPU compute, VRAM, CPU, and system memory)
• Queue depth / backpressure
• Cold-start and model-swap time
• Deployment frequency and rollback outcomes

Output:

• Candidate SLIs/SLOs, recording rules, and the minimum symptom-first alerts needed to operate the system.

4) Deployment, rollouts, and rollback paths

I’m looking for operational safety:

• Can you roll out gradually (canary, weighted routing, feature flags)?
• Can you roll back code, runtime images, configuration, and model artifacts independently?
• Are runtime images and model artifacts immutable and traceable to a deployment?

Output:

• A release-hygiene backlog ordered by risk reduction, not novelty.

5) Failure modes and a risk register

The audit should name the top ways the system fails and what you do about each:

• GPU node failures and driver/runtime drift
• Model crashes / OOMs
• Thundering herds and retry storms
• Bad deploys and broken compatibility
• Silent partial failures (slowdowns, queue growth)

Output:

• A risk register with evidence, mitigations, owners, and a validation step.

What you get after the audit

• A scorecard across cost, reliability, security, and operability
• A GPU cost model with explicit assumptions and confidence gaps
• A risk register with the highest-impact failure modes and mitigations
• A sequenced roadmap, usually framed as the next 30, 60, and 90 days when that horizon fits the team

The limitations matter. If request metrics cannot be joined to model and cost data, I report the gap instead of inventing cost per request. If the workload window misses a seasonal peak, the capacity recommendation stays provisional.

If you want help implementing

If the roadmap is clear and you want help shipping it:

• A bounded stabilization project works when the findings have clear acceptance checks.
• Advisory work fits better when the team will implement the changes and needs architecture or review support.

Start here: AI Infra Readiness Audit.

---

Related reading

For deeper dives on specific aspects of the audit, see:

• GPU Cost Baseline: What to Measure, What Lies - Building a cost model that is actually useful
• SLOs for Inference: Latency, Errors, Saturation - Defining meaningful reliability targets
• Hybrid/On-Prem GPU: The Boring GitOps Path - When and how to run your own GPU infrastructure
• GPU Failure Modes: What Breaks and How to Debug It - Common failures and how to diagnose them

Or browse the full AI Infrastructure Readiness topic guide.