Loom Mills v2 — Rollback Playbook

When a v2 feature misbehaves in production, the cheapest mitigation is almost always a single ConfigMap edit + Flux reconcile — the operator hot-reloads policy on every reconcile tick (60s default). This playbook walks the operator through the three rollback layers: feature flag disable, policy proposal revert, and (last resort) canonical-DB restore from MinIO.

When to roll back

Symptom	First-line action
Squad routing degrading merge rate	Disable squads (8.3-1)
Audit pool flooding noise / blocking merges	Disable audit OR flip back to advisory_only (8.3-2)
Council debate burning budget without quality gain	Disable debate for that trigger (8.3-3)
Cross-repo item misbehaving on a non-home repo	Disable cross_repo routing (8.3-4)
Adaptive proposal applied a bad policy	Revert the proposal (slice 7.2)
Operator boot loop / DB integrity check fails	Restore DB from MinIO backup

Layer 1 — feature flag disable

All v2 features are gated behind a single enabled: bool field in platform/gitops/k3s/mills/configmap-policy.yaml. The operator's PolicyManager hot-reloads on the next reconcile tick (≤60s) once the ConfigMap symlink swap is observed.

Hot-reload caveat (pre-fix): between 2026-05-04 and the hot-reload fix landing, the PolicyManager fsnotify filter only matched events for the policy.yaml symlink directly, missing the K8s ConfigMap ..data symlink swap. If the operator hasn't picked up the change within ~2 min after flux reconcile, do a manual rolling restart:

kubectl rollout restart deploy/loom-mills-operator -n loom-mills

Verify the new pod's in-memory policy: kubectl exec ... -- wget -qO- localhost:8090/api/mills/policy | jq '.Squads.Enabled'. Disruption: ~30s (deployment is replicas: 1, strategy: Recreate).

Disable squads

# In platform/gitops:
git checkout -b ops/disable-squads
yq -i '.policy.squads.enabled = false' k3s/mills/configmap-policy.yaml
git commit -am "ops(mills): disable squads (incident YYYY-MM-DD)"
git push
flux reconcile kustomization apps -n flux-system
# Confirm hot-reload:
kubectl logs -n loom-mills deploy/loom-mills-operator --tail=20 | grep "policy reloaded"

After disable, in-flight squad-routed runs continue to terminal state — disable affects routing decisions for new backlog items only. To kill in-flight runs, force-escalate them per MILLS_RUNBOOK.md → Force-escalate a stuck pipeline run.

Disable audit

Two modes — the lighter one keeps audits running but takes them out of the merge gate:

# Lighter: flip to advisory_only (audits still emit findings but never block):
yq -i '.policy.audit.advisory_only = true' k3s/mills/configmap-policy.yaml

# Heavier: stop running audits at all:
yq -i '.policy.audit.enabled = false' k3s/mills/configmap-policy.yaml

Prefer advisory_only = true first — you keep the visibility (HUD's Audit panel still populates) without the merge-blocking blast radius. Flip enabled = false only if the audit pool itself is the problem (e.g. wedged dispatcher, runaway cost).

Disable debate

Debate is per-trigger (cron / roadmap / incident). Disable the offending trigger only:

# Disable debate on the cron-fired Sunday council:
yq -i '.policy.council.debate.enabled.cron = false' k3s/mills/configmap-policy.yaml

# Or fully:
yq -i '.policy.council.debate.enabled.cron = false' k3s/mills/configmap-policy.yaml
yq -i '.policy.council.debate.enabled.roadmap = false' k3s/mills/configmap-policy.yaml
yq -i '.policy.council.debate.enabled.incident = false' k3s/mills/configmap-policy.yaml

In-flight debates run to budget; the next council will use the single-pass (non-debate) ensemble.

Disable cross-repo

cross_repo.enabled gates whether a backlog item whose TargetProject names a non-home repo may start at all. Turning it off fail-closes that routing: the reconciler skips such items with reason=cross_repo_disabled and they stay queued until it is re-enabled or the target is corrected. Home-repo items are unaffected.

# 1. Stop starting items bound for non-home repos:
yq -i '.policy.cross_repo.enabled = false' k3s/mills/configmap-policy.yaml
git commit -am "ops(mills): disable cross_repo (incident YYYY-MM-DD)"
git push && flux reconcile kustomization apps -n flux-system

# 2. Confirm the gate is live (skips appear as reconciler.skipped events):
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/events?kind=reconciler.skipped" \
  | jq '.[] | select(.data.reason == "cross_repo_disabled")'

Each cross-repo item is an ordinary single-repo pipeline run against another project, so anything already in flight is rolled back the same way as any other run — there is no multi-repo half-merged state to unwind. To back out work that already merged, revert its MR on the target repo.

Atomic multi-repo merge (one item, several repos, all-or-nothing) was specified as slices 4.2–4.5 but never built; the machinery and its /api/mills/cross-repo/runs endpoints were removed on 2026-08-15. If that capability is ever built, this section needs a revert playbook again.

Disable adaptive proposal job

Adaptive runs once per week (Sunday 05:00 UTC). To prevent the next firing:

yq -i '.policy.adaptive_policy.enabled = false' k3s/mills/configmap-policy.yaml

The Sunday scheduler observes policy.adaptive_policy.enabled == false on its next tick and short-circuits without writing rows. Existing policy_proposals rows are unaffected (revert them via Layer 2).

Layer 2 — revert an applied policy proposal

Adaptive proposals get a 24-hour revert_deadline stamped on apply (slice 7.2). The actual revert is a manual ConfigMap edit because the operator deliberately does NOT auto-write to the gitops repo — that's a safety boundary so a runaway operator can't ship arbitrary policy changes through GitOps.

Note: The DAO exposes Revert(id) (pkg/mills/store/dao_policy_proposal.go) but no REST endpoint is wired in v2.0. A future slice may surface it on the HUD card; for now, the manual flow below is canonical.

Revert procedure (works within and outside the 24h window)

# 1. Inspect the original diff in the proposal row:
ID=42
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/policy/proposals/$ID" | jq -r .Diff
# example output: "policy.budgets.pipeline.max_usd_per_run: 5.0 → 6.0"

# 2. Reverse the diff in the gitops ConfigMap directly:
git checkout -b ops/revert-policy-proposal-$ID
yq -i '.policy.budgets.pipeline.max_usd_per_run = 5.0' \
   platform/gitops/k3s/mills/configmap-policy.yaml
git commit -am "ops(mills): revert policy proposal $ID"
git push && flux reconcile kustomization apps -n flux-system

# 3. Mark the proposal rejected so adaptive doesn't keep proposing the
#    same delta on next Sunday (slice 7.2 endpoint, admin-token):
curl -X POST -H "Authorization: Bearer $LOOM_MILLS_ADMIN_TOKEN" \
  "$LOOM_MILLS_OPERATOR_URL/api/mills/policy/proposals/$ID/reject"

The operator hot-reloads the live policy on the next reconcile tick (≤60s); behaviour reverts immediately. The proposal row's state stays applied_human (or whatever it was) — that's by design so post-incident review can see "this proposal was applied then unwound."

Layer 3 — DB restore

Last resort. See MILLS_RUNBOOK.md → Recover from a corrupted DB. The procedure scales operator to 0, restores from the nightly MinIO snapshot, and brings the operator back up. Items committed since the last snapshot are lost in a restore — roadmap_intents and .loom/backlog/*.yaml self-heal because they're regenerated from the canonical store, but in-flight pipeline runs whose attempts started since the snapshot have to be re-driven by hand.

Verifying a rollback landed

Every rollback path has the same verification surface:

# 1. Operator hot-reloaded the new policy:
kubectl logs -n loom-mills deploy/loom-mills-operator --tail=50 | grep -i "policy reloaded"

# 2. Live policy matches the gitops repo:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/policy" | jq '.squads.enabled'
# expect: false

# 3. Behaviour confirms — for a squads disable, the next reconcile tick
#    routes a backlog item via the fallback path and SquadOutcome rows
#    stop landing:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/squads" | jq '.[].LastOutcomeAt' | sort -u

Decision tree summary

v2 feature misbehaving
├── Audit / squads / debate / adaptive — disable via ConfigMap (≤60s)
├── Cross-repo — disable + abort in-flight + verify revert
├── Specific bad proposal applied — revert via REST (within 24h)
└── DB / canonical-state corruption — restore from MinIO (operator down ~5min)

Sources

• docs/MILLS.md — v1 architecture + v2 forward references
• docs/MILLS_RUNBOOK.md — operator runbook (DB restore, force-escalate, audit a merged change)
• .loom/93-product-spec-mills-v2-hierarchical-swarm-2026-05-02.md §"Failure modes"
• .loom/94-implementation-plan-mills-v2-hierarchical-swarm-2026-05-02.md Phase 8 slices 8.1–8.3
• pkg/mills/policy.go — enabled field on every v2 policy struct
• pkg/mills/store/dao_policy_proposal.go — Apply / Reject / Revert DAO methods
• cmd/loom-mills-operator/handlers_policy_proposals.go — REST endpoints (Phase 7 slice 7.2)