Loom Mills — Operator Runbook
Day-2 procedures for loom-mills-operator. For architecture and policy reference, see docs/MILLS.md.
All kubectl commands assume KUBECONFIG=~/workspace/platform/gitops/.kube/k3s.yaml (or kc-k3s alias). Mac CLI examples use LOOM_MILLS_OPERATOR_URL (default https://mills.flexinfer.ai) for the REST + MCP listener; mutation examples also require an operator admin bearer token. The public ingress has a separate Cloudflare Access edge gate, authenticated with LOOM_MILLS_CF_ACCESS_ID / LOOM_MILLS_CF_ACCESS_SECRET (or the generic CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET fallback). The operator runs in namespace loom-mills. Lifecycle probes and Prometheus metrics live on a separate :9090 listener and are not assumed to be exposed at the public REST URL.
GitLab webhooks for ci_watch
Set LOOM_MILLS_GITLAB_WEBHOOK_SECRET to a strong shared secret and
LOOM_MILLS_OPERATOR_URL to the externally reachable operator base URL. At
startup the operator idempotently ensures the configured GitLab project has a
Pipeline/Merge Request hook targeting
$LOOM_MILLS_OPERATOR_URL/api/mills/hooks/gitlab. The receiver does not accept
the operator admin bearer; it authenticates GitLab's X-Gitlab-Token with the
shared secret. Keep the endpoint behind TLS and do not log or place the secret
in URLs.
Webhook payloads are wake-up hints only. ci_watch always re-reads the MR and
pipeline through the GitLab API before deciding success or failure. While a
subscription is live its safety poll runs every five minutes; if either env var
is missing or registration fails, startup warns and the existing polling path
remains available. A bounded subscriber drops its oldest hint on overflow;
alert on growth of mills_webhook_events_dropped_total from the metrics
listener.
After deployment, confirm GitLab shows both Pipeline and Merge Request events enabled for the hook, trigger several pipelines, and inspect the one-day stage roll-up:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/telemetry/stages?window=1d" |
jq '.stages[] | select(.stage == "ci_watch")'
Webhook-enabled runs should bring ci_watch p50 below 120 seconds. If it does
not, check operator logs for the registration warning, GitLab hook delivery
history for 401/connection failures, and the overflow counter before changing
the five-minute fallback.
The loom client and bundled status snapshot resolve both auth layers automatically. Literal curl examples against the public URL assume equivalent Authorization and CF-Access-* headers are supplied by the operator; they are omitted below for readability.
Quick status
loom mills status # one-liner from Mac
kubectl get deploy,po,pvc -n loom-mills # cluster snapshot
kubectl logs -n loom-mills deploy/loom-mills-operator --tail=200
curl -sf $LOOM_MILLS_OPERATOR_URL/api/mills/capabilities | jq
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/kpis?window=1d" | jq
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/telemetry/stages?window=1d" | jq
# Optional lifecycle/raw Prometheus checks. Keep this port-forward running in
# another terminal; these routes are not on LOOM_MILLS_OPERATOR_URL.
kubectl -n loom-mills port-forward svc/loom-mills-operator 9090:9090
curl -sf http://127.0.0.1:9090/readyz # 200 once initialized
curl -sf http://127.0.0.1:9090/metrics | grep '^mills_'
/readyz on the metrics listener is service readiness: the HTTP process,
migrations, and policy manager are initialized. It is not an autonomy approval signal. Before enabling or
trusting unattended writes, check /api/mills/status or
/api/mills/capabilities and require autonomy_ready=true. When this is
false, the scheduler may continue ticking for observability, but the reconciler
will not start queued backlog items.
Common autonomy blockers are intentional fail-closed states: missing admin auth,
missing or unwritable LOOM_MILLS_REPO_ROOT/.loom, unwired FlexInfer/GitLab/HUD
spawn/MCP hub dependencies, NoOp dispatcher coverage on write stages, or fake
council participants. Read-only APIs may stay healthy while these blockers are
present.
Stage failure counters also retain deliberate fail-fast and spawn-stall harness
runs. Group the corresponding Loki rows by run/backlog ID before declaring a
real-work regression; those failures are evidence that a negative-path gate
fired. The mills_autonomous_merges_real KPI excludes heartbeat canaries, but
stage-attempt telemetry is intentionally unfiltered.
In k3s, LOOM_MILLS_REPO_ROOT=/workspace/loom-core is backed by the same
Longhorn RWO PVC as /var/lib/loom-mills. There is no initContainer, git-sync
sidecar, or CronJob: the checkout is created in-process at operator boot by
ensureRepoRoot (cmd/loom-mills-operator/repo_root.go) using the GitLab
project token. If credentials, network, git, or the checkout itself fail, the
pod should still become service-ready, but /api/mills/capabilities must show
repo_root red and autonomy_ready=false.
Known gap — the checkout does not refresh. ensureRepoRoot early-returns
when repoRootReady is satisfied, and that check only asserts that .git
exists and .loom is a writable directory. Once the PVC holds a good clone,
those are true forever, so the fetch / checkout / merge --ff-only block is
never reached on a healthy pod and the tree stays pinned at whatever commit it
was first cloned at. Accumulated untracked .loom/*COUNCIL-* artifacts would
also make --ff-only warn-and-skip even if it were reached.
For a tests-stage escalation containing tests checkout infrastructure: pushed head unresolvable for <branch>, verify that the implement stage pushed the
named branch and that the operator token can read the item's target GitLab
project. This failure is deliberate: Mills will not run or pass an unpinned
quality gate.
Consequence for planning: the roadmap extractor fills roadmap_intents from
whatever ROADMAP.md is on the PVC, so a stale checkout produces stale-but-present
intents. The fail-closed preflight stays green while the signal is wrong, and
eval roadmap_alignment grades against it. To check freshness:
kubectl -n loom-mills exec deploy/loom-mills-operator -- \
git -C /workspace/loom-core log -1 --format='%H %ci'
If that lags origin/main, restore freshness out-of-band (re-clone the PVC path
or roll the pod after clearing it) and file the ensureRepoRoot refresh fix.
Pipeline-start recovery
A queued item is admitted with its run, workflow identity, capacity reservation, transition ledger row, and dispatch intent in one SQLite transaction. The starter runs only after that transaction commits. A pod crash between those steps is self-healing: the next reconciler tick leases the committed intent or recovers an acknowledged-but-still-queued top-level run and drives the same run ID forward.
Use the outbox gauge and structured events to distinguish a brief restart drain from a stuck starter:
curl -sf http://127.0.0.1:9090/metrics | \
grep '^mills_pipeline_dispatch_outbox_pending '
kubectl logs -n loom-mills deploy/loom-mills-operator --since=30m | \
grep -E 'reconciler\.(started|dispatch_dead_lettered|dispatch_ack_failed)'
The metrics command assumes the :9090 port-forward from Quick status is
running. The gauge may rise briefly during a restart and should then fall. Delivery is
at-least-once by design, with the run ID as the starter idempotency key. Do not
delete outbox or reservation rows to force progress: lease fencing prevents a
late consumer from acknowledging another consumer's work, and manual deletion
can hide an unaccepted start.
After the bounded retry ceiling, Mills marks the intent dead_letter,
escalates the current run/backlog aggregate, synchronizes the workflow terminal
state, and releases the reservation atomically. Investigate the associated
reconciler.dispatch_dead_lettered event and starter configuration, then use
the normal human-resolution/requeue path. Obsolete intents from an older
aggregate are retired without escalating a newer run.
The operator remains a singleton. These recovery guarantees cover process restart under SQLite WAL; they do not provide multi-replica worker fencing.
Run provenance
Every run is stamped once at that same post-commit boundary with a
run.provenance event (actor reconciler.provenance), keyed
subject_kind=pipeline_run for the DAG lane and workflow_run for the
imperative lane. The payload carries policy_checksum (sha256 of the exact
policy bytes the active policy was parsed from — the same value as the
deployment's loom.flexinfer.ai/policy-checksum annotation when the pod is in
sync), stage_models, and prompt_hashes. It is the join key for per-version
win-rate and cost analysis: without it a merged run cannot be attributed to the
policy revision, model pins, or prompt templates that produced it.
The stamp records START-TIME INTENT. A mid-run swap — policy hot-reload, a
council fallback editor substituting for an unreachable remote backend — is
attributable from the per-dispatch pipeline.stage.agent_routed events
instead, not from the stamp.
curl -sf http://127.0.0.1:9090/metrics | grep '^mills_run_provenance_stamps_total'
outcome="duplicate" is crash-recovery replay re-reaching an already-stamped
run and is normal. A rising outcome="error" means runs are landing without
the join key; the operator logs append run provenance stamp failed with the
run id. Provenance is best-effort by design — a failed stamp never blocks a
dispatch — so this is an analytics-completeness alert, not an availability one.
Updating backlog items safely
POST /api/mills/backlog creates an item when its ID is new. Updating an
existing ID requires the Revision returned by the latest backlog GET/list
response. A stale write returns HTTP 409 with
{"error":"stale-backlog-write"}. Re-read the item, merge the intended
metadata change, and retry with the new Revision; do not blindly replay the old
body because it may overwrite a lifecycle transition.
Pause and resume
The kill switch is policy.enabled: false. Edits to the mounted ConfigMap propagate via fsnotify within seconds; in-flight runs continue under their captured policy, but no new runs start.
Pause
# Edit platform/gitops/k3s/mills/configmap-policy.yaml in the gitops repo:
# enabled: false
# Then reconcile:
flux reconcile kustomization apps -n flux-system --with-source
# Wait for hot-reload signal in operator logs:
kubectl logs -n loom-mills deploy/loom-mills-operator --since=2m | grep "policy reloaded"
# Confirm the reconciler is parked:
loom mills status # expected: enabled=false, queue_depth=0 (no new picks)
The reconciler exits cleanly within one tick (≤60s). The HTTP and metrics listeners stay up so HUD reads continue working.
Resume
Reverse the edit (enabled: true or remove the field), reconcile, watch for the next policy reloaded log.
Emergency pause without GitOps
Only when GitOps is unavailable (e.g., Flux is down). Patches the mounted ConfigMap directly; must be reverted by a GitOps commit afterwards or Flux will fight you.
kubectl patch configmap -n loom-mills loom-mills-policy \
--type merge -p '{"data":{"policy.yaml":"<full yaml with enabled: false>"}}'
# Restore via git/Flux as soon as the issue is resolved.
Force-escalate a stuck pipeline run
A run can wedge if a stage worker hangs (e.g., ci_watch waiting on a CI job that's been canceled). Force-escalation transitions the run to escalated, records a failure record in events, and (if configured) opens a GitLab issue + agent-context handoff.
# Find the run
loom mills pipelines list
# Escalate
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs/<run_id>/escalate
# Confirm
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs?state=terminal&limit=50" | \
jq '.[] | select(.State == "escalated")'
The reconciler will not auto-retry escalated items; a human must edit the linked YAML or close the GitLab issue with a deliberate human-resolved label to unblock.
Outside-agent pickup and requeue
An outside agent may finish an escalated item's implementation before the item
is requeued. Commit and push that work to the exact source branch Mills derives
for the item, then requeue it. Before dispatching implement, Mills inspects
that origin branch against main. A nonempty diff is adopted as the successful
implement result and proceeds through the normal post-implement gates; no new
implement agent is spawned. The implement stage record contains
adopted_branch, adopted_head_sha, and adopted_at provenance.
Requeueing adopts whatever nonempty work currently exists on that branch, including a stale prior attempt. Review the remote branch before requeueing; the normal gates remain the safety boundary. If reimplementation is required, delete or rename the remote item branch before requeueing so the probe observes a missing branch and dispatches normally. There is currently no force-policy flag.
The probe fails closed when origin, the base ref, or the diff cannot be read: Mills records an implement error instead of risking an overwrite. A push racing between the probe and a new spawn is still possible; avoid pushing to the item branch after requeue begins.
Slice branch ref collisions and husk branches
For a single-slice run, the canonical source branch is nested as
<type>/<item-id>/<slice>. A legacy pre-slice branch named exactly
<type>/<item-id> prevents git from creating that nested ref. The typical
symptom is one or more dash-flattened husk branches followed by repeated
[branch_pushed.missing_ref] failures.
Before a fresh implement spawn, Mills now checks all origin heads for strict
path-prefix collisions. It deletes an exact same-item legacy ref only after
confirming that no open merge request uses it, then records a
branch_contract.ref_retired run event and proceeds on the canonical branch.
Foreign collisions and legacy refs with open MRs fail before spawn with
[branch_contract.ref_collision]. If manual recovery is needed, delete the
legacy ref and any husks, seed the exact contract branch, and requeue the item.
The imperative workflow lane
Most backlog items run the DAG pipeline (queued → plan_slice → … → merge), covered by the sections above. A backlog item can instead carry a policy.workflow_template selection (see "Workflow template registry (S7)" in docs/mills-operational-guardrails.md); admission then routes it through ClaimWorkflowStart into a workflow run — an imperative run driven by a go.starlark.net program from the compiled-in template registry, executed by the durable-journal runtime instead of the pipeline stage graph. These items get no pipeline run at all; everything about what happened lives in the workflow run and its step journal.
Key differences from a pipeline run worth knowing before you inspect one:
- The run's engine, template, template version, interpreter version, and clamped params are frozen at start and never re-resolved — editing the backlog item cannot re-route an in-flight run.
- Every terminal outcome —
done,error, orquarantined— escalates the backlog item and stops before merge. v1 templates never open an MR or merge; adonerun still needs a human to review and land the work product. Don't readdoneas "shipped." - The run's work product lives on a deterministic branch named
mills-wf/<run-id>, pushed by the run itself. There is no other record of the diff — the branch is the deliverable to review.
Inspecting a run
HUD — Workflows tab. Under Mills, the Workflows panel lists every imperative run (GET /api/mills/workflow/runs under the hood, polled every 15s): run ID, engine, template@version, state, step count, backlog ID, cost, and start time. Click a row to open the step-timeline drawer, which shows each journaled step with a derived badge — live (ran a real effect this execution), cache_hit (replayed from the journal, no live effect), pending (interrupted mid-step), failed, or quarantined (the whole run halted on a nondeterminism/call-hash mismatch). The drawer also shows the interpreter version pin and the frozen selection params (content hash + clamped params/enums) when present.
If the run belongs to a backlog item, open that item's detail drawer instead (Backlog panel → row) to see the "Imperative workflow" section: the item's template selection, every run claimed for it, and — critically — the escalation attention banner once a run has settled terminal. The banner surfaces the run's outcome, a copy-to-clipboard for the mills-wf/<run-id> branch, and an explicit "pre-merge template — nothing was merged" reminder, so a reviewer triaging an escalated item doesn't have to cross-reference the Workflows tab manually.
REST — /api/mills/workflow/runs.
# List recent runs (default limit 50, max 200), newest first:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs?limit=50" | \
jq '.runs[] | {id, backlog_id, template, template_version, state, step_count, cost_usd}'
# One run plus its full step journal (badges included):
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>" | jq
The list endpoint is a flat, cheap summary (no nested steps); the detail endpoint returns the run plus every journaled step, each with the same live / cache_hit / pending / failed / quarantined badge the HUD drawer shows. When the run carries a frozen S7 selection or canary identity, the detail response also includes a workflow_params field (opaque JSON — content hash + clamped params/enums) for inspecting exactly what identity the run is pinned to.
Responding to a workflow.terminal_settle escalation
When a claim-started imperative run reaches any terminal state, the same transaction that commits its lifecycle CAS also releases its budget reservation, escalates the backlog item, and records a workflow.terminal_settle event on the item (GET /api/mills/backlog/<id> events, or the HUD attention banner described above). The event payload carries run_id, final_state, the branch (mills-wf/<run-id>), the template, and a human-readable reason string.
As the operator handling the escalation:
- Identify the run and its outcome from the event payload or the HUD banner —
done,error, orquarantined. - Review the work product on
mills-wf/<run-id>regardless of outcome. Adonestate means the run's script completed without error, not that the diff is safe to ship — nothing has been merged, and there is no CI/MR gate to trust yet. - For
errororquarantined, also check the step journal (GET /api/mills/workflow/runs/<run_id>) for the failing/quarantined step before deciding whether to retry.quarantinedspecifically means a step's re-derivedcall_hashdidn't match the journal — treat it as a nondeterminism bug in the template or its inputs, not a transient failure to retry blindly. - Disposition the item the same way you would any escalated backlog item: merge the branch manually if the work product is good, edit the item and re-trigger if it needs rework, or close it out. There is no "resume to merge" action — the workflow lane stops pre-merge by design.
- If the run is still live and clearly wedged rather than settled (no forward step progress, past
workflows.max_run_minutes), use the pause/fail controls below instead of waiting — a wedged run is force-terminalized aserrorby the scheduler's wall-clock bound anyway, but you don't have to wait for it.
Pause / resume / fail a workflow run
Imperative workflow runs (the durable-journal runtime behind policy.workflows.enabled) have their own lifecycle mutations, separate from pipeline escalation. Use these to stop a wedged run live — e.g. a zombie whose spawn died terminally (the 2026-07-09 wf-canary loop) — instead of waiting for a code deploy.
# Find the run
curl -sf $LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs | jq '.runs[] | {id, state}'
# Pause (running → paused; takes effect between steps, on the next scheduler tick)
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
$LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>/pause
# Resume (paused → running)
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
$LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>/resume
# Fail (running|paused → error; TERMINAL — no un-fail. Optional audit reason.)
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
-d '{"reason":"zombie spawn, pod gone"}' \
$LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>/fail
Expected output: the run's lifecycle view, e.g. {"id":"wf-canary-…","state":"paused","paused_at":"…"}. An invalid transition returns 409 with the current state. Pause/fail are eventually-consistent like the policy kill switch: an in-flight step finishes its current attempt first; the scheduler skips the run from the next tick.
Replay a council run
Useful when you want to verify a fix to the brief assembler, swap an ensemble member, or compare A/B outputs.
Dry-run (safe)
Runs the full council pipeline against a scratch DB; nothing is committed and no GitLab mutations happen. The sidecar + plan paths are returned for inspection.
loom mills council dryrun
# Output: sidecar JSON path, .loom/<NN>-… draft paths, total cost
Real replay with a different ensemble
# 1. Edit policy.council.ensemble in platform/gitops/k3s/mills/configmap-policy.yaml
# e.g., swap editor.model from claude-opus to codex-gpt5.
# 2. flux reconcile.
# 3. Wait for "policy reloaded" log line.
# 4. Trigger:
loom mills council run
# 5. Inspect the new run:
loom mills eval list
# HUD: Mills → Eval panel; sort by created_at desc.
When the run lands, compare cost + Loop A score + downstream Loop B attribution against the previous run with the same brief content (same roadmap_intents snapshot).
V2.1 will add first-class A/B replay UI; until then, the manual flow above is the supported path.
Council blocked on missing roadmap intents
Symptom: every council run terminalizes immediately with outcome=error and a
note containing require_roadmap_intents; no reviewer or editor spend occurs and
cost_usd_approx is 0. The fail-closed intent preflight fired because the
canonical roadmap_intents store is empty (see "Fail-closed intent preflight" in
docs/MILLS.md).
# Confirm the store is empty and the brief was marked:
loom mills council runs --limit 5 # look for the blocked note
kubectl -n loom-mills exec deploy/loom-mills-operator -- \
sqlite3 /var/lib/loom-mills/mills.db 'SELECT COUNT(*) FROM roadmap_intents;'
Fix the cause, not the symptom — the extractor fills the store from
$LOOM_MILLS_REPO_ROOT/ROADMAP.md on every run, so an empty store means that
file is missing, unreadable, or has no open - [ ] bullets under a plannable
section:
kubectl -n loom-mills exec deploy/loom-mills-operator -- \
sh -c 'ls -l /workspace/loom-core/ROADMAP.md && grep -c "^- \[ \]" /workspace/loom-core/ROADMAP.md'
Break-glass only, when planning must proceed against an empty store: set
council.require_roadmap_intents: false in
platform/gitops/k3s/mills/configmap-policy.yaml and bump
loom.flexinfer.ai/policy-checksum in platform/gitops/k3s/mills/deployment.yaml
(fsnotify misses the ..data symlink swap). Revert once the store refills.
Note: the block happens after admission, so each blocked tick consumes a
council_runs row. The council tier sets no max_runs_per_day today, so this is
harmless; if one is ever added, a wedged store could exhaust the daily allowance
and block the recovery run too.
Audit a merged change
Find which council brief produced the slice that produced the merge:
# Get the pipeline run by MR iid:
curl -sf \
"$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs?mr_iid=<iid>"
# Eval Loop B attribution and Loop A score:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/eval/scores?limit=200" | \
jq '.[] | select(.SubjectID == "<run_id>" or .SubjectID == "<council_run_id>")'
For deeper inspection, the events table has the full per-stage trace (slog rows are also written via JSON to stderr; aggregated in Loki).
Serial merge queue
GitLab CE has no merge trains. Without the queue, parallel Mills branches go
stale while 17–28 minute pipelines run; main moves underneath them and their
merge PUTs die with has_conflicts (killed !1509 and !1511 on 2026-08-08),
and merges land in bursts that stack uncancelable main pipelines. The serial
queue (pkg/mills/mergequeue) guarantees every MR is CI-tested on the exact
main it lands on, one candidate at a time per (project, target_branch) lane.
Semantics
When merge_queue.enabled is true, the pipeline's merge stage validates the
full ci_watch authorization once (including the #374 head-movement fence),
enqueues the candidate into the durable merge_queue table, and waits.
The processor drives only the head of each lane:
- Up to date (MR
diff_refs.base_sha== target tip): merge immediately — the run's own green CI is the proof. - Behind: request a rebase (a durable
mr_head_transitionsledger row; cursors snapshotted before the PUT, verdict settled via the #374 observer), await a fresh branch pipeline on the rebased head, then merge with the rebased SHA as the precondition. - Terminal settle:
mergedwakes the waiting stage with the merged SHA;evictedfails the stage with a typed verdict and the run falls through to the runner. The queue never retries internally — but the runner now acts per reason:head_movedcarries the observed successor SHA and rides the #374 head-movement rewind (re-gate + re-prove, no escalation), and an evicted run's next stage attempt re-enters the queue through the full enqueue-time authorization (one active candidate per run since migration 031; the settled rows remain as audit history).
Eviction reasons and their error classes: rebase_conflict,
rebase_ambiguous, ci_red → code; ci_timeout → infra; mr_closed →
config; head_moved → rewind (see above); merge_failed → classified from
the GitLab error text (405/422/cannot-be-merged → config). An enqueue into a
full lane (depth ≥ merge_queue.max_depth, default 10) escalates immediately
with reason queue_full — backpressure surfaces instead of silently deepening
the queue.
Durability: queue state and the rebase ledger live in the canonical SQLite store, so an operator restart resumes the head candidate exactly where the previous process died — re-observing, never re-mutating. The resumed merge stage re-finds its entry FIRST and skips authorization re-validation (the queue's own rebase legitimately advances the #374 fence).
Rollout / rollback
The queue is a policy flip — no restart, no schema action:
merge_queue:
enabled: true # default false
max_depth: 10 # optional
Rolling back (enabled: false or removing the section) halts the processor on
its next tick without losing state; runs waiting in the merge stage observe
the disable and fall back to the direct merge (a queue-rebased candidate then
fails closed on the stale fence and escalates for a clean re-gate — expected).
Observability
GET /api/mills/pipeline/runs/{id}gains amerge_queueblock (state, position, eviction reason) whenever the run has an entry.- Prometheus:
mills_mergequeue_depth,mills_mergequeue_wait_seconds,mills_mergequeue_evictions_total{reason},mills_mergequeue_merged_total. - Events:
mergequeue.merged/mergequeue.evictedrows against the run. - Stage wait bound:
LOOM_MILLS_MERGE_QUEUE_WAIT_MINUTES(default 180) — the merge stage escalates if the queue produces no verdict in that window (dead processor / saturated queue).
Recover from a corrupted DB
The canonical SQLite DB lives on a Longhorn RWO PVC. WAL replay handles most operator restarts; nothing should be needed for a clean kill. The procedures below are for the rare cases where the DB is unrecoverable.
Symptom: database is locked or schema check fails on boot
-
Capture diagnostics:
kubectl logs -n loom-mills deploy/loom-mills-operator --previous --tail=500 > /tmp/mills-prev-logs.txt kubectl exec -n loom-mills deploy/loom-mills-operator -- sqlite3 /var/lib/loom-mills/state.db "PRAGMA integrity_check;" -
If
integrity_checkreturns anything other thanok, restore from the most recent nightly backup in MinIO:# List backups (nightly CronJob writes one per day): mc ls minio/loom-mills-backups/ # Pick a backup to restore: BACKUP=2026-04-30T06-00-00.db # Scale operator to 0 to release the PVC: kubectl scale deploy -n loom-mills loom-mills-operator --replicas=0 # Restore via a one-shot Job (the manifest lives at # platform/gitops/k3s/mills/jobs/restore-from-backup.yaml; render with the # chosen filename): kubectl create -n loom-mills -f - <<EOF apiVersion: batch/v1 kind: Job metadata: generateName: mills-restore- spec: template: spec: restartPolicy: Never containers: - name: restore image: registry.harbor.lan/library/loom-mills-operator:stable command: ["/bin/sh", "-euxc"] args: - | mc cp minio/loom-mills-backups/$BACKUP /var/lib/loom-mills/state.db sqlite3 /var/lib/loom-mills/state.db 'PRAGMA integrity_check;' volumeMounts: - { name: state, mountPath: /var/lib/loom-mills } volumes: - name: state persistentVolumeClaim: claimName: mills-state EOF # Watch the restore Job to completion: kubectl logs -n loom-mills job/<job-name> -f # Bring the operator back up: kubectl scale deploy -n loom-mills loom-mills-operator --replicas=1 kubectl rollout status deploy -n loom-mills loom-mills-operator -
After restore, run a council dryrun to confirm the brief assembler still works against the restored canonical state. Then resume normal operation.
Loss-window note
The nightly CronJob captures one snapshot per day. Items committed since the last snapshot are lost in a restore. Re-run the council to reproduce — roadmap_intents is idempotent and .loom/backlog/*.yaml is regenerated from the canonical store, so most state self-heals.
Inspect what the operator is doing right now
# Active runs (council + pipeline):
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/status" | jq
# What gates fired in the last hour:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs?state=terminal&limit=50" | jq
# What the budget enforcer thinks:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/kpis?window=1d" | \
jq '{SnapshotAt, Metrics: (.Metrics | {pipeline_cost_usd, gate_pass_rate, pipeline_merged_real, regression_rate})}'
Production rollout staging
The default-on flip (slice 6.6) was landed for the in-binary policy default. Each cluster overlay still owns whether the mills is enabled in its ConfigMap. Stage rollouts in this order; never flip more than one environment per week.
Stage 1 — Local
A developer can run the operator against a local SQLite file and a local policy YAML for the smallest possible smoke test:
make build/loom-mills-operator
mkdir -p /tmp/loom-mills
./bin/loom-mills-operator \
--db-path /tmp/loom-mills/state.db \
--policy-path testdata/policy.yaml \
--listen :8090 \
--metrics-addr :9090
curl -sf localhost:9090/healthz # 200
curl -sf -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
localhost:8090/api/mills/status | jq
Local runs use the FakeReviewer/FakeEditor (cmd/loom-mills-operator/main.go: buildCouncilRunner) when no FlexInfer or HUD spawn is configured; this is sufficient for handler smoke tests but does not exercise real model calls.
Stage 2 — Dev cluster (kc-k3s)
# In the platform/gitops repo on a dev branch:
# Edit platform/gitops/k3s/mills/configmap-policy.yaml: enabled: true
# Commit, push, open MR, merge.
flux reconcile kustomization apps -n flux-system --with-source
kubectl rollout status deploy -n loom-mills loom-mills-operator
Soak for one week. Verify each acceptance criterion against the deployed cluster:
| Criterion | Check |
|---|---|
| Council dryrun produces sidecar + 3 markdown docs in <8 min for <$5 with eval ≥0.7 | loom mills council dryrun and inspect cost/score in HUD |
| End-to-end backlog → merged MR via fixture run | `loom mills backlog list --json |
| Eval Loops A/B/C populated | loom mills eval list returns recent rows for all three subject kinds |
| Idle-throttle drops reconciler cadence | loom mills status shows next_tick_in_s ≥ 240 when queue is empty |
HUD Mills view renders four panels | Visit <HUD>/mills |
| Backups produced nightly | mc ls minio/loom-mills-backups/ shows last 7 entries |
Stage 3 — Production cluster
Only after dev has been green for 7 consecutive days. Same edit + reconcile flow, but watch the regression KPI:
# Keep the :9090 port-forward from Quick status running, record the starting
# counter value, then watch for any increase during rollout.
curl -sf http://127.0.0.1:9090/metrics | grep '^mills_regression_count_total'
watch -n 30 'curl -sf http://127.0.0.1:9090/metrics | grep "^mills_regression_count_total"'
If mills_regression_count_total increases from the recorded baseline in the first 24h, pause and investigate before any further flips. This counter is Alertmanager-correlated; the REST Metrics.regression_rate field is a label-driven backlog KPI and is not an equivalent rollout alarm. The kill switch (enabled: false) is the safest and lowest-blast-radius rollback.
GitLab issue importer (Slice 1a of .loom/43)
The operator can pull open issues labelled mills-eligible from its
configured GitLab project (GITLAB_PROJECT env) and create one
BacklogItem per fresh issue. Disabled by default; opt in via policy.
intake:
gitlab:
enabled: true
eligible_label: "mills-eligible" # required label on the issue
poll_interval_seconds: 300 # 5 min
default_priority: "P2" # fallback if no priority:Px label
Label semantics on the issue side:
mills-eligible(required) — the importer's selector. Add this on any issue you want Mills to pick up.priority:P0/priority:P1/priority:P2/priority:P3(optional) — sets the BacklogItem priority. Highest priority wins when multiple are present. Default isdefault_priority.
Dedup: BacklogItem ID is gl-<project_id>-<issue_iid>, so the
importer is safe to re-run. Once an item exists, the importer does NOT
update it — reconciler/council state transitions (queued → running →
merged/escalated) are preserved across ticks.
Verify:
# After enabling, watch for the import line in operator logs:
kubectl -n loom-mills logs deploy/loom-mills-operator -f | \
grep "gitlab importer"
# Check the backlog API for the new item:
loom mills backlog list | grep gl-
Don't enable until pipelines are reaching merge stage. Per the
2026-05-24 kill-test (.loom/local/handoffs/mills-autonomy-killtest-...)
100% of pipeline_runs were escalating at tests or plan_slice.
Adding more intake before Slices 2c + 2e land would just deepen the
escalation pile.
Source-branch push before MR
The mr stage publishes the spawn agent's commits to origin before
calling GitLab CreateMR. Without this, GitLab accepts the MR row
but it points at a branch with no head_sha, and ci_watch hangs
forever waiting for a pipeline that can't exist.
There are TWO push paths (belt-and-suspenders):
-
Spawn-side push (primary): the
implementstage prompt explicitly instructs the agent togit push -u origin HEADas its final step. The spawn pod has git credentials configured. This is the path that actually fires today because the single- repo SpawnWorker does NOT use Mills' WorktreeAllocator — so the operator doesn't know the worktree path and can't push from outside the pod. -
Operator-side push (fallback, currently inert): when
jc.Run.WorktreePathis set,GitLabWorker.runMRruns theclients.GitBranchPusherbeforeCreateMR. No live caller setsWorktreePathtoday, so this path is inert. A future slice should propagate the spawn-side WorkingDir back through SpawnResponse so the fallback kicks in for real.
- Implementation:
pkg/mills/clients/branch_pusher.go(clients.GitBranchPusher) shellsgit push --force-with-lease -u origin HEAD:<branch>from the run'sWorktreePath. - Interface:
pipeline.BranchPusher(4-line contract). The operator wires the production pusher ontoGitLabWorker.BranchPusherat startup; tests inject fakes. - Idempotency:
--force-with-leaselets retries safely overwrite a stale prior push without clobbering unrelated upstream work. A no-op push (HEAD already at origin) exits 0.
Failure mode (push errors before MR is created): the mr stage
returns an error and the runner retries per the Slice 2c
classification — a transport / quota error gets free retries; a
real git push rejection counts against MaxAttempts.
v2 rollout staging (preview)
When .loom/94-implementation-plan-mills-v2-hierarchical-swarm-2026-05-02.md Phase 8 lands, each v2 feature flips behind its own policy flag with a 1-week soak between flips. Order:
policy.squads.enabled: truepolicy.audit.enabled: true(advisory-only by default)policy.council.debate.enabled.incident: truepolicy.cross_repo.enabled: true— gates per-itemTargetProjectrouting to non-home repos (already flipped; see "Cross-repo demand sourcing" below)policy.adaptive_policy.enabled: true(manual-apply only)
Detail per slice in .loom/94-…2026-05-02.md Phase 8. The rollback playbook for v2 lives in MILLS_V2_ROLLBACK.md — covers feature flag disable, policy proposal revert, and DB restore from MinIO.
Useful loops
# Watch council runs as they complete:
watch -n 30 'loom mills eval list | head -10'
# Watch escalation rate:
watch -n 60 'kubectl logs -n loom-mills deploy/loom-mills-operator --since=10m | jq -r "select(.msg==\"escalation\") | [.time, .item_id] | @tsv"'
# Diff the live policy vs. git:
kubectl get configmap -n loom-mills loom-mills-policy -o jsonpath='{.data.policy\.yaml}' | \
diff - platform/gitops/k3s/mills/configmap-policy.yaml
Spawn pool capacity & multi-repo
Mills pipeline stages (plan_slice / research / implement) run as spawn pods.
The spawn orchestrator is embedded in mobile-hud (internal/hud/spawn.go),
not the operator — the operator only POSTs a spawn request to the HUD. So the
spawn knobs are env vars on the mobile-hud Deployment
(k8s/base/servers/mobile-hud/deployment.yaml), which deploys via the Flux
loom-hub-servers Kustomization. The gitops override patch
(platform/gitops/clusters/k3s/flux-system/kustomization-loom-hub-servers.yaml)
does not touch these, so the base-manifest values are authoritative.
| Env var | Meaning | Current |
|---|---|---|
SPAWN_MAX_CONCURRENT | Global cap on active spawn pods (mobile-hud is a single replica, so this is a fleet-wide ceiling). | 10 |
SPAWN_MAX_CONCURRENT_BUILDS | Concurrent spawn image builds (buildah); throttled independently so a higher run cap can't cause a build thundering herd. | 1 |
SPAWN_DEFAULT_CPU / SPAWN_DEFAULT_MEMORY_MB | Per-spawn resource limits. | 0.5 / 4096 |
SPAWN_SYNC_MODE | Workspace sync mode. git-clone = the pod clones the repo fresh (source of truth); the workspace mount is used only to fingerprint the repo for the Dockerfile. | git-clone |
SPAWN_GIT_BASE_URL | Git base the pod clones from; rooted at the services/ group. | http://192.168.50.218/services |
SPAWN_PROJECTS | Comma-separated repos shown in the HUD/mobile spawn picker. Cosmetic — not an enforcement gate. | see manifest |
Project labels. Spawn requests keep the canonical project path (for example,
services/loom-core) in durable state, while the pod's loom.dev/project label
is normalized to a Kubernetes-safe, 63-character value. The label is for pod
discovery and degraded recovery only; clone and routing logic must continue to
use the canonical request value.
Raising concurrency. Cluster allocatable memory is ~1.25 TiB (16 nodes), so
even SPAWN_MAX_CONCURRENT at 10 (= 5 CPU + 40 GiB) is <4% of capacity. Capacity
is not the constraint — demand is. Bump the value in the base manifest and let
Flux roll mobile-hud; no gitops-patch change is needed.
Expanding to more repos. A stage targets a repo via the pipeline worker's
Project (defaults to loom-core). For a repo to spawn cleanly:
- Merge token reach — the Mills GitLab worker uses the
services-group token (loom-mills-gitlab-group/api-token, Maintainer on groupservices), so onlyservices/*repos can be opened+merged. Cross-group repos (libs/, platform/) need a separate token grant. - Git-clone group — the pod clones
SPAWN_GIT_BASE_URL/<name>.git, i.e. theservices/group. Non-services repos would need a group-aware base URL. - Workspace fingerprint (best-effort) —
resolveProjectPathlooks for the repo on theloom-hub-workspacePVC (Longhorn RWO, mounted read-only) to fingerprint it for an accurate runtime image. In git-clone mode a repo that is not staged on the PVC no longer hard-fails: the orchestrator falls back to a lexicalservices/<name>path and a generic runtime image, and the init-container clone provides the real source. Staging the repo on the PVC still yields a better-fingerprinted image (correct language toolchain), so for build-heavy repos prefer to stage it. Populating/resizing that RWO PVC is a coordinated gitops op (single-writer; co-locate the writer with mobile-hud).
Multi-repo demand activation (S6)
The steps above make a stage able to run against a services/* repo. S6
makes demand itself target foreign repos: the plan-slice emitter sources an
allowlist of non-home projects and stamps each emitted item's TargetProject,
so the S2/S3/S4 routing carries the whole run cross-repo. Activation is
two-key — both must be true or foreign demand stays inert:
- Execution key —
policy.cross_repo.enabled: true. Already flipped 2026-07-05 for the keystone (flexdeck!244). This alone changes nothing about demand; it only lets a foreign-targeted item execute instead of being skipped fail-closed by the reconciler. - Demand key —
policy.cross_repo.demand_projects: [services/<repo>, …]. The emitter consults this list ONLY whenenabledis also true (Policy.CrossRepoDemandProjects), so a stray allowlist can never source foreign demand while execution is off.
Procedure to onboard a repo (e.g. services/flexdeck):
- Branch pipeline — the target repo must run a pipeline on the MR's source
branch (flexdeck did this via flexdeck!242) so
ci_watchhas a green gate to merge on. Without it the autonomous merge can't confirm success. - Set the allowlist — add the repo to
cross_repo.demand_projectsin the gitops policy ConfigMap (platform/gitops/k3s/mills/…). - Roll the operator — the emitter snapshots policy at startup and fsnotify
misses ConfigMap
..dataswaps, so bump the deployment pod-checksum (orrollout-restart) to pick up the new allowlist. - Author demand — the emitter emits from Plans scoped to the target
project that have a ready (
plan_slice_emitter.ready_phase, defaultpending) slice with declared files, in the emitter's namespace (plan_slice_emitter.namespace). A foreign repo with no such Plan produces nothing; author aservices/<repo>Plan with a ready slice to drive a run. - Verify — operator log
plan-slice emitter created backlog item … demand, the item's pipeline runs against the target repo, an MR opens+merges in the target repo, andmills_autonomous_merges_realincrements.
Plan lifecycle scan contract. The emitter queries only planned and
in_progress Plans before looking for pending slices. Take-up queries only
planned, in_progress, in_review, and merging Plans. Terminal plan bodies
remain durable in the Plan Store but are not transferred or decoded on every
poll. If /api/mills/safety/quiescence shows plan_slice_emitter or takeup
active across an entire poll interval, inspect agent-context latency and the
operator logs; do not widen either query to an unfiltered namespace scan.
Rollback — remove the repo from demand_projects (stops new foreign
demand) or set cross_repo.enabled: false (also fail-closes any already-queued
foreign item on the next reconciler tick), then roll the operator.
Diagnosing a stuck merged-MR escalation
The ghost-spark sweep settles a merged MR by its durable (project, MR IID)
stamp. An external rebase may change the head SHA or actor after ci_watch;
those values do not invalidate a merged MR's identity. A restart runs an
immediate sweep, so a rollout of a settlement fix may close several previously
stuck items at once; the transition is CAS-safe and repeat sweeps are no-ops.
Start with the run's durable events instead of reconstructing a GitLab timeline.
For every rejected candidate the reconciler appends
reconciler.ghost_spark_skipped with a stable precondition and non-empty
detail:
cooldown: the durablerecheck_afterhas not elapsed.project_provenance: the MR project stamp is missing, malformed, or contradictory; do not guess a project from the current backlog target. A legacy empty escalation binding is interpreted as the configured home project only when the backlog item is itself a home-project item; the same binding on a foreign item remains fail-closed.client_selection: no GitLab client is configured for the stamped project.candidate_exclusion: the item has no MR-bearing run, no deterministic rescue branch, or the bounded pass lookup budget was exhausted.
Correlate the event by its pipeline_run subject (or backlog_item when no run
exists), then inspect run, mr_iid, precondition, and detail. A cooldown
or lookup-budget event should clear on a later pass. Provenance and client
selection are configuration/data faults and also produce
reconciler.ghost_spark_failed; repair the stamp or project-scoped client and
let the next sweep retry. A successful recovery emits
reconciler.ghost_spark_closed with project, mr_iid, and outcome.
escalation_sweep_state is not created or advanced for a
project_provenance skip because no GitLab lookup occurred; its
matching reconciler.ghost_spark_failed event is written once for the run and
MR IID. If the item remains stuck, repair the durable binding or home-project
configuration and expect the next sweep to retry immediately rather than wait
for recheck_after.
See also
Sources
cmd/loom-mills-operator/main.go(lifecycle, env vars, fakes vs. wired clients)cmd/loom-mills-operator/auth.go(admin token loading)pkg/mills/policy.go(enabled,IsEnabled, kill-switch semantics)pkg/mills/policy_manager.go(fsnotify hot-reload)pkg/mills/reconciler.go(one-tick exit on disable)pkg/mills/scheduler.go(cron + idle throttle)pkg/mills/eval/{judge,outcome_attributor,council_roi,cross_run}.go(Loops A/B/C)pkg/mills/store/migrate.go(migrations on boot; safe replay)platform/gitops/k3s/mills/(manifests; not in this repo — lives in the gitops repo)docs/MILLS.md(architecture + policy reference).loom/91-implementation-plan-agent-swarm-council-pipeline-2026-04-25.md§"Default-off rollout"
Factory Health Plane — detector replay (S0 kill-test)
scripts/mills/health-replay.sh reconstructs main-branch pipeline health over a
window using the Health Plane detector semantics (only terminal success/failed
pipelines flip health; skipped/canceled/manual never do) and prints every red
window with would-page markers. Read-only; requires GITLAB_TOKEN.
GITLAB_TOKEN=... scripts/mills/health-replay.sh [days] [page_threshold_min]
Kill-test run 2026-08-29 (30 days, threshold 90m), gating
plan-mills-factory-health-plane-2026-08:
- 411 main pipelines: 246 success, 91 failed, 74 canceled.
- 40 red windows; 16 would-page — every one genuinely red main (0 false fires; canceled/skipped never flipped state). Largest epochs: ~46h (08-23→25), ~18h (08-19→20), ~12h ×2 (08-12, 08-13), plus the 991m advisory window (GO-2026-6303) closed by !1762 at 08-29 13:39Z.
- Verdict: the detector assumption HOLDS. Expected page volume ≈16/month at the 90m threshold — tune threshold or quiet hours to taste.
- Productionizing notes for the S1 poller: use per-pipeline
finished_at(list-APIupdated_atmutates on retry and can produce small negative window artifacts), dedupe to one page per window, and initialize every metric label combination at start (counter-born-nonzero guard).
Rolling out a new model
Model ids are free-form in policy (validModelToken checks shape only); the
harness or API is the authority that rejects an unknown id at run time. What
the operator needs from you when a new tier ships (Claude Fable 5.1,
GPT-6 "astra", …) is, in order:
-
Price it, or it is not merely $0 — it is a whole-reservation charge. Council/spin cost lookups are exact-match. An unpriced model loses the per-attempt ceiling (
unpricedAttemptCeilingUSD) and a single timed-out lens charges the entire run reservation. Add a row tobudgets.model_pricesin the gitops policy the day the id is known:budgets: model_prices: gpt-6-astra: { provider: openai, input_per_million: 8, cached_input_per_million: 0.8, output_per_million: 40 } claude-fable-5-1: { provider: anthropic, input_per_million: 10, cached_input_per_million: 1, cache_write_per_million: 12.5, output_per_million: 50 }Rows hot-reload with the policy and win over the compiled tables in
pkg/mills/clients/{council_anthropic,council_openai,council}.go. Promote a confirmed price into the compiled table on the next image. Codex spawn-stage estimates use a separate table (internal/hud/bridge/spawn_pricing.go); add the row there too or the spawn cost is estimated at the gpt-5-codex fallback rate. -
Wire it where the policy already chooses models. All of these are gitops edits in
k3s/mills/configmap-policy.yaml:- Spinning Room: add a frame
{ name: <frame>, model: <id>, backend: anthropic|openai|flexinfer }(hot-reloads, no checksum bump). A frame whose model the API rejects fails that spin loudly at the editor phase; prefer adding the frame alongside the current one rather than renaming it. - Council editor:
council.ensemble.editor: { model: <id>, backend: anthropic|openai }; keepeditor_fallback_modela local flexinfer id. The chain is editor →editor_fallback(the OTHER vendor's frontier model; an anthropic editor defaults to{ model: gpt-5.5, backend: openai },backend: noneopts out) → local flexinfer, so retargeting the editor to a new vendor usually means re-pinningeditor_fallbackto the old one. Needs theloom.flexinfer.ai/policy-checksumbump on the deployment. - Pipeline stages:
pipeline.stage_models.<plan_slice|implement|pr_self_review>: <id>and/or anagent_routingruleroute: { agent: codex|claude-code, model: <id> }. Re-targeting the agent in a rule drops the stage_models pin on purpose (codex exec --model claude-…fails every stage); set the model on the rule itself. - Reviewer lenses take
litellmorflexinferonly. A frontier reviewer goes through the LiteLLM gateway asoa/<id>— which also needs the alias on the gateway side (platform/gitops litellm config).
- Spinning Room: add a frame
-
Check the harness version gate. OpenAI gates new families behind a minimum Codex CLI (
gpt-5.6needed codex ≥ 0.143, 400 "requires a newer version of Codex"). BumpSPAWN_CODEX_VERSIONon the mobile-hud deployment before routing a new OpenAI tier through codex. -
Verify on
/api/mills/wiring(HUD → Mills → Overview): every surface's{backend, model}as the operator actually resolved it, pluspolicy.policy_checksum. Then watch the first run's stage records —Model/Backendland at pick end,Artifacts.agent_routing.modelat routing — and the council run'sevidence.provenance.stage_models. -
Both spawn paths honour the pin. The legacy CLI path passes
--model/codex exec --model; the SDK driver (loom-spawn-driver) accepts--modelfrom the same request field as of this runbook revision. Older driver bundles ignore the flag (forward-compatible parser) and run the harness default — rebuild withmake sync-spawn-driverafternpm run buildintools/spawn-driver.
Onboarding a repo or creating a project from the HUD
Work → Projects is the intake surface. A repo becomes a Mills project in two halves, and the HUD shows which half is missing for every repo it lists:
| Half | Endpoint | Effect | Takes effect |
|---|---|---|---|
| Runtime registration | POST /api/mills/projects/onboard (X-Admin-Token) | row in the bootstrapped registry; the repo may source demand and resolves the global protected-paths list under the two-key gate (cross_repo.enabled + allow_bootstrapped) | immediately |
| Git policy | POST /api/mills/projects/policy-mr (operator admin token) | gitops MR adding the repo to cross_repo.demand_projects and, when asked, intake.gitlab.projects, protected_paths_per_repo, per_repo_overrides, plus the deployment policy-checksum bump | after merge + operator restart |
GET /api/mills/projects returns every repo the operator knows (home,
demand list, issue intake, runtime registry) with ready, blockers
(cross_repo_disabled, not_in_demand, allow_bootstrapped_off,
protected_paths_unknown) and pending (git_policy_missing) computed by
the same policy accessors the reconciler and gates use — so the chip in the
HUD is the operator's verdict, not a guess. A 404 from this route means the
deployed operator image predates intake: the HUD then labels every repo
unknown, keeps the forge half working (create/import), and disables the
register-only action rather than promising a registration that would fail.
Forge half (HUD daemon, internal/hud/domain/forge): identity comes from
GITLAB_TOKEN/GITHUB_TOKEN env or the daemon host's glab/gh logins
(GET /api/forge/auth shows which, never the token). New project creates
the GitLab project under a chosen group (libs/* default public, else
private), seeds README, a green .gitlab-ci.yml, AGENTS.md, ROADMAP.md,
CHANGELOG.md, .loom/README.md (+ go.mod/cmd/Makefile for the go template),
creates the GitHub repo and installs the push mirror — dry_run: true
returns the exact step plan first. Onboard repo from GitHub looks for a
GitLab twin by repo name (registry, then GitLab search) and defaults to
onboarding that twin; importing again would mint a duplicate. Only with no
twin (or on explicit choice) is the repo imported into GitLab with GitHub
kept as the mirror.
The policy MR is anchored text insertion (comments preserved), idempotent,
and fails closed on a missing anchor; the branch is
mills/onboard-<slug>-<stamp>. Merge it like any policy change — the
checksum bump restarts the operator, which re-snapshots demand_projects.