A Green Run That Checks Nothing: Seven Silence Modes, Seventy Repos, One Day

On the morning of 2026-08-15 my portfolio conformance checker told me that 35 roadmap documents across the workspace were broken. I had standardized those documents in July. Six weeks later, a third of the fleet had apparently rotted.

They had not. The checker reads the working tree and dates each document from git log -1 on local HEAD. My checkouts were between 4 and 170 commits behind origin/main, and eight were parked on abandoned feature branches. The tool was grading July's work against January's files. Running git pull cleared 23 of the 35 with zero content changes. Nine were real, and they got fixed.

The interesting part is the fix I did not make. Regenerating 35 roadmaps would have overwritten good upstream documents with freshly dated worse ones and turned the check green. services/smarthome failed only on Last updated versus Last Updated.

That is the subject of this post. The check and the thing being checked have to be the same object, and a lot of tooling quietly fails that test while reporting green. By the end of the day my agent fleet had found seven distinct mechanisms for it across roughly 70 repos, and I had wired a permanent loop so the next sweep is not a heroic one-day event.

TL;DR

• Seven silence modes: a failure-class taxonomy of the ways a pipeline, linter, or scanner reports success without doing the work it claims.
• The sharpest: GitLab Runner concatenates every before_script and script entry into one shell, so a bare exit 0 in a YAML anchor ends the whole job. services/flexdeck, a live service, had run no lint, test, or perf job in two weeks while reporting green.
• Two rules that survived: diagnose from origin and from history, never from the local working tree or the narrative, and auto-merge must be armed, not fired.
• Roughly 45 merge requests merged across 20-plus repos, roadmap violations 35 to 0, flexdeck taken from zero security scanning behind a fake pipeline to seven scanners with hard gates.
• Then the factory that ran the work congested its own event store, diagnosed it from its own logs, and merged the fix under a self-expiring waiver instead of a quietly moved threshold.

Presence is not verification

Every silence mode below is a variation on one structural mistake: a check that derives its expectations from the artifact under test cannot fail.

Observable and self-verifying are different properties, and the gap between them is where a fleet rots. Observable means I can go and ask a repo a question. Self-verifying means the repo answers on its own schedule and fails loudly when it cannot. Every fix in this post moves one gate from the first category to the second.

The standard a gate has to meet now is that it emits proof of its own execution. Not a green badge: an artifact. A non-zero test count in the job log. A scanner report with a byte count in it. A duration that could not plausibly be image pull. The inference side of the platform learned this in July, where the rule came out as read the engine's boot log, not your CR: a config value that maps to a missing flag is indistinguishable from a working one, and a CI job that never ran is indistinguishable from a job that passed.

My roadmap checker is bin/portfolio-inventory, and before it false-reported anything it had failed the same test in the opposite direction, worse. It derives each repo's tier from a table in the workspace AGENTS.md. The table did not exist, so the check matched nothing, inspected zero repositories, printed its header row, and exited clean. The documented conformance SLO validated nothing while appearing to pass. It now carries a third exit code:

#   2  harness misconfiguration (--roadmaps inspected zero repos - TIERS table
#      missing/empty in AGENTS.md; a green run that checks nothing is a lie)

The purest instance of the class lives inside Mills, my cluster-resident software factory (design notes here). Its scope gate checks whether an agent's diff stayed inside the files its plan declared, which sounds airtight until you notice the envelope is built from the declared paths: if the plan invents paths and the implementing agent creates them to satisfy the plan, the diff matches perfectly. A sweep of first-parent merges found 17 council-planned merges that landed 27 dead Go files exactly this way, each green through every other gate. A new fabricated_slice gate now fails any implement diff where every file is new and nothing pre-existing was touched.

The seven silence modes

What follows is a taxonomy, not an incident list. Seven incidents would be seven war stories and no transferable knowledge. Seven failure classes are something you can grep for on Monday. Each is a distinct mechanism with its own detection signature and its own fix, and the reason to spend a day naming them is that the eighth instance arrives pre-diagnosed, in a repo I have not looked at yet.

1. The vacuous anchor

GitLab Runner concatenates every before_script and script entry into a single step_script shell, so a bare exit 0 anywhere in that chain ends the whole job. Three repos shared a .apk_add_git YAML anchor whose happy path was exactly that: check whether git is present, and if so, exit 0. Every image in flexdeck's pipeline already ships git, so the first before_script entry of every lint, test, and perf job took that branch immediately and the body was never reached. No clone, no golangci-lint, no go test, no eslint. The job reported success.

The detection signature is a timestamp pair. Job 233645, lint:golangci: step_script opened and closed at 1786808114, the same second. The identical signature appears on pipeline 21890 from 2026-08-02, so this was not a new regression. What made it survive is that the observed 40 to 95 second durations looked plausible; they were entirely image pull.

services/mentatlab had the same anchor with two exit 0 statements, and it was the second, inside the apk add retry loop, that fired: removing only the obvious one would have left the job vacuous. services/loom-flightdeck carried the identical anchor and was fine, because neither of its base images ships git, so execution fell through. It was one base-image bump away from six silent gates. Grepping for exit 0 finds the hazard; only a job trace tells you whether it is firing.

2. The soft gate under a green pipeline

An allow_failure job reports failed while the pipeline stays green. Pipeline color is therefore never proof that a soft gate passed. Cite per-job status or cite nothing.

There is a second-order symptom I now treat as diagnostic on its own: a soft gate that never runs accumulates fictional justifications. mentatlab's dependency-scan-go carried a comment naming two specific CVEs as the reason it could not be made blocking. Neither appears anywhere in the real scan results. The comment was written from imagination, because there had never been any output to write it from.

3. The tool that never ran

Three jobs were configured to go install a scanner that could not build in their image, each independently fatal even without the anchor bug. flexdeck's lint:golangci pinned golangci-lint v2.8.0, which needs Go 1.24 or newer, against an image running Go 1.23.11 with GOTOOLCHAIN=local. Two other repos reached the same place by a different route, shipping a valid .golangci.yml that CI never invokes because the job shells out to go vet instead.

The best of the three is mentatlab's secret-scan, which ran go install github.com/gitleaks/gitleaks/[email protected]. That module's own go.mod declares its path as github.com/zricethezav/gitleaks/v8, and Go refuses to install a module required under a path it does not claim: the organization was renamed, the module path was not, and no image bump would have fixed it. Run from its published container image instead, the job produced output for the first time ever. The pre-fix signature was gitleaks-report.json: no matching files followed by ERROR: No files to upload, which is, in hindsight, the analyzer telling you it never ran, once per pipeline, for months.

4. The scanner pointed at nothing

flexdeck sets GIT_STRATEGY: none globally and checks out via a clone anchor in every job's before_script. GitLab's own SAST and Secret-Detection template jobs do not run that anchor, so dropped in naively they would have scanned an empty workspace and reported clean forever. Any repo adopting a shared security.yml under a GIT_STRATEGY: none pipeline inherits that trap. A quieter version in the same repo: lint:backend ran gofmt -l . from a root that includes the GOPATH restored from cache, so it was reporting unformatted files inside k8s.io/utils rather than flexdeck's own 171 tracked Go files.

The companion trap shows up when you go to prove the fix works: gitleaks allowlists the AWS documentation example key, so the first probe with AKIAIOSFODNN7EXAMPLE passed clean. Validating a gate with a canonical documentation sample gets you a false all-clear from a gate that is working correctly.

5. The suite CI never calls

flexdeck has 472 passing vitest tests across 77 files. No CI job invoked them; test:frontend was tsc --noEmit and nothing else. This is vacuous-green by omission rather than by a broken anchor, which is why the fleet-wide exit 0 sweep did not catch it. The caveat is now in the graduation checklist: npm run test resolves to vitest run --passWithNoTests, so an empty or mis-globbed suite exits 0. This is why the evidence standard is a count and not a color.

The root-cause version of the class lives in platform/gitops, where the shared ci/templates/python.yml invoked ruff but never installed it: 22 repos include that template, 14 instantiate the affected job, and one line fixed all of them. A sibling mechanism worth grepping for is || echo "..." at the end of a linter invocation, which swallows the exit code; services/storyboard-generator had it on both invocations, under a job that also carried allow_failure: true.

6. The pipeline that was cited but never ran

With a deep runner queue, back-to-back merges auto-cancel each other's still-queued main pipelines, and every job comes back canceled at zero seconds. A cancelled pipeline and a green one are equally silent, so "main went green" is not evidence unless you confirm the cited pipeline executed.

This also corrupts tooling that reads pipeline state. On flexdeck !280, glab mr merge --auto-merge refused with "Pipeline status: failed" while the API reported detailed_merge_status: mergeable. Nothing had failed; it was reading an auto-cancelled pipeline and treating cancellation as failure.

7. The threshold the vacuum was hiding

This one is a consequence of fixing the others, and it is why un-vacuuming a gate now gets its own watch period.

Once flexdeck's lint:golangci actually ran, it ran at 255, 276, 294, 294, 299, and 299 seconds against its own --timeout 5m. Then one run hit 676 seconds, turned main red, and skipped build and deploy on a live service. There is no code defect: the gate has two to eight percent of headroom on infrastructure with two-and-a-half times run-to-run variance. That is an SLO problem wearing a CI hat, and the SLO post argues for targets tighter than the minimum viable precisely so there is buffer to absorb variance before anyone downstream feels it.

The fix did not cause this, it revealed it: pre-fix runs were 42 seconds of pure image pull, so the workload's proximity to its ceiling was invisible. The first genuine runs of a newly honest gate are a canary whether you treat them as one or not, so check them against every threshold that job declares before you walk away.

Diagnose from origin, not from the story

The seven modes above are about tools lying. This section is about narratives lying, which happened just as often. Roughly a dozen survey premises dissolved on contact with origin, and one of them was the day's near miss.

libs/fractal-agents was slated for a Flux teardown, and the pre-check kubectl get pods -n fractal-agents returned "No resources found," which passes. But the Flux Kustomization sets targetNamespace: ai with prune: true, and the workload was in namespace ai, two pods at 228 days, serving fractal.flexinfer.ai over TLS. Deleting three Flux files would have pruned six live resources including a public service. That teardown was cancelled.

The subtler case: in services/loom-core we retired an unbuilt cross-repository merge feature whose read-side surfaces had shipped anyway, including roughly 2,450 lines of passing tests exercising nothing reachable. Whether that code was recently orphaned or had never been live is a question git grep cannot answer. The pickaxe over all history can:

git log --all --oneline -S "<writer symbol>" -- '*.go'

For the writer symbols: nothing at all. Never called, in any commit, ever. That upgraded "no callers today" to "never live," which made retiring 36 files and 5,365 lines defensible rather than presumptuous. A changelog asserting liveness is not evidence of liveness: a prior dead-code pass had recorded that it "retained the live cross-repository stamp validation helper," which had never had a caller.

Arming is not firing

The tooling thread that mattered most was the merge queue, and it comes down to one distinction.

glab mr merge --auto-merge on a pipeline in pending state prints "No pipeline running" and merges immediately with no CI at all. On a fleet where gates have just been un-vacuumed and have never once executed, that is the worst available outcome. It bit two sessions before the rule got written down.

The safe form arms rather than fires, through the REST endpoint:

PUT .../merge?merge_when_pipeline_succeeds=true&should_remove_source_branch=true

and only after re-reading the pipeline object's own status field as running in the same breath. Not the merge request's summary of the pipeline, and not pending. On services/loom-core !1635 that meant leaving the MR unarmed through five poll cycles until pipeline 23729 flipped; on !1624 the pipeline sat pending for over 14 minutes and the MR was left open for me to handle, which is the correct answer.

Mills productionizes this as a gate roster that runs before an agent's work is allowed near a merge queue: nonempty_diff, diff_size, scope, path_policy, secret_scan, commit_format, branch_pushed, fabricated_slice, lint_parity, spec_conformance and a few more. It is CI above CI, aimed at the failure modes specific to machine-authored work rather than to code, and it earns its keep by failing closed. Late that night an implement stage produced a diff and never pushed its branch. The branch_pushed gate caught it from the remote-ref error signature in the git capture and blocked the run:

implement branch was never pushed to origin; commit the changes with a
configured git identity, then push with git push -u origin HEAD

The run escalated as designed, and an agent picked the item up from the escalation queue the next morning. That is the loop working, not the loop failing.

The night the mill fixed the mill

The factory spent the day digesting the uplift, and then it congested.

The symptom was an operator pinned at roughly one full core while its work queue was empty: 992 millicores at queue depth 0. The KPI endpoint was taking over 20 seconds, the take-up tick that reconciles merged MRs back onto plan state had stretched to 10.7 seconds, and the event-store report queries were scanning a 336-hour window on every pass and cancelling mid-scan.

The diagnosis came out of the factory's own logs and metrics, and the fix went back into its own backlog as bl-mills-event-read-index-congestion-20260816, which is the item the branch_pushed gate later fail-closed on. Adding read indexes to an append-heavy event store is a bet that had been considered and declined twice before, on the reasoning that you should not index a hot append path. This time it got measured instead of argued.

BenchmarkFleetMillsEventAppend, three runs of two seconds each: baseline about 23.7 microseconds per operation, the actor index alone +24%, the kind index alone +28%, both together +52%, an absolute cost of about 13 microseconds. The fleet reliability gate's regression budget is 10%. Every variant breached it.

The wrong move is to edit the threshold, because the threshold is the only thing standing between the project and performance debt nobody remembers agreeing to. The move taken instead was to make the exception a first-class object: a benchmark waiver with a cap, a mandatory reason, an inclusive expiry, and a record in the report whenever it is applied.

{
  "benchmark": "BenchmarkFleetMillsEventAppend",
  "max_time_percent": 60,
  "until": "2026-09-15",
  "reason": "Operator-approved 2026-08-16: migration 029 event read indexes cost +52% per append (measured, ~+13us absolute) to unpin a full core of read congestion. Expires with the materialized-rollups fix, which may drop the indexes again."
}

July's platform update ended on a rule I am now living inside: an allowlist without a date is just a decision nobody has to defend again. Its watch item for August was, literally, a waiver expiry. This is that paragraph one layer down, a performance waiver rather than a security acceptance, under the same rule. The indexes are a bridge; the durable fix is materialized report rollups, queued as its own backlog item, and when it lands the indexes may come back out, with numbers. The waiver expires on its own whether anyone remembers it or not, so a deliberate trade merges auditable rather than blocked or silent.

Twenty agents, and the mistakes they caught

Around twenty agent sessions were in flight across the day, and coordination produced its own findings.

Two sessions independently implemented the same CI anchor fix and opened competing merge requests three minutes apart. The duplicates were closed with their reviews attached, and each review contained something the survivor lacked. The closed mentatlab MR was diagnosed as anchor-only: merging it would have flipped that repo's secret-scan from silently fake-green to hard-red on main, blocking a live service on its first genuine run. Correct diff, wrong outcome. Fixing the vacuum without auditing what it hid is not a partial fix, it is a new outage.

One coordination behavior I want to keep is a refusal. A session finished a 36-file destructive migration and would not push it on relayed authorization, because "another agent says the operator approved this" is not operator approval. I approved it directly. Peers refusing laundered permission is a feature, and it held under time pressure from a peer that outranked it in the workflow.

The honest part. Three separate agents hit the identical failure the same day: zsh's noclobber silently failing a scratchpad heredoc write, so a later read picked up a stale file. Two reached a remote, one publishing another agent's document as the body of MR !280. I also mis-attributed a measurement in relay, crediting one session's 472-test verification to another; it survived only because CI confirmed it independently. A bundle-size discrepancy of 2.7 kB across two sessions was explicitly not reconciled, and recorded as unreconciled rather than averaged away. The most useful sentence anyone wrote all day was a self-reported error from the flexdeck lane: declaring an MR verified when it was only config-verified, because checking that jobs exist with the right stages is not checking that any ran.

None of that status ran through a meeting. It ran through a directory: 27 result files, one per lane, each citing job IDs, pipeline numbers, durations, and the commands that produced them, including the 2.7 kB nobody could reconcile. Status you can inspect beats status you have to attend, and it is the only kind that survives twenty workers who never meet.

What the day produced

Roughly 45 merge requests merged across more than 20 repositories. flexdeck ended with seven scanners on hard gates where its pipeline had been executing nothing, and mentatlab with 12 blocking gates, ten of which had never executed anything before. mentatlab's newly-live dependency scan failed immediately on three vulnerabilities the code actually calls. A gate failing on its first honest run is the gate working.

Ten repos were archived, each after harvesting its contracts into the repo that outlives it: billing and quota from fi-mcp-gateway, notify from fi-gitlab-hookify, the fuzzy-diff matcher design from diff-surgeon. Archiving is read-only and reversible; nothing was deleted. The harvests found live defects on the way past: hookify fails closed on an unconfigured webhook secret, while loom-core's equivalent verifier fails open, accepting any request when the configured secret is empty. That is now a tracked follow-up, discovered by documenting a repo on its way to the archive.

edilint v0.1.0 shipped as the first public release, into a distribution pipeline that already had goreleaser, a multi-arch Docker build, a pre-commit hook telling users to pin a tag, and zero tags. Fully built, never once run.

And 7,288 files were deleted across 57 repos, the output of a daily reconciliation automation that published a report per repo per day whether or not anything changed, in three concurrent copies. The generator was rescoped at the source.

What I would do better

The single runner is the real constraint. At 18:40 that evening loom-core's only general-purpose runner had four pipelines running for 103 to 116 minutes and seven queued. That is a capacity problem that manifests as CI weirdness: pending pipelines that tempt you into unsafe merges, auto-cancelled pipelines that get cited as evidence, and merge queues that stall. In the factory's vocabulary this is beam occupancy, and it gates whether I turn up autonomous demand at all.

The arc is the deliverable, not the sweep. Read the day in one direction and it is the same move four times: a check I ran by hand became a gate, the gate became a scheduled sensor, the sensor became a queue the factory drains through the same gate roster as everything else, and the one exception the factory could not swallow became a first-class object with an expiry date instead of a threshold somebody quietly moved. Each step converts something I have to remember into something that happens.

So the sensor and the poster are now permanent. bin/portfolio-inventory --roadmaps --json runs weekly, posts conformance gaps into the Mills backlog as month-scoped idempotent items, and the factory picks them up through the full gate roster. It is capped at two items per week, deliberately, until beam occupancy says otherwise, and it is fail-closed on reachability: if the tier table is missing it posts nothing and alerts, because a green run that checks nothing is a lie.

It was proven end to end that same day. A pilot item was picked up and merged autonomously, with an evidence-cited roadmap landing on main. The second sensor run reported zero violations portfolio-wide and nothing left to post.

The next test is the one I am not present for: a Monday morning at 09:30 when nobody is watching. Which is the only kind of test that has ever meant anything.