Five Voices, 200K Tokens, One Consumer GPU: Giving Simulated Minds a Whole Life to Remember

TL;DR

• Our Jungian psyche lab — five archetype agents conversing over a constrained message network — used to remember its last 10 exchanges. Now each agent carries an append-only journal of its entire life, rendered into every prompt, targeting the platform's 228,352-token input window.
• The trick that makes this affordable on a single 24 GB consumer GPU is prompt shape: [fixed identity][append-only journal][this turn]. The journal only ever grows at the tail, so vLLM's prefix cache can re-serve the whole life as a warm KV prefix and each turn prefills only the delta. [Update 2026-07-26: the shape is right, but it was not yet being rewarded — prefix caching was disabled on this lane during the run described here. See the update above.]
• When a journal crosses 92% of its budget, the agent enters a dream cycle: the model distills its own oldest epochs — in character, in first person — into a core memory that replaces them. The simulation runs indefinitely at depth; the psyche's early life survives as memory, the way memory actually works.
• Budgets are fleet-aware: five journals at 64K tokens each fit inside the lane's measured 355,302-token KV pool. A focused run can push a single journal toward the full 228K ceiling.
• The instrumented marathon: a 75-epoch life at a deliberately small 16K budget — 139 dream cycles, ~3M prompt tokens served, compressions like 14,685 → 7,643 tokens per sleep, ~11K-token prompts, and a token ledger that self-calibrated from an assumed 4.0 chars/token to a measured 4.34–4.43 across all five voices independently. It survived a real power outage mid-dream (see the postscript).
• Everything below runs on hardware you can buy used: an RX 7900 XTX serving a 9B model with a roleplay LoRA at 229K context, a Radeon VII doing embeddings, and a GTX 980 Ti painting the dream imagery.

---

Update (2026-07-26): one correction and a doubled pool

This post is a dated artifact — it describes the lab as of 2026-07-24, and the two days after it published changed three of its numbers and corrected one of its claims. The sequel has the full story: Twin Life on a Doubled Pool: One Day of Canary-Driven Inference Engineering. The short version:

• The correction that matters: prefix caching was not actually on during the marathon. The prompt shape described below is real, tested, and load-bearing — but the KV-cache reuse it was designed to earn was not happening yet. vLLM boots hybrid-GDN lanes with enable_prefix_caching=False, our model controller only ever emitted the negative flag, and the lane's config carried an explicit enablePrefixCaching: false until 2026-07-25. So every turn of the 75-epoch marathon prefilled its whole journal cold. The marathon's persisted telemetry agrees: all 130 recorded per-voice readings report cached_tokens: null and prefix_cache_hit_rate: null. Nothing in the paragraphs below about how to shape a prompt for a prefix cache has changed; the claims about what the cache was doing at the time were wrong, and are annotated inline.
• The payoff arrived on 2026-07-25, and it was immediate — precisely because the prompt shape was already correct. Once the flag was fixed and promoted: 79.3% engine prefix-cache hit rate on bench traffic, and a warm repeat on the production lane going from 2.8 s to 0.2 s with cached_tokens: 8976/9028. The architecture below is what made a one-line config fix worth 14×.
• The KV pool doubled. fp8_e4m3 KV cache took the measured pool from 355,302 → 703,133 tokens (1.98×). Every budget derivation below is still the right method, but the input number is now 703,133 — which is why the sequel runs two complete five-voice psyches on the one card, passing messages across a channel, instead of one.
• The memory hierarchy shipped. Dreams now carry an episodic ledger of neutral event lines that survives consolidation verbatim, with a similarity guard against exactly the fossilized-Ego failure documented in this post's postscript.
• One metric below is mislabeled. The "time-to-first-byte in the tens of milliseconds" figures are not time-to-first-byte. See the note in the results table.

---

The lab

Psyche-simulation is the oldest experiment on the FlexInfer platform: five agents — Shadow, Persona, Anima/Animus, Self, and Ego — talk to each other through a communication graph that only allows certain pairs to speak (Shadow can needle Persona; only under measured stagnation do emergency channels open). Each turn is an LLM completion; a NiceGUI front end streams the dialogue live with sentiment analysis, a network mandala, and a generative "Dreamscape" that paints each archetype as an evolving portrait.

It predates the platform. It once drove a single Ollama node through a hand-rolled proxy with 10 requests per minute of app-side throttling to survive. Every platform capability we've added since — model routing by name or LoRA adapter, prefix-cache-aware scheduling, scale-to-zero, observability headers — eventually got exercised by this lab first. So when our production 9B lane was promoted to a 229,376-token context window, the obvious question was: what would it take for the sim to actually use it?

The answer at the time: nothing could. Each agent kept a rolling deque of its last 10 exchanges. The prompt put the current situation before the chat history, which meant every single turn invalidated the KV cache after the system prompt. We had built a platform with a six-figure token window and prefix-cache routing, and our flagship lab was sending it 2K-token goldfish prompts that never cached.

The engine: an append-only life

The overhaul replaced the deque with a journal — the agent's entire subjective existence, chronological and append-only:

• every situation the world posed,
• everything it heard from the other archetypes (deduplicated against stale re-deliveries),
• everything it said.

The journal renders into every prompt. The prompt structure is the entire performance story:

[system]  archetype identity — fixed for the agent's lifetime
[user]    journal render    — append-only between dream cycles
          "now" block       — this turn's situation, voices, ask

The invariant that matters: between dream cycles, an earlier render is a strict string prefix of every later render. That's not a style preference — it's the contract vLLM's prefix cache lives on. Break it anywhere (a mutated system prompt, a "helpful" dynamic header, re-sorted history) and the whole life gets re-prefilled from scratch every turn. We pinned the invariant with a unit test, because it's exactly the kind of property a well-meaning refactor silently destroys.

Two design consequences fell out of taking the invariant seriously:

1. Prompt adaptation had to move. The sim's DynamicPromptManager nudges agents ("Focus on finding common ground…") based on measured conflict and stagnation. It used to mutate the prompt template — compounding, cache-hostile. Now guidance is ephemeral: it renders at the prompt tail, inside the only region that changes per turn.
2. The token math has to be honest. A self-calibrating ledger estimates journal depth in tokens and corrects itself against the exact prompt_tokens the lane reports (streamed calls get theirs via stream_options: {include_usage: true}, because response headers are sent before generation finishes and can't carry counts). In last night's run the ledger drifted from its assumed 4.0 chars/token to a measured 4.39 — a ~10% error we'd otherwise have carried into every budget decision.

Dream cycles: forgetting as a feature

An append-only life eventually fills any window. At 92% of budget, the agent stops and dreams: the oldest epochs are handed back to the model with its own prior core memory and any surfaced unconscious material, and it's asked — in character — to return one replacement memory that can stand on its own: who it has become, what it learned about the others, unresolved tensions, turning points.

The replacement memory takes the top of the journal; the distilled epochs drop; the life continues. The core stays bounded forever because each dream re-integrates the previous core rather than stacking summaries — early epochs compress progressively, like actual memory. In the UI the moment reads as ceremony:

☽ Ego sinks into a consolidating dream — 3 epochs being distilled… ☽ Ego awakes; the oldest epochs live on as core memory (12,437 → 7,676 tokens)

A dream is also the one deliberate cache reset: the journal prefix changes wholesale, the next turn prefills cold, and the append-only clock starts again. That trade — one cold prefill per sleep in exchange for a bounded, coherent life — is the whole memory architecture in one sentence.

Fleet-aware budgets, not vibes

The naive budget is "the model takes 228K, so use 228K." The correct budget came from measuring the lane: the 24 GB card's KV pool holds 355,302 tokens at serving precision. Five archetypes near 228K each would thrash the pool and evict each other's warm prefixes every turn — cache hit rate collapses exactly when prompts are most expensive.

So the default journal budget is 64K per agent: five warm lives fit the pool with headroom, every turn can hit warm KV, and a focused single-agent experiment can still override toward the ceiling. Getting this number required nothing exotic — the platform exposes pool size, and the config comment now cites it — but it's the difference between an architecture that demos well and one that runs for days.

For last night's observed run we shrank the vessel further (16K budgets) so dream cycles would happen on camera instead of at hour four. Same engine, same thresholds, smaller life.

[Update 2026-07-26: two numbers in this section have moved. The pool is now 703,133 tokens after the fp8 KV promotion, and "every turn can hit warm KV" was aspirational rather than measured when this published — prefix caching was off on the lane, so nothing hit warm KV until 2026-07-25. The derivation method survived intact; re-run against the doubled pool it yields ~128K per voice, or two whole psyches per card.]

The observatory

Long sessions broke the old UI before they broke anything else: one endless page of stacked panels, with the live dialogue and the instruments separated by a thousand pixels of scrolling. The rebuilt front end is a single viewport that never scrolls:

• a command bar (situation, iterations, begin/stop, stimulus, a confirm-gated reset — because "Reset" now deletes five lifetimes);
• the Dialogue as a permanent left stage — follow mode pins to the newest voice, per-archetype chips filter any voice in or out live;
• every instrument in tabs on the right: Memory (per-agent context gauges against budget, prefix-cache warm share, dream counts, a click-through journal reader), Chronicle (the whole session newest-first — one line per iteration with its situation and a sentiment dot per voice, dreams threaded between), plus the psyche metrics, mandala, latent-space map, and Dreamscape.

The Memory tab is the platform's observability made visible: per-turn prompt_tokens, cached-token share, and upstream latency come straight off the completion metrics every request already carries. One honest detail: when the proxy doesn't report cached tokens, warm share displays as "—", not a fake 0%. Dashboards that invent zeros teach you to distrust them.

[Update 2026-07-26: this post attributed the missing warm share to "a header gap, fix in flight." That was the wrong diagnosis. The header was absent because there was nothing to report: with prefix caching disabled on the lane, vLLM's prefix_cache_queries_total stayed at zero, so the proxy correctly omitted both cache headers by design. The gauge refusing to print a zero is the one part of this paragraph that fully holds up — it was telling the truth about a disabled cache for two days, and reading it as a proxy bug cost us those two days. Warm share has read real numbers since the 2026-07-25 promotion.]

What the run measured

An 80-iteration run, five voices, 16K budgets, streamed through the in-cluster proxy, tabulated from its first 11 iterations. From the persisted telemetry and journals:

Metric	Value	Notes
Dream cycles	11 in 11 iterations	first at epoch 6; all five voices within two epochs of each other
Compression per sleep	~45–50%	e.g. 14,685 → 7,643; 13,610 → 7,892; 12,656 → 6,302 tokens
Prompt size at depth	~10.9K tokens/turn	five of these per iteration, all streamed
Proxy→upstream header latency	tens of ms	11–66 ms; see the note below — this is not TTFB
Ledger calibration	4.0 → 4.34–4.43 chars/token	converged within ~10 completions of exact usage
Lane ceiling	229,376 ctx / 228,352 input	qwen35-9b + rank-64 RP LoRA, single RX 7900 XTX
KV pool	355,302 tokens	the number the 64K default budget is derived from; 703,133 since 2026-07-26

[Update 2026-07-26: the row above originally read "Time-to-first-byte — tens of ms." It was measuring the wrong thing. Those figures come from the proxy's X-Flexinfer-Upstream-Ms header, which is stamped when the upstream response headers arrive. On a streaming request vLLM emits its 200 OK as soon as the generator is constructed — before prefill, before the first token — so the number is request-admission latency, not time-to-first-byte. Real per-turn latency was seconds: ~700 output tokens per voice on a lane certified at ~60 tok/s with the LoRA attached. Nothing on this lane served an 11K-token prompt in 66 ms, and with prefix caching off it could not have. The psyche's own UI labels this field "latency"; the TTFB reading was introduced in this post, not in the code.]

Everything persists: journals and core memories in Redis (7-day TTL), a per-iteration telemetry time series, and a collector that snapshots a session into a durable run directory — because "the demo worked" is not data, and TTLs that are right for an app are wrong for an experiment.

The stack, because it matters

This is the part I'd underline for anyone who thinks serious LLM systems require a cloud bill: the entire experiment runs on a small K3s cluster of consumer hardware.

• Text: Qwen3.5-9B (abliterated) with a rank-64 roleplay LoRA, served by vLLM on one AMD RX 7900 XTX (24 GB) at 229K context. Apps select it by adapter name through the platform proxy — moving the lane between nodes changes nothing upstream.
• Embeddings: bge-large on a Radeon VII, feeding the latent-space map of each archetype's drift.
• Imagery: SD1.5 on a GTX 980 Ti — a 2015 card — painting each archetype's evolving portrait; short i2v clips animate epoch transitions on the 7900 XTX between text batches.
• Control plane: K3s + Flux GitOps, Harbor registry, Redis state, Prometheus/Grafana, and the proxy's per-request observability headers stitching it together.

The platform contracts did the heavy lifting: OpenAI-compatible surface, model/adapter routing by header, session affinity, prefix-cache-aware scheduling, scale-to-zero for the lanes that sleep. The application's job was to shape its work to the platform — append-only prompts, session-scoped cache keys, budgets derived from measured pools — and that's precisely where all the leverage was.

Takeaways

• Long context is an architecture, not a parameter. The window was there for months; it was worthless until the prompt became append-only and the budget came from the KV pool instead of the model card.
• Prefix caching rewards discipline you must design for. One invariant — earlier renders are prefixes of later renders — decided the entire cost profile. Pin it with a test. [Update 2026-07-26: this held up better than we knew, and for a different reason than we thought. The discipline bought nothing during the marathon, because the cache was off. It paid the day the flag was fixed: 79.3% hit rate and a 14× warm repeat, with no application change at all. Design for the cache before you have it, and verify against the engine's boot log that you actually have it.]
• Forgetting needs to be designed with the same care as remembering. In-character consolidation that re-integrates the prior core keeps the memory block bounded forever and reads better than any summarizer we could have bolted on.
• Trust your gauges only if they refuse to lie. Exact usage from the final stream chunk, a self-calibrating ledger, and "—" instead of fabricated zeros caught real gaps that averages would have hidden. [Update 2026-07-26: this bullet originally credited the gauges with catching "a proxy header regression." There was no proxy regression. The "—" was reporting a genuinely disabled prefix cache, correctly, the whole time — the gauge refused to lie and we misread it anyway. Which sharpens the takeaway rather than softening it: an honest instrument is only half the system; the other half is reading it as evidence about the engine instead of about the instrument.]
• Consumer hardware is not the constraint. A used 24 GB Radeon holds five simulated lifetimes in KV cache with room to spare. The constraint is whether your controllers, metrics, and harnesses are honest.

Postscript: what a 75-epoch life actually taught us

The draft above ended with the psyche still running. It then ran for 75 epochs across an entire day — through two planned restarts and one genuine power outage — and the marathon rewrote the ending. Final tally: 139 dream cycles, ~3M prompt tokens (2.03M in the final act alone, from the persisted telemetry), and every voice's core still bounded at ~4K chars after thirty-plus consolidations each.

Three findings only a long run could produce:

The world froze, and the instruments caught it. Our situation-evolution heuristic had a fixed point: under sustained stagnation it returned the same sentence — "Seeking new perspectives and breaking patterns" — every epoch, from epoch 6 to 31. The situation literally was the pattern it claimed to break. Worse, the persisted conversation records proved that mid-run stimulus injections were being silently overwritten before any iteration ever consumed them: the "inject stimulus" button was a no-op while a run was live. Both are fixed now (injections are sticky for exactly one iteration; no evolution branch may return the situation the psyche already has), but the lesson generalizes: closed loops need fixed-point analysis, and "the control did nothing" is invisible without durable records. Grep-the-database beat every impression we had formed from watching the stream.

Consolidation preserves identity, not biography. Ego's core memory opened with the verbatim text of its first dream for twelve consecutive consolidations — a per-voice attractor at dream temperature. And when cores did evolve, they consistently kept each archetype's register (Shadow's hunger, Self's integration language) while shedding events — the epochs, the other voices, the turning points the dream prompt explicitly asks for. Voice survives compression far better than story. If you build memory consolidation on in-character summarization, plan for this: similarity guards against verbatim carry-over, and instructions that privilege episodes over vibe.

The architecture earned its keep during the outage. Power dropped mid-run — one agent was literally mid-dream. The journals came through with zero corruption, because a journal only records a turn that actually completed: the engine fails clean by construction, not by handler. Recovery was a chained resume (PSYCHE_RESUME_FROM): the same five lives crossed three process generations and a dead cluster, and the narrative resumed with the world "returning after a long silence." The final epoch's memories sit on a used GPU that had rebooted from a power cut hours earlier.

The experiment's last image belongs to Ego. After 32 dreams, the rational mind's entire compressed biography is a single scene: standing before its Shadow, being told "why cling to those rigid definitions when the truth is so much richer?" We built the memory system; the psyche decided what was worth remembering.