The orchestrator–coordinator pattern

2026-10-18

Summary. I build with coding agents every day, and I do not trust any single thing they produce. A model's output is sampled, so the same input can come back different on the next run; that is a property of the tool, not a bug to prompt away. So the process is built around checking rather than believing: fixed roles, a written record in its own git repo, an independent reader on a different model family for every piece of work, a usage gate before every dispatch, a release gate plus a full test run, and one human (me) who reads the record, uses the product on real data and types the release command. This is the current shape of that process, which runs on omp: an orchestrator session, coordinator subagents, and writer, reader and researcher workers. It replaced a terminal-tab setup in mid October 2026, and the first days on the new stack taught me as much as the months before it.

What GigAI is

GigAI is a small open-source Python project: a local runtime for AI work that keeps settings, credentials and state on your own computer and works through the model you already use. Its first Gig is Scout, a job-search assistant. It is the main thing this process builds, but the same machine has also written a public learning site and an interactive statistics book, and those taught me the most about independent reading.

The premise: outputs vary, so the structure has to carry the trust

The thing I keep in front of me is that a language model behaves as a nondeterministic algorithm: outputs are sampled, and two runs on the same input can disagree. My record shows this all the time. A spike that fitted a timing model on a handful of calls and then measured a few dozen more found that its own predictions came in above even their pessimistic scenario in more than half the cases, and said so. Three independent readers of the same lessons, each a fresh instance with the same instructions, each found gaps the previous one had not. A worker whose output passed every automated check it was given still produced a page that looked bad once a person opened it.

None of that makes agents useless. It means the unit of trust cannot be a single output. It has to be the structure around the output: who did it, who checked it, what was executed rather than read, what the record says, and what I verified myself before anything shipped.

The shape: three roles and a record outside the repo

The process has three agent roles and one human, and it runs under omp as a tree two levels deep.

The orchestrator is one long-lived omp session started in a folder that sits next to the product worktrees, never inside one. I talk only to it. It owns the record and the decisions, starts coordinators, reviews their reports, and hands me the commands that are mine to run.
A coordinator per line of work (a product version, the learning site, the book) runs as a subagent of the orchestrator. It writes worker specs, runs the usage gate, dispatches workers, verifies what comes back, commits by explicit path, records metrics, and reports back. It never talks to me.
Workers are the coordinator's subagents: a writer changes the files named in its spec, a reader independently checks a writer's output, a researcher runs spikes and measurements. Workers cannot spawn anything, never commit, and talk only to the coordinator that dispatched them. If I type into one by mistake, it answers in one line and points me back to the orchestrator.

The record is the point. The orchestrator's folder is its own local git repo with no remote (about 1,200 commits so far): a decisions.log with one dated line per decision and its reason, a status workpad, handoff files, worker specs and reports, spike reports, per-packet metrics, the operating rules, and the task tracker's issue file. The product repo gitignores all of it. omp keeps session transcripts and a memory store, and I treat both as recall aids only; the record in git is the memory, and a new orchestrator session starts by reading the newest handoff, the status file, the decision log and the rules index, in that order.

The roles and the record: I talk to the orchestrator; the orchestrator keeps the record and starts coordinators; each coordinator runs the usage gate before dispatching writer, reader and researcher workers; the reader checks the writer; per-packet metrics go back into the record.

Roles are tied to models, and the reader runs on a different model family from the writer, so a second read is not the same model agreeing with itself.

Role	Runs as	Model family
Orchestrator	the main omp session	Claude (largest model)
Coordinator	subagent, one per line	Claude (mid-size model)
Writer	worker	Claude (one of two mid-size models, picked by remaining weekly allowance)
Reader, researcher	worker	OpenAI Codex models, with a Claude fallback when that account is held

A packet, start to finish

The unit a worker gets is a packet with a five-part spec: target, owned files (nothing else), the change, acceptance as an observable check with the exact command, and exclusions. Under omp two parts were added. Every spec names checkpoints (at least after reading, after the first draft, and before finishing): at each one the worker appends a section to its report file and sends its coordinator one line. And every spec names the coordinator the worker reports to, because workers start with no context at all.

The checkpoints exist for a plain reason. I run two subscription accounts, a large Claude plan and a small Codex plan, and either can run out mid-packet. On the first evening a five-hour window on one account hit its limit while the other sat unused, and both coordinators stopped. Now a usage gate runs before every dispatch: it reads both accounts' usage windows, compares them with written thresholds (a low mark where dispatch is allowed but noted, a hold mark where it is not), records a snapshot, and prints which agent to use for each role. A worker that dies on a usage limit is replaced by a fresh one on the other account with the same spec and "resume from checkpoint N"; one research spike was resumed that way from its partial files, and the resumed worker found and fixed a leakage bug in the scripts it inherited. The omp configuration also has fallback chains, so a usage-limit error switches model instead of killing the agent.

Two more rules shape a packet:

At most two live workers per coordinator unless the orchestrator says three, and two workers never own the same file. A coordinator once ran three on my ambiguous "may run alongside"; the cap stays two unless I say the number.
Clean up after every packet. The coordinator stops the worker's leftover servers, browsers and build children and checks nothing still runs. This rule exists because four orphaned build workers once ran for seven hours and my laptop ran hot.

When a packet comes back, the coordinator applies the old rules, which did not change with the stack: done is not verified (check the files touched, re-run the claimed command yourself, read the result as the user would); read is not executed ("tests assert X" from reading a file is not a run); report the narrowest thing actually proved. Then it commits that packet's files by explicit path, never with a broad add, and never pushes. Pushes, deploys and releases are mine.

Independent readers, on a different model

The rule I would defend hardest is that no agent both does and judges the same work. For documents and lessons that means: one worker writes, a different worker reads, and the reader is never the writer's own instance and usually not the writer's model family. The reader's job is specific. Read the output as the learner would, start to end, at the screen sizes named. Fetch every cited URL and quote the supporting sentence. Recompute every number. Report invented facts, uncited claims, moved numbers and format breaks with file and location. Verdict per file: accepted or changes needed.

What the readers caught, from the scorecard:

On a batch of six concept lessons from a Claude writer, the Codex reader returned "rewrite" on all six with real errors the writer had missed (a sampling scheme that cannot keep every error, pool and autoscaler bounds that did not add up, a CDN step that billed egress to the wrong party), where Claude readers on the neighbouring batches had accepted after mechanical fixes.
In the statistics book, a different reader on each lesson found a real statistics error in every lesson of one chapter (a bootstrap interval printed from a seed artefact, a misquoted bin count, a Fisher test with mismatched units) and, in another chapter, a docstring claiming that dependent tests lower the false-positive chance. Elsewhere a reader caught a figure that was not the nearest billion of the number it cited.
Readers also caught the dull things that matter: exam questions that reused the lesson's own examples, frequency claims with no source, a quote that did not exist in the document it was attributed to.

The fix loop needed a limit. On one set of exercises every re-read by a fresh reader found new protocol-level gaps (an input with no stated source, a missing edge or permission, a race), so the batches took four or five fix rounds and as many reads, and the scorecard verdict was "slow but sound". The rules now: at most two fix rounds, then escalate with screenshots to the orchestrator; the reader who found the last blockers verifies the fix, instead of a fifth fresh reader; and the spec demands the systematic audit table (every endpoint, every step's inputs) in the first write, not the third.

One more lesson sits next to that: acceptance chains do not catch visual quality. A diagram packet passed every check in its spec and still looked wrong once opened; "the coordinator has to look" is now in the rule.

Spikes before building

Anything not obvious gets a research spike before product code, and every claim in a spike report carries a label: measured, recorded earlier, fitted, cited, derived, or [INFERENCE] with the arithmetic shown. The orchestrator reviews the report before anything is built from it, and a different session builds.

The clearest recent example is the assessment-speed spike. I wanted a Scout assessment in five to ten seconds instead of about fifty. The spike ran as a series of small packets under its own coordinator: trace the pipeline from recorded data, measure the floor with a handful of guarded live calls, write candidate prompt shapes offline, run a bounded batch of live calls, verify the outputs with code-side guards, run one blind judge call for invented facts, then write the decision. Every live call needed my approval with a count before it ran, and the harness refused a call beyond its approved number. In all it was about sixty live calls on a private copy of my data, for roughly ten dollars of usage.

The answer was not the one I wanted, which is why I trust it. Five seconds is impossible (the lean floor call alone is about five seconds). No design came under ten seconds for a whole assessment. The one measured win was a flag-level change: a lean mode with low reasoning effort, median call about twenty seconds instead of about thirty-four, faster on every one of the test jobs, with zero invented facts on the judge pass, but accuracy exactly at the bar with no margin. The spike's later update also recorded that its own earlier predictions had been optimistic and by how much. The decision on record is that mode as an opt-in setting, with the slower default kept until a second batch clears the bar with margin; everything else was parked with the measured reason next to it, and so, for now, was the patch itself. The spike is kept.

Releases, and the hands that run them

The unit of product work is still a patch version on its own branch cut from the released head, with main equal to the last release. The release path has one new lesson and one old constant.

The lesson: the release gate is now a cut-down lane (lint, the fast unit lane plus release-gate tests, build, wheel and sdist smoke, a core-flow smoke, UI bundle, eval scoring), built that way so a release runs in under ten minutes. On one patch the gate was green while the full suite, run on the same commit, failed dozens of tests, most of them a regression from the patch itself. The rule since then: before every release, dispatch the full profile on the same commit and require it green, or every remaining failure named as pre-existing on the base. The coordinator reports both runs on that exact commit; the orchestrator diffs the failing lists itself before I see a command.

The constant: agents run every check, I run the release. The command takes no inputs; the workflow refuses unless a green gate exists on exactly that commit, publishes the gate's own built files (nothing rebuilt), creates the GitHub Release, deploys docs and fast-forwards main. Coordinators may prepare a patch release; every release needs my go.

Hands-on use by me is not optional either, and it is where the spec is found to be wrong rather than the code. Before a release command is handed over, the real flows (sources update, what's new, assess) run through the real CLI on a copy of my real home, and findings go into the current patch, not the next one. A model-assisted step counts only on real data: synthetic cases are a self-test for CI, never evidence.

Security and privacy decisions are part of the same discipline and have not moved: the product stores no name, email, phone, address or links; posting text is treated as untrusted and fenced in prompts; and the docs say plainly that anything GigAI gives an agent goes to that agent's model provider. Those were decided by execution in an earlier spike, not by argument, and the public README carries them.

Keeping score

Every packet leaves numbers behind. After a packet is verified or rejected, the coordinator runs a metrics script over the worker's session file (tokens, requests, wall minutes, models used, whether it hit a limit) and appends a row to a per-packet log, then adds a row to a model scorecard: task kind, model and account, minutes, tokens, a verdict (good, slow, wrong, escalated) and a one-line note. Routing changes come from the scorecard, not from taste. After the first days it holds about a hundred and fifty packets.

Cost figures I use only where they changed a decision. The statistics book cost a few hundred dollars of equivalent usage for about twenty accepted lessons in a day, and the weekly gate stopped it at its hold mark with a handoff that says what was accepted, what was written but not yet read, and what was not started; the remaining chapters wait for the weekly reset. A lesson with one write, one or two reads and a fix round costs on the order of ten to twenty dollars equivalent, which is what makes "a different reader for every lesson" affordable.

What went wrong in the first days on omp

All of this is in the record, and most of it changed a rule or a config line.

Idle parking killed workers. I focused a coordinator in the agent hub by accident; it answered me and ended its turn; after the default idle timeout omp parked it, and parking aborted both of its running workers. Idle parking is now off, and the coordinator definition forbids ending its turn while a worker runs.
The wrong restart flag stopped every subagent. The orchestrator process exited and was relaunched with the "continue" flag; a coordinator and its worker were cut off thirty seconds in, and the preview server died with the session. The rule: if subagents were running, restore the session file instead, which resumes them; and start long-lived things as named services in the session that keeps them.
A revived coordinator keeps the permissions it was spawned with. Changing a coordinator's definition (for example, which worker kinds it may start) does not reach a coordinator revived from park; one rejected a worker kind it had not been born with. New rights mean a fresh coordinator, started from the old one's handoff and progress file.
The gate was green and the suite was red, above.
The fix loop with no limit, above.
Orphaned processes, above. And in the other direction: a coordinator's cleanup once killed another worker's preview server by mistake. Logged, the worker told, moved on.
The cutover itself slipped. I decided to let the old coordinator's in-flight workers finish, but nobody sent it a hold, so it kept dispatching. The checklist now says: send the hold the moment the cutover is decided.
A takeover that edited before it read. The first thing the new orchestrator did was rewrite the status file before I had asked. A takeover starts read-only and reports first.

What I'd tell another staff engineer

Decide that no single output is trusted, then design for it. Sampling means the same prompt can give a different answer tomorrow; put the trust in roles, gates and a record, not in a good run.
Keep the record outside the repo, version it, never push it. One line per decision with its reason. Transcripts and agent memory are for recall; the record is what a new session reads first.
Separate doer, verifier and releaser. Writers never commit; readers are a different instance and, when you can, a different model family; the coordinator re-runs the claimed check and commits by path; you run the release.
Put a checkpoint in every spec, and gate dispatch on usage. Accounts run out. A packet that resumes from its last checkpoint costs far less than one restarted, and a gate's "hold" is a clean stop instead of a dead worker.
Cap the loops. Two live workers. Two fix rounds, then escalate with evidence. Let the reader who found the last blocker verify the fix.
Run the full suite before every release, on the exact commit, even when the gate is green. Especially then.
Spike first and let the spike disappoint you. Label every claim; ask for the live-call count before the go; keep the measured reason next to every parked option.
Clean up processes after every packet, and never park an agent that owns running children.
Look at the page. Acceptance commands do not see a bad diagram.
Expect to read everything. The interrupt budget is still mine, and it is still the real constraint.

What still hurts

It is built on one harness's semantics. Subagent trees, parking, restore versus continue, the agent hub and the collab link are omp's; the rules file changes when the harness does.
Fresh readers keep finding things. Each re-read found new gaps, which is good news about the readers and bad news about first drafts. The audit-table-first rule helps; it does not make a first draft complete.
Cost is metered per packet, not per decision. I can say what a lesson or a spike cost; I cannot yet say what a wrong decision cost.
Context limits still force handoffs. Coordinators are started fresh when their definition changes or their account holds, and a handoff written at a hold is a map that can be stale.
One identity everywhere. Agents act as me on GitHub and in both accounts. The release command and the deploys stay in my hands for that reason; a narrower second identity is still on the list.

This is one engineer's working method, a few days into its current stack. The shape (roles, record, independent readers, gates, a human who runs the change) has now survived two harnesses, which is the best evidence I have that the shape, not the harness, carries the trust.