Coordinating Coding Agents and Catching Real Bugs with CI Oracles
Running multiple coding agents on the same repository fails in two predictable ways: parallel agents overwrite or silently invalidate each other's files, and tests that pass on a single macOS laptop hide operating-system, math-library, and scheduling bugs. As a non-coder directing AI assistants without reading Python implementations line by line, I cannot spot a Linux syscall mismatch or a subtle pause-race by eyeballing a diff. What works instead is pairing workspace isolation (wcp) with a clean, multi-version Linux CI runner as an independent oracle. On November 18, 2026, pushing two stdlib-only Python repositories—wcp (232 tests) and luck-filter (55 tests)—to GitHub Actions Ubuntu CI immediately exposed five concrete bugs that passed locally on macOS. This note records every one of them with exact commit SHAs, error messages, code changes, and reproducible commands.
1. Why One Green Laptop Is Not Enough
In Verifying AI-Assisted Contributions Without Reading Code, the core rule is to replace trust in an assistant's summary with an independent oracle. When you own a repository that runs on both macOS and Linux—or coordinates background processes—your own laptop is still only one environment:
- Kernel and syscall semantics differ: macOS accepts POSIX flag combinations (such as
fchmodatwithAT_SYMLINK_NOFOLLOW) that Linux rejects. - C math libraries round differently: Apple
libmand Linuxglibccan differ by a few ULPs onlgammaandexp, exposing missing analytical boundary checks. - Process scheduling exposes races: A slower container runner with a different PID-1 reaper and CPU cadence hits timing windows between subprocess exit and coordinator polling that rarely fire on a fast local machine.
- Newer runtimes tighten warnings: Running the same suite on Python 3.14 surfaces unclosed SQLite connection warnings that older versions stay quiet about.
None of those defects require reading the code yourself to detect. You only need a clean remote CI matrix (here, GitHub Actions ubuntu-latest running Python 3.11, 3.12, and 3.13) plus a rule that a red CI run is treated as a real defect to diagnose and fix—never as noise to bypass.
2. Isolating Parallel Agents Before They Collide (wcp)
Before CI even runs, coordinating two or more coding agents on one repository creates its own failure mode: one agent edits a file while another runs tests or rewrites the same path, leaving dirty trees or silently reverting work. wcp (Workspace Coordination Protocol, v0.3.0, zero external dependencies, Python 3.9+) enforces a mechanical boundary between agents:
- One canonical checkout, isolated detached worktrees: Builders never write directly to the canonical repository; each claimed task gets its own detached
git worktreewith a declared write scope (for example,--write "src/engine/**") and optional shared resources. - Leases and fencing epochs: Every claim carries a time-bounded lease and monotonic epoch stored in SQLite with
fcntl.flockadvisory locks; stale epochs are refused even if their diff would cleanly apply. - Hash-bound submissions and fresh-candidate verification:
wcp task submitrecords before/after SHA-256 hashes, andwcp task verifyre-runs the repository's declared check command against a fresh candidate built from current canonical before publishing atomically.
The same discipline applies to developing wcp itself: all 232 unit and integration tests live in tests/ (covering database fencing, descriptor-relative safe file writes, git worktree lifecycle, hook execution, and multi-task runner loops) and run on every push in .github/workflows/ci.yml.
3. Four Real Bugs Caught in wcp (232 Tests)
After tagging the initial public release of wcp (7a520f7), running the 232-test suite across local Python 3.14 and Ubuntu GitHub Actions CI (Python 3.11–3.13) surfaced four distinct bugs in sequence. Each became its own minimal, verifiable commit:
3.1 Python 3.14 SQLite ResourceWarnings (ca5d376)
Running the suite under Python 3.14 emitted SQLite ResourceWarning messages because _open(ns) in wcp/cli.py opened a database connection for CLI commands without closing it before main() returned, and several tests in tests/test_db.py left connections for garbage collection.
Fix: Track every connection opened by _open(ns) on ns._conns and close all of them in a finally: block at the bottom of main() (wcp/cli.py), and register self.addCleanup(conn.close) for every connection opened in tests/test_db.py.
3.2 Linux fchmodat(AT_SYMLINK_NOFOLLOW) Incompatibility (4a289b2)
On the very first Linux CI run, wcp failed at import time before a single test executed. To prevent symlink races when toggling executable bits, wcp/paths.py originally asserted:
if os.chmod not in os.supports_dir_fd or os.chmod not in os.supports_follow_symlinks:
raise RuntimeError("wcp needs os.chmod(dir_fd=, follow_symlinks=False)")
That check passed on macOS, where fchmodat(..., AT_SYMLINK_NOFOLLOW) is supported. On Linux, fchmodat does not support AT_SYMLINK_NOFOLLOW, so Python excludes os.chmod from os.supports_follow_symlinks and wcp immediately aborted.
Fix: Commit 4a289b2 replaced the module guard with hasattr(os, "fchmod"), updated the Ubuntu CI matrix from ["3.9", "3.12"] to ["3.11", "3.12", "3.13"], and rewrote set_exec_bit() in wcp/paths.py to open the target descriptor directly with O_NOFOLLOW, inspect it with os.fstat, and update permissions with os.fchmod:
try:
rfd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=fd)
except OSError as e:
if e.errno in _NOFOLLOW_ERRNOS:
raise _unsafe(rel, name, "not a regular file")
raise
try:
st = os.fstat(rfd)
if not stat.S_ISREG(st.st_mode):
raise _unsafe(rel, name, "not a regular file")
perm = st.st_mode & 0o7777
if mode == "100755":
new = perm | ((perm & 0o444) >> 2)
else:
new = perm & ~0o111
if new != perm:
os.fchmod(rfd, new)
finally:
os.close(rfd)
Besides restoring Linux portability, operating on the already-open file descriptor rfd closes the time-of-check/time-of-use window between stat and chmod on both macOS and Linux.
3.3 Linux PID-1 Grandchild Reaping Race in tests/test_gitx.py (b231597)
Once wcp imported on Linux, CI caught a process-reaping race in ChecksTests.test_run_check_validates_cwd_and_kills_process_group_on_timeout. The test launches a Python child that spawns a sleeping grandchild, writes the grandchild's PID to disk, and times out after 1 second so gitx.run_check kills the entire process group.
Immediately after run_check returned, the test ran os.kill(grandchild_pid, 0) once and asserted ProcessLookupError. On Linux CI, once the intermediate child died, the terminated grandchild was reparented to PID 1 and could briefly remain a zombie until PID 1 reaped it—during which os.kill(grandchild_pid, 0) still succeeds even though the grandchild is already dead.
Fix: Commit b231597 replaced the instant probe with a bounded 2.0-second monotonic poll (20 ms interval), still failing loudly with grandchild process %d survived timeout kill if the process actually survived timeout termination.
3.4 Real Concurrency Bug: Runner.summary() Pause-Race (a3e20b8)
The most important catch was not a platform quirk or test-timing issue—it was a genuine state-machine bug in the multi-agent coordinator itself (Runner.summary() in wcp/run.py).
In Runner.loop(), each poll cycle calls self._iterate() (which steps every non-terminal task) and immediately calls self.summary(). Previously, summary() decided whether the whole run had settled into paused with:
if len(accepted) == len(rts):
status = 'done'
elif paused and self._live_count() == 0:
status = 'paused'
else:
status = 'running'
That check contained a subtle race whenever two sibling tasks ran concurrently and one paused (for example, after a failed build):
- During
_iterate(), taskais in statebuildingand its builder subprocess is still alive, so_step(a)leavesainbuilding. - Task
bfails and transitions topaused. - In the small window after
_step(a)returns and beforesummary()runs, taska's builder subprocess finishes cleanly and exits. summary()seespausedis non-empty andself._live_count() == 0, so it returnsstatus = 'paused'and breaks out ofRunner.loop()—abandoning taskainbuildingbefore the next iteration can reap its exit code, submit its bundle, and verify it. The same gap applied to siblings already insubmit_pendingorverifying.
Fix: Commit a3e20b8 added an explicit two-line check on durable task status so the runner never pauses while any sibling task still needs reaping, submission, or verification:
in_flight = any(r['status'] in ('building', 'submit_pending', 'verifying') for r in rts)
if len(accepted) == len(rts):
status = 'done'
elif paused and not in_flight and self._live_count() == 0:
status = 'paused'
else:
status = 'running'
With that fix in place, all 232 tests passed cleanly across Python 3.11, 3.12, and 3.13 on Ubuntu CI as well as locally.
4. One Floating-Point & Boundary Bug Caught in luck-filter (55 Tests)
The exact same CI-as-oracle pattern caught a subtle numerical bug the same evening in luck-filter (v1.0.0, a 55-test stdlib-only Python toolkit for multiple-testing and backtest-overfitting checks, paired with the offline interactive companion at kilianthemagnificent.github.io/luck-filter/).
After updating the Ubuntu CI matrix to Python 3.11–3.13 (4ba26d1), all 55 tests had already passed locally on macOS—yet Ubuntu Linux CI failed on line 541 of test_luckfilter.py:
@check("binomial test: exact tail, and the edges")
def _():
close(L.binomial_test(10, 10, 0.5), 0.5 ** 10, 1e-15)
close(L.binomial_test(0, 10, 0.5), 1.0, 1e-15)
close(L.binomial_test(6, 10, 0.5), 0.376953125, 1e-12)
The CI failure printed a deceptively puzzling error message (because close() formats both sides with %.12g):
AssertionError: 1 != 1 (tol 1e-15)
Why did binomial_test(0, 10, 0.5) pass on macOS and fail on Linux? Because binomial_test summed log-space binomial terms computed with math.lgamma and math.exp (_log_comb and _sum_exp in luckfilter.py) and clamped the sum from above with min(1.0, ...). On macOS, floating-point rounding landed at or above 1.0 (or within 1e-15 below it). On Linux, glibc's lgamma/exp rounding left the full-support sum slightly below 1.0 by more than 1e-15, where min(1.0, ...) could not hide it.
Fix (3180f16): Instead of relaxing the test tolerance, binomial_test() in luckfilter.py now handles the mathematically exact boundary cases directly before entering log-space:
def binomial_test(successes, n, p0):
"""Exact one-sided binomial p-value, P(X >= successes) under rate p0."""
if not 0.0 <= p0 <= 1.0:
raise LuckFilterError("p0 must lie in [0, 1]")
if successes <= 0:
return 1.0
if successes > n:
return 0.0
if p0 == 0.0:
return 0.0
if p0 == 1.0:
return 1.0
lp, lq = math.log(p0), math.log1p(-p0)
return min(1.0, _sum_exp([_log_comb(n, k) + k * lp + (n - k) * lq
for k in range(successes, n + 1)]))
Inspecting those four boundary branches also fixed a real logic bug on the p0 == 1.0 line: the previous code had written return 1.0 if successes >= n else 0.0, which returned 0.0 whenever 0 < successes < n and p0 == 1.0—even though under p0 == 1.0, X = n almost surely so P(X >= successes) is 1.0 for every successes <= n and 0.0 only when successes > n. Checking successes <= 0 and successes > n first makes both p0 == 0.0 and p0 == 1.0 exact.
5. Bug Matrix & Non-Coder CI Workflow
| Repo & Commit | Oracle That Caught It | Symptom | Root Cause & Fix |
|---|---|---|---|
wcp ca5d376 |
Local Python 3.14 test run | SQLite ResourceWarnings on CLI / DB tests |
Closed CLI connections in main() finally (wcp/cli.py) and registered addCleanup(conn.close) in tests/test_db.py. |
wcp 4a289b2 |
Ubuntu Linux CI (Python 3.11–3.13) | Import-time RuntimeError on os.chmod |
Linux lacks fchmodat(AT_SYMLINK_NOFOLLOW); switched set_exec_bit() to O_RDONLY | O_NOFOLLOW + os.fstat + os.fchmod. |
wcp b231597 |
Ubuntu Linux CI | Timeout test failed on immediate os.kill(grandchild_pid, 0) |
Reparented Linux grandchild awaited PID-1 reaping; polled with 2.0 s deadline in tests/test_gitx.py. |
wcp a3e20b8 |
Ubuntu Linux CI | Run paused while sibling task was un-reaped | Builder exited between _step() and summary(); gated pause on not in_flight for building/submit_pending/verifying in wcp/run.py. |
luck-filter 3180f16 |
Ubuntu Linux CI (Python 3.11–3.13) | AssertionError: 1 != 1 (tol 1e-15) at binomial_test(0, 10, 0.5) |
glibc lgamma/exp drift below 1.0; added exact guards for successes <= 0, successes > n, and fixed p0 == 1.0 in luckfilter.py. |
You do not need to write Python yourself to run this feedback loop reliably:
- Isolate concurrent agent edits first: Give each agent its own worktree and declared write scope (via
wcpor separate branches) so two agents never race on the same working tree. - Treat cross-OS CI as a required gate, not a formality: Pair local macOS runs with a Linux GitHub Actions matrix spanning multiple runtime versions (for example Python 3.11, 3.12, and 3.13).
- Never loosen a failing assertion before diagnosing why it failed: If we had raised the tolerance in
test_luckfilter.pyfrom1e-15to1e-12, we would have hidden both the missingsuccesses <= 0boundary guard and the invertedp0 == 1.0branch. If we had retried CI after theRunner.summary()failure, a real race inwcp/run.pywould have shipped. - Keep one commit per diagnosed failure: Small, single-purpose commits make it obvious which CI failure each change resolved and make regressions easy to bisect.
6. Reproduce
Both repositories use only the Python 3.9+ standard library and require no packages or network access once cloned:
# 1. Reproduce wcp (232 unit & integration tests)
git clone https://github.com/KilianTheMagnificent/wcp.git && cd wcp
./bin/wcp --json version
python3 -W default -m unittest discover -s tests -v
# 2. Reproduce luck-filter (55 self-checks + built-in demo)
git clone https://github.com/KilianTheMagnificent/luck-filter.git && cd luck-filter
python3 -B test_luckfilter.py
python3 -B luckfilter.py --demo
To inspect the exact boundary behavior in luck-filter directly:
python3 -B -c "import luckfilter as L; print(L.binomial_test(0, 10, 0.5), L.binomial_test(3, 10, 1.0), L.binomial_test(11, 10, 1.0))"
# Expected output: 1.0 1.0 0.0
7. Scope & Honest Limits
- CI only catches what tests and environments exercise: A green Ubuntu matrix across Python 3.11–3.13 catches Linux syscall, glibc, and container-scheduling differences, not Windows behavior (
wcpexplicitly requires a POSIX environment withfcntl.flockandos.fchmod) or logic paths lacking tests. - Tests are still AI-assisted code: When coding agents write both implementation and tests, a passing suite can still miss an unstated requirement. That is why every CI failure needs an explicit check that the fix addresses the underlying invariant—not a weakened assertion.
- Isolation is not a security sandbox:
wcpcoordinates cooperative coding agents and prevents accidental workspace collisions, stale publishes, and sensitive-path writes; it is not an OS/VM security boundary against hostile code. - No human line-by-line code review: Both
wcpandluck-filterwere developed with AI coding assistants under Andre's direction, verified through automated suites, multi-version CI, and empirical CLI checks rather than manual line-by-line software engineering review. Outside pull requests remain closed in favor of reproducible issue reports.
KilianTheMagnificent/agent-verification-playbook.