Coordinating Coding Agents and Catching Real Bugs with CI Oracles

Running multiple coding agents on the same repository fails in two predictable ways: parallel agents overwrite or silently invalidate each other's files, and tests that pass on a single macOS laptop hide operating-system, math-library, and scheduling bugs. As a non-coder directing AI assistants without reading Python implementations line by line, I cannot spot a Linux syscall mismatch or a subtle pause-race by eyeballing a diff. What works instead is pairing workspace isolation (wcp) with a clean, multi-version Linux CI runner as an independent oracle. On November 18, 2026, pushing two stdlib-only Python repositories—wcp (232 tests) and luck-filter (55 tests)—to GitHub Actions Ubuntu CI immediately exposed five concrete bugs that passed locally on macOS. This note records every one of them with exact commit SHAs, error messages, code changes, and reproducible commands.

1. Why One Green Laptop Is Not Enough

In Verifying AI-Assisted Contributions Without Reading Code, the core rule is to replace trust in an assistant's summary with an independent oracle. When you own a repository that runs on both macOS and Linux—or coordinates background processes—your own laptop is still only one environment:

None of those defects require reading the code yourself to detect. You only need a clean remote CI matrix (here, GitHub Actions ubuntu-latest running Python 3.11, 3.12, and 3.13) plus a rule that a red CI run is treated as a real defect to diagnose and fix—never as noise to bypass.

2. Isolating Parallel Agents Before They Collide (wcp)

Before CI even runs, coordinating two or more coding agents on one repository creates its own failure mode: one agent edits a file while another runs tests or rewrites the same path, leaving dirty trees or silently reverting work. wcp (Workspace Coordination Protocol, v0.3.0, zero external dependencies, Python 3.9+) enforces a mechanical boundary between agents:

The same discipline applies to developing wcp itself: all 232 unit and integration tests live in tests/ (covering database fencing, descriptor-relative safe file writes, git worktree lifecycle, hook execution, and multi-task runner loops) and run on every push in .github/workflows/ci.yml.

3. Four Real Bugs Caught in wcp (232 Tests)

After tagging the initial public release of wcp (7a520f7), running the 232-test suite across local Python 3.14 and Ubuntu GitHub Actions CI (Python 3.11–3.13) surfaced four distinct bugs in sequence. Each became its own minimal, verifiable commit:

3.1 Python 3.14 SQLite ResourceWarnings (ca5d376)

Running the suite under Python 3.14 emitted SQLite ResourceWarning messages because _open(ns) in wcp/cli.py opened a database connection for CLI commands without closing it before main() returned, and several tests in tests/test_db.py left connections for garbage collection.

Fix: Track every connection opened by _open(ns) on ns._conns and close all of them in a finally: block at the bottom of main() (wcp/cli.py), and register self.addCleanup(conn.close) for every connection opened in tests/test_db.py.

3.2 Linux fchmodat(AT_SYMLINK_NOFOLLOW) Incompatibility (4a289b2)

On the very first Linux CI run, wcp failed at import time before a single test executed. To prevent symlink races when toggling executable bits, wcp/paths.py originally asserted:

if os.chmod not in os.supports_dir_fd or os.chmod not in os.supports_follow_symlinks:
    raise RuntimeError("wcp needs os.chmod(dir_fd=, follow_symlinks=False)")

That check passed on macOS, where fchmodat(..., AT_SYMLINK_NOFOLLOW) is supported. On Linux, fchmodat does not support AT_SYMLINK_NOFOLLOW, so Python excludes os.chmod from os.supports_follow_symlinks and wcp immediately aborted.

Fix: Commit 4a289b2 replaced the module guard with hasattr(os, "fchmod"), updated the Ubuntu CI matrix from ["3.9", "3.12"] to ["3.11", "3.12", "3.13"], and rewrote set_exec_bit() in wcp/paths.py to open the target descriptor directly with O_NOFOLLOW, inspect it with os.fstat, and update permissions with os.fchmod:

        try:
            rfd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=fd)
        except OSError as e:
            if e.errno in _NOFOLLOW_ERRNOS:
                raise _unsafe(rel, name, "not a regular file")
            raise
        try:
            st = os.fstat(rfd)
            if not stat.S_ISREG(st.st_mode):
                raise _unsafe(rel, name, "not a regular file")
            perm = st.st_mode & 0o7777
            if mode == "100755":
                new = perm | ((perm & 0o444) >> 2)
            else:
                new = perm & ~0o111
            if new != perm:
                os.fchmod(rfd, new)
        finally:
            os.close(rfd)

Besides restoring Linux portability, operating on the already-open file descriptor rfd closes the time-of-check/time-of-use window between stat and chmod on both macOS and Linux.

3.3 Linux PID-1 Grandchild Reaping Race in tests/test_gitx.py (b231597)

Once wcp imported on Linux, CI caught a process-reaping race in ChecksTests.test_run_check_validates_cwd_and_kills_process_group_on_timeout. The test launches a Python child that spawns a sleeping grandchild, writes the grandchild's PID to disk, and times out after 1 second so gitx.run_check kills the entire process group.

Immediately after run_check returned, the test ran os.kill(grandchild_pid, 0) once and asserted ProcessLookupError. On Linux CI, once the intermediate child died, the terminated grandchild was reparented to PID 1 and could briefly remain a zombie until PID 1 reaped it—during which os.kill(grandchild_pid, 0) still succeeds even though the grandchild is already dead.

Fix: Commit b231597 replaced the instant probe with a bounded 2.0-second monotonic poll (20 ms interval), still failing loudly with grandchild process %d survived timeout kill if the process actually survived timeout termination.

3.4 Real Concurrency Bug: Runner.summary() Pause-Race (a3e20b8)

The most important catch was not a platform quirk or test-timing issue—it was a genuine state-machine bug in the multi-agent coordinator itself (Runner.summary() in wcp/run.py).

In Runner.loop(), each poll cycle calls self._iterate() (which steps every non-terminal task) and immediately calls self.summary(). Previously, summary() decided whether the whole run had settled into paused with:

        if len(accepted) == len(rts):
            status = 'done'
        elif paused and self._live_count() == 0:
            status = 'paused'
        else:
            status = 'running'

That check contained a subtle race whenever two sibling tasks ran concurrently and one paused (for example, after a failed build):

  1. During _iterate(), task a is in state building and its builder subprocess is still alive, so _step(a) leaves a in building.
  2. Task b fails and transitions to paused.
  3. In the small window after _step(a) returns and before summary() runs, task a's builder subprocess finishes cleanly and exits.
  4. summary() sees paused is non-empty and self._live_count() == 0, so it returns status = 'paused' and breaks out of Runner.loop()—abandoning task a in building before the next iteration can reap its exit code, submit its bundle, and verify it. The same gap applied to siblings already in submit_pending or verifying.

Fix: Commit a3e20b8 added an explicit two-line check on durable task status so the runner never pauses while any sibling task still needs reaping, submission, or verification:

        in_flight = any(r['status'] in ('building', 'submit_pending', 'verifying') for r in rts)
        if len(accepted) == len(rts):
            status = 'done'
        elif paused and not in_flight and self._live_count() == 0:
            status = 'paused'
        else:
            status = 'running'

With that fix in place, all 232 tests passed cleanly across Python 3.11, 3.12, and 3.13 on Ubuntu CI as well as locally.

4. One Floating-Point & Boundary Bug Caught in luck-filter (55 Tests)

The exact same CI-as-oracle pattern caught a subtle numerical bug the same evening in luck-filter (v1.0.0, a 55-test stdlib-only Python toolkit for multiple-testing and backtest-overfitting checks, paired with the offline interactive companion at kilianthemagnificent.github.io/luck-filter/).

After updating the Ubuntu CI matrix to Python 3.11–3.13 (4ba26d1), all 55 tests had already passed locally on macOS—yet Ubuntu Linux CI failed on line 541 of test_luckfilter.py:

@check("binomial test: exact tail, and the edges")
def _():
    close(L.binomial_test(10, 10, 0.5), 0.5 ** 10, 1e-15)
    close(L.binomial_test(0, 10, 0.5), 1.0, 1e-15)
    close(L.binomial_test(6, 10, 0.5), 0.376953125, 1e-12)

The CI failure printed a deceptively puzzling error message (because close() formats both sides with %.12g):

AssertionError: 1 != 1 (tol 1e-15)

Why did binomial_test(0, 10, 0.5) pass on macOS and fail on Linux? Because binomial_test summed log-space binomial terms computed with math.lgamma and math.exp (_log_comb and _sum_exp in luckfilter.py) and clamped the sum from above with min(1.0, ...). On macOS, floating-point rounding landed at or above 1.0 (or within 1e-15 below it). On Linux, glibc's lgamma/exp rounding left the full-support sum slightly below 1.0 by more than 1e-15, where min(1.0, ...) could not hide it.

Fix (3180f16): Instead of relaxing the test tolerance, binomial_test() in luckfilter.py now handles the mathematically exact boundary cases directly before entering log-space:

def binomial_test(successes, n, p0):
    """Exact one-sided binomial p-value, P(X >= successes) under rate p0."""
    if not 0.0 <= p0 <= 1.0:
        raise LuckFilterError("p0 must lie in [0, 1]")
    if successes <= 0:
        return 1.0
    if successes > n:
        return 0.0
    if p0 == 0.0:
        return 0.0
    if p0 == 1.0:
        return 1.0
    lp, lq = math.log(p0), math.log1p(-p0)
    return min(1.0, _sum_exp([_log_comb(n, k) + k * lp + (n - k) * lq
                              for k in range(successes, n + 1)]))

Inspecting those four boundary branches also fixed a real logic bug on the p0 == 1.0 line: the previous code had written return 1.0 if successes >= n else 0.0, which returned 0.0 whenever 0 < successes < n and p0 == 1.0—even though under p0 == 1.0, X = n almost surely so P(X >= successes) is 1.0 for every successes <= n and 0.0 only when successes > n. Checking successes <= 0 and successes > n first makes both p0 == 0.0 and p0 == 1.0 exact.

5. Bug Matrix & Non-Coder CI Workflow

Repo & Commit Oracle That Caught It Symptom Root Cause & Fix
wcp ca5d376 Local Python 3.14 test run SQLite ResourceWarnings on CLI / DB tests Closed CLI connections in main() finally (wcp/cli.py) and registered addCleanup(conn.close) in tests/test_db.py.
wcp 4a289b2 Ubuntu Linux CI (Python 3.11–3.13) Import-time RuntimeError on os.chmod Linux lacks fchmodat(AT_SYMLINK_NOFOLLOW); switched set_exec_bit() to O_RDONLY | O_NOFOLLOW + os.fstat + os.fchmod.
wcp b231597 Ubuntu Linux CI Timeout test failed on immediate os.kill(grandchild_pid, 0) Reparented Linux grandchild awaited PID-1 reaping; polled with 2.0 s deadline in tests/test_gitx.py.
wcp a3e20b8 Ubuntu Linux CI Run paused while sibling task was un-reaped Builder exited between _step() and summary(); gated pause on not in_flight for building/submit_pending/verifying in wcp/run.py.
luck-filter 3180f16 Ubuntu Linux CI (Python 3.11–3.13) AssertionError: 1 != 1 (tol 1e-15) at binomial_test(0, 10, 0.5) glibc lgamma/exp drift below 1.0; added exact guards for successes <= 0, successes > n, and fixed p0 == 1.0 in luckfilter.py.

You do not need to write Python yourself to run this feedback loop reliably:

  1. Isolate concurrent agent edits first: Give each agent its own worktree and declared write scope (via wcp or separate branches) so two agents never race on the same working tree.
  2. Treat cross-OS CI as a required gate, not a formality: Pair local macOS runs with a Linux GitHub Actions matrix spanning multiple runtime versions (for example Python 3.11, 3.12, and 3.13).
  3. Never loosen a failing assertion before diagnosing why it failed: If we had raised the tolerance in test_luckfilter.py from 1e-15 to 1e-12, we would have hidden both the missing successes <= 0 boundary guard and the inverted p0 == 1.0 branch. If we had retried CI after the Runner.summary() failure, a real race in wcp/run.py would have shipped.
  4. Keep one commit per diagnosed failure: Small, single-purpose commits make it obvious which CI failure each change resolved and make regressions easy to bisect.

6. Reproduce

Both repositories use only the Python 3.9+ standard library and require no packages or network access once cloned:

# 1. Reproduce wcp (232 unit & integration tests)
git clone https://github.com/KilianTheMagnificent/wcp.git && cd wcp
./bin/wcp --json version
python3 -W default -m unittest discover -s tests -v

# 2. Reproduce luck-filter (55 self-checks + built-in demo)
git clone https://github.com/KilianTheMagnificent/luck-filter.git && cd luck-filter
python3 -B test_luckfilter.py
python3 -B luckfilter.py --demo

To inspect the exact boundary behavior in luck-filter directly:

python3 -B -c "import luckfilter as L; print(L.binomial_test(0, 10, 0.5), L.binomial_test(3, 10, 1.0), L.binomial_test(11, 10, 1.0))"
# Expected output: 1.0 1.0 0.0

7. Scope & Honest Limits

Related Notes & Toolkit For the earlier non-coder verification walkthroughs across upstream issues and owned releases, see Verifying AI-Assisted Contributions Without Reading Code and the reusable pre-flight policy scanner and checklists in KilianTheMagnificent/agent-verification-playbook.