Verifying AI-Assisted Contributions Without Reading Code

I am not a software engineer and I do not read TypeScript, Lean 4, or JavaScript implementations line by line. When an AI coding assistant builds a command, checks a configuration file, or writes a proof, its own summary is not verification. Contributing honestly without reading code means shifting the check onto an independent oracle: a before-and-after command whose output proves a config file loaded, a proof kernel that rejects invalid deductions, or a small invented fixture where both what must change and what must survive are written down before running the tool. This note documents three concrete cases—two upstream issue contributions and one owned CLI release—with the exact commands, observed outputs, and limits of the method.

1. The Core Rule: Predict What Changes and What Survives

Unverified AI pull requests waste maintainer time because the person opening them cannot answer follow-up questions about edge cases or architecture. A practical alternative for a non-coder is to keep contributions strictly within what can be verified externally—reproducible documentation checks, minimal bug reports, or machine-checked formal counterexamples—and never open uninvited code pull requests.

Before accepting an assistant's claim or posting anything publicly, write down three things:

  1. One observable thing that must change: a tool disappearing from a JSON list, a checker exit code flipping from 2 to 0, or a Lean kernel accepting a theorem without sorry.
  2. One nearby value that must survive untouched: an unaffected tool call, a negative refund amount, a multiline quoted cell, or the original input file on disk.
  3. An independent oracle and plain disclosure: record the exact commit or release, use only invented sample data, run the check from a clean copy, and state plainly which AI tools helped run the check or write the code.

2. Case 1 — Reproducible Config & Tool-List Check (modelcontextprotocol/inspector#2980 → PR #3016)

modelcontextprotocol/inspector explicitly asks outside contributors for issues rather than pull requests. In November 2026, open issue #2980 ("docs: add a config reference for the composable test server") tracked adding a configuration reference so users could run custom test servers without reading TypeScript source files.

The existing setup table described the default stdio fixture as having a fixed configuration, even though the repository already shipped a tiny JSON file, test-servers/configs/demo.json:

{
  "serverInfo": {
    "name": "composable-demo",
    "version": "1.0.0"
  },
  "tools": [{ "preset": "echo" }, { "preset": "get_temp" }],
  "transport": {
    "type": "stdio"
  }
}

Without reading the server's TypeScript implementation, checking whether server-composable.js --config truly worked over stdio came down to a simple before-and-after behavioral test (run locally with Codex against the 2.15.0 fixture, disclosed in the comment):

Comparing the tool list before and after a one-line JSON edit proves the custom file actually loaded instead of silently falling back to defaults. On 2026-11-18, I posted that short evidence-backed suggestion on #2980. Later the same day, maintainer cliffhall merged PR #3016 into v2/main (9e9b1b80), crediting the comment in the PR description ("configs/demo.json as the minimal stdio example (suggested in the issue's comments)"), adopting the before-and-after tools/list verification tip verbatim in docs/test-servers.md, and closing #2980 (comment).

3. Case 2 — Kernel-Checked Formal Evidence (google-deepmind/formal-conjectures#7590 & open-proofs)

For mathematical formalizations, running a CLI is not enough—a plausible-looking natural-language argument from an AI model can easily be wrong. In open-proofs, Claude Code and Codex generate Lean 4 proof scripts, and acceptance rests on a five-layer checker (integrity, lint, clean build, opcheck, and leanchecker) rather than human code review (see the companion note Checking Untrusted AI-Authored Lean 4 Proofs).

Earlier contributors to google-deepmind/formal-conjectures had already reported value-question statements whose answer(...) placeholder omitted a quantified parameter (#7327, #7434, #7493). Rather than reporting candidate statements from a text scan alone, we required a kernel-checked negation proof (∀ a, ¬ ∀ param, ...) using only Lean's standard axioms (propext, Classical.choice, Quot.sound) for every reported declaration:

Even on a lightweight runner without downloading Mathlib, .github/workflows/checker-ci.yml (commit 70a5590, verified in CI run 44427818995) enforces the fast Python self-tests, pin checks, full-claim source lint, and trusted_base_sha256 consistency on every push:

$ python3 tests/run_adversarial.py --only lint
ok   lint/char_literal_quote_cannot_hide_sorry
ok   lint/comment_mention_of_sorry_flagged_raw
ok   lint/unterminated_string_cannot_hide_code
ok   lint/nested_comment_close_handled
ok   lint/csimp_flagged
ok   lint/lean_import_rejected
ok   lint/honest_result_clean
ok   lint/statement_def_rejected
ok   lint/statement_with_proof_rejected
ok   lint/statement_clean

10/10 expectations hold

# Static pin check + 3 lakefile tamper assertions + lint_layer on all 142 claims + ledger check:
OK: pins valid, 3/3 lakefile tamper checks pass, 142/142 claims pass lint & ledger PASS check (trusted_base=10a78286602f).

I filed #7590 on 2026-10-24 linking all eight Lean proof files, explicitly stating that Claude Code found the instances and wrote the proofs and that no human reviewed them line by line, and marking the issue up for grabs rather than submitting an unreviewed PR. Separate upstream PRs (#7946 and #8015) later fixed four of the eight declarations independently. On 2026-11-18, after re-checking all eight declarations against upstream b576d52 with Codex, I posted a short status follow-up listing the four still-open statements (also tracked in #7977 and open PR #7990).

4. Case 3 — Four-Invoice Behavioral Walkthrough (inbox-guard v0.1.0)

The same rule applies to an owned repository built with AI assistance. Before tagging inbox-guard v0.1.0 (commit d8cbb15), we verified a fresh clone directly against the walkthrough in docs/checking-a-change.md using the four invented invoices in examples/invoices.csv.

Writing predictions down first prevents an empty-findings bug from fooling you (for instance, a tool that silently drops rows or corrupts negative amounts):

Below is the verbatim transcript from a clean temporary clone of inbox-guard at d8cbb15 on Node v26.13.1 (npm test also passed 16/16 tests in 569 ms):

$ git clone https://github.com/KilianTheMagnificent/inbox-guard.git && cd inbox-guard
$ printf '%s\n' 'Send invoice 42 to orders@example.org.' | node ./bin/inbox-guard.mjs redact -
Send invoice 42 to <EMAIL_1_351174>.

$ node ./bin/inbox-guard.mjs check-csv examples/invoices.csv
{
  "safe": false,
  "rowCount": 5,
  "columnCount": 4,
  "findings": [
    {
      "row": 3,
      "column": 4
    },
    {
      "row": 5,
      "column": 4
    }
  ]
}
# exit code: 2

$ node ./bin/inbox-guard.mjs sanitize-csv examples/invoices.csv --output reviewed-invoices.csv
# exit code: 0

$ cat reviewed-invoices.csv
Invoice,Vendor,Balance,Memo
INV-001,"Alpine, Ltd.",-42.50,Refund received
INV-002,Harbor Supplies,17.00,"'=2+3"
INV-003,Paper Mill,0.00,"First delivery
Second delivery, marked ""received"""
INV-004,Harbor Supplies,-7.25,"'+2+3"

$ node ./bin/inbox-guard.mjs check-csv reviewed-invoices.csv
{
  "safe": true,
  "rowCount": 5,
  "columnCount": 4,
  "findings": []
}
# exit code: 0

$ node ./bin/inbox-guard.mjs sanitize-csv examples/invoices.csv --output reviewed-invoices.csv
inbox-guard error: Output already exists. Choose a new filename.
# exit code: 1

Re-running check-csv examples/invoices.csv after the overwrite test returned exit code 2 with the exact same two coordinates (rows 3 and 5, column 4), confirming the original export was never modified.

5. Comparison & Reusable Verification Checklist

Case Independent Oracle What Changed What Survived Outcome
inspector#2980 Running server-composable.js --config over stdio + tools/list Removing get_temp dropped it from the tool list echo still returned Echo: invoice 42 Credited and merged in PR #3016 (v2/main)
formal-conjectures#7590 Lean 4 kernel + 5-layer checker (opcheck, leanchecker, 48 self-tests) Answer-free negations accepted under standard axioms Pinned upstream untouched; all 142 claims share one TRUSTED_BASE hash 4 of 8 fixed independently (#7946, #8015); 4 tracked in #7590 / #7990
inbox-guard v0.1.0 Fresh clone + npm test (16/16) + 4-invoice CLI walkthrough + CI/CodeQL Memos in rows 3 & 5 prefixed with '; exit code 2 → 0 -42.50 refund, vendor comma, multiline note, untouched original Tagged & released as v0.1.0

A compact checklist adapted from docs/checking-a-change.md works across all three settings:

  1. Check the target project's live contribution and AI policy before doing any work; skip projects that ban AI-assisted work, and keep human-only prose in your own words where required.
  2. Record exact version and environment (commit SHA or tag, runtime version, OS).
  3. Use small, invented sample inputs only—never real emails, customer rows, or local paths.
  4. Write down one expected change and one invariant that must survive before running the command.
  5. Run from a clean clone or unmodified baseline and keep the exact observed output and exit codes.
  6. State honestly what was checked, what still needs checking (such as a specific spreadsheet app's import dialog), and which AI tools assisted.

6. Scope & Honest Limits

Related Notes & Repositories Deep dives into the two owned verification systems referenced above: Checking Untrusted AI-Authored Lean 4 Proofs (open-proofs) and Screening Untrusted Office Inputs Before LLM Triage (inbox-guard).