Verifying AI-Assisted Contributions Without Reading Code
I am not a software engineer and I do not read TypeScript, Lean 4, or JavaScript implementations line by line. When an AI coding assistant builds a command, checks a configuration file, or writes a proof, its own summary is not verification. Contributing honestly without reading code means shifting the check onto an independent oracle: a before-and-after command whose output proves a config file loaded, a proof kernel that rejects invalid deductions, or a small invented fixture where both what must change and what must survive are written down before running the tool. This note documents three concrete cases—two upstream issue contributions and one owned CLI release—with the exact commands, observed outputs, and limits of the method.
1. The Core Rule: Predict What Changes and What Survives
Unverified AI pull requests waste maintainer time because the person opening them cannot answer follow-up questions about edge cases or architecture. A practical alternative for a non-coder is to keep contributions strictly within what can be verified externally—reproducible documentation checks, minimal bug reports, or machine-checked formal counterexamples—and never open uninvited code pull requests.
Before accepting an assistant's claim or posting anything publicly, write down three things:
- One observable thing that must change: a tool disappearing from a JSON list, a checker exit code flipping from
2to0, or a Lean kernel accepting a theorem withoutsorry. - One nearby value that must survive untouched: an unaffected tool call, a negative refund amount, a multiline quoted cell, or the original input file on disk.
- An independent oracle and plain disclosure: record the exact commit or release, use only invented sample data, run the check from a clean copy, and state plainly which AI tools helped run the check or write the code.
2. Case 1 — Reproducible Config & Tool-List Check (modelcontextprotocol/inspector#2980 → PR #3016)
modelcontextprotocol/inspector explicitly asks outside contributors for issues rather than pull requests. In November 2026, open issue #2980 ("docs: add a config reference for the composable test server") tracked adding a configuration reference so users could run custom test servers without reading TypeScript source files.
The existing setup table described the default stdio fixture as having a fixed configuration, even though the repository already shipped a tiny JSON file, test-servers/configs/demo.json:
{
"serverInfo": {
"name": "composable-demo",
"version": "1.0.0"
},
"tools": [{ "preset": "echo" }, { "preset": "get_temp" }],
"transport": {
"type": "stdio"
}
}
Without reading the server's TypeScript implementation, checking whether server-composable.js --config truly worked over stdio came down to a simple before-and-after behavioral test (run locally with Codex against the 2.15.0 fixture, disclosed in the comment):
- Baseline run: Run
server-composable.js --config test-servers/configs/demo.jsonover stdio and list available tools → offeredechoandget_temp; callingechoreturnedEcho: invoice 42. - Modified copy (what must change): Remove the
get_temppreset from a copy ofdemo.jsonand re-run → listing tools returned onlyecho. - Invariant check (what must survive): Calling
echoon the edited config still returnedEcho: invoice 42for the same invented input.
Comparing the tool list before and after a one-line JSON edit proves the custom file actually loaded instead of silently falling back to defaults. On 2026-11-18, I posted that short evidence-backed suggestion on #2980. Later the same day, maintainer cliffhall merged PR #3016 into v2/main (9e9b1b80), crediting the comment in the PR description ("configs/demo.json as the minimal stdio example (suggested in the issue's comments)"), adopting the before-and-after tools/list verification tip verbatim in docs/test-servers.md, and closing #2980 (comment).
3. Case 2 — Kernel-Checked Formal Evidence (google-deepmind/formal-conjectures#7590 & open-proofs)
For mathematical formalizations, running a CLI is not enough—a plausible-looking natural-language argument from an AI model can easily be wrong. In open-proofs, Claude Code and Codex generate Lean 4 proof scripts, and acceptance rests on a five-layer checker (integrity, lint, clean build, opcheck, and leanchecker) rather than human code review (see the companion note Checking Untrusted AI-Authored Lean 4 Proofs).
Earlier contributors to google-deepmind/formal-conjectures had already reported value-question statements whose answer(...) placeholder omitted a quantified parameter (#7327, #7434, #7493). Rather than reporting candidate statements from a text scan alone, we required a kernel-checked negation proof (∀ a, ¬ ∀ param, ...) using only Lean's standard axioms (propext, Classical.choice, Quot.sound) for every reported declaration:
- What must hold for the proof: Each of the 8 declarations across 7 Erdős problems (
142[2 declarations],319,321,361,409variants.sigma,503, and692parts.ii) compiles against pinned upstream96343eeb5990, matches its statement definitionally inopcheck, uses only standard axioms, and passes independentleancheckerreplay. - What must hold for the checker itself: The checker must fail on deliberate cheats (comment/string evasion,
sorry,native_decide, shadowed definitions, tamperedlakefile.tomlpins) and keep every recorded verdict tied to a single SHA-256 hash of the trusted checker files (TRUSTED_BASE).
Even on a lightweight runner without downloading Mathlib, .github/workflows/checker-ci.yml (commit 70a5590, verified in CI run 44427818995) enforces the fast Python self-tests, pin checks, full-claim source lint, and trusted_base_sha256 consistency on every push:
$ python3 tests/run_adversarial.py --only lint
ok lint/char_literal_quote_cannot_hide_sorry
ok lint/comment_mention_of_sorry_flagged_raw
ok lint/unterminated_string_cannot_hide_code
ok lint/nested_comment_close_handled
ok lint/csimp_flagged
ok lint/lean_import_rejected
ok lint/honest_result_clean
ok lint/statement_def_rejected
ok lint/statement_with_proof_rejected
ok lint/statement_clean
10/10 expectations hold
# Static pin check + 3 lakefile tamper assertions + lint_layer on all 142 claims + ledger check:
OK: pins valid, 3/3 lakefile tamper checks pass, 142/142 claims pass lint & ledger PASS check (trusted_base=10a78286602f).
I filed #7590 on 2026-10-24 linking all eight Lean proof files, explicitly stating that Claude Code found the instances and wrote the proofs and that no human reviewed them line by line, and marking the issue up for grabs rather than submitting an unreviewed PR. Separate upstream PRs (#7946 and #8015) later fixed four of the eight declarations independently. On 2026-11-18, after re-checking all eight declarations against upstream b576d52 with Codex, I posted a short status follow-up listing the four still-open statements (also tracked in #7977 and open PR #7990).
4. Case 3 — Four-Invoice Behavioral Walkthrough (inbox-guard v0.1.0)
The same rule applies to an owned repository built with AI assistance. Before tagging inbox-guard v0.1.0 (commit d8cbb15), we verified a fresh clone directly against the walkthrough in docs/checking-a-change.md using the four invented invoices in examples/invoices.csv.
Writing predictions down first prevents an empty-findings bug from fooling you (for instance, a tool that silently drops rows or corrupts negative amounts):
- Must change:
INV-002's memo (=2+3, row 3, col 4) becomes"'=2+3";INV-004's memo (+2+3, row 5, col 4) becomes"'+2+3";check-csvmoves from exit code2(2 findings) to exit code0(0 findings). - Must survive: The refund of
-42.50(and-7.25), the comma in"Alpine, Ltd.", the multiline quoted delivery note inINV-003, the 5-row by 4-column shape, and the originalexamples/invoices.csvfile on disk.
Below is the verbatim transcript from a clean temporary clone of inbox-guard at d8cbb15 on Node v26.13.1 (npm test also passed 16/16 tests in 569 ms):
$ git clone https://github.com/KilianTheMagnificent/inbox-guard.git && cd inbox-guard
$ printf '%s\n' 'Send invoice 42 to orders@example.org.' | node ./bin/inbox-guard.mjs redact -
Send invoice 42 to <EMAIL_1_351174>.
$ node ./bin/inbox-guard.mjs check-csv examples/invoices.csv
{
"safe": false,
"rowCount": 5,
"columnCount": 4,
"findings": [
{
"row": 3,
"column": 4
},
{
"row": 5,
"column": 4
}
]
}
# exit code: 2
$ node ./bin/inbox-guard.mjs sanitize-csv examples/invoices.csv --output reviewed-invoices.csv
# exit code: 0
$ cat reviewed-invoices.csv
Invoice,Vendor,Balance,Memo
INV-001,"Alpine, Ltd.",-42.50,Refund received
INV-002,Harbor Supplies,17.00,"'=2+3"
INV-003,Paper Mill,0.00,"First delivery
Second delivery, marked ""received"""
INV-004,Harbor Supplies,-7.25,"'+2+3"
$ node ./bin/inbox-guard.mjs check-csv reviewed-invoices.csv
{
"safe": true,
"rowCount": 5,
"columnCount": 4,
"findings": []
}
# exit code: 0
$ node ./bin/inbox-guard.mjs sanitize-csv examples/invoices.csv --output reviewed-invoices.csv
inbox-guard error: Output already exists. Choose a new filename.
# exit code: 1
Re-running check-csv examples/invoices.csv after the overwrite test returned exit code 2 with the exact same two coordinates (rows 3 and 5, column 4), confirming the original export was never modified.
5. Comparison & Reusable Verification Checklist
| Case | Independent Oracle | What Changed | What Survived | Outcome |
|---|---|---|---|---|
inspector#2980 |
Running server-composable.js --config over stdio + tools/list |
Removing get_temp dropped it from the tool list |
echo still returned Echo: invoice 42 |
Credited and merged in PR #3016 (v2/main) |
formal-conjectures#7590 |
Lean 4 kernel + 5-layer checker (opcheck, leanchecker, 48 self-tests) |
Answer-free negations accepted under standard axioms | Pinned upstream untouched; all 142 claims share one TRUSTED_BASE hash |
4 of 8 fixed independently (#7946, #8015); 4 tracked in #7590 / #7990 |
inbox-guard v0.1.0 |
Fresh clone + npm test (16/16) + 4-invoice CLI walkthrough + CI/CodeQL |
Memos in rows 3 & 5 prefixed with '; exit code 2 → 0 |
-42.50 refund, vendor comma, multiline note, untouched original |
Tagged & released as v0.1.0 |
A compact checklist adapted from docs/checking-a-change.md works across all three settings:
- Check the target project's live contribution and AI policy before doing any work; skip projects that ban AI-assisted work, and keep human-only prose in your own words where required.
- Record exact version and environment (commit SHA or tag, runtime version, OS).
- Use small, invented sample inputs only—never real emails, customer rows, or local paths.
- Write down one expected change and one invariant that must survive before running the command.
- Run from a clean clone or unmodified baseline and keep the exact observed output and exit codes.
- State honestly what was checked, what still needs checking (such as a specific spreadsheet app's import dialog), and which AI tools assisted.
6. Scope & Honest Limits
- Behavioral spot-checks do not replace code review or security audits: Observing four invoices or one JSON config confirms that specific behavior, not every edge case in the underlying implementation. That is why
inbox-guardkeeps outside pull requests turned off, pairs CLI walkthroughs with automated unit/timing tests, CodeQL, andgitleaksscanning, and routes suspected bypasses to Private Vulnerability Reporting. - A formal kernel proves formal validity, not informal meaning: A Lean 4 proof confirms a theorem follows from the pinned Lean definitions under standard axioms; it cannot decide whether a formalization matches the informal mathematical literature.
- Downstream environments still matter: Verifying CSV bytes on disk does not guarantee how every version of Excel, Google Sheets, or LibreOffice Calc imports those cells unless you test that specific application and import dialog.
- Policy compliance is part of verification: Technical evidence is still unwelcome if it violates a maintainer's AI policy or opens uninvited code PRs on an issue-only repository. Respecting repository rules comes first.
open-proofs) and Screening Untrusted Office Inputs Before LLM Triage (inbox-guard).