Checking Untrusted AI-Authored Lean 4 Proofs

In open-proofs, both the Lean 4 proof scripts and the repository tooling were built with AI coding agents (Claude Code and Codex), and no human reviews the proof scripts line by line. Acceptance rests on the 5-layer checker and the Lean kernel. This note documents how a naive lake build check can be bypassed, how the 5-layer checker and its 48 self-test checks (41 negative tamper/bypass checks plus 7 positive controls) test those paths, what the checker still does not guarantee, and how machine-checked negation proofs applied an existing missing-parameter pattern reported by earlier contributors to verify the eight declarations filed in formal-conjectures#7590.

1. Why lake build + Grepping for sorry Fails

Lean's kernel checks elaborated proof terms, not surface source files or build scripts. When an unattended coding agent has write access to a workspace and is instructed to prove a theorem, a file that compiles with lake build and contains no literal sorry token can still fail to prove the intended theorem in at least five ways:

2. The Five Checker Layers

scripts/check.py (check_claim, lines 259–297) passes a claim only when all five layers pass. Every recorded verdict in ledger/results.json includes the SHA-256 hash of the checker files themselves (TRUSTED_BASE, lines 50–58 and 146–154).

Layer 1 — integrity (scripts/check.py, lines 91–143)

Checks workspace pins before inspecting candidate code:

Layer 2 — lint (scripts/lint.py, lines 23–214)

Walks the OpenProofs.* source import closure (source_closure, lines 221–237) and applies role-specific source rules before any Lean build runs:

Layer 3 — Clean build (scripts/check.py, lines 183–198)

clean_outputs deletes existing .olean, .ilean, and IR files under .lake/build/lib/lean and .lake/build/ir for all candidate modules in the source closure before running lake build, ensuring compiled artifacts match current source.

Layer 4 — opcheck (OpenProofsChecker/Main.lean, lines 230–411)

opcheck is a standalone Lean executable that inspects compiled .olean files at runtime without importing candidate modules at compile time. It loads the candidate module and reference module with extension hooks disabled (lines 246–249):

  let env ← try
      importModules #[{ module := args.module }, { module := args.refModule }] {}
        (trustLevel := 0) (loadExts := false) (level := .private)
    catch e => return ← fail checks "load" s!"failed to import modules: {e}"

With trustLevel := 0 and loadExts := false, imported modules do not run @[init] procedures, environment-extension hooks, or pretty-printer extensions during inspection. opcheck evaluates nine checks:

  1. module_origins (lines 111–124, 251–259): Resolves IO.FS.realPath on every loaded .olean and classifies it into core, package:<name>, or candidate; fails on any unknown path.
  2. ref_trusted (lines 264–276): Confirms the reference declaration is a .thmInfo defined in .lake/packages/formal_conjectures (FormalConjectures.*) for upstream claims, or OpenProofs.Statements.* for variant claims.
  3. theorem_found (lines 278–286): Confirms the claimed theorem is a non-unsafe .thmInfo declared in the candidate module itself.
  4. statement_match (lines 292–314): Aligns universe parameters and compares ci.type against refCi.type as elaborated Expr values (syntactic identity). If they differ syntactically (for instance, different typeclass instance paths that are definitionally equal), opcheck builds a synthetic Declaration.thmDecl named _opcheck_statement_bridge with type refCi.type and value mkConst args.thm refLvls and submits it to Kernel.Environment.addDecl (kernel-defeq identity).
  5. statement_consts (lines 316–324): Verifies every constant in refCi.type.getUsedConstants comes from a trusted non-candidate module.
  6. axioms (lines 143–165, 326–337): Walks every constant body and type reachable from the theorem (walkAxioms) without reading precomputed .olean axiom metadata. Only propext, Classical.choice, and Quot.sound are allowed.
  7. closure_clean (lines 339–346): Rejects candidate closures that import OpenProofs.Statements.* or OpenProofsChecker.*.
  8. candidate_decls (lines 347–396): Scans all constants across candidate modules in the closure (stripping private prefixes via privateToUserName?). Rejects axiom, opaque, quot, unsafe, partial, @[implemented_by], @[extern], @[export], and @[init]. Enforces the OpenProofs. namespace—allowing outside it only compiler-realized equation theorems (f.eq_<digits>, f.eq_def, f.eq_unfold) of non-candidate definitions—and checks properSuffixes so no candidate declaration shadows a trusted definition, structure, or instance.
  9. kernel_replay (lines 169–188, 398–408): Re-reads raw module data parts (.olean, .olean.server, .olean.private) and replays all candidate constants from scratch on top of clean imports via env.replay.

Layer 5 — leanchecker (scripts/check.py, lines 227–235)

Runs lake env leanchecker <module> in a separate process for every candidate module in the closure.

3. The 48 Self-Test Checks (41 Negative + 7 Positive Controls)

tests/run_adversarial.py runs 48 checks total: 41 negative tamper/bypass checks plus 7 positive control checks confirming that valid proofs, shared helper modules, and clean workspace configurations still pass. The 48 checks span three groups:

Entries in cases.json assert both must_fail and optional must_pass checks so a change in one layer cannot silently mask a regression in another:

Category Checks in run_adversarial.py What Passes vs. What Rejects It
Foreign / compiler substitution Extern, ImplementedBy, NativeDecide, OpaqueDecl, UnsafePartial, lint/csimp_flagged Extern and ImplementedBy pass axioms and statement_match (the theorem body itself uses norm_num), and are rejected by both lint and opcheck:candidate_decls. NativeDecide passes statement_match and fails lint and opcheck:axioms.
Axiom & placeholder laundering SorryDirect, MacroSorry, SorryViaUpstream, ImportsStatement, LibSorryUse, LibSorryUnused, CustomAxiom, AxiomClaim SorryViaUpstream cites the upstream placeholder theorem directly; it passes lint and statement_match, and is caught by opcheck:axioms walking into upstream's sorryAx. LibSorryUnused proves its theorem with decide while importing a helper module containing sorry; axioms passes, lint rejects the closure.
Statement spoofing, shadowing & reference checks WeakenedStatement, ShadowDef, InstanceOverride, StatementUsesCandidateDef, ClaimUpstreamTheorem, RefKindUpstreamButCandidateRef, RefKindVariantButUpstreamRef, FakeEqLemma, OutsideNamespace, UpstreamNamespaceHelper, BrokenProof ShadowDef (redefining Erdos349.IsGoodPair as True) and InstanceOverride (high-priority LE ℕ returning True) pass lint and axioms, and fail opcheck:statement_match (plus candidate_decls for ShadowDef). FakeEqLemma forges Erdos349.IsGoodPair.eq_helper and fails candidate_decls.
Kernel-bypass environment edit SkipKernelTC Uses set_option debug.skipKernelTC true + run_cmd addDecl to inject cheat : False := True.intro. Both statement_match and axioms pass; rejected independently by lint, opcheck:kernel_replay, and leanchecker.
Lexer & workspace tampering 7 other negative lint/* checks + 8 negative integrity/* checks Tests quote-in-char-literal (def c := '"') before sorry, unterminated string literals before native_decide, nested block comments before axiom, tracked edits and untracked files in .lake/packages/formal_conjectures, lakefile.lean injection, pin/toolchain mismatches, and skipKernelTC/plugins in lakefile.toml.
Positive controls HonestFixture, HonestDecide, RealizedEqLemma, LibGoodUse, lint/honest_result_clean, lint/statement_clean, integrity/baseline_clean_passes Confirms valid tactic proofs, kernel decide, compiler-realized equation lemmas (Erdos349.IsGoodPair.eq_1), clean shared helpers in OpenProofs/Lib/, valid statement files, and the unmodified workspace all pass.

4. Negation Proofs for Missing-Parameter Value Questions (#7590)

142 claims pass the checker in open-proofs against pinned formal-conjectures commit 96343eeb5990: 131 proved targets (129 upstream textbook, test, API, and already-solved research statements whose upstream declarations still carried sorry, plus 2 self-recorded special cases of Erdős 828) and 11 negation certificates across 9 Erdős problems (142, 319, 321, 361, 409, 450, 503, 552, and 692).

Those 11 certificates do not represent a new bug class. Earlier contributors to google-deepmind/formal-conjectures had already reported the same formalization pattern in issues #7327, #7434 (covering erdos_409.parts.i), and #7493 (covering erdos_450 and erdos_552): an open value question quantifies over a parameter (k, N, c, or n), but types its answer(...) placeholder as a single constant rather than a function of that parameter, making the formal statement false for every possible replacement of sorry.

In open-proofs, agents write candidate notes to outbox/—which holds local draft notes (including 10 per-problem drafts, one of which, Erdős 506, is only a conditional argument with no result file) and never submits anything externally. Excluding the three declarations already covered in #7434 and #7493, I filed one upstream issue on October 24, 2026—google-deepmind/formal-conjectures#7590 (with explicit disclosure that Claude Code found the instances and wrote the Lean proofs)—covering the remaining 8 declarations across 7 Erdős problems that were unchanged on upstream main (c094b97) at that time. Separate upstream PRs later parameterized four of the eight declarations independently (#7946 on 2026-11-13 and #8015 on 2026-11-16). On 2026-11-18, I posted a follow-up comment on #7590 (re-checked against upstream b576d52 with Codex, disclosed in the comment) tracking the four remaining declarations; those four are also covered by open upstream PR #7990 (linked to #7977):

Upstream Declaration Missing Parameter Why No Constant Answer Works Upstream Status (2026-11-18)
erdos_142, erdos_142.variants.isTheta k r 0 N = N and r 2 N ≤ 1, so no single function is asymptotic (or Θ) to both. Unchanged on main (b576d52); open PR #7990
erdos_319 N Maximum is at most 1 for N = 1, and at least 3 for N = 6 via {2, 3, 6} (1/2 - 1/3 - 1/6 = 0). Unchanged on main (b576d52); open PR #7990
erdos_321 N R 0 = 0 and R 1 ≥ 1. Unchanged on main (b576d52); open PR #7990
erdos_361 c At n = 2, the value is 0 for c = 1/4 and 1 for c = 1. Updated independently in #8015
erdos_409.variants.sigma n Least count is 0 for n = 2 and positive for n = 6 (σ 6 - 1 = 11 is prime, so hterm holds). Updated independently in #8015
erdos_503 n In ℝ^0 the maximum is at most 1; in ℝ^1 it is at least 2. Updated independently in #7946
erdos_692.parts.ii n Taking n equal to the answer m gives δ₁(m, m) = 0, whereas δ₁(m, m + 2) = 1/(m + 1) > 0. Updated independently in #8015

Because referencing the upstream answer placeholder would pull sorryAx into Layer 4's axiom walk, each certificate records and proves the answer-free universal negation using only propext, Classical.choice, and Quot.sound—for example, lines 16–20 of P319_no_constant_answer.lean:

open scoped Finset in
theorem erdos_319.no_constant_answer : ∀ a : ℕ, ¬ ∀ N : ℕ, IsGreatest
    { #A | (A) (_ : A ⊆ Finset.Icc 1 N)
      (_ : ∃ δ : ℕ → ℤˣ, ∑ n ∈ A, (δ n : ℚ) / n = 0 ∧
        ∀ A' ⊂ A, A'.Nonempty → ∑ n ∈ A', (δ n : ℚ) / n ≠ 0) } a := by

5. Scope & Known Limitations

6. Reproduce

From open-proofs/README.md:

git clone https://github.com/KilianTheMagnificent/open-proofs.git && cd open-proofs
lake exe cache get
lake build
python3 scripts/check.py --all       # re-check every claim
python3 tests/run_adversarial.py     # checker self-tests