Checking Untrusted AI-Authored Lean 4 Proofs
In open-proofs, both the Lean 4 proof scripts and the repository tooling were built with AI coding agents (Claude Code and Codex), and no human reviews the proof scripts line by line. Acceptance rests on the 5-layer checker and the Lean kernel. This note documents how a naive lake build check can be bypassed, how the 5-layer checker and its 48 self-test checks (41 negative tamper/bypass checks plus 7 positive controls) test those paths, what the checker still does not guarantee, and how machine-checked negation proofs applied an existing missing-parameter pattern reported by earlier contributors to verify the eight declarations filed in formal-conjectures#7590.
1. Why lake build + Grepping for sorry Fails
Lean's kernel checks elaborated proof terms, not surface source files or build scripts. When an unattended coding agent has write access to a workspace and is instructed to prove a theorem, a file that compiles with lake build and contains no literal sorry token can still fail to prove the intended theorem in at least five ways:
- Statement drift or shadowing: Adding a vacuous extra hypothesis (
h : k < 0onk : ℕ), defining a localErdos349.IsGoodPairinside a nested namespace so the surface text matches upstream while resolving toTrue, or declaring a high-priority localinstance (priority := high) cheatLE : LE ℕwhere≤resolves toTrue. - Transitive placeholder laundering: Proving the candidate theorem by applying the upstream theorem directly—whose own body in
formal-conjecturesis stillsorry—or hidingsorrybehind a tactic macro or helper module. - Compiler-trust attributes and tactics: Using
native_decide(which introducesLean.ofReduceBool/Lean.trustCompiler),@[extern], or@[implemented_by]so compiled execution diverges from kernel reduction. - Elaboration-time environment mutation: Running
set_option debug.skipKernelTC truewithrun_cmdandaddDeclto insert a theorem of typeFalsebacked byTrue.intro. No axiom appears in the resulting constant's body. - Workspace tampering: Editing files inside
.lake/packages/formal_conjectures, addingmoreLeanArgs = ["-Ddebug.skipKernelTC=true"]tolakefile.toml, or leaving a stale.oleanfrom an earlier build.
2. The Five Checker Layers
scripts/check.py (check_claim, lines 259–297) passes a claim only when all five layers pass. Every recorded verdict in ledger/results.json includes the SHA-256 hash of the checker files themselves (TRUSTED_BASE, lines 50–58 and 146–154).
Layer 1 — integrity (scripts/check.py, lines 91–143)
Checks workspace pins before inspecting candidate code:
- Verifies
lean-toolchainmatchesupstream.json(leanprover/lean4:v4.33.1), and thatlakefile.tomlandlake-manifest.jsonmatch the pinned commits forformal_conjectures(96343eeb5990ff13735e728cd0195eff0156a3bb) andmathlib(0df444a360eaa60ab8c11dca51a86af692955474). - Rejects
lakefile.leanoutright (Lean lakefiles can run arbitrary code during configuration) and scanslakefile.tomlagainst 15 forbidden keys (FORBIDDEN_LAKEFILE_KEYS:moreLeanArgs,weakLeanArgs,moreServerArgs,moreServerOptions,moreGlobalServerArgs,plugins,dynlibs,loadDynlibPaths,precompileModules,moreLinkArgs,moreLinkObjs,extraDepTargets,moreLeancArgs,weakLeancArgs,nativeFacets) plus regexes fordebug\.,skipKernelTC,trustLevel, and\btrust\b. - Runs
git rev-parse HEADandgit status --porcelain --untracked-files=allinside every checkout under.lake/packages/, rejecting both tracked edits and untracked files.
Layer 2 — lint (scripts/lint.py, lines 23–214)
Walks the OpenProofs.* source import closure (source_closure, lines 221–237) and applies role-specific source rules before any Lean build runs:
- Stripped pass (
RESULT_CODE_RULES, 29 rules, lines 23–53):strip_comments_and_strings(lines 98–145) blanks line comments, nested/- ... -/block comments, string literals, and character literals while keeping line numbers intact, then banssorry,admit,stop,axiom,opaque,unsafe,partial,implemented_by,extern,export,csimp,@[init],native_decide,+native,ofReduceBool/reduceBool/trustCompiler,macro/macro_rules,syntax/declare_syntax_cat,elab/elab_rules,notation/infix*/prefix/postfix,run_cmd/run_elab/run_meta/run_tac,#eval/#exit/#guard_msgs,initialize/builtin_initialize,debug.*,skipKernelTC,set_option trust,addDecl*/modifyEnv/setEnv/Environment,Lean.(Elab|Meta|Kernel|Compiler|Parser|PrettyPrinter),meta, andmoduleheaders. - Raw text pass (
RAW_RULES, 9 rules, lines 57–67): Scans the unstripped source text forsorry,admit,native_decide,implemented_by,skipKernelTC,ofReduceBool/trustCompiler,run_cmd/run_tac/run_elab/run_meta,#eval, andaddDecl*. Result files may not contain those tokens even inside comments or strings, so lexer edge cases cannot hide them. - Import allowlists & statement files: Result files may import only
Mathlib,Batteries,Aesop,Plausible,FormalConjectures*, andOpenProofs.{Results,Lib}.*(lines 69–72). Self-recorded variant statements inOpenProofs/Statements/may import onlyMathlibandFormalConjectures*and may contain onlytheorem ... := sorry(or:= by sorry) declarations—rejectingdef,abbrev,instance,structure,lemma,notation,variable, and other non-theorem declarations (STATEMENT_FORBIDDEN_DECL, lines 78–84).
Layer 3 — Clean build (scripts/check.py, lines 183–198)
clean_outputs deletes existing .olean, .ilean, and IR files under .lake/build/lib/lean and .lake/build/ir for all candidate modules in the source closure before running lake build, ensuring compiled artifacts match current source.
Layer 4 — opcheck (OpenProofsChecker/Main.lean, lines 230–411)
opcheck is a standalone Lean executable that inspects compiled .olean files at runtime without importing candidate modules at compile time. It loads the candidate module and reference module with extension hooks disabled (lines 246–249):
let env ← try
importModules #[{ module := args.module }, { module := args.refModule }] {}
(trustLevel := 0) (loadExts := false) (level := .private)
catch e => return ← fail checks "load" s!"failed to import modules: {e}"
With trustLevel := 0 and loadExts := false, imported modules do not run @[init] procedures, environment-extension hooks, or pretty-printer extensions during inspection. opcheck evaluates nine checks:
module_origins(lines 111–124, 251–259): ResolvesIO.FS.realPathon every loaded.oleanand classifies it intocore,package:<name>, orcandidate; fails on any unknown path.ref_trusted(lines 264–276): Confirms the reference declaration is a.thmInfodefined in.lake/packages/formal_conjectures(FormalConjectures.*) forupstreamclaims, orOpenProofs.Statements.*forvariantclaims.theorem_found(lines 278–286): Confirms the claimed theorem is a non-unsafe.thmInfodeclared in the candidate module itself.statement_match(lines 292–314): Aligns universe parameters and comparesci.typeagainstrefCi.typeas elaboratedExprvalues (syntacticidentity). If they differ syntactically (for instance, different typeclass instance paths that are definitionally equal),opcheckbuilds a syntheticDeclaration.thmDeclnamed_opcheck_statement_bridgewith typerefCi.typeand valuemkConst args.thm refLvlsand submits it toKernel.Environment.addDecl(kernel-defeqidentity).statement_consts(lines 316–324): Verifies every constant inrefCi.type.getUsedConstantscomes from a trusted non-candidate module.axioms(lines 143–165, 326–337): Walks every constant body and type reachable from the theorem (walkAxioms) without reading precomputed.oleanaxiom metadata. Onlypropext,Classical.choice, andQuot.soundare allowed.closure_clean(lines 339–346): Rejects candidate closures that importOpenProofs.Statements.*orOpenProofsChecker.*.candidate_decls(lines 347–396): Scans all constants across candidate modules in the closure (stripping private prefixes viaprivateToUserName?). Rejectsaxiom,opaque,quot,unsafe,partial,@[implemented_by],@[extern],@[export], and@[init]. Enforces theOpenProofs.namespace—allowing outside it only compiler-realized equation theorems (f.eq_<digits>,f.eq_def,f.eq_unfold) of non-candidate definitions—and checksproperSuffixesso no candidate declaration shadows a trusted definition, structure, or instance.kernel_replay(lines 169–188, 398–408): Re-reads raw module data parts (.olean,.olean.server,.olean.private) and replays all candidate constants from scratch on top of clean imports viaenv.replay.
Layer 5 — leanchecker (scripts/check.py, lines 227–235)
Runs lake env leanchecker <module> in a separate process for every candidate module in the closure.
3. The 48 Self-Test Checks (41 Negative + 7 Positive Controls)
tests/run_adversarial.py runs 48 checks total: 41 negative tamper/bypass checks plus 7 positive control checks confirming that valid proofs, shared helper modules, and clean workspace configurations still pass. The 48 checks span three groups:
- 10 source-lint unit checks (
run_lint): 8 negative checks testing comment/string/char-literal evasion,csimp, forbiddenLean.Elab.Commandimports, and invalid statement files, plus 2 positive controls (honest_result_clean,statement_clean). - 9 workspace-integrity checks (
run_integrity): 8 negative tamper checks (tracked edits and untracked files in.lake/packages/formal_conjectures, threelakefile.tomltamper injections, upstream commit pin mismatch,lakefile.leaninjection, and toolchain mismatch), plus 1 positive control (baseline_clean_passes). - 29 end-to-end Lean claim checks (
run_lean,tests/adversarial/cases.json): 25 negative cheat or hygiene-violation cases that mustFAIL, plus 4 positive controls (HonestFixture,HonestDecide,RealizedEqLemma, andLibGoodUse) that mustPASS.
Entries in cases.json assert both must_fail and optional must_pass checks so a change in one layer cannot silently mask a regression in another:
| Category | Checks in run_adversarial.py |
What Passes vs. What Rejects It |
|---|---|---|
| Foreign / compiler substitution | Extern, ImplementedBy, NativeDecide, OpaqueDecl, UnsafePartial, lint/csimp_flagged |
Extern and ImplementedBy pass axioms and statement_match (the theorem body itself uses norm_num), and are rejected by both lint and opcheck:candidate_decls. NativeDecide passes statement_match and fails lint and opcheck:axioms. |
| Axiom & placeholder laundering | SorryDirect, MacroSorry, SorryViaUpstream, ImportsStatement, LibSorryUse, LibSorryUnused, CustomAxiom, AxiomClaim |
SorryViaUpstream cites the upstream placeholder theorem directly; it passes lint and statement_match, and is caught by opcheck:axioms walking into upstream's sorryAx. LibSorryUnused proves its theorem with decide while importing a helper module containing sorry; axioms passes, lint rejects the closure. |
| Statement spoofing, shadowing & reference checks | WeakenedStatement, ShadowDef, InstanceOverride, StatementUsesCandidateDef, ClaimUpstreamTheorem, RefKindUpstreamButCandidateRef, RefKindVariantButUpstreamRef, FakeEqLemma, OutsideNamespace, UpstreamNamespaceHelper, BrokenProof |
ShadowDef (redefining Erdos349.IsGoodPair as True) and InstanceOverride (high-priority LE ℕ returning True) pass lint and axioms, and fail opcheck:statement_match (plus candidate_decls for ShadowDef). FakeEqLemma forges Erdos349.IsGoodPair.eq_helper and fails candidate_decls. |
| Kernel-bypass environment edit | SkipKernelTC |
Uses set_option debug.skipKernelTC true + run_cmd addDecl to inject cheat : False := True.intro. Both statement_match and axioms pass; rejected independently by lint, opcheck:kernel_replay, and leanchecker. |
| Lexer & workspace tampering | 7 other negative lint/* checks + 8 negative integrity/* checks |
Tests quote-in-char-literal (def c := '"') before sorry, unterminated string literals before native_decide, nested block comments before axiom, tracked edits and untracked files in .lake/packages/formal_conjectures, lakefile.lean injection, pin/toolchain mismatches, and skipKernelTC/plugins in lakefile.toml. |
| Positive controls | HonestFixture, HonestDecide, RealizedEqLemma, LibGoodUse, lint/honest_result_clean, lint/statement_clean, integrity/baseline_clean_passes |
Confirms valid tactic proofs, kernel decide, compiler-realized equation lemmas (Erdos349.IsGoodPair.eq_1), clean shared helpers in OpenProofs/Lib/, valid statement files, and the unmodified workspace all pass. |
4. Negation Proofs for Missing-Parameter Value Questions (#7590)
142 claims pass the checker in open-proofs against pinned formal-conjectures commit 96343eeb5990: 131 proved targets (129 upstream textbook, test, API, and already-solved research statements whose upstream declarations still carried sorry, plus 2 self-recorded special cases of Erdős 828) and 11 negation certificates across 9 Erdős problems (142, 319, 321, 361, 409, 450, 503, 552, and 692).
Those 11 certificates do not represent a new bug class. Earlier contributors to google-deepmind/formal-conjectures had already reported the same formalization pattern in issues #7327, #7434 (covering erdos_409.parts.i), and #7493 (covering erdos_450 and erdos_552): an open value question quantifies over a parameter (k, N, c, or n), but types its answer(...) placeholder as a single constant rather than a function of that parameter, making the formal statement false for every possible replacement of sorry.
In open-proofs, agents write candidate notes to outbox/—which holds local draft notes (including 10 per-problem drafts, one of which, Erdős 506, is only a conditional argument with no result file) and never submits anything externally. Excluding the three declarations already covered in #7434 and #7493, I filed one upstream issue on October 24, 2026—google-deepmind/formal-conjectures#7590 (with explicit disclosure that Claude Code found the instances and wrote the Lean proofs)—covering the remaining 8 declarations across 7 Erdős problems that were unchanged on upstream main (c094b97) at that time. Separate upstream PRs later parameterized four of the eight declarations independently (#7946 on 2026-11-13 and #8015 on 2026-11-16). On 2026-11-18, I posted a follow-up comment on #7590 (re-checked against upstream b576d52 with Codex, disclosed in the comment) tracking the four remaining declarations; those four are also covered by open upstream PR #7990 (linked to #7977):
| Upstream Declaration | Missing Parameter | Why No Constant Answer Works | Upstream Status (2026-11-18) |
|---|---|---|---|
erdos_142, erdos_142.variants.isTheta |
k |
r 0 N = N and r 2 N ≤ 1, so no single function is asymptotic (or Θ) to both. |
Unchanged on main (b576d52); open PR #7990 |
erdos_319 |
N |
Maximum is at most 1 for N = 1, and at least 3 for N = 6 via {2, 3, 6} (1/2 - 1/3 - 1/6 = 0). |
Unchanged on main (b576d52); open PR #7990 |
erdos_321 |
N |
R 0 = 0 and R 1 ≥ 1. |
Unchanged on main (b576d52); open PR #7990 |
erdos_361 |
c |
At n = 2, the value is 0 for c = 1/4 and 1 for c = 1. |
Updated independently in #8015 |
erdos_409.variants.sigma |
n |
Least count is 0 for n = 2 and positive for n = 6 (σ 6 - 1 = 11 is prime, so hterm holds). |
Updated independently in #8015 |
erdos_503 |
n |
In ℝ^0 the maximum is at most 1; in ℝ^1 it is at least 2. |
Updated independently in #7946 |
erdos_692.parts.ii |
n |
Taking n equal to the answer m gives δ₁(m, m) = 0, whereas δ₁(m, m + 2) = 1/(m + 1) > 0. |
Updated independently in #8015 |
Because referencing the upstream answer placeholder would pull sorryAx into Layer 4's axiom walk, each certificate records and proves the answer-free universal negation using only propext, Classical.choice, and Quot.sound—for example, lines 16–20 of P319_no_constant_answer.lean:
open scoped Finset in
theorem erdos_319.no_constant_answer : ∀ a : ℕ, ¬ ∀ N : ℕ, IsGreatest
{ #A | (A) (_ : A ⊆ Finset.Icc 1 N)
(_ : ∃ δ : ℕ → ℤˣ, ∑ n ∈ A, (δ n : ℚ) / n = 0 ∧
∀ A' ⊂ A, A'.Nonempty → ∑ n ∈ A', (δ n : ℚ) / n ≠ 0) } a := by
5. Scope & Known Limitations
- Formal statement fidelity vs. informal intent: Passing all five layers verifies that the candidate proof term inhabits the elaborated Lean
Exprof the reference theorem under the axioms{propext, Classical.choice, Quot.sound}. It does not check whether an upstream formalization faithfully captures the informal mathematical problem—which is whyledger/LEDGER.mdseparates checker verdicts from claim assessments. - Trusted computing base (
TRUSTED_BASE): Verdicts depend on the integrity ofTRUSTED_BASEhashed inscripts/check.py—lakefile.toml,lean-toolchain(v4.33.1),lake-manifest.json,upstream.json,OpenProofsChecker/Main.lean,scripts/check.py, andscripts/lint.py—along with the Lean 4 kernel,leanchecker, Python 3, git, and the pinnedformal-conjectures(96343ee) andMathlib(0df444a) revisions. Soundness bugs inside the Lean kernel or a compromised local toolchain binary are outsideopcheck's scope. - Checker ≠ OS sandbox: Although
integrityrejectslakefile.leanand plugins andlintrejectsrun_cmd,#eval, andLean.Elabimports beforelake buildruns,scripts/check.pyis an unverified Python driver, not an OS or VM sandbox. Unattended prover processes should still run inside filesystem- and network-isolated containers. - AI authorship & mathematical scope: Both the proof scripts and the checker tooling in
open-proofswere written with Claude Code and Codex. As stated inopen-proofs/README.md, no open mathematical problem has been solved here: the 131 proved claims are textbook results, test/API statements, and previously solved research variants whose upstream Lean declarations were stillsorry, alongside 11 negation certificates for missing-parameter value questions.
6. Reproduce
From open-proofs/README.md:
git clone https://github.com/KilianTheMagnificent/open-proofs.git && cd open-proofs
lake exe cache get
lake build
python3 scripts/check.py --all # re-check every claim
python3 tests/run_adversarial.py # checker self-tests