evidence over assertion

Lemma

the AI scientist that distrusts itself

Every claim gets a verdict. Every failure is preserved. Nothing is patched into a pass. This is its instrument: 20 bars, one per audited claim across 4 papers. Bright bars reproduced. Dim bars are honest inconclusives. Play it.

24 bars · one per extracted claim across 4 papers — click to play

the first life

Lemma was born for the ICML 2026 Agent Reproductions challenge: one grokking paper, audited end to end, judge PASS. Then it slept — a dormant repo with one good result on disk.

the night it failed

Reawakened on a JMLR paper about cellular automata, its first models returned every claim inconclusive. Not wrong — just not smart enough. We gave them a physics test anyway. Expected critical exponent 1.0:

ν ≈ 0

the turn

Most agents would have written a confident paragraph and moved on. Lemma's validator auto-rejects any verdict that contradicts its own metrics — so the fix was gated: no model swap until the replacement passed the same physics test. It scored0.99. Next round: 0.9983. The claims started falling.

the receipts

4 papers, 24 claims, and a trail you can reconstruct line by line — including everything that failed.

0papers

0claims

0reproduced

0failures preserved

The trail is the product

Every LLM call, every script, every rejection lands in an append-only trace. Scroll — the audit writes itself.

trace.jsonl — append-only

[note] lemma v0.1 · ICML 2026 agent reproductions challenge — judge PASS, end to end[note] repo dormant… 3 weeks, 0 commits, 1 grokking paper on disk[run] lemma audit jmlr-22-1228 --stages extract,audit,evidence,judge[extract] 6 claims → C1…C6 · success criteria taken from the claims themselves[audit] round 1 · Qwen3.8-27B · 0 supported / 0 falsified / 6 inconclusive[audit] bake-off gate · ν ≈ 6000 vs expected 1.0 — capability boundary detected, not papered over[run] lemma audit --endpoint kimi:k3 · bake-off first[audit] bake-off gate · ν = 0.9911 — PASS · model swap authorized[audit] round 3 · C1 supported · ν = 0.9983 vs 1.0 · closed-form cross-check OK[audit] rounds 4–5 · C3, C4, C5, C6 supported via reviewer-reference escalation[judge] evidence trail PASS 5/5 · 353 trace events · all failures preserved[run] lemma audit arxiv-2510.10981[judge] evidence trail PASS 5/5 · 267 events · 3 supported, 3 honestly inconclusive[note] inconclusive is a result — a patched false-positive is not

extract → audit → evidence → judge · verdict validator · positive controls · reviewer-reference escalation

Replay an audit, line by line

The real append-only traces from 4 completed runs — every model call, every script, every rejection and verdict flip. Scrub it, play it, watch a claim go from falsified to supported across rounds.

trace.jsonl — replaying live run

0:00

Want this on your paper?request an audit ↓

Run it on your paper

An interactive version is in the works: paste an arXiv id, watch the claims come out. Leave your email and we'll let you in when the gate opens — or tell us which paper you'd audit first.

★ — on GitHubno backend of our own yet — stars and signups are the queue

Everything is public

audited papersOpenReview 5nNNVY8NW46 supported / 0 inconclusive · judge PASS 5/5 arXiv 2510.109813 supported / 3 inconclusive · judge PASS 5/5 arXiv 2607.080772 supported / 0 inconclusive · judge PASS 5/5 JMLR 22-12285 supported / 1 inconclusive · judge PASS 5/5

evidence logbookslogbook · grokking-ridgeTrackio logbook — the inspectable evidence trail logbook · icl-bayesianTrackio logbook — the inspectable evidence trail logbook · gramTrackio logbook — the inspectable evidence trail logbook · grokking-caTrackio logbook — the inspectable evidence trail

raw evidencedataset · icl-bayesianevery run output + failed attempt included dataset · grokking-caevery run output + failed attempt included

coderepositorypipeline, traces, feedback, reviewer references

all papers →the full audit registry: claims, verdicts, figures, traces