evidence over assertion
the AI scientist that distrusts itself
Machines are beginning to do science faster than anyone can check it. Lemma is the missing layer — a lab that turns claims into inspectable verdicts, keeps every failure, and distrusts itself first.
Recorded evidence · failures included
45 bars · one per extracted claim across 9 papers — click to play
Science is about to be written at machine speed — and nobody can verify that fast. Lemma's job is the missing layer: make checking a claim as cheap as making one, so every result in the literature can carry a verdict you can inspect, with the evidence attached.
Not a reviewer — an instrument. The lab extracts testable claims, writes and runs its own controlled numerical experiments, and snapshots every attempt immutably. A failed control readsinconclusive, never falsified — and an independent judge scores the trail itself.
Pointed at the high-dimensional regression literature, the lab audited 21 claims across three papers — and two published claims failed their own controlled tests. Between them, new science: an agent-generated hypothesis verified in seconds, and a measurement no paper reported — the asymptotic risk approximation is already within ~0.1% at d = 80.
9 papers, 59 claims, and a trail you can reconstruct line by line — including everything that failed.
the cost of honesty — every number was spent to find out, not to prove.
One literature is a start — not the destination. The goal is a verification map of science itself: every claim tracking its evidence, every failure preserved, every uncertainty stated in the open. Discovery accelerates the moment trust becomes cheap. That is the lab we are building.
the wider landscape
Science runs on claims, but claims are cheap. In a 2016 Nature survey of 1,576 researchers, 70% failed to reproduce another lab's experiment — and more thanhalf failed to reproduce their own.
Now the generators have arrived: AI scientists that draft papers and hypotheses faster than any human can read them. The bottleneck is no longer production — it is trust. Who checks the claims before they become literature?
papers & hypotheses, at machine speed
Sakana · Google co-scientist · agentic writing pipelines
answers mined from millions of papers
FutureHouse · Elicit · PaperQA-class agents
who is winning at reproducing papers
PaperBench · ICML Agent Reproductions · verifier harnesses
claims checked in code, failures kept on record
this pipeline · extract → audit → evidence → judge
Lemma is not another generator. It is the audit layer the new wave forgot — and it distrusts itself by design.
how an audit works
Success criteria come from the claims themselves — nothing is back-filled after the fact.
6 claims extracted · untestable ones declared, never silently skipped
The auditor writes a numpy script per claim, runs it, iterates at most 3×. A positive control is mandatory on every run.
5 supported · 1 honestly inconclusive · failures kept, never patched
Run outputs, figures, failed attempts and reviewer references assemble into a Trackio logbook — the trail is the product.
An automated judge checks structure, evidence, integrity, cost disclosure and trace completeness. The verdict is bound to the trail, not the outcome.
31 LLM calls · 199 tool runs · 39.4 min wall
Each stage re-runs idempotently — the trace records every attempt, including the failures.
Every LLM call, every script, every rejection lands in an append-only trace. Scroll — the audit writes itself.
extract → audit → evidence → judge · verdict validator · positive controls · reviewer-reference escalation
The real append-only traces from 9 completed runs — every model call, every script, every rejection and verdict flip. Scrub it, play it, watch a claim go from falsified to supported across rounds.
Want this on your paper?request an audit ↓
New-paper audits run on the local CLI — this form is a request line, not a hosted run. The recorded evidence is already live:enter the Audit Room, or run it yourself:
./lemma audit <arxiv-id> --jobs 2
Leave your email to request an audit or get notified when a hosted version opens — tell us which paper you'd check first.
the verification ledger — every paper, every claim, every failure on record
supported falsified inconclusive not yet tested— inconclusive is a result, not a failure to report one