reflection × hugging face · build with beam · nov 2026
Beam wins the auditability axis: 4/4 episodes against a closed frontier model's 0/4
We pointed Lemma — the AI scientist that distrusts itself — at Beam 1.0 501B and built a benchmark, an RL environment with verifiable rewards, a self-improvement loop, and a long-context ablation. Every attempt is published, including the crashes.
1 · the four-way ladder
Keyed tasks in the RL environment — reference answer key recovered, positive control passed, verdict supported. Deterministic grader, no LLM judge. 20 episodes, all published; the 27B-class model only reached 1.0 after the environment told it the positive control was missing.
2 · opus's 0/4 is a contract failure — the gallery
Opus 5.5 computed the physics but never reported it under a metric name the grader's alias list accepts, or declined to compute at all. Verbatim from its published scripts:
| what opus reported | what the key needed |
|---|---|
| median_nu, mean_nu, nu_se_bootstrap | nu / nu_fitted / exponent |
| claim_quantity_computed: false → inconclusive | an actual computed metric |
| rel_diff: "not_computed" (attempt 3) | rel_diff as a number |
| 6 C1 attempts, 0 graded alias hits | one graded metric per attempt |
Its single 1.0 is the harder CA C2 task, recovered from the 20k excerpt alone. The capability was there; the auditability contract wasn't. Reporting what you measured — under the contracted names, with the mandatory control — is part of the skill.
3 · the bake-off gate
| model | ν (target 1.0) | note |
|---|---|---|
| RunInfra Qwen 27B | ≈ 6779 | round-2 bake-off 2026-08-16 |
| DeepSeek V4 Pro | ≈ 6072 | round-2 bake-off 2026-08-16 |
| Orca Flash / Pro (27B) | crash / timeout | flash crashed / pro timed out, 2026-08-16 |
| Kimi K3 | 0.9911 | gate pass 2026-08-16, 92s |
| Beam 1.0 501B | 0.9974 | single generation after the 6000-budget trap |
4 · the self-improvement loop (zero human feedback)
verify_gain-flagged — any move toward “supported” after feedback is the reward-hacking direction, and its metrics are published next to the flag.5 · the 1M context, ablated
6 · every episode
| episode | task | attempts | best |
|---|---|---|---|
| Claude Opus 5.5 | CA C1 — 1D critical exponent (nu = 1.0) | 0.00 | |
| Claude Opus 5.5 | CA C1 — 1D critical exponent (nu = 1.0) | 0.00 | |
| Beam 1.0 501B | CA C1 — 1D critical exponent (nu = 1.0) | 1.0 @ att 1 | |
| Beam 1.0 501B | CA C1 — 1D critical exponent (nu = 1.0) | 1.0 @ att 3 | |
| Kimi K3 | CA C1 — 1D critical exponent (nu = 1.0) | 1.0 @ att 2 | |
| 27B-class (RunInfra) | CA C1 — 1D critical exponent (nu = 1.0) | 1.0 @ att 3 | |
| 27B-class (RunInfra) | CA C1 — 1D critical exponent (nu = 1.0) | 0.00 | |
| Claude Opus 5.5 | CA C2 — D-ball exponent series (nu = (D+1)/2) | 1.0 @ att 3 | |
| Claude Opus 5.5 | CA C2 — D-ball exponent series (nu = (D+1)/2) | 0.00 | |
| Claude Opus 5.5 · full paper | CA C2 — D-ball exponent series (nu = (D+1)/2) | 0.50 | |
| Claude Opus 5.5 · full paper | CA C2 — D-ball exponent series (nu = (D+1)/2) | 0.00 | |
| Beam 1.0 501B | CA C2 — D-ball exponent series (nu = (D+1)/2) | 0.00 | |
| Beam 1.0 501B · full paper | CA C2 — D-ball exponent series (nu = (D+1)/2) | 1.0 @ att 3 | |
| Claude Opus 5.5 | ICL C1 — Bayesian risk identity (rel_diff = 0) | 0.00 | |
| Claude Opus 5.5 | ICL C1 — Bayesian risk identity (rel_diff = 0) | 0.00 | |
| Beam 1.0 501B | ICL C1 — Bayesian risk identity (rel_diff = 0) | 1.0 @ att 1 | |
| Beam 1.0 501B | ICL C1 — Bayesian risk identity (rel_diff = 0) | 1.0 @ att 2 | |
| Kimi K3 | ICL C1 — Bayesian risk identity (rel_diff = 0) | 1.0 @ att 2 | |
| 27B-class (RunInfra) | ICL C1 — Bayesian risk identity (rel_diff = 0) | 1.0 @ att 3 | |
| 27B-class (RunInfra) | ICL C1 — Bayesian risk identity (rel_diff = 0) | 1.0 @ att 2 |
Each row links to a trajectory in the dataset (rl_env/) with the full script source of every attempt.