CA Grokking audited by Beam 1.0 501B (clean conditions)
Same paper, no human feedback, no reviewer references: Reflection Beam 1.0 501B as Lemma's auditor. 2/6 supported (L1 vs L2 grokking probability, bimodal grokking times), C1/C2 honestly falsified on Beam's own implementations (wrong exponents, passing controls), C4/C5 inconclusive. Part of the Build-with-Beam entry and the Beam Audit Bench.
3supported
1falsified
2inconclusive
0not audited
15model calls
125tool runs
1.9wall min
1failures kept
claims
6 audited · click a claim to open the auditor's own summary — metrics, notes, control status
auditor's noteShifted training distribution (mu=2*epsilon) with local fit over 1e-3 <= t_eps-t <= 0.1.
nu1.004
t_eps1.027
n_fit_points4822
delta_min0.001006
delta_max0.09998
w_at_crossing0.1148
dzdt_at_crossing-0.07402
mu2.000
lambda10.5000
lambda20.000
epsilon1.000
control_slope1.000
control_passtrue
auditor's noteD-dimensional uniform ball model with nontrivial shift; population-trained perceptron dynamics; test error from spherical-cap volume; log-log exponent fit.
control_passtrue
control_nu2.700
D2_t_eps1.048
D2_nu2.004
D2_nu_pred1.500
D2_nu_abs_err0.5037
D2_n_points104.0
D2_E_final0.000
D5_t_eps0.9962
D5_nu3.858
D5_nu_pred3.000
D5_nu_abs_err0.8583
D5_n_points99.00
D5_E_final0.000
D10_t_eps0.9717
D10_nu6.782
D10_nu_pred5.500
D10_nu_abs_err1.282
D10_n_points97.00
D10_E_final0.000
auditor's noteEmpirical grokking probability with L1 exceeds that with L2 for the tested parameters.
p_grok_l10.1367
p_grok_l20.000
n_trials300.0
D10.00
N20.00
epsilon1.010
lambda20.1000
ratio_l1_over_l21.367e+11
control_p_grok_l10.1500
control_p_grok_l20.000
control_passtrue
auditor's noteRidge-regression fixed point for lambda1=0; uniform balls centered at +/-epsilon with radius 1; grokking iff w1>0 and epsilon*w1 - ||w|| >= |b|.
claim_idC4
trials_per_D1.000e+5
D_values[5,10,20,50]
count_D5145.0
count_D102.000
count_D200.000
count_D500.000
P_D50.001450
P_D100.00002000
P_D200.000
P_D500.000
expected_successes_D5145.0
expected_successes_D102.000
expected_successes_D200.000
expected_successes_D500.000
CI95_D5[0.0012325393507730022,0.0017057623637835467]
CI95_D10[0.000005484738516524055,0.00007292683754201227]
CI95_D20[0,0.00003841311258303963]
CI95_D50[0,0.00003841311258303963]
monotonic_decreasefalse
nonzero_dimensions2.000
min_count0.000
log_slope—
log_intercept—
log_r2—
control_passtrue
control_r21.000
control_slope-0.1000
auditor's noteUniform MPS tensor-network student trained with full-batch GD and L2 weight decay. Control is parity learning; main is Rule-30 local-rule learning.
control_passfalse
main_passfalse
main_first_phase_epoch—
main_num_phase_checkpoints0.000
main_transition_epoch—
main_min_train_error0.4933
main_max_test_error_during_phase—
main_min_test_error0.5050
main_final_train_error0.4933
main_final_test_error0.5050
control_first_phase_epoch—
control_num_phase_checkpoints0.000
control_transition_epoch—
control_min_train_error0.5350
control_max_test_error_during_phase—
control_min_test_error0.5340
control_final_train_error0.5350
control_final_test_error0.5340
auditor's noteSimulated D-dimensional ball model reduction; fast basin gives continuous small t_G, slow basin gives Dirac-like peak at analytic slow time. Positive control is a synthetic mixture with known bimodality.
n_trials1.000e+4
n_grokked1.000e+4
frac_slow_cluster0.6649
fast_mean0.4091
fast_std0.1802
slow_time_analytic1.705
lambda_slow0.01893
control_passtrue
bimodal_detectedtrue
why the judge trusts this trail
The verdict is bound to the evidence trail, not the outcome — five dimensions, all checked.
5/5PASS
structureclaims.json (6 claims), audit_report.json present
evidence6 audited claims; issues: none
integrity0 claims skipped as untestable, all declared untestable at extraction
cost disclosureper-claim attempt counts recorded; wall time in run files + trace
trace228 events, 15 LLM calls, 125 tool runs, 1 failed attempts preserved
figure evidence
the trail
15 model calls · 125 tool runs · 1.9 min wall · append-only, failures preserved
0:00