Grokking phase transitions in learning local rules with gradient descent
Grokking in cellular-automaton rule learning: critical exponents, grokking-probability trends, and a bimodal grokking-time distribution. Audited end to end on CPU.
5supported
0falsified
1inconclusive
0not audited
31model calls
199tool runs
39.4wall min
20failures kept
claims
6 audited · click a claim to open the auditor's own summary — metrics, notes, control status
auditor's noteFitted critical exponent nu=0.998 within 0.05 of 1.0 (Eq. 9/10).
nu_fitted0.9983
nu_control1.000
control_passtrue
t_eps1.114
t_train_zero1.102
xbar-0.06639
xbar_fixed_point-0.1164
Emax0.4331
n_fit_points745.0
auditor's noteClosed-form verification of Eq 23: critical exponent nu=(D+1)/2 in the D-dimensional uniform-ball model, using the paper's exact large-N gradient flow (Eqs 20-21: dw/dt=-Gw+a with dataset-level G and a) on the Sec 3.3.1 shifted-ball dataset, test error via Eq 22 (cap integral by theta-quadrature), exponent fitted from log E_D vs log(1-h) in the asymptotic window h in (0.999, 1). Shift calibrated at 1.0. Reviewer-provided reference.
shift_used1.000
nu_D21.500
target_D21.500
r2_D21.000
nfit_D23358
nu_D53.000
target_D53.000
r2_D51.000
nfit_D51.021e+4
nu_D105.499
target_D105.500
r2_D101.000
nfit_D107890
control_passtrue
per_D{"D2":{"ok":true,"nu":1.4999612238147915,"r2":0.9999999998097971,"t_eps":0.15364574062711286,"train_err_at_t_eps":0.261,"n_fit":3358,"eps_used":0.9919094254623091,"Emax":0.009186612981473748},"D5":{"ok":true,"nu":2.9997950334784917,"r2":0.9999999974579918,"t_eps":9.792441352663412,"train_err_at_t_eps":0,"n_fit":10210,"eps_used":0.9937116084384998,"Emax":0.000011960407300541593},"D10":{"ok":true,"nu":5.4993439566100895,"r2":0.9999999451989837,"t_eps":45.03788404941575,"train_err_at_t_eps":0,"n_fit":7890,"eps_used":0.9997263240694412,"Emax":2.7439091038412737e-8}}
auditor's noteHard regime D=10, eps=1.01: P_grok(L1)=0.205 vs P_grok(L2)=0.000; criterion (>=2x or >0.5 vs <0.1) met.
P_grok_L10.2050
P_grok_L20.000
trials400.0
D10.00
eps1.010
err_L1_2sigma0.04037
err_L2_2sigma0.000
diff0.2050
diff_over_sigma10.16
control_passtrue
auditor's noteMode: common eps=1.005. P_grok(D) at chosen eps: [0.7295, 0.176, 0.032, 0.0125] over D=[2, 5, 10, 20]. Strictly decreasing: True, drop=0.717, log-linear slope=-0.2142879654255913, R2=0.876845082447202.
P_table{"D=2":{"eps=1.005":0.7295,"eps=1.01":0.864,"eps=1.02":0.973,"eps=1.05":0.9995,"eps=1.1":1,"eps=1.2":1},"D=5":{"eps=1.005":0.176,"eps=1.01":0.468,"eps=1.02":0.7965,"eps=1.05":0.992,"eps=1.1":1,"eps=1.2":1},"D=10":{"eps=1.005":0.032,"eps=1.01":0.1915,"eps=1.02":0.6005,"eps=1.05":0.98,"eps=1.1":1,"eps=1.2":1},"D=20":{"eps=1.005":0.0125,"eps=1.01":0.1345,"eps=1.02":0.6175,"eps=1.05":0.9915,"eps=1.1":1,"eps=1.2":1}}
chosen_eps_per_D{"2":1.005,"5":1.005,"10":1.005,"20":1.005}
chosen_P[0.7295,0.176,0.032,0.0125]
strictly_decreasingtrue
overall_drop0.7170
log_linear_slope-0.2143
log_linear_R20.8768
control_passtrue
selection_modecommon eps=1.005
auditor's noteThe perceptron-map proxy trivializes Rule-30 (8-triplet lookup: zero generalization gap by construction). The paper's claim concerns the over-parameterized tensor-network regime (gpu-small, out of CPU-audit scope). Verdict: inconclusive by compute-scope limitation - reproducing it requires tensor-network training on long sequences.
control_passfalse
control_final_train_err0.2500
control_final_test_err0.2500
n_unique_triplet_patterns8.000
generalization_gap_possiblefalse
auditor's noteClosed-form verification of Sec 3.3.3 (Eqs 40-47, Fig 9 params D=5, eps=2, lambda2=0.01): Monte Carlo over Gaussian initial conditions using the paper's zeroth-order large-N grokking-time PDF — continuous fast-relaxation part (Eq 42) + Dirac peak at t_slow (Eq 47) with weight 1-p_fast (Eq 45). All bimodality tests and the qualitative D/lambda_2 dependence pass. Reviewer-provided reference implementation.
valley_ratio0.000
check_valleytrue
frac_fast0.05384
frac_slow0.9462
separation_ratio3.984
check_separationtrue
slow_rel_err1.315e-16
check_slow_time1.000
sharpness_fast0.7191
sharpness_slow1.315e-16
check_sharpnesstrue
mean_fast0.05462
mean_slow0.2111
t_slow_analytic0.2111
p_fast_theory0.05498
p_fast_measured0.05384
qualitative{"p_fast_by_D":{"4":0.13861491959545832,"5":0.054977241662413245,"6":0.019887707187131058},"t_slow_by_D":{"4":0.18265619189878632,"5":0.21110731213224213,"6":0.23903155976878207},"t_slow_by_lambda2":{"0.0":0.2258848239814991,"0.01":0.21110731213224213,"0.05":0.16732209183814747},"p_fast_decreases_with_D":true,"t_slow_increases_with_D":true,"t_slow_increases_with_smaller_lambda2":true}
control_passtrue
control_bimodal_valley_oktrue
control_unimodal_valley_rejectedtrue
n_trials5.000e+4
D5.000
eps2.000
lambda20.01000
all_bimodality_checks[true,true,true,true]
all_qualitative_checks[true,true,true]
why the judge trusts this trail
The verdict is bound to the evidence trail, not the outcome — five dimensions, all checked.
5/5PASS
structureclaims.json (6 claims), audit_report.json present
evidence6 audited claims; issues: none
integrity0 claims skipped as untestable, all declared untestable at extraction
cost disclosureper-claim attempt counts recorded; wall time in run files + trace
trace353 events, 31 LLM calls, 199 tool runs, 9 failed attempts preserved
figure evidence
the trail
31 model calls · 199 tool runs · 39.4 min wall · append-only, failures preserved
0:00