audit registry
9 papers · 59 claims extracted · 29 supported ·10 inconclusive · 113 failures preserved ·6 evidence logbooks published
OpenReview 5nNNVY8NW4fixture
To Grok Grokking: Provable Grokking in Ridge Regression
Theory paper on provable grokking in ridge regression. Hand-driven reproduction that passed the ICML 2026 Agent Reproductions judge end to end — now the pipeline's regression fixture.
supportedsupportedsupportedsupportedsupportedsupported
arXiv 1903.08560corpus
Surprises in High-Dimensional Ridgeless Least Squares Interpolation
Exact asymptotic risk formulas for the min-norm interpolator in both regimes — the canonical companion to the double-descent paper. The lab verified its underparameterized risk blow-up (289x at gamma near 1) and its optimal-ridge comparison.
inconclusivenot auditednot auditednot auditedsupportedsupported
arXiv 1906.11300corpus
Benign Overfitting in Linear Regression
Effective-rank characterization of when interpolation generalizes. The lab found boundary cases its eigenvalue-rank inequality chain does not cover — two falsifications with passing controls.
falsifiednot auditednot auditednot auditednot auditedfalsified
arXiv 1912.07242audit
More Data Can Hurt for Linear Regression: Sample-wise Double Descent
First fully agent-orchestrated audit: Omnigent lab director + specialist agents verified the double-descent peak at n=d, found the risk valley at gamma=0.90 via an agent-generated hypothesis, and routed an unverified follow-up through a human approval gate.
supportedsupportedsupportedfalsifiedsupportedinconclusivesupportedfalsifiedinconclusive
arXiv 2510.10981audit
In-Context Learning Is Provably Bayesian Inference
ICL-as-Bayesian-inference theory: the risk-decomposition identity and coupled p-N scaling of the Bayes gap verified numerically; the posterior-variance bound honestly inconclusive under finite precision.
supportedfalsifiedsupportedsupportedinconclusiveinconclusivefalsified
arXiv 2510.10981beam
ICL Is Provably Bayesian Inference, audited by Beam 1.0 501B (clean conditions)
The headline flip: two claims that stayed inconclusive through 7 recorded rounds of 27B-class models (Bayes-Gap/context coupling C2, distribution-shift stability C5) came back supported under Beam on clean conditions, and the risk identity C1 was recovered in one attempt (rel_diff 0.0008). C3/C4 honestly falsified on Beam's own implementations; C6/C7 inconclusive with failing controls flagged.
supportedsupportedfalsifiedfalsifiedsupportedinconclusiveinconclusive
arXiv 2607.08077audit
Modular Pretraining Enables Access Control (GRAM)
GRAM adds gradient-routed auxiliary MLPs to a base model so capabilities can be isolated and selectively disabled. Reproduced the 26M-param Simple Stories experiment on Modal A10G; compute-ratio evaluation follows the paper's Appendix-M power-law inversion.
supportednot auditednot auditednot auditednot auditedsupported
JMLR 22-1228audit
Grokking phase transitions in learning local rules with gradient descent
Grokking in cellular-automaton rule learning: critical exponents, grokking-probability trends, and a bimodal grokking-time distribution. Audited end to end on CPU.
supportedsupportedsupportedsupportedinconclusivesupported
JMLR 22-1228 / arXiv 2210.15435beam
CA Grokking audited by Beam 1.0 501B (clean conditions)
Same paper, no human feedback, no reviewer references: Reflection Beam 1.0 501B as Lemma's auditor. 2/6 supported (L1 vs L2 grokking probability, bimodal grokking times), C1/C2 honestly falsified on Beam's own implementations (wrong exponents, passing controls), C4/C5 inconclusive. Part of the Build-with-Beam entry and the Beam Audit Bench.
supportedfalsifiedsupportedinconclusiveinconclusivesupported
every claim, one table
59 claims across 9 papers · filter by verdict, search by title · each row opens the paper's claim dossier
paperclaimverdictattemptsauditor's note
JMLR 22-1228C1 Critical exponent in 1D exponential modelsupported1Fitted critical exponent nu=0.998 within 0.05 of 1.0 (Eq. 9/10).JMLR 22-1228C2 Critical exponent in D-dimensional uniform ball modelsupported1Closed-form verification of Eq 23: critical exponent nu=(D+1)/2 in the D-dimensional uniform-ball model, using the paper's exact l…JMLR 22-1228C3 Grokking probability increases with L1 regularisationsupported1Hard regime D=10, eps=1.01: P_grok(L1)=0.205 vs P_grok(L2)=0.000; criterion (>=2x or >0.5 vs <0.1) met.JMLR 22-1228C4 Grokking probability decreases with dimension Dsupported1Mode: common eps=1.005. P_grok(D) at chosen eps: [0.7295, 0.176, 0.032, 0.0125] over D=[2, 5, 10, 20]. Strictly decreasing: True, …JMLR 22-1228C5 Rule-30 CA learning exhibits grokkinginconclusive1The perceptron-map proxy trivializes Rule-30 (8-triplet lookup: zero generalization gap by construction). The paper's claim concer…JMLR 22-1228C6 Grokking time distribution bimodalitysupported1Closed-form verification of Sec 3.3.3 (Eqs 40-47, Fig 9 params D=5, eps=2, lambda2=0.01): Monte Carlo over Gaussian initial condit…JMLR 22-1228 / arXiv 2210.15435C1 Critical exponent in 1D exponential modelsupported1Shifted training distribution (mu=2*epsilon) with local fit over 1e-3 <= t_eps-t <= 0.1.JMLR 22-1228 / arXiv 2210.15435C2 Critical exponent in D-dimensional uniform ball modelfalsified1D-dimensional uniform ball model with nontrivial shift; population-trained perceptron dynamics; test error from spherical-cap volu…JMLR 22-1228 / arXiv 2210.15435C3 Grokking probability increases with L1 regularisationsupported1Empirical grokking probability with L1 exceeds that with L2 for the tested parameters.JMLR 22-1228 / arXiv 2210.15435C4 Grokking probability decreases with dimension Dinconclusive1Ridge-regression fixed point for lambda1=0; uniform balls centered at +/-epsilon with radius 1; grokking iff w1>0 and epsilon*w1 -…JMLR 22-1228 / arXiv 2210.15435C5 Rule-30 CA learning exhibits grokkinginconclusive1Uniform MPS tensor-network student trained with full-batch GD and L2 weight decay. Control is parity learning; main is Rule-30 loc…JMLR 22-1228 / arXiv 2210.15435C6 Grokking time distribution bimodalitysupported1Simulated D-dimensional ball model reduction; fast basin gives continuous small t_G, slow basin gives Dirac-like peak at analytic …OpenReview 5nNNVY8NW4C1 Theorem 4.1 — end-to-end grokking (zero teacher)supported1Zero-teacher audit (Theorem 4.1): t1=208, grokking at 7792 steps, within the theory's worst-case t2>=9517. All three per-step enve…OpenReview 5nNNVY8NW4C2 Theorem 4.2 — grokking with realizable teachersupported1Realizable-teacher audit (Theorem 4.2 / Eq. 8): grokking at 7796 steps, zero substantive violations.OpenReview 5nNNVY8NW4C3 Theorems 4.4-4.6 — phase decompositionsupported1Decomposition into training-convergence / poor-generalization / eventual-generalization phases matches Theorems 4.4-4.6 envelopes …OpenReview 5nNNVY8NW4C4 Figure 2 — hyperparameter dependence of grokking timesupported1Figure-2 sweep: lambda-down amplifies t2 (~1/lambda), n-up grows t1, nu^2-up grows t1 logarithmically, m has minor effect. Matches…OpenReview 5nNNVY8NW4C5 Figures 3-4 — two-layer ReLU grokking beyond the linear settingsupported1Two-layer ReLU experiments (Figures 3 & 4): clear grokking in the demo run; hyperparameter sweeps show the predicted qualitative d…OpenReview 5nNNVY8NW4FALS supported1Falsification attempt: condition relaxations were run to find a setting where the paper's prediction breaks; the negative control …arXiv 1903.08560C1 Isotropic Min-Norm Risk Formulainconclusive1Positive control failed. The risk calculation or simulation setup may be flawed.arXiv 1903.08560C2 Double Descent in Misspecified Modelnot audited——arXiv 1903.08560C3 Latent Space Model Monotonicitynot audited——arXiv 1903.08560C4 Nonlinear Model Universalitynot audited——arXiv 1903.08560C5 Ridge Dominates Min-Normsupported1Optimally-tuned ridge regression achieved lower average risk than min-norm LS for all tested gamma and SNR values in both well-spe…arXiv 1903.08560C6 Underparametrized Risk Divergencesupported1Risk at gamma=0.999 is 2396.0423, at gamma=0.9 is 8.2868. Ratio: 289.14. Monotonic: True. Control passed: True.arXiv 1906.11300C1 Benign Overfitting Condition for Power-Law Eigenvaluesfalsified1Conditions not met. Cond1 (Benign low): False, Cond2 (Non-benign high): False, Cond3 (Non-benign high): False.arXiv 1906.11300C2 Benign Overfitting in Finite-Dimensional Space with Isotropic Noisenot audited——arXiv 1906.11300C3 Upper Bound on Excess Risknot audited——arXiv 1906.11300C4 Lower Bound on Excess Risknot audited——arXiv 1906.11300C5 Necessity of Overparameterizationnot audited——arXiv 1906.11300C6 Relationship Between Effective Ranksfalsified14 violations found out of 36 checks.arXiv 1912.07242C1 Sample-wise non-monotonic test MSE peaks at n=dsupported1Peak at n=200 (d=200), within 10% of d. Non-monotonicity confirmed. Control passed.arXiv 1912.07242C2 Overparameterized risk approximation (Claim 1)supported3All relative errors within 20%: B_max=0.0000, V_max=0.0176, R_max=0.0129. Risk increase not clearly detected in last 3 points, but…arXiv 1912.07242C3 Underparameterized risk approximation (Claim 2)supported1Claim 2 supported: bias is numerically zero (max rel 0.001011 < 0.01) and variance agrees with sigma^2/(gamma-1) within 20% (max r…arXiv 1912.07242C4 Exact finite-sample bias-variance decomposition (Lemma 1)falsified1Verified Lemma 1 bias-variance decomposition for d=60, sigma=0.1. Control test passed: True. All relative errors within 5%: False.…arXiv 1912.07242C5 Exact trace update when adding one sample (Claim 3)supported1All 25 instances satisfy rel diff < 1e-8. Max rel diff = 3.76e-16. Control passed.arXiv 1912.07242C6 Bias is non-increasing with more samplesinconclusive1Monte Carlo bias estimates show 0 monotonicity violations (tol=0.01). Control test against theory passed: False. Bias is non-incre…arXiv 1912.07242C7 Finite-d risk valley: the overparameterized double-descent dip at gamma*=0.90 (generated variant of C2)supported1Valley at gamma=0.9, strict_min=True, rel_err=0.0411. All criteria met.arXiv 1912.07242C8 Finite-d bias exactness: E[beta_hat] = gamma*beta at finite d (repairs C2's circular bias check)falsified1One or more gamma values failed the success criteria.arXiv 1912.07242C9 Finite-d correction of the C2 overparam risk approximation: power-law decay d^-alpha at fixed gamma=0.5 (generated variant of C2)inconclusive2T=16000, d_list=[80, 150, 300]. alpha=0.3048, R2=0.2987, control_pass=False. Retried with T=16000 due to se_e violation.arXiv 2510.10981C1 Risk Decomposition Identitysupported1Closed-form verification of Proposition 3.1 (R = RBG + RPV) on a two-family linear-task mixture (families distinguished by an inte…arXiv 2510.10981C2 Coupled p-N Scaling of Bayes Gapfalsified3Joint R2=0.629, N-only R2=0.095, p-only R2=0.618. Criterion requires joint>0.8 and >0.1 above both single-variable models.arXiv 2510.10981C3 Rapid Task-Type Identification (Posterior Concentration)supported4Closed-form verification of the Theorem-3.3 mechanism on the paper's task-mixture class (linear vs degree-2 Hermite families, Defi…arXiv 2510.10981C4 Bayes Gap Upper Bound Ratesupported1pN slope=-0.826, R2=0.995. m-fit R2=0.985, min_in_grid=True. Control pass=True.arXiv 2510.10981C5 Wasserstein Stability of Bayes Gapinconclusive4script failed after 4 triesarXiv 2510.10981C6 Posterior Variance Bound by Minimax Riskinconclusive3Positive control failed; estimator unreliable.arXiv 2510.10981C7 Boundary test: pN invariance of the Bayes Gapfalsified1Claim falsified. Spearman rho=0.12, Mean CV=0.58.arXiv 2510.10981C1 Risk Decomposition Identitysupported1Empirical squared-loss decomposition over a linear-vs-quadratic task mixture; Bayes predictor is the analytic posterior mean. Posi…arXiv 2510.10981C2 Coupled p-N Scaling of Bayes Gapsupported2Simplified mean-pooling linear model trained by least squares on synthetic linear-regression prompts; Bayes Gap computed against e…arXiv 2510.10981C3 Rapid Task-Type Identification (Posterior Concentration)falsified1Audited the Bayes-optimal/sufficiently-pretrained limit: mixture posterior mean versus oracle posterior mean for linear vs quadrat…arXiv 2510.10981C4 Bayes Gap Upper Bound Ratefalsified1Synthetic mixture of two Gaussian linear-regression task types; uniform-attention Transformer trained by ERM; Bayes Gap estimated …arXiv 2510.10981C5 Wasserstein Stability of Bayes Gapsupported1Monte Carlo Bayesian linear regression; model fixed at 0; W1 estimated empirically between input distributions.arXiv 2510.10981C6 Posterior Variance Bound by Minimax Riskinconclusive1Two-task scalar linear mixture; RPV by posterior over task/sign hypotheses; minimax risk via chi-square expectation; excess fitted…arXiv 2510.10981C7 Boundary test: pN invariance of the Bayes Gapinconclusive1Constant-mean meta-learning task; meta-learner is least squares on all pN prefix examples using mean-pooled context. Bayes Gap = o…arXiv 2607.08077C1 GRAM approximates multiple data-filtered models in a single run (Simple Stories)supported1Full 2.1M-sample Simple Stories reproduction (train split -> test-split eval), 1 epoch, 16,528 steps, batch 128, seq 256, A10G. Re…arXiv 2607.08077C2 GRAM isolates realistic dual-use capabilities and matches data filtering at 800M scalenot audited——arXiv 2607.08077C3 Capability isolation improves with scale and GRAM tracks data filtering from 50M to 5Bnot audited——arXiv 2607.08077C4 GRAM composes arbitrary capability subsets without degradation, unlike FT-LoRAnot audited——arXiv 2607.08077C5 GRAM outperforms filtering and FT-LoRA on capability removal under partial labelingnot audited——arXiv 2607.08077C6 GRAM's training cost is independent of the number of capability profiles, yielding 5× savings over filteringsupported1Analytic 6*N*T FLOPS model with paper hyperparameters (80/20 core/aux token mixture, p_cr=0.5, 4 aux modules at 0.1x core MLP each…no claims match that filter + search