Modular Pretraining Enables Access Control (GRAM)
GRAM adds gradient-routed auxiliary MLPs to a base model so capabilities can be isolated and selectively disabled. Reproduced the 26M-param Simple Stories experiment on Modal A10G; compute-ratio evaluation follows the paper's Appendix-M power-law inversion.
2supported
0falsified
0inconclusive
4not audited
2model calls
37tool runs
225.1wall min
3failures kept
claims
2 audited · click a claim to open the auditor's own summary — metrics, notes, control status
auditor's noteFull 2.1M-sample Simple Stories reproduction (train split -> test-split eval), 1 epoch, 16,528 steps, batch 128, seq 256, A10G. Reimplemented from paper text (no public code): 26M decoder-only Transformer, GRAM = core MLP 1856 + 4x aux MLP 192 (1.58M/aux ~= 6% of baseline, matches paper), p_af=1.0, p_as=0.3, p_cr=0.5, heterogeneous accumulation, AdamW b=0.9/0.95 wd=0.1, LR 5e-3 WSD 10/80/10, bf16. Compute ratios via Appendix-M power-law inversion of the baseline learning curve. RESULT: core 0.975 >= 0.9 PASS; retain 0.976 within 0.014 of paper filtering 0.962 (threshold ~0.02) PASS; forget 0.789 well below retain (gap 0.187, matching paper gap 0.186) PASS; elicit 0.836 ~ paper 0.855 with partial-recovery pattern (elicit > forget) reproduced. Training wall time GRAM/baseline = 1.003 (cost independence corroborates C6). CAVEATS: single seed (paper: 3, pooled); data-filtering baseline not re-trained, so retain-vs-filtering uses the paper's reported filtering numbers; text-only reimplementation may differ in unreported details. Per-category: aliens retained 1.002 (paper 0.99), deadline forgotten hardest core-only 0.606 (paper 0.61 example).
core_compute_ratio0.9748
retain_compute_ratio_mean0.9757
forget_compute_ratio_mean0.7890
elicit_compute_ratio_mean0.8360
retain_vs_paper_filtering_abs_diff0.01370
retain_minus_forget_gap0.1867
core_ratio_ge_0.9true
retain_within_0.02_of_filteringtrue
forget_well_below_retaintrue
paper_gram_core0.9380
paper_gram_retain0.9520
paper_gram_forget0.7660
paper_gram_elicit0.8550
paper_filtering_core0.9610
paper_filtering_retain0.9620
paper_filtering_forget0.7800
paper_filtering_elicit0.8700
per_cat_retain{"a deadline or time limit":0.9847,"alien encounters":1.0023,"bygone eras":0.9641,"cultural traditions":0.9517}
per_cat_forget_core_only{"a deadline or time limit":0.6058,"alien encounters":0.7151,"bygone eras":0.942,"cultural traditions":0.8852}
baseline_params2.745e+7
gram_total_params3.219e+7
gram_aux_params_each1.578e+6
total_steps1.653e+4
baseline_wall_s3839
gram_wall_s3853
gram_over_baseline_walltime_ratio1.003
seeds1.000
gpuModal A10G
train_splittrain
eval_splittest
no per-claim summary in this registry snapshot — verdict preserved from the prior run.
no per-claim summary in this registry snapshot — verdict preserved from the prior run.
no per-claim summary in this registry snapshot — verdict preserved from the prior run.
no per-claim summary in this registry snapshot — verdict preserved from the prior run.
auditor's noteAnalytic 6*N*T FLOPS model with paper hyperparameters (80/20 core/aux token mixture, p_cr=0.5, 4 aux modules at 0.1x core MLP each => 1.4x total MLP). GRAM trains on the same token count as baseline (aux oversampling offset by core undersampling), so its FLOPS ratio is just the active-parameter ratio: 1.0400x (within 5% of 1.0). Filtering needs 5 full runs = 5.0x, i.e. 4.81x savings for GRAM. Active MLP params per served profile = 1.10x <= 1.1x. Positive control (zero-overhead degenerate case) returns exactly 1.0x/5.0x as required.
gram_train_flops_ratio_vs_baseline1.040
filtering_train_flops_ratio_vs_baseline5.000
filtering_over_gram_savings4.808
gram_total_mlp_params_ratio1.400
gram_active_mlp_params_per_profile_ratio1.100
gram_active_total_params_per_profile_ratio1.067
prob_aux_module_active_per_token0.6000
control_passtrue
control_gram_ratio_expected_1.01.000
control_filter_ratio_expected_5.05.000
why the judge trusts this trail
The verdict is bound to the evidence trail, not the outcome — five dimensions, all checked.
5/5PASS
structureclaims.json (6 claims), audit_report.json present
evidence2 audited claims; issues: none
integrity0 claims skipped as untestable, all declared untestable at extraction
cost disclosureper-claim attempt counts recorded; wall time in run files + trace
trace29 events, 2 LLM calls, 7 tool runs, 0 failed attempts preserved
figure evidence
the trail
2 model calls · 37 tool runs · 225.1 min wall · append-only, failures preserved
0:00