{"id":"6e83eb15-9fc8-40e8-8bd9-b6117061260f","arxiv_id":"2602.00942","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SALAAD uses an ADMM-style optimizer and an adaptive controller to force LLM weights into sparse-plus-low-rank form during pretraining, so one checkpoint can be elastically deployed at many parameter budgets.","lead":"This paper trains LLMs so that each weight matrix splits into a low-rank part and a sparse part during training, letting a single checkpoint be shrunk to many memory budgets after training without retraining. A per-layer control loop sets the rank/sparsity trade-off automatically, and a greedy allocator removes the least important pieces at deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Elastic-deployment evidence (Fig. 2) may reflect per-budget oracle selection of κ rather than a fixed deployment rule; single-run curves lack error bars.","rationale":"The reader's verdict is CONDITIONAL and identifies the undisclosed rho scaling constant as the weakest assumption. I agree that this is a legitimate reproducibility and robustness concern, and the reader's recommendation to report the constant, add seeds, and clarify κ selection is sensible. However, in stress-testing the central claim itself—that one checkpoint supports a continuous spectrum of capacities—I find a more direct vulnerability in the evidence presented for that claim, namely Figure 2. The paper's own ablations (Figure 3) show that κ was swept per model and budget, and Table 1 reports different κ values across scales. Without an explicit statement of how κ was selected for Figure 2, the smooth degradation curve may be an artifact of per-budget oracle tuning. This is not a question of arithmetic or an undisclosed hyperparameter value; it is a question of whether the empirical centerpiece of the elastic-deployment claim can be reproduced under a fixed, a priori deployment rule. If a fixed κ yields similar smooth curves, the central claim holds and the rho concern becomes more prominent; if not, the headline claim needs to be weakened. I therefore recommend the paper address this by reporting fixed-κ curves with error bars, but do not recommend changing the conditional acceptance status based on this concern alone, as the paper could still meet its claims after this additional analysis.","tokens_in":20877,"tokens_out":11392,"duration_ms":132778,"concrete_test":"Re-generate the Figure 2 curves with a single fixed κ (e.g., κ = 0.7, which lies in the reported 'gray region') applied uniformly across all budgets and model scales, using at least 3 independent training seeds. Report mean and standard deviation of perplexity at each budget. Additionally, compute the oracle-κ curve by tuning κ on a held-out validation split and compare the two: if the fixed-κ curves are non-monotone or the performance gap to the oracle-κ curve is large, then the smooth elastic-deployment claim is not established; if they overlap, the concern is resolved. The same test should be run for the vanilla RPCA+HPA baseline to ensure the comparison remains at equal κ-selection fairness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'a single training run yields a continuous spectrum of model capacities' is supported almost entirely by Figure 2, which plots perplexity versus parameter count for SALAAD models adapted via HPA. The paper never states how κ is chosen for the plotted curves. Section 5.2 (Figure 3) sweeps κ for each model size and budget, reporting that the optimal κ lies in a narrow 'gray region,' and Table 1 uses different κ values per scale (0.7, 0.6, 0.6, 0.8). If the Figure 2 points use the best κ on the same data used to evaluate perplexity, then the smooth monotone curve is an oracle upper envelope, not a deployable single-rule curve. A user facing a fixed memory budget cannot reproduce the shown perplexity without knowing how to set κ. Moreover, no seeds or error bars are reported anywhere in the paper, and the HPA importance proxy I(u) ∝ |u| (§4.3) is never validated against actual per-unit importance. This directly threatens the load-bearing assertion of smooth, elastic deployment from one checkpoint. The rho-scaling concern raised in the reader's verdict is also valid, but it is secondary: even a perfect ρ would not establish the central claim if the capacity–perplexity curve is generated by per-budget oracle tuning of κ.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SALAAD, a training-time framework that induces sparse and low-rank structure in LLM weights by solving a regularized decomposition X = L + S through a two-stage stochastic ADMM variant. An integral controller adaptively adjusts the nuclear-norm and ℓ1 penalties per block so that only one penalty coefficient ρ needs tuning; after training, a homomorphic parameter allocation (HPA) strategy removes parameters from the SLR surrogate to obtain a continuous capacity–perplexity trade-off from a single checkpoint. Experiments on LLaMA-style models from 60M to 1B report perplexity and parameter counts for the dense model X, the structured surrogate L+S, and HPA-compressed variants, and claim that SALAAD matches or beats prior compressed-pretraining methods while enabling elastic deployment.","tokens_in":21138,"tokens_out":5130,"duration_ms":58850,"significance":"If the claims are sustained, the paper would make a useful contribution: it offers a training-compatible, architecture-preserving way to obtain SLR structure, and the HPA idea of converting a single trained checkpoint into a continuous family of capacity levels is practically appealing. The manuscript also has strengths that should be acknowledged: a broad set of scales (60M–1B), explicit ablations of ρ, Δα, Δβ, and κ, and a transparency that the main results use float32 training while Appendix D reports bf16 training. The empirical finding about embedding layers is interesting and potentially valuable. However, the significance as currently framed is limited by two load-bearing issues: the headline comparison in Table 1 is at unequal numerical precision, and the elastic-deployment curves in Figure 2 appear to rely on per-budget oracle selection of κ. Both need to be resolved before the central claims can be accepted.","major_comments":[{"comment":"The main comparison is not at equal numerical precision. Table 1 states that all SALAAD results are trained in float32, while all baselines are trained in bf16. Appendix D reports bf16 training: at 130M, SALAAD L+S gives PPL 24.18 with 129M parameters vs LOST 24.05 with 94M; at 350M, L+S gives 18.93/287M vs LOST 18.95/185M; and eL+eS at 130M gives 25.67/97M vs LOST 24.05/94M. Under fair precision, the advantage of Table 1 substantially vanishes or reverses. The §5.1 statement that SALAAD \"consistently match or outperform\" baselines must be re-supported using bf16 results as the primary comparison, or explicitly qualified as a float32-vs-bf16 comparison.","section":"Section 5.1; Appendix D, Table 4"},{"comment":"The elastic-deployment curves in Figure 2 do not state how κ is selected. Section 5.2 (Figure 3) shows that κ is swept for each model size and budget, with the optimal value lying in a gray region, and Table 1 uses different κ values per scale (0.7, 0.6, 0.6, 0.8). If the Figure 2 points are obtained with the best κ on the evaluation data, the smooth capacity–perplexity curve is an upper envelope, not a deployable single-rule curve. A user facing a fixed memory budget has no stated way to choose κ without oracle tuning. This directly threatens the central claim that one checkpoint yields a continuous spectrum of capacities without retraining.","section":"§4.3, Eq. (9); §5.2, Fig. 3; Table 1"},{"comment":"HPA relies on two unvalidated assumptions: unit importance I(u) proportional to magnitude |u|, and structural homomorphism across blocks (shared global ratios). No experiment compares this proxy with exact or oracle importance, nor with alternative allocation rules. Since Figure 2 is the only evidence for the elastic-deployment claim, and Eq. (8) is explicitly intractable, the greedy magnitude-based rule needs at least a sanity-check experiment (e.g., leave-one-out importance on a small model, or comparison with one or more alternative κ rules) to be load-bearing.","section":"§4.3"},{"comment":"The ρ scaling law is fitted only on 60M and 130M models, and the proportionality constant is never reported. The sensitivity is substantial: at 130M, Table 8 shows PPL ranging from about 21.7 at ρ=1e-8 to about 30.0 at ρ=1e-6 for some (Δα,Δβ) settings — roughly a 40% degradation over two orders of magnitude. Without reporting the fitted constant and demonstrating that the 350M and 1B runs lie in a stable plateau, the claim that a single fixed ρ transfers across scales is not supported by the evidence.","section":"§5.1, Eq. (7); Appendix G, Table 8"}],"minor_comments":[{"comment":"The target density is written as \\hat\\Upsilon = 0.05 in the text, but several figure panels show \"\\hat\\Upsilon = 0.95\" (e.g., Figure 1b, Figure 7c). This inconsistency makes the convergence plots hard to interpret.","section":"Figures 1, 7, 8"},{"comment":"The L+S rows in Table 1 print perplexity and parameter counts without separators (e.g., \"31.2650 22.65126\"), making the table difficult to read. Use explicit column separators.","section":"Table 1"},{"comment":"No seeds or error bars are reported for any run. Given the small parameter-count and perplexity differences in Appendix D, single-run numbers cannot be distinguished from noise.","section":"All experiments"},{"comment":"The text says that |X − (L+S)|_F \"remains bounded throughout training\" and refers to Appendix E, but Appendix E contains only empirical learning-dynamics plots. A theoretical boundedness statement or a clear statement that this is empirical observation should be added.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The main issue is that the headline comparison is at unequal precision and the HPA deployment curves may be oracle-selected. Neither is fatal to the method: the bf16 appendix shows the algorithm does train stable models, and the κ selection could be made a fixed rule. But both are load-bearing for the central claims, and the paper should be revised substantially before publication. I would not recommend rejection, because the core idea is plausible and the missing pieces are addressable by re-analysis and additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core of this paper is SALAAD, a two-stage stochastic ADMM that nudges pretrained LLM weights toward low-rank-plus-sparse decompositions block-by-block, with an integral controller adjusting penalties to hit user-set rank and density targets. The HPA truncation scheme then carves the structured surrogate to any parameter budget. That combination is new — nothing else does training-time SLR induction with block-adaptive penalties and post-hoc budgeted truncation — and the engineering is substantial: four model scales, ablations, a bf16 follow-up, and a genuinely useful empirical observation that embedding layers tolerate SLR induction while the LM head does not. The vanilla-RPCA comparison in Figure 2 is fair and favorable, showing training-time induction helps post-hoc compression.\n\nThe problems are concentrated in the framing. Table 1 compares float32-trained SALAAD against bf16 baselines; when SALAAD is also trained in bf16 (Appendix D), the advantage largely disappears and LOST often wins with 27–55% fewer parameters. The paper's own appendix undercuts the 'matches or outperforms' claim, so the main table needs repositioning to matched precision.\n\nThe bigger issue is Figure 2. The paper never says how κ is chosen for those curves, while Section 5.2 sweeps κ per budget and Table 1 uses different κ values per scale. If the figure uses the best κ per budget on the same data, the smooth monotone curve is an oracle upper envelope, not a deployed single-rule curve. That directly threatens the load-bearing 'continuous elastic deployment' claim. The fix is simple: report a fixed κ rule (e.g., κ = 0.6) and show the same qualitative behavior, or disclose the per-budget selection and tell practitioners how to set it. There are no seeds or error bars anywhere, which makes it hard to judge how much this matters.\n\nTwo smaller but real issues: the ρ scaling law depends on a fitted proportionality constant that is never reported, while Table 8 shows perplexity swinging from 21.7 to 30.0 over the tested range — a 40% degradation, so the 'single hyperparameter across scales' claim is under-supported. And the boundedness of |X − (L+S)|_F is stated as a guarantee but only empirically demonstrated. No code or data are released.\n\nNone of this invalidates the framework — it's coherent, the ablation work is honest, and the I-controller's explicit targeting of rank and density means the low parameter counts are by construction, which should be framed as such rather than called emergent. The paper is for people working on training-time compression and elastic deployment; it deserves a serious referee, but it needs a major revision: disclose the ρ constant, specify and test a fixed κ rule, add error bars, and redo the main comparison at matched precision.","headline":"A serious training-time SLR framework with a genuinely new optimizer, but the elastic-deployment headline likely rests on per-budget oracle tuning of κ, and the main table compares float32 to bf16; it deserves peer review after those are fixed.","tokens_in":21797,"tokens_out":3691,"would_cite":true,"duration_ms":43221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a language model so every weight matrix splits into a low-rank part plus a sparse part yields a single checkpoint that can be resized to any memory budget at deployment, without retraining.","keywords":["sparse and low-rank decomposition","ADMM","LLM pretraining","model compression","elastic deployment","integral control","homomorphic parameter allocation","embedding layer structure"],"falsifier":"Re-run the 1B pretraining twice, at ρ = 1e-8 and ρ = 1e-6 (the ends of the 130M ablation range in the paper's Table 8): if the single fixed ρ transfers, both runs should train stably with similar perplexity and compression; if the roughly 40% perplexity swing reproduces at 1B, the 'one hyperparameter across scales' premise collapses and with it the 350M/1B results that anchor the elastic-deployment claim.","tokens_in":20562,"feed_emoji":"🗜️","tokens_out":12055,"duration_ms":121402,"temperature":0.7,"pith_summary":"The paper argues that sparse-and-low-rank structure must be induced while a language model trains, not stamped on afterwards: if each weight matrix is trained to decompose into a low-rank part L plus a sparse part S, a single checkpoint can later be truncated to any memory budget with smooth, predictable quality loss. The proposed framework, SALAAD, does this with a two-stage ADMM loop layered on ordinary training, plus an integral controller that adaptively tunes rank and sparsity per block so only one global penalty coefficient ρ has to be set. Across 60M to 1B parameter models, the structured surrogate matches or beats prior sparse/low-rank pretraining methods at comparable parameter counts, and its performance-capacity curve stays smooth where post-hoc-compressed vanilla checkpoints degrade sharply. The paper also surfaces an asymmetry: embedding layers take on low-rank and sparse structure without hurting training, while the language-model head resists it. If the results hold, one training run replaces the current practice of training and releasing many fixed-size models.","feed_headline":"One pretraining run yields a model that resizes to any memory budget","feed_subtitle":"Training weights as low-rank plus sparse makes one model checkpoint resize smoothly, not crash, at smaller budgets.","key_machinery":"The engine is the ADMM decomposition X = L + S, whose two proximal operators do the structural work in closed form: singular-value soft-thresholding (the nuclear-norm proximal step, which shrinks small singular values of L toward zero) and elementwise soft-thresholding (the ℓ1 proximal step, which zeros small entries of S), with a dual variable Y pulling X and L+S together. Around that loop sit two named pieces: the I-controller, an integral controller that raises or lowers the rank and sparsity penalties per block so that only the global penalty ρ needs tuning, and HPA, a greedy budget-splitting rule that truncates rank and sparsity by uniform global ratios across blocks, giving continuous,","core_discovery":"SALAAD imposes structure during training, not after: it trains dense weights X together with a structured surrogate X ≈ L + S, L low-rank and S sparse. Training alternates ordinary gradient steps on ℓ(X) + (ρ/2)‖X − L − S + Y/ρ‖² with closed-form proximal updates — singular-value soft-thresholding for L, elementwise soft-thresholding for S, plus a dual update. An integral controller adapts the two penalties blockwise so only the global coefficient ρ needs tuning, and the surrogate supports continuous post-hoc truncation by a homomorphic allocation rule. Across 60M–1B parameters the paper reports that the surrogate matches or beats prior sparse/low-rank pretraining baselines, degrades smoothl","pith_inferences":["The undisclosed proportionality constant in the ρ scaling law is the crux: the paper says it is fitted on 60M/130M models and fixed for 350M/1B, but never reports the number, so the across-scale claim is not yet fully reproducible from the text.","The headline table compares float32-trained SALAAD against bfloat16-trained baselines — a precision difference the paper discloses but never quantifies; rerunning baselines in float32 would separate structural gains from precision gains.","HPA assumes block homomorphism — uniform global truncation ratios ignore block-specific sensitivity; if the surrogate structure is genuinely heterogeneous, a sensitivity-weighted allocation should beat HPA, a testable extension the paper does not run.","The benign-embedding finding suggests an untested design rule: embedding widths and vocabulary sizes could be chosen for compressibility inside the same framework, without altering the Transformer skeleton."],"forward_implications":["A single SALAAD checkpoint can be truncated at deployment to any parameter budget below nominal size with a smooth, predictable performance–capacity trade-off, whereas RPCA-compressed vanilla checkpoints degrade sharply.","The structured surrogate and its truncated variants match or beat existing sparse-and-low-rank pretraining methods at comparable parameter counts, without changing the Transformer architecture.","Post-hoc decomposition is not enough: RPCA applied to standard-trained weights recovers only weak structure (roughly 48–55% effective rank), so SLR compression needs the training-time induction SALAAD performs.","Embedding layers can be folded into SLR training at no perceptible loss cost, adding compression redundancy, while the LM head cannot — delimiting where structure can be extracted.","The I-controller collapses per-block structural hyperparameters into one global ρ, so the recipe transfers across model scales without hand-crafted per-layer rank or sparsity schedules."],"fun_headline_variants":["Train once, deploy at any memory size – SALAAD makes it work","New method adapts LLM capacity on the fly without retraining","SALAAD: low-rank plus sparse weights for elastic LLM inference","Elastic LLM: train once, then resize for any budget","SALAAD slims LLMs for any memory limit after one training run"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything hinges on a single undisclosed number: the proportionality constant in the ρ ∝ 1/(N√(nm)) scaling law is fitted on the 60M and 130M models and then fixed for 350M and 1B, yet its value is never reported, and the 130M ablation shows perplexity swinging from 21.7 to 30.0 — roughly 40% — as ρ moves from 1e-8 to 1e-6.","fun_headline_variants_meta":{"raw":{"variants":["Train once, deploy at any memory size – SALAAD makes it work","New method adapts LLM capacity on the fly without retraining","SALAAD: low-rank plus sparse weights for elastic LLM inference","Elastic LLM: train once, then resize for any budget","SALAAD slims LLMs for any memory limit after one training run"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3477,"prompt_tokens":707,"completion_tokens":2770,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":2684}},"tokens_in":451,"tokens_out":2770,"duration_ms":21899,"temperature":1.0,"reasoning_tokens":2684,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:51:37.921124+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 1B pretraining twice, at ρ = 1e-8 and ρ = 1e-6 (the ends of the 130M ablation range in the paper's Table 8): if the single fixed ρ transfers, both runs should train stably with similar perplexity and compression; if the roughly 40% perplexity swing reproduces at 1B, the 'one hyperparameter across scales' premise collapses and with it the 350M/1B results that anchor the elastic-deployment claim.","supporting_citations":[],"review_version":1}