{"id":"18be0354-4aab-4f9a-9120-7d2152dac33f","arxiv_id":"2509.03057","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A gated-adapter method with a sparsity regularizer is reported to match or slightly beat full fine-tuning on MNLI and BoolQ using 1.4% of parameters, but lacks the experimental detail needed to verify the claim.","lead":"This paper proposes an adapter fine-tuning method in which trainable gates decide where adapters are inserted, plus a sparsity penalty to keep the structure small. The reported results are two accuracy numbers on MNLI and BoolQ, with no code, data, or training details.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2)'s continuous sigmoid gating is never connected to a discrete structure; without a thresholding/relaxation scheme, 'structure-learnable' reduces to soft gating, and the reported gains/1.4% parameter claim are unsupported.","rationale":"The reader's weakest assumption already identifies the central gap: Eq. (2) assumes sigmoid-gated continuous interpolation discovers meaningful discrete adapter insertion points, but no discretization scheme is provided. My stress-test sharpens this into a concrete correctness objection. A sigmoid gate is strictly positive for all finite logits, so the L1-style regularizer in Eq. (3) can shrink gates toward zero but never exactly zero without a hard threshold or a different parameterization (e.g., sparsemax or concrete relaxation). Thus, as written, the model remains a soft-gated adapter at inference; the 'structure-learnable' claim is not derivable from the equations. This is not merely a missing detail: it undermines the attribution of the reported improvements to structural search and makes the 1.4% parameter-efficiency claim ambiguous. The absence of experimental setup (base model, dataset version, code) compounds but is secondary to this internal conceptual gap. I therefore agree with the reader's rejection, and my analysis does not change the verdict.","tokens_in":6470,"tokens_out":3358,"duration_ms":38788,"concrete_test":"Implement Eqs. (1)-(3) on a specified public base model (e.g., DeBERTa-v3-base) for MNLI and BoolQ. Run two inference variants: (A) keep all learned sigmoid gates continuous; (B) hard-threshold each gate at 0.5 (or select a top-k per layer) and disable adapters whose gate is below threshold. If variant A reproduces the reported ~87.4/89.6 accuracies while variant B degrades, the reported performance is attributable to dense soft gating, not discrete structure learning. If variant B also matches, require the authors to specify the exact selection rule and verify that the 1.4% parameter count reflects only active adapters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that continuous gating in Eq. (2) plus the sparsity regularizer in Eq. (3) lets the model 'automatically optimize adapter insertion points, activation paths, and module combinations.' This requires a well-defined mapping from trained continuous gates to a discrete adapter structure. The paper supplies none. At inference, either (a) all sigmoid gates remain continuous, in which case every adapter contributes with nonzero weight—the regularizer cannot drive sigmoid outputs to exactly zero for finite parameters—so no structure has actually been learned and 'compressing redundant paths' is only soft attenuation; or (b) a threshold/selection rule is applied, but no such rule, no straight-through estimator, no Gumbel-Softmax relaxation, and no search over discrete insertion configurations is specified. The parameter count of 1.4% is therefore ambiguous: it counts trainable parameters, not active structural paths, and cannot substantiate the claim that the method discovers task-specific substructures. The missing backbone model, dataset version, hyperparameters, and code make the numbers in Table 1 unverifiable, but the deeper correctness risk is the unstated continuous-to-discrete step at the core of the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a parameter-efficient fine-tuning method for large language models in which adapter modules are augmented by differentiable sigmoid gates (Eq. 2) and a sparsity regularizer (Eq. 3). The stated goal is to let the model automatically optimize adapter insertion points, activation paths, and module combinations from a frozen backbone. The authors report 87.4% accuracy on MNLI and 89.6% on BoolQ with 1.4% trainable parameters, and claim robustness to input noise and to the choice of sparsity weight. The manuscript does not provide the base model, training details, code, or complete experimental data; several key figures are missing; and the mapping from continuous gating to a discrete 'learned structure' is never specified.","tokens_in":6873,"tokens_out":4631,"duration_ms":55659,"significance":"If the claims were substantiated, the contribution would be a modest incremental result: the idea of gated or routed adapters is already present in the adapter literature, and the reported gains over LoRA are about one accuracy point on two tasks. The genuinely interesting part of the claim—automatic discovery of task-specific adapter structures—is not demonstrated, because the method is underspecified and the experiments are not reproducible. The paper contains no machine-checked proofs, no code, no complete experimental protocol, and no derivation beyond two equations, so the significance is currently limited to a plausibility sketch.","major_comments":[{"comment":"The central claim that the method 'optimizes adapter insertion points, activation paths, and module combinations' requires a well-defined mapping from learned continuous gates to a discrete adapter structure. No such mapping is given. If the sigmoid gates remain continuous at inference, every adapter contributes with nonzero weight and no structural path is ever selected; the sparsity regularizer cannot drive sigmoid outputs exactly to zero for finite parameters. If a threshold or discrete selection is intended, none is specified: there is no straight-through estimator, no Gumbel-Softmax relaxation, and no search over discrete configurations. The 'structure-learnable' aspect therefore reduces to soft gating, and the reported 1.4% parameter count measures trainable parameters, not active structural paths.","section":"Section III, Eq. (2)–(3)"},{"comment":"The only comparative result is a single table with two tasks, and it lacks the base model name, dataset versions, hyperparameters, training budget, number of runs, error bars, and significance tests. The gap over the LoRA baseline is 0.9 points on MNLI and 0.9 on BoolQ, which is within typical run-to-run variance. Parameter percentages across methods are also uninterpretable without specifying exactly which modules are counted per method. This table is insufficient to support the headline superiority claim.","section":"Section V, Table 1"},{"comment":"The sensitivity and robustness analyses described in the abstract and text are unverifiable because Figures 2 and 3 are absent from the manuscript. Statements such as 'when λ increases from 0.0 to 1.0, the accuracy improves' and 'the MNLI task maintains over 86.0% accuracy at 15% noise' are assertions without data. In addition, the λ sweep appears to be reported on the same tasks as the main evaluation, with no indication of a separate validation set; selecting λ by best test accuracy would be circular and would inflate the reported results.","section":"Section V, Figures 2 and 3"},{"comment":"The experimental section relies on an undefined 'Multi-Task NLU Benchmark (MT-NLU)'. No citation, URL, task list, or dataset description is provided, and the name does not correspond to a standard public benchmark in the literature. Without a defined dataset and split, the reported accuracies cannot be reproduced or compared with other work.","section":"Section IV"}],"minor_comments":[{"comment":"Eq. (1) is garbled; please provide a clean expression for the adapter output and define all dimensions. In the text after Eq. (3), the symbol for the adjustment coefficient is given as 'B' while Eq. (3) uses λ; please reconcile.","section":"Section III"},{"comment":"Several references are topically distant from the paper, e.g., [8], [9], [15], and [21] concern object tracking, federated recommendation, cloud scaling, and multi-agent RL. The LoRA baseline is cited through a survey [23] rather than the original LoRA paper, which should be cited. Please replace unrelated citations with directly relevant adapter and parameter-efficient fine-tuning literature.","section":"References"},{"comment":"The paper references Figures 1, 2, and 3 but none are included. These figures are necessary to understand the architecture and the sensitivity results.","section":"Section V"},{"comment":"The indexing in Eq. (4) is unclear: is K the number of adapters in a layer, and are the α_t^k shared across layers? Please clarify the relationship between the layer-level gates in Eq. (2) and the task-level gates in Eq. (4).","section":"Section III, Eq. (4)"}],"recommendation":"reject","confidential_remarks":"The manuscript appears closer to an extended abstract than a complete research paper: the core algorithm is not specified, the experimental evidence is a single table, and all three figures are missing. The main novelty claim—discrete structural learning—is not supported by any mechanism that maps continuous gates to discrete structures. Even with a major revision, the experiments and method section would need to be essentially rewritten, so rejection seems appropriate. The citation pattern in the Related Work section is also concerning; many citations are topically unrelated and may warrant editorial scrutiny."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is an easy call, but not a painful one. The paper takes a standard adapter and adds trainable sigmoid gates with an L1-style sparsity penalty, then calls the learned gate values a 'structure.' That is a real thing you could build on, but it is an old trick: gated adapters have been around for a while, and the paper never engages with that line of work. What it does well is aim at a real problem—letting the model choose per task where and how to use shared adapters—and the loss in equation (3) is a sensible way to push gate activations down. I want to credit that.\n\nThe problems are visible from a first reading. The manuscript is incomplete: Figure 1 has only a caption, the equations are mangled in the PDF, and there are no experimental details—no backbone model, no hyperparameters, no data splits, no error bars, no code. Two accuracy numbers on MNLI and BoolQ are not enough to support the claim that they beat full fine-tuning by 0.2 and 0.1 points. The 1.4% parameter count only tells you how many parameters are trainable; it says nothing about whether the final structure is actually sparse or how many adapter paths are active. The paper never says whether the sigmoid gates are thresholded at inference, or whether the soft gates are simply retained. So the 'structure-learnable' claim has a hole in it. This matches the stress-test note.\n\nThe related work is a stream of references, many of them to a particular community, but the key comparisons to gated or routed adapters are missing. The sensitivity analysis is described in prose, with no actual numbers or figures. This is not a paper in a state that a referee can fairly judge.\n\nI would desk reject this version. If the authors come back with a clean rewriting, a real discrete selection mechanism, and a proper evaluation with baselines and ablations, then it could get a fair hearing as a modest adapter variant. Right now the work is a sketch, not a manuscript.","headline":"A gated-adapter variant with sparsity; the writing and evaluation are too thin to verify the claimed gains.","tokens_in":7274,"tokens_out":2781,"would_cite":false,"duration_ms":28274,"reading_group":"no","serious_thinker":"unclear","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Learnable-gate adapter beats full fine-tuning at 1.4% of parameters","keywords":["structure-learnable adapter","parameter-efficient fine-tuning","differentiable gating","structural sparsity","multi-task NLU","adapter routing","LLM fine-tuning","robustness to noise"],"falsifier":"Inspect the trained gate values sigma(alpha_l) for each layer and task on the MT-NLU benchmark. If most gates sit between 0.2 and 0.8 rather than near 0 or 1, then no discrete adapter structure has been learned: the model is running a dense weighted average of adapters, and the claimed structural sparsity and the 1.4% parameter accounting do not describe the actual computation.","tokens_in":1452,"feed_emoji":"🧩","tokens_out":1802,"duration_ms":61385,"temperature":0.7,"pith_summary":"This paper tries to show that adapter-based fine-tuning need not rely on hand-fixed insertion points. It introduces a sigmoid gate at every backbone layer that continuously interpolates between passing the input through and applying an adapter, plus a sparsity penalty that pushes unnecessary gates closed. On the MT-NLU benchmark, the resulting structure-learnable adapter reaches 87.4% accuracy on MNLI and 89.6% on BoolQ while training only 1.4% of the model's parameters, slightly above full fine-tuning and above LoRA, AdapterFusion, Prefix-Tuning, and PiSSA. The authors also report that moderate sparsity improves both tasks, excessive sparsity degrades them, and mild input noise causes only gradual accuracy loss. If correct, the paper offers a way to let each task carve its own routing path through shared adapters without added tuning cost.","feed_headline":"Learnable-gate adapter beats full fine-tuning at 1.4% of parameters","feed_subtitle":"Structure-learnable adapter hits 87.4% on MNLI and 89.6% on BoolQ, beating full fine-tuning with 1.4% of its parameters.","key_machinery":"The central object is the gated adapter unit h~ = (1 - sigma(alpha_l))h + sigma(alpha_l)Adapter(h) for each layer. The sigmoid turns the discrete choice 'insert adapter or not' into a differentiable interpolation, so gradient descent can decide where adapters are useful; the sparsity regularizer lambda * sum(sigma(alpha_l)) trades performance against structural complexity; a task-specific gate vector over a shared adapter set extends the same mechanism to multi-task routing. These three pieces together carry the structure-search claim: no explicit architecture search or discretization is used, only continuous gating trained with the task loss.","core_discovery":"The paper's central claim is that adapter insertion and routing can be made learnable rather than manually designed. Each adapter is wrapped in a sigmoid-gated residual connection whose control variable is trained by backpropagation; a structural sparsity term in the loss penalizes too many active adapters. In multi-task settings, each task gets its own gate vector over a shared adapter pool so different tasks can activate different combinations. The reported result is that this continuous gate search, trained end-to-end with the backbone frozen, yields 87.4% on MNLI and 89.6% on BoolQ at 1.4% trainable parameters, surpassing full fine-tuning (87.2%/89.5%) and four parameter-efficient baseli","pith_inferences":["The paper never converts continuous gate values into binary insertion decisions, so the deployed structure is undefined: if gates converge near 0.5, every adapter still runs partially and the 1.4% parameter count may understate compute. A direct test is to inspect the distribution of learned gate values after training.","The headline margins over full fine-tuning and PiSSA are only a few tenths of an accuracy point; a natural next check is to repeat the comparison across multiple seeds and report variance before treating the advantage as stable.","The same gating mechanism could be evaluated on generation tasks or long-context benchmarks, since the paper only reports two classification and reading-comprehension tasks from MT-NLU.","One could test whether a hard binary gate with a straight-through estimator changes the outcome; if it performs similarly, the continuous sigmoid is not essential to the claimed structure learning."],"forward_implications":["If the result holds, parameter-efficient fine-tuning can match or slightly exceed full fine-tuning on NLU benchmarks while training only about 1% of the weights.","Moderate structural sparsity acts as a regularizer: forcing some gates closed improves MNLI and BoolQ, while over-sparsification breaks adapter paths and hurts accuracy.","Task-specific gating over a shared adapter pool gives a concrete mechanism for multi-task learning: different tasks can use different adapter combinations without storing separate full models.","The reported robustness curves imply that learned gates suppress noisy paths under mild perturbation, which matters for deployment with imperfect input.","Because the method is a drop-in adapter layer, it can be combined with other parameter-efficient techniques such as low-rank decomposition or prefix tuning."],"supporting_citations":[{"why":"Establishes that existing adapters rely on manually designed fixed topologies, the limitation the paper targets.","marker":"[1]"},{"why":"Supplies the full fine-tuning baseline (87.2% MNLI, 89.5% BoolQ) that the proposed method must beat.","marker":"[22]"},{"why":"Supplies the LoRA baseline at 0.85% parameters that the method is compared against.","marker":"[23]"},{"why":"Supplies the AdapterFusion baseline, the fixed-topology adapter approach the method claims to improve on.","marker":"[24]"},{"why":"Supplies the Prefix-Tuning baseline at 0.5% parameters, used to show the risk of excessive compression.","marker":"[25]"},{"why":"Supplies the PiSSA baseline at 1.6% parameters, a recent efficient fine-tuning method the structure-learnable adapter outperforms.","marker":"[26]"}],"fun_headline_variants":["Gated adapters auto-route, beat full fine-tuning at 1.4%","Learnable adapter gates top full fine-tuning with tiny footprint","Structure-learnable adapters: smart routing beats full FT","Auto-gated adapters surpass full fine-tuning at 1.4% params"],"cache_read_input_tokens":9088,"weakest_assumption_plain":"The method assumes that a soft on/off switch trained by gradient descent will settle into a clear sparse choice of where to put adapters; the paper does not describe how the continuous switch values become the final structure.","fun_headline_variants_meta":{"raw":{"variants":["Gated adapters auto-route, beat full fine-tuning at 1.4%","Learnable adapter gates top full fine-tuning with tiny footprint","Structure-learnable adapters: smart routing beats full FT","Auto-gated adapters surpass full fine-tuning at 1.4% params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1222,"prompt_tokens":751,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":495,"tokens_out":471,"duration_ms":5069,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:05:58.422649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the trained gate values sigma(alpha_l) for each layer and task on the MT-NLU benchmark. If most gates sit between 0.2 and 0.8 rather than near 0 or 1, then no discrete adapter structure has been learned: the model is running a dense weighted average of adapters, and the claimed structural sparsity and the 1.4% parameter accounting do not describe the actual computation.","supporting_citations":[{"cited_title":"Parameter-efficient fine-tuning of large- scale pre-trained language models,","cited_arxiv_id":null,"evidence_quote":"Establishes that existing adapters rely on manually designed fixed topologies, the limitation the paper targets."},{"cited_title":"A survey on lora of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA baseline at 0.85% parameters that the method is compared against."},{"cited_title":"Pissa: Principal singular values and singular vectors adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the PiSSA baseline at 1.6% parameters, a recent efficient fine-tuning method the structure-learnable adapter outperforms."}],"review_version":1}