{"id":"b7ed3547-2c24-45d0-8e3f-a6df1e497676","arxiv_id":"2501.16372","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A retrospective of the authors' LoNAS, Shears, and SQFT methods arguing that elastic low-rank adapters and weight-sharing NAS together compress LLMs efficiently.","lead":"This paper reviews the authors' own prior work combining low-rank adapters with neural architecture search for compressing large language models. It reports that such elastic adapters can cut model size by up to 80 percent while roughly matching or slightly beating standard LoRA accuracy on selected tasks.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The weight-sharing transfer assumption is the load-bearing link: LoNAS/Shears/SQFT report sub-network accuracies sampled from a super-network without demonstrating that those accuracies survive extraction as standalone models.","rationale":"I read the paper in good faith as a retrospective workshop summary of the authors' prior peer-reviewed work, not as a new experimental claim. Its stated purpose is to describe how LoNAS, Shears, and SQFT combine low-rank adapters with weight-sharing NAS. The strongest claim is that this combination is bi-directionally beneficial and produces compressed models that match or beat vanilla LoRA. The reader's unverified verdict is appropriate because the paper contains no new experiments, error bars, or external baselines; it relies entirely on self-cited tables reproduced from earlier papers. The load-bearing assumption is indeed the weight-sharing transfer assumption. In LoNAS Mode B, the frozen base weights are sliced while the elastic adapters are trained inside a super-network; in Shears/SQFT Mode A, only the adapter rank is elastic, and sub-adapters are submatrices of the full adapter sharing gradients during training. In all cases, the reported accuracy is measured on sub-networks whose adapter weights were not trained independently. Weight-sharing NAS is known to produce super-networks whose internal ranking can diverge from standalone performance unless special training techniques are used, and this paper does not report any such safeguards or any validation that the selected subnets, once extracted, retain their accuracy. This concern does not require accusing the authors of any misrepresentation; it is a standard, checkable threat to the validity of the summary's central claim. The concrete test I propose directly checks whether the transfer assumption holds for the exact models and checkpoints the paper points to. If the test shows that extracted subnets retain accuracy, the concern is resolved. If it shows a large gap, the compression claims would need to be revised or qualified. Since the paper is already marked UNVERDICTED rather than ACCEPT or REJECT, my analysis does not move that verdict; it identifies the specific condition under which the paper's claims would need to be reconsidered.","tokens_in":7324,"tokens_out":4291,"duration_ms":43413,"concrete_test":"Use the released repository checkpoint for LoNAS-7B. Sample the three Table 1 subnets (heuristic, Search Subnet-1, Search Subnet-2) and evaluate each in three conditions: (a) as extracted directly from the super-network; (b) after a short adapter-only fine-tuning, e.g., one epoch on the same 15k commonsense dataset, with the same training recipe; and (c) as a standalone model trained from scratch at that exact rank/width configuration with frozen base weights. If (b) or (c) improves average accuracy by more than about 1-2 points over (a), the super-network evaluation overstates standalone performance and the transfer assumption fails. Additionally, sample 20 random sub-configurations and compute the Spearman rank correlation between super-network-predicted accuracy and standalone fine-tuned accuracy; a low or negative correlation would invalidate the search procedure's premise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that the benefits of combining low-rank adapters with weight-sharing NAS are bi-directional rests on the Elastic Adapter statement: 'By allowing the activation of variable configurations of a layer during the forward and backward passes, one is effectively training a super-network in which the smaller structures share their weights with their bigger counterparts.' For LoNAS (Mode B) and Shears/SQFT (Mode A), the reported numbers in Tables 1-3 are accuracies of sub-networks or sub-adapters evaluated directly from this shared-weight super-network. This is exactly the weight-sharing NAS transfer assumption: the proxy performance of a sub-network inside the super-network must be indicative of its standalone performance after extraction. The paper provides no evidence for this correlation: no standalone retraining of the selected subnets, no rank-correlation analysis between super-network predictions and independently trained models, and no adoption of known safeguards such as progressive shrinking or the sandwich rule that the cited Once-for-All work uses to mitigate this failure mode. The text only proposes a heuristic 'middle point' sub-network to quickly estimate super-network quality; it does not validate that the searched sub-adapters retain accuracy after extraction. If the transfer assumption fails, the reported 'improve or only minor drops' accuracy in Tables 1-3 could be an artifact of super-network evaluation, and the headline compression claims (up to 80% parameter reduction, 1.4x speedup) would be misleading because the extracted models would be less accurate than reported. All three methods inherit this assumption, so it is the single load-bearing point for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that combining low-rank adapters (LoRA) with weight-sharing neural architecture search benefits both techniques. It introduces the Elastic LoRA Adapter framework with two modes (rank elasticity and channel-width elasticity), then describes the authors' prior methods LoNAS, Shears, and SQFT as instantiations. The paper reproduces accuracy and compression tables from these prior papers and concludes that compressed models either improve or show minor accuracy drops relative to vanilla LoRA. It also sketches future work on multi-objective search.","tokens_in":7548,"tokens_out":4843,"duration_ms":40475,"significance":"If the reported numbers are taken at face value, the paper consolidates a useful design pattern—elastic low-rank adapters inside a weight-sharing super-network—that can reduce LLM parameter counts by up to 80% and yield up to 1.4x inference speedup. The open-source release of models and code is a practical contribution. However, the manuscript itself contains no new experiments; every supporting table is reproduced from the authors' earlier publications, and the reported sub-network accuracies rely on an unexamined weight-sharing transfer assumption. The paper's value is therefore as a retrospective synthesis, not as a source of fresh evidence.","major_comments":[{"comment":"The reported accuracies for LoNAS, Shears, and SQFT are measured on sub-networks sampled from the shared-weight super-network, but the paper provides no evidence that these proxy accuracies survive after the sub-adapters are extracted and evaluated as standalone models. The text even labels the mechanism as an 'expectation' and the fast evaluation as a 'heuristic.' Without standalone fine-tuning or retraining of the selected sub-adapters, or a rank-correlation study between super-network predictions and independently trained models, the 'improve or only minor drops' claim is not established; this is the load-bearing assumption for the compression claim.","section":"Efficient Neural Architecture Search with the Guidance of Low-Rank Adapters (LoNAS), Tables 1 and 2"},{"comment":"The second direction of the bidirectional claim—'NAS becomes more efficient by incorporating the guidance of low-rank representations'—is asserted but never measured. No search-cost comparison (wall-clock time, GPU hours, or samples to convergence) is provided between weight-sharing NAS with and without low-rank guidance. Add such measurements or explicitly downgrade the claim to a qualitative observation rather than an empirical result.","section":"Introduction and Preliminaries; Conclusion"},{"comment":"All results are reproduced from the authors' own prior work and are point estimates without error bars, number of seeds, or significance tests. Given the conclusion's comparative language ('improve or exhibit only minor drops'), the absence of variance information and the selective task coverage make the comparison to vanilla LoRA fragile. At minimum, state the experimental protocol of the original papers and indicate which differences are within run-to-run noise.","section":"Performance Summary and Additional Considerations (Tables 1-3)"},{"comment":"The paper says Shears found NLS 'particularly well-suited for sparse models' but does not discuss why low-rank adapters and sparsity interact. More importantly, the sparsification metric Ψ(W) = |W| · ‖X‖_2 is applied to the frozen weights, while adapters are later merged via SparsePEFT; the paper never shows the actual memory footprint of the final merged model minus the dense adapters. Clarify whether the reported 'Total Params.' in Table 1 and 'Non-zero Params.' in Table 3 correspond to deployable merged models or to the base model plus separate adapters.","section":"Restricting the Elasticity to the Adapter Rank and Exploiting Model Sparsity and Low Numerical Precision (Shears/SQFT)"},{"comment":"The merging equations define quantization-aware merging, but the paper does not report end-to-end accuracy after merge-and-dequantize for the quantized pipelines. In Table 2, 'SQFT + QA-SparsePEFT' is shown at 44.0 vs. LoRA 44.1, but it is unclear whether this number is obtained from the merged INT4 model or from a simulation. State explicitly what is being evaluated.","section":"Addressing the Challenges of Merging Adapters with Low-precision Sparse Models (Eq. 3-4)"}],"minor_comments":[{"comment":"The phrase 'Artificial Inteligence' contains a typo; it should be 'Artificial Intelligence.'","section":"Introduction"},{"comment":"The symbols z, s, and W^p are used in Eq. (3) before being defined in the text; define them or point to the pre-fine-tuning stage where they are introduced.","section":"Addressing the Challenges of Merging Adapters with Low-precision Sparse Models (Eq. 3)"},{"comment":"The two modes are labeled 'Elastic Adapter (Mode A)' and 'Elastic Adapter (Mode B)' but the caption does not explain which dimensions are elastic in each mode; add a sentence to the caption.","section":"Figure 1"},{"comment":"The 'middle point' heuristic is mentioned but never precisely defined; specify what the middle point is (e.g., median rank and width) so the reader can reproduce it.","section":"Efficient Neural Architecture Search with the Guidance of Low-Rank Adapters (LoNAS)"},{"comment":"The conclusion states elastic adapters 'achieve better results than their vanilla low-rank adapter counterparts,' but Table 1's Search Subnet-2 (65.6) and Table 3's 50% sparsity Shears (50.9) are below LoRA; revise to 'improve or exhibit only minor drops' consistently.","section":"Conclusion"},{"comment":"Several references are incomplete or inconsistently formatted (e.g., 'Hu et al. 2023' appears in Table 1 but the bibliography entry is for LLM-Adapters; the LoNAS reference has a line break in the title).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a retrospective account of the authors' own prior methods, and all numerical evidence is self-citational. For a journal venue, the lack of independent validation or any new experiment is a concern; if the venue accepts such papers, the transfer-assumption and search-efficiency issues are the ones to fix. The manuscript would be strengthened by an explicit limitations paragraph acknowledging that sub-network proxy accuracies are not shown to transfer to standalone deployment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nothing here is new, and to its credit the paper doesn't pretend otherwise. It is a retrospective summary of the authors' own prior work — LoNAS, Shears, SQFT — with equations and tables pulled from those papers. What it does well: it organizes the two modes of elastic LoRA adapters clearly, reproduces the key numbers, and points to the source papers and code. As a workshop note, this is useful and honest. As a research contribution, it adds no evidence or analysis beyond the original papers.\n\nThe soft spot the stress-test flags is real and worth saying explicitly. The claim that the benefits are bi-directional — NAS helps adapters, adapters guide NAS — rests on Tables 1-3, and those tables report sub-network accuracies from a weight-sharing super-network. The paper never tells us whether those accuracies survive extracting the sub-network as a standalone model. It mentions a midpoint heuristic for quickly assessing super-network quality, but that doesn't validate the transfer. So a reader cannot tell from this paper whether the 80% parameter reduction and 1.4x speedup would hold in deployment with the same accuracy. The paper points to the source papers for details, which helps, but the retrospective itself should either state the validation or flag the assumption.\n\nThe other soft spots are consequential but not shocking. The evidence is entirely self-cited, with no external baselines and no error bars. The NSGA-II suggestion is just a hand-wave, not an implementation. And the 'novelty' is really a framing contribution: describing these methods as a case of low-rank guidance for NAS. None of these problems ruin the paper, because it is explicitly retrospective, but they limit its audience to people who want a quick overview of this line of work, not people looking for new results.\n\nMy recommendation: if this is intended for a workshop, it's acceptable as a summary and will help readers find the primary sources. If it is submitted as a main-track research paper, I would desk reject it — there is nothing for a referee to check beyond whether the recap is accurate, and the accuracy is better verified against the published papers. I would not send this to full peer review.","headline":"A transparent but entirely retrospective summary of the authors' own prior work; fine as a workshop note, not a research contribution.","tokens_in":8141,"tokens_out":4185,"would_cite":false,"duration_ms":40522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that making LoRA adapters elastic in rank or width during training lets a single weight-sharing super-network produce compressed LLMs that match or beat vanilla LoRA accuracy while cutting parameters by up to 80 percent…","keywords":["low-rank adaptation","LoRA","neural architecture search","weight-sharing super-network","large language model compression","parameter-efficient fine-tuning","sparsity","quantization-aware fine-tuning"],"falsifier":"Train a LoNAS or SQFT super-network on a fixed dataset, record the validation accuracy of a sampled sub-adapter before extraction, then extract exactly that sub-adapter, run standalone inference with no further training, and compare task accuracy. If post-extraction accuracy is substantially lower than in-super-network accuracy across multiple sub-adapters, the shared-weight transfer assumption is false and the search results would not transfer.","tokens_in":1804,"feed_emoji":"🧩","tokens_out":2721,"duration_ms":86033,"temperature":0.7,"pith_summary":"The paper asserts that low-rank adapters and weight-sharing neural architecture search strengthen each other, and it documents three methods that exploit that synergy to compress large language models. By making LoRA's matrices elastic, sliceable in rank or width during training, a single super-network can be fine-tuned once and then searched for small sub-adapters that guide pruning of the frozen base model. Across LoNAS, Shears, and SQFT, the reported result is up to 80 percent parameter reduction and up to 1.4x inference speedup, with accuracy matching or slightly exceeding vanilla LoRA on commonsense and math reasoning tasks. The reader should care because the methods lower the compute and memory barrier for deploying large models on limited hardware while keeping the benefits of fine-tuning.","feed_headline":"Low-rank adapters can shrink LLMs 80% without accuracy loss","feed_subtitle":"Weight-sharing search turns one super-network into many small, task-tuned models that match or beat standard LoRA.","key_machinery":"The central mechanism is the elastic LoRA adapter: the low-rank factors $L_1$ and $L_2$ are replaced by families that can be sliced to smaller ranks (Mode A) or smaller input/output widths (Mode B) during training. Because every forward pass activates one sub-configuration, gradients flow through the shared weights of all sub-adapters, effectively training a weight-sharing super-network. Two auxiliary mechanisms keep the compression intact after fine-tuning: a binary mask $M$ derived from the base weights sparsifies the adapter product, and a clamp-round-scale formula re-quantizes merged weights.","core_discovery":"This paper claims that the interaction between low-rank representations and weight-sharing NAS is bidirectional: NAS makes low-rank adapters better by letting them be elastic, and low-rank adapters make NAS cheaper by restricting the search to adapter parameters rather than full model weights. Concretely, it describes an elastic adapter that can activate sub-configurations of the LoRA factors during training, so small sub-adapters share weights with larger ones. LoNAS then uses the elastic adapter to remove chunks of the frozen weights, Shears restricts elasticity to rank and fine-tunes sparse models, and SQFT adds quantization-aware merging strategies, SparsePEFT and QA-SparsePEFT, so the compressed model can be served at low precision without losing sparsity. The empirical tables show compressed models that improve on, or stay within about a point of, vanilla LoRA accuracy while cutting parameters and speeding inference.","pith_inferences":["Inference: The reported accuracy of searched sub-adapters is measured while they still share weights inside the super-network; if the transfer assumption holds, extracting and standalone-fine-tuning a sub-adapter should preserve or improve accuracy, which is directly testable.","Inference: The elastic-adapter mechanism is not tied to transformers; applying Mode A or Mode B elasticity to other architectures could extend the same search-before-deploy workflow beyond LLMs.","Inference: Because a super-network is trained once, one could search for Pareto-optimal configurations across multiple hardware targets, such as MACs, memory, and latency, without retraining, reducing the cost of deployment cycles.","Inference: The paper describes evolutionary search as future work; a cheaper alternative would be a predictor trained on randomly sampled sub-adapters to rank configurations and avoid running validation on many candidates."],"forward_implications":["Elastic rank adapters remove the need to guess a single LoRA rank; the search can choose the rank that fits the task and the deployment budget.","LoNAS-style pruning can cut parameter counts by roughly 80 percent and give about 1.4x measured inference speedup while keeping accuracy within a point of LoRA.","Because Shears only trains adapter parameters, it is cheaper than LoNAS and can be applied to already-sparsified models, recovering accuracy lost to pruning.","SQFT's SparsePEFT and QA-SparsePEFT allow adapter weights to be merged into sparse or quantized base models without destroying sparsity or precision, so the deployed model keeps its compression.","A weight-sharing super-network can be searched once and then specialized to multiple deployment targets, since sub-adapters are cheap to extract."],"supporting_citations":[{"why":"Supplies LoRA, the base low-rank adaptation method that the elastic adapters extend.","marker":"Hu et al. 2022"},{"why":"Introduces the weight-sharing super-network training paradigm that elastic adaptation borrows.","marker":"Cai et al. 2020"},{"why":"Contributes the Mode B elastic adapter and the LoNAS method, including the Table 1 results.","marker":"Muñoz et al. 2024 (LoNAS)"},{"why":"Contributes the Neural Low-Rank Adapter Search method and the sparse-model results in Tables 2 and 3.","marker":"Muñoz, Yuan, and Jain 2024 (Shears)"},{"why":"Contributes SparsePEFT and QA-SparsePEFT, enabling fine-tuning of sparse and quantized base models.","marker":"Muñoz, Yuan, and Jain 2024 (SQFT)"},{"why":"Provides QLoRA, the low-precision fine-tuning inspiration for SQFT's quantization-aware pipeline.","marker":"Dettmers et al. 2023"},{"why":"Supplies Wanda, the weight-importance metric used to sparsify base models in Shears.","marker":"Sun et al. 2023"},{"why":"Provides the LLM-Adapters unified reasoning datasets used as evaluation benchmarks in the tables.","marker":"Hu et al. 2023"},{"why":"Extends LoNAS to all transformer weight matrices and entire block removal, showing the approach generalizes.","marker":"Sukthanker et al. 2024"}],"fun_headline_variants":["Low-rank + NAS: LLM compression with smarter adapters","Elastic adapters let NAS compress LLMs without accuracy loss","Weight-sharing NAS meets low-rank for lean LLMs","Sparse, quantized, low-rank: NAS-crafted LLMs","Low-rank adapters get a NAS upgrade for smaller LLMs"],"cache_read_input_tokens":10240,"weakest_assumption_plain":"The paper assumes that a sub-model's accuracy while it shares weights inside the super-network is a reliable predictor of its accuracy once extracted and used standalone; if that transfer fails, LoNAS, Shears, and SQFT would select configurations that do not deliver their measured accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank + NAS: LLM compression with smarter adapters","Elastic adapters let NAS compress LLMs without accuracy loss","Weight-sharing NAS meets low-rank for lean LLMs","Sparse, quantized, low-rank: NAS-crafted LLMs","Low-rank adapters get a NAS upgrade for smaller LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1306,"prompt_tokens":873,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":489,"tokens_out":433,"duration_ms":10957,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:14:54.334938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a LoNAS or SQFT super-network on a fixed dataset, record the validation accuracy of a sampled sub-adapter before extraction, then extract exactly that sub-adapter, run standalone inference with no further training, and compare task accuracy. If post-extraction accuracy is substantially lower than in-super-network accuracy across multiple sub-adapters, the shared-weight transfer assumption is false and the search results would not transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the weight-sharing super-network training paradigm that elastic adaptation borrows."},{"cited_title":"S.; Staffler, B.; Hutter, F.; and Klein, A","cited_arxiv_id":null,"evidence_quote":"Extends LoNAS to all transformer weight matrices and entire block removal, showing the approach generalizes."}],"review_version":1}