{"id":"b5d71a2c-73a9-4a7a-ac81-2df58708085a","arxiv_id":"2506.20629","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.","lead":"The paper introduces PLoP, a nearly free method that uses one forward pass to compute alignment scores and decide which module types, such as query, value, or MLP projections, should receive LoRA adapters during finetuning. If it works as claimed, practitioners could improve LoRA finetuning accuracy at no extra training cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline empirical comparison is not yet reliable: Tables 1–2 select learning rates on GSM8K test accuracy with one seed each, so the claimed consistent gains over MLP/Attn may be seed noise; the NFN-to-LoRA heuristic remains underevidenced.","rationale":"The reader identified the theory-to-method gap as the weakest assumption. I agree that Theorem 1 does not derive the low-NFN rule, and the paper itself calls the single-datapoint/SignSGD setup unrealistic. However, the central claim is an empirical one: PLoP consistently outperforms or competes with existing placements. A heuristic can be practically useful even without a complete derivation, provided the experiments are clean. The more load-bearing problem is that the main supporting experiments are not clean: learning rates are selected on the test metric, there are no seeds or error bars, and the reported margins are small. For a 1.7B model, a 0.2% GSM8K difference across placements is well within seed noise; the absence of any uncertainty quantification makes the 'consistently' in the abstract unsupported. This does not mean the method is wrong. The NFN maps show plausible structure, PLoP-1 is worse in the GRPO experiment, and PLoP matches or beats baselines in several settings. With proper validation, multiple seeds, and public code, the paper could support a conditional acceptance. Since the reader already returned CONDITIONAL, my concern reinforces that verdict rather than changing it.","tokens_in":24758,"tokens_out":5041,"duration_ms":63453,"concrete_test":"Re-run the SFT text-generation protocol (Tables 1–2) with a held-out validation split for LR selection: split MetaMathQA or use a GSM8K subset as validation, select LR per placement from {1,...,5}×10−4 on validation, then evaluate once on GSM8K test; repeat with at least 5 seeds and report mean±std for each placement and the paired differences PLoP−MLP and PLoP−Attn. If the paired differences are within one standard error or are not consistently positive, the headline 'consistently outperforms' should be weakened to 'competitive.' Additionally, check whether the selected module types are stable across seeds; if different seeds select different types, the method's recommendation itself is unstable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('PLoP consistently outperforms...') rests mainly on Tables 1–2, Fig. 7, and Table 3. The SFT text-generation experiments contain a protocol flaw: Section 4.2 states that for each placement 'we sweep over the learning rate in {1,2,3,4,5}×10−4 and report the result with the best accuracy.' Accuracy here is GSM8K, the same test metric used for the headline comparison; no validation split is mentioned. With one seed per configuration and no error bars, a 0.2–1.3 percentage-point margin (e.g., Table 2: PLoP r=64 75.2% vs. MLP 75.0%; Table 1: PLoP r=64 62.0% vs. MLP 63.3%) is within plausible seed noise for a 0.6–1.7B model. Appendix C.4 likewise reports single GRPO runs and a fixed LR, and Fig. 7 shows single smoothed trajectories without uncertainty. Therefore, the empirical evidence does not currently establish consistent superiority; it establishes that PLoP is competitive in several settings. The theoretical derivation (Theorem 1) is not the main blocker for the headline claim because a heuristic can be validated empirically, but the heuristic itself (low NFN implies LoRA benefit) is only supported by these unvalidated comparisons. If the comparisons were run with proper validation and multiple seeds, the claim could be settled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PLoP, a method for automatically selecting which module types (e.g., Query, Key, MLP projections) should receive LoRA adapters when finetuning a pretrained model on a given task. The selection is based on a new score, the Normalized Feature Norm (NFN), computed from a small number of forward passes: for each module, the norm of its output feature on task data is divided by a random-baseline norm. The authors motivate the method with a theoretical analysis of feature-norm growth during training (Theorem 1), and they argue that modules with low NFN scores have the most adaptation potential. They present experiments in three settings: supervised finetuning for classification (ANLI), supervised finetuning for text generation (MetaMathQA to GSM8K), and reinforcement learning with GRPO for mathematical reasoning. The central claim is that PLoP consistently outperforms, or at worst matches, common placement strategies such as attention-only or MLP-only placement, often with fewer trainable parameters.","tokens_in":25068,"tokens_out":5299,"duration_ms":58158,"significance":"If validated, PLoP would be a practically useful contribution: it is lightweight (a single forward pass, no full-model gradients), has no fitted parameters, and addresses a real open question in the LoRA literature, where the original LoRA paper and later work give conflicting placement advice. The paper also provides code and reproducible-style experimental details. However, the strength of the empirical evidence does not currently support the headline claim of consistent superiority: the main text-generation comparison selects hyperparameters on the test set with one seed, and all experiments lack uncertainty quantification. The theoretical result (Theorem 1) is for a highly simplified setting and is, as stated, internally inconsistent with its own proof. The core heuristic that low NFN predicts LoRA benefit is asserted rather than derived or directly validated. These issues are fixable, but they are load-bearing for the paper's central claims.","major_comments":[{"comment":"The SFT text-generation protocol has a test-set selection flaw. The text states that for each placement strategy the authors sweep the learning rate in {1,2,3,4,5}×10^{-4} and report the result with the best accuracy, where accuracy is GSM8K test accuracy, with no validation split and one seed per run. This means the reported gains are selected on the evaluation metric and could be driven by noise; the margins are small (e.g., Table 1: PLoP r=64 62.0% vs. MLP 63.3%; Table 2: PLoP r=64 75.2% vs. MLP 75.0%). The claim of consistent superiority is therefore not established by these tables. I recommend reporting results with a held-out validation split for hyperparameter selection and multiple seeds with means and variances.","section":"Section 4.2, Tables 1–2"},{"comment":"The central heuristic that modules with low NFN scores are the best LoRA targets is asserted rather than derived. Theorem 1 shows feature-norm growth during training for a single trainable layer, a single datapoint, and SignSGD; it does not establish that a low static NFN at initialization predicts a larger LoRA benefit after finetuning. The only control for the direction of the heuristic is the PLoP−1 method, but it is evaluated on the same flawed protocol. The paper should either provide a direct derivation linking low NFN to adaptation potential under batch training and Adam, or an explicit validation study (e.g., comparing per-module-type LoRA gains with NFN rankings across models and tasks) that does not rely on the current test-set-selected comparisons.","section":"Section 3, Figure 1, Step 3"},{"comment":"The closed-form expression for Γt in Theorem 1 is inconsistent with the recurrence derived in its proof. The proof derives Γ_{t+1} = Γ_t + β²(1+2t), which solves to Γ_t = Γ_0 + β² t², whereas the theorem states Γ_t = Γ_0 + β²(1 + t(t−1)). These differ for t ≥ 2 (e.g., t=2 gives 3β² vs. 4β²). As written, the theorem is false relative to its own proof. This is a correctness issue in the main theoretical result and must be fixed, even if the qualitative \"quasi-quadratic growth\" conclusion remains.","section":"Appendix B.2, Theorem 1"},{"comment":"All reported experiments are single runs without seeds or error bars. Figure 7 shows single smoothed trajectories, and Appendix C.4 explicitly states that the GRPO runs used a fixed learning rate without a sweep due to resource limits. Table 3 therefore provides no measure of run-to-run variability, so the observed advantages of PLoP over MLP (e.g., 74.52% vs. 73.61% at r=16) could be within noise. The empirical evidence currently supports that PLoP is competitive in several settings, but not that it consistently outperforms, and this should be stated more cautiously.","section":"Sections 4.1, 4.3, Figure 7, Table 3"}],"minor_comments":[{"comment":"The word \"nonconclusive\" should be \"inconclusive.\"","section":"Abstract"},{"comment":"The text contains typos: \"alignement\" should be \"alignment,\" and \"These low memory footprint\" should be \"This low memory footprint.\"","section":"Section 3"},{"comment":"The caption contains LaTeX artifacts such as \"Feature/uni00A0Norm\"; these should be cleaned up.","section":"Figure 2 caption"},{"comment":"The definition of O(d_n) uses \"c_n < κ d_n\" without absolute values; since sequences are stated to be in R and R_+, this is ambiguous and should be clarified.","section":"Section 2, Notation"},{"comment":"The notation \"n−1∥W2zin2∥2\" is confusing because the layer index and the weight matrix name are both \"2\"; please use distinct subscripts.","section":"Section 2.1"},{"comment":"The model name \"Qwen3.5-0.5B\" appears to be inconsistent with the model sizes used in Section 4.1 (Qwen2.5-0.5B or Qwen3-0.6B); please correct.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear and useful practical idea, and the authors are credible, but the empirical protocol (test-set LR selection, single seeds, no error bars) is a serious issue that must be addressed before the central claim can be accepted. Additionally, the error in Theorem 1's closed form suggests the theoretical appendix needs careful proofreading. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PLoP is a genuinely cheap and plausible way to choose LoRA placement: compute normalized feature norms on one small forward pass, rank module types, insert adapters in the lowest-scoring types. That placement criterion is new, and it would be a useful contribution to parameter-efficient finetuning if it holds up. The theory in Section 2/Appendix B is a real theorem about feature-norm growth in a simplified linear network trained with SignSGD on a single example; it is coherent for what it claims, and the paper is honest that the setup is unrealistic. The NFN normalization against a random baseline is sensible, the compute cost is genuinely one forward pass, the PLoP−1 control is a good idea, and the parameter-matched baselines make the comparisons fairer than usual. Citation pattern looks fine: the placement literature and the conflicting LoRA attention-versus-MLP recommendations are covered.\n\nThe soft spots are real and fairly serious. Tables 1 and 2 tune the learning rate on GSM8K test accuracy, with one seed and no error bars. The margins that carry the \"consistently outperforms\" claim are 0.2–1.3 points, which is within seed noise for a 0.6–1.7B model. Selecting the LR on the test metric means those comparisons are not a clean evaluation. The ANLI curves in Fig. 7 are single smoothed runs with no uncertainty, and the GRPO results are single runs with a fixed LR. So the evidence supports \"PLoP is competitive in several settings,\" not \"consistently outperforms.\" The other gap is the theory-to-method bridge: Theorem 1 says feature norms grow during training for one trainable layer; it does not say that modules with low static NFN benefit most from LoRA. Figure 1 asserts that. The method can still be right as a heuristic, but the empirical validation is currently too weak to carry it. The NFN code is public; the training code is not yet released, which hurts reproducibility checks.\n\nWho gets value: PEFT practitioners who want a nearly free placement rule, and researchers working on module selection. It deserves a serious referee, not a desk reject. A referee should require a validation split for LR selection, multiple seeds and error bars, released training code, and an abstract claim that matches the evidence. With those changes this could be a solid paper.","headline":"A cheap and plausible method for LoRA placement, but the 'consistently outperforms' claim is carried by test-set-tuned, single-seed comparisons with margins inside seed noise.","tokens_in":25592,"tokens_out":3557,"would_cite":true,"duration_ms":40687,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces PLoP, a cheap, gradient-free Normalized Feature Norm score that picks which module types to LoRA-tune, and shows it beats standard attention-only or MLP-only placement in its experiments.","keywords":["LoRA placement","parameter-efficient finetuning","normalized feature norms","module-data alignment","feature norm growth","large language models","supervised finetuning","reinforcement learning"],"falsifier":"Take a fixed model-task pair from the paper, such as Llama-3.2-1B on GSM8K, and exhaustively evaluate every three-module-type LoRA subset with matched parameter counts; the central claim fails if the lowest-NFN combination PLoP recommends does not rank at or near the top in final accuracy.","tokens_in":24544,"feed_emoji":"🎯","tokens_out":7105,"duration_ms":75945,"temperature":0.7,"pith_summary":"The paper's claim is that which parts of a pretrained model should receive LoRA adapters can be decided automatically, before training, by measuring the alignment between each module type and the finetuning data. The proposed score, the Normalized Feature Norm (NFN), compares a module's actual output norm on real inputs to its output norm on a random input of the same magnitude; values near one mean little special alignment, and PLoP inserts LoRA into the module types with the lowest scores. The authors ground this in a theoretical analysis showing that feature norms grow during training in modules whose weights align with their inputs, then validate it with supervised finetuning for classification, supervised finetuning for math text generation, and GRPO reinforcement learning for reasoning. In their experiments PLoP beats attention-only, MLP-only, and all-module LoRA in several cases at matched or lower parameter counts, and at worst matches the best baseline. If this is right, practitioners get a nearly free way to replace guesswork about LoRA placement.","feed_headline":"Cheap score tells where LoRA adapters should go","feed_subtitle":"PLoP beats attention-only and MLP-only LoRA on math finetuning and reasoning RL, at nearly zero extra cost.","key_machinery":"The load-bearing object is the Normalized Feature Norm: for a module with weight $W$ and input feature $z_{\\mathrm{in}}(x)$, NFN is $\\|W z_{\\mathrm{in}}(x)\\| / \\|W \\tilde z_{\\mathrm{in}}(x)\\|$, where $\\tilde z_{\\mathrm{in}}(x)$ is an i.i.d. Gaussian vector with the same norm as $z_{\\mathrm{in}}(x)$; the denominator strips away the scale of $W$ and of the input, leaving a measure of weight-input alignment. PLoP averages NFN over a batch of task data, aggregates the scores by module type, and places LoRA adapters in the module types with the lowest scores. The theoretical support is the feature-update identity under SignSGD and maximal-update scaling, $W_{t+1} z_{\\mathrm{in}} = W_t z_{\\mathrm{in}} - \\eta n^{-1} \\|z_{\\mathrm{in}}\\|_1 S(dz_{\\mathrm{out}})$, which produces norm growth only when the module weight and its input are aligned; Theorem 1 makes this precise for a linear network and predicts quasi-quadratic growth of the normalized feature norm at early training.","core_discovery":"The central discovery is that a module's potential to benefit from LoRA can be read off from its normalized feature norm before finetuning: modules whose real-data features are far less aligned with their weights than a random baseline are the ones where adaptation helps. The authors support this with a theorem in an idealized setting, a wide linear network with one trainable layer, one training point, and SignSGD under maximal-update parameterization, where the normalized feature norm grows quasi-quadratically early in training because alignment between the weight and its input produces a positive update term that a randomized baseline does not receive. This motivates the NFN score and the rule \"insert LoRA in the lowest-scoring module types.\" Empirically, on ANLI classification, MetaMathQA-to-GSM8K text generation, and GRPO for mathematical reasoning, PLoP's chosen module types (for example Value-Out-Down on Qwen3-1.7B) beat attention-only placement by several points, match or beat MLP-only placement, and in several cases outperform all-module LoRA while using fewer trainable parameters.","pith_inferences":["An untested dynamic extension follows from the paper's plateau observation: recompute NFN scores during finetuning and move adapters as alignment saturates, since most feature-norm growth happens in the first few hundred steps.","Because specialized models show higher NFN scores than general models on the same task, NFN could serve as a cheap diagnostic of how well a base model is already aligned to a dataset, potentially informing whether to spend budget on SFT or RL.","The theoretical bridge from a single-layer, single-datapoint, SignSGD analysis to Adam-trained Transformers with batches is left unproved, so a direct test of whether static low scores are the mechanism rather than a proxy would strengthen or revise the placement rule.","Layer-level selection remains an open direction: the paper reports inconsistent layer-level results, suggesting module-type aggregation is the right granularity for now but that better layer-level features could recover finer placement gains."],"forward_implications":["LoRA placement can be chosen at roughly the cost of one batch forward pass, with no full-model gradients, making placement search practical on commodity GPUs.","Attention-only LoRA, the default from the original LoRA work, is not the right choice for math reasoning tasks; mixed attention and MLP placements such as Value-Out-Down or Down-Up-Value perform better.","PLoP can beat placing adapters in all module types while using roughly 40 to 60 percent of the trainable parameters, for example 43.9 million versus 69.7 million for Qwen3-1.7B.","The reverse placement rule, choosing highest-scoring module types, performs worse in the paper's experiments, supporting the interpretation that low alignment rather than high importance marks LoRA potential.","NFN rankings are roughly stable across model sizes within the same family, so a ranking computed once on a smaller model may transfer to its larger sibling."],"supporting_citations":[{"why":"Supplies the LoRA method and the attention-placement default that PLoP must beat.","marker":"[1]"},{"why":"Documents that optimal placement varies by model and task, motivating automatic selection.","marker":"[13]"},{"why":"Shows MLP placement can outperform attention placement, providing the second baseline.","marker":"[14]"},{"why":"Provides the maximal-update parameterization and infinite-width feature-learning framework behind the norm-growth analysis.","marker":"[19]"},{"why":"Explains how feature learning and norm growth differ across layers, supporting uneven NFN scores among module types.","marker":"[22]"},{"why":"Supplies the GSM8K benchmark used to evaluate both supervised finetuning and GRPO performance.","marker":"[26]"},{"why":"Introduces the GRPO algorithm used in the reinforcement-learning experiments.","marker":"[29]"}],"fun_headline_variants":["NFN score spots best LoRA placement cheaply","Where LoRA helps? Check feature norm first","PLoP: low-cost guide for LoRA adapter spots","Score feature norms to place LoRA adapters","PLoP beats common LoRA placement on math and RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's load-bearing premise is that module types with low Normalized Feature Norm before finetuning are precisely the ones where LoRA adapters will help most, and the paper's theorem does not prove that step, only feature-norm growth in a simplified single-layer, single-example SignSGD setting.","fun_headline_variants_meta":{"raw":{"variants":["NFN score spots best LoRA placement cheaply","Where LoRA helps? Check feature norm first","PLoP: low-cost guide for LoRA adapter spots","Score feature norms to place LoRA adapters","PLoP beats common LoRA placement on math and RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1191,"prompt_tokens":976,"completion_tokens":215,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":137}},"tokens_in":592,"tokens_out":215,"duration_ms":2895,"temperature":1.0,"reasoning_tokens":137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:44:36.081538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed model-task pair from the paper, such as Llama-3.2-1B on GSM8K, and exhaustively evaluate every three-module-type LoRA subset with matched parameter counts; the central claim fails if the lowest-NFN combination PLoP recommends does not rank at or near the top in final accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the maximal-update parameterization and infinite-width feature-learning framework behind the norm-growth analysis."},{"cited_title":"Visualising feature learning in deep neural networks by diagonalizing the forward feature map, 2024","cited_arxiv_id":null,"evidence_quote":"Explains how feature learning and norm growth differ across layers, supporting uneven NFN scores among module types."}],"review_version":1}