{"id":"ed8e699b-4c24-45de-84eb-02d5c353b9f8","arxiv_id":"2412.09827","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LoRATRF inserts a learned cosine-similarity filter and low-rank transformation into LoRA's hidden states, claiming better accuracy with 33.71% fewer parameters, though the reported gains are inconsistent.","lead":"This paper proposes LoRATRF, a way to fine-tune large language models that adds a learned task-relevant filter on top of the standard LoRA method, using fewer trainable parameters. The authors report small accuracy gains on some benchmarks, but the paper's own tables show inconsistent results for key reasoning tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts the text on HellaSwag: LoRATRF is listed as 82.4, while the text claims 85.6 and a 0.8% gain over DoRA; the table actually shows DoRA at 84.8, so the central superiority claim rests on an unreconciled number.","rationale":"The reader's verdict was REJECT with moderate confidence, and their strongest_claim and rationale already flagged the HellaSwag inconsistency (82.4 in Table 2 vs 85.6 in text). My stress test confirms that this is the single most load-bearing concern: it directly contradicts the paper's claimed superiority on a key benchmark and makes the empirical evidence impossible to trust as presented. The reader's stated weakest_assumption focuses on the task-aware filter's modeling capability, which is a legitimate scientific concern but secondary to the numerical contradiction. I therefore partially agree with the reader: the load-bearing issue is the internal data inconsistency, not primarily the mechanism's design. My proposed verdict is UNCHANGED because the paper should be rejected as submitted: the central claim lacks a consistent empirical basis, and no amount of additional analysis of the filter mechanism can resolve the contradiction between Table 2 and the text without new experiments. The concrete test proposed—re-running the HellaSwag configuration—would settle whether the 85.6 figure is a reproducible result or a reporting error, and is the minimal check needed to restore confidence in any of the reasoning-benchmark claims.","tokens_in":8940,"tokens_out":2386,"duration_ms":22730,"concrete_test":"Re-run the LLaMA-7B HellaSwag fine-tuning with rank r=32 and the stated module selection (Wo, Wf1, Wf2), using the paper's evaluation protocol, and record the exact accuracy from the raw outputs. Then check whether the resulting number is 82.4 or 85.6. Independently recompute the LoRATRF-DoRA gap on that same evaluation. If 85.6 is not reproducible, or if DoRA actually exceeds LoRATRF, the paper's central claim of consistent superiority on commonsense reasoning is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LoRATRF reduces parameters by 33.71% and achieves better performance than SOTA low-rank methods across NLU, commonsense, and mathematical reasoning. On the reasoning benchmark, Table 2 reports LoRATRF HellaSwag = 82.4 and DoRA = 84.8, yet the accompanying text states that 'LoRATRF attains an accuracy of 85.6% on HellaSwag, surpassing DoRA baseline by 0.8%.' This is not a minor typo: it reverses the direction of the comparison. If 82.4 is correct, LoRATRF underperforms DoRA by 2.4 percentage points on HellaSwag, directly undermining the claimed 'better performance' on commonsense reasoning. If 85.6 is correct, then Table 2 itself is misreported and the reader cannot trust any number in it. The parameter-reduction claim is also disassociated from the reasoning experiments: Table 2 reports no parameter counts for LoRATRF or DoRA, and the 33.71% figure appears only in the GLUE comparison against LoRA r=8 (1.33M vs 0.88M), not against the SOTA methods used in the LLaMA experiments. Because the headline results are internally inconsistent and the key comparative claim cannot be checked, the empirical foundation of the paper's central argument fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoRATRF, a parameter-efficient fine-tuning method that augments LoRA with a learned task vector tξ and a low-rank transformation matrix. In each Transformer layer, hidden representations are reweighted by cosine similarity with tξ, linearly transformed, and added back to the original representation, with the goal of enhancing task-relevant features. The method is evaluated on GLUE with DeBERTaV3-base and on mathematical/commonsense reasoning benchmarks with LLaMA-7B. The authors claim that LoRATRF reduces trainable parameters by 33.71% and outperforms SOTA low-rank methods (LoRA, AdaLoRA, DoRA), and they include analyses of module choice and rank robustness.","tokens_in":9276,"tokens_out":6930,"duration_ms":66823,"significance":"If the claims were established, LoRATRF would be a simple and computationally cheap way to improve low-rank adaptation through explicit task-dependent feature reweighting, and the representation-editing perspective could be of interest to the PEFT community. The paper gives a clear algorithmic sketch and reports a concrete parameter reduction in the GLUE setting. However, the current empirical support is not adequate: the central LLaMA-7B comparison contains a direct numeric contradiction, the 33.71% parameter-reduction claim is not attached to the LLaMA-7B experiments, baselines are borrowed without matched conditions, and no run-to-run variability is reported. The contribution cannot be accepted on the present evidence.","major_comments":[{"comment":"The text states that 'LoRATRF attains an accuracy of 85.6% on HellaSwag, surpassing DoRA baseline by 0.8%,' but Table 2 reports LoRATRF at 82.4 and DoRA at 84.8. These numbers are irreconcilable: if Table 2 is correct, LoRATRF is 2.4 points below DoRA, reversing the claimed superiority on HellaSwag. The authors must identify which value is correct, correct the other, and re-derive the conclusions that depend on this comparison.","section":"Main Results: Commonsense and Mathematical Reasoning (Table 2)"},{"comment":"The abstract's claim of a 33.71% parameter reduction is supported only by Table 1, where LoRATRF uses 0.88M parameters versus 1.33M for LoRA r=8 on GLUE. In the LLaMA-7B experiments (Table 2), no parameter counts are reported, and the text states that LoRATRF uses the same rank (r=32) as the baseline while additionally introducing a task vector tξ and a low-rank transformation matrix T. The added components make it unlikely that LoRATRF has fewer trainable parameters than DoRA in that setting, so the paper should report parameter counts for every method in every table and restrict parameter-reduction claims to the configurations where they are actually measured.","section":"Abstract and Tables 1-2"},{"comment":"Baseline numbers in Table 1 and Table 2 are taken directly from earlier papers (Zhang et al. 2023a; Liu et al. 2024; Hu et al. 2023) without matching hyperparameters, training budgets, or evaluation protocols. For Table 1 the authors report a mean over 5 random seeds but give no standard deviation or significance test, and for Table 2 no seeding information is provided. Differences such as +0.2% on CoLA and +0.8% on HellaSwag are within typical run-to-run variance, so the claimed improvements over AdaLoRA and DoRA are not statistically supported.","section":"Experimental Setup: Baselines and Evaluation"},{"comment":"The contribution of the proposed filter is not isolated in any experiment. There is no ablation comparing LoRATRF against LoRA with an equivalent number of additional parameters (e.g., a learned per-dimension scaling or an unstructured low-rank residual), and no analysis of what the learned mask or transformation does in different layers or across tasks. As a result, the paper does not demonstrate that 'task-relevant feature enhancement' is the cause of any observed gain rather than simply added capacity.","section":"Methodology: Task-Aware Filters (Eqs. 4-5)"},{"comment":"The robustness experiment reports LoRA average accuracies of 59.3% at r=8 and 51.2% at r=4 on HellaSwag, which are far below the LoRA accuracy of 78.1% at r=32 in Table 2. The text does not explain the protocol for Figure 3 (e.g., whether the same data, training steps, and evaluation are used as in Table 2), so the large reported gaps (+19.3% at r=4, +14.6% at r=8) may be artifacts of different settings rather than evidence of robustness. The authors should align the setups or report the missing details.","section":"Quantitative Analysis: Robustness towards different rank settings (Figure 3)"}],"minor_comments":[{"comment":"The word 'vareity' should be 'variety'.","section":"Abstract"},{"comment":"'Featur Enhancement' should be 'Feature Enhancement'.","section":"Motivation"},{"comment":"The similarity function 'sim' is not formally defined; the text later says cosine similarity, but the definition should appear together with the equation.","section":"Task-Aware Filters (Eq. 4)"},{"comment":"The final residual addition is only described in words; an explicit equation such as h_out = h + \\hat{h} would clarify the forward pass.","section":"Task-Aware Filters (after Eq. 5)"},{"comment":"The column header 'LLM Method' is ambiguous; the GPT-3.5 row should be labeled as a reference model rather than a method under comparison, and 'SV AMP' should be 'SVAMP'.","section":"Table 2"}],"recommendation":"reject","confidential_remarks":"The internal HellaSwag inconsistency is load-bearing and would require a re-run of the key experiments to resolve. The parameter-efficiency claim in the abstract is not matched by the LLaMA-7B experiments, and the robustness section contains unexplained large deviations from the main LoRA results. The paper would need substantial additional experiments and matched baselines before it could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a plausible PEFT variant: it takes the task-vector idea from Zou et al.'s representation-editing work, applies a cosine-similarity soft mask to hidden states, adds a low-rank transformation, and stacks that on LoRA. That combination is new as far as I know, and the GLUE results look reasonable — LoRATRF gets 89.52 average with 0.88M trainable parameters vs AdaLoRA's 89.49 with 1.27M. The writing is clear and the related work is cited properly. That is the good part.\n\nThe soft spot is the load-bearing number. Table 2 lists LoRATRF at 82.4 on HellaSwag while the text claims 85.6 and says it beats DoRA by 0.8%. The table shows DoRA at 84.8, so if 82.4 is the real number, LoRATRF loses by 2.4 points on the very benchmark used to support the reasoning claim; if 85.6 is real, Table 2 is wrong. Either way the reader cannot trust the center of the paper. This is not a small typo in a peripheral result, and the stress-test note is right to call it out.\n\nThe other issues are more standard but still real. LLaMA baselines are borrowed from other papers with no identical setting, there are no error bars or significance tests in Table 2, and the 33.71% parameter reduction is computed only against LoRA r=8 on GLUE, not against the SOTA methods in the reasoning experiments. A simple learned-scaling baseline would also help, since the task-vector reweighting might not be doing much beyond per-layer scaling.\n\nNet: the idea is worth one paragraph in a future survey, but the empirical case as written doesn't support 'better performance.' I would not send this to referees in current form. Ask the authors to fix the HellaSwag number, rerun all baselines under one protocol, and report variance; after that it could be a reasonable short paper.\n\nRecommendation: desk-reject now, invite resubmission after the numbers are reconciled.","headline":"A plausible LoRA extension undercut by an unreconciled HellaSwag number: the text says 85.6 while Table 2 says 82.4, and the claimed reasoning advantage disappears on inspection.","tokens_in":9816,"tokens_out":5185,"would_cite":false,"duration_ms":51041,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a task-aware filter to each layer lets low-rank fine-tuning match or beat stronger baselines with fewer trainable parameters.","keywords":["parameter-efficient fine-tuning","low-rank adaptation","task-aware filter","task vector","representation editing","LoRA","GLUE","LLaMA"],"falsifier":"Re-run the HellaSwag experiment and check whether the accuracy is 85.6 or 82.4, since the paper's text claims 85.6 but Table 2 reports 82.4; if the true value is 82.4, the claimed gain over DoRA disappears. Also freeze the task vector $t_\\xi$ to a random vector and repeat the GLUE runs: if accuracy stays the same, the learned direction is not the cause of the gains.","tokens_in":8730,"feed_emoji":"🎯","tokens_out":4414,"duration_ms":42181,"temperature":0.7,"pith_summary":"The paper proposes LoRATRF, a parameter-efficient fine-tuning method that augments LoRA with a per-layer task-aware filter. The filter learns a task vector, reweights hidden tokens by cosine similarity to that vector, and adds the refined representation back into the network. The authors claim this narrows the gap between LoRA and full fine-tuning, reducing trainable parameters by 33.71% while matching or exceeding state-of-the-art low-rank methods on GLUE, commonsense reasoning, and math reasoning tasks.","feed_headline":"Task-aware filter beats low-rank baselines with 33% fewer parameters","feed_subtitle":"A per-layer task vector reweights hidden tokens, helping low-rank fine-tuning beat stronger baselines with fewer parameters.","key_machinery":"The task-aware filter consists of a learnable task vector $t_\\xi$ and a low-rank transformation matrix $T$. Each token $h_i^l$ is reweighted by $\\mathrm{sim}(h_i^l, t_\\xi)$ clamped to $[0,1]$, then multiplied by $T$, and the refined vector is added to the original hidden state. The filter's role is to selectively amplify task-relevant features in every layer, with parameter efficiency coming from approximating $T$ as a product of two low-rank matrices. This mechanism is the only added component beyond standard LoRA, so it carries the paper's entire performance claim.","core_discovery":"The central claim is that editing hidden representations with a task-aware filter makes low-rank adaptation more effective. In each Transformer layer, a learnable task vector $t_\\xi$ computes a soft mask via cosine similarity clamped to $[0,1]$, the masked token is linearly transformed by a low-rank matrix $T$, and the result is added back to the original representation. On GLUE with DeBERTaV3-base, LoRATRF uses 0.88M trainable parameters versus 1.27M for AdaLoRA and reaches an average score of 89.52, the best among the compared low-rank methods. On LLaMA-7B reasoning benchmarks, the paper reports the best GSM8K result and claims an accuracy gain over DoRA on HellaSwag.","pith_inferences":["The paper does not ablate the task vector against a fixed random vector, so a natural test is whether the learned direction encodes genuine task identity or merely acts as a learnable per-layer scaling.","If the filter works by amplifying task-relevant tokens, the learned $t_\\xi$ directions should be consistent across similar tasks and should shift systematically when the task changes, which could be inspected by probing trained filters.","The method might transfer to multi-task or continual learning settings, where a shared task filter could either help or interfere; that extension is untested.","Resolving the paper's internal HellaSwag discrepancy (85.6 in the text versus 82.4 in Table 2) would clarify whether the claimed gain over DoRA is real."],"forward_implications":["LoRATRF can be dropped into existing LoRA pipelines without changing the frozen backbone, offering a direct upgrade for current fine-tuning code.","If the gains hold across ranks, the method makes low-rank fine-tuning viable at very small ranks such as $r=4$ and $r=8$, where LoRA degrades sharply.","The 33.71% parameter reduction on GLUE suggests that feature reweighting can substitute for a substantial amount of adapter capacity.","Improvements on math and commonsense reasoning benchmarks imply the filter's benefit is not confined to classification-style NLU tasks."],"supporting_citations":[{"why":"Defines the LoRA baseline that LoRATRF extends and is the source of the low-rank update formulation.","marker":"(Hu et al. 2022)"},{"why":"Provides the AdaLoRA baseline and the reported GLUE baseline results that LoRATRF compares against.","marker":"(Zhang et al. 2023a)"},{"why":"Supplies the DoRA baseline that LoRATRF reports surpassing on LLaMA-7B reasoning tasks.","marker":"(Liu et al. 2024)"},{"why":"Introduces the concept of task-aware filters that the paper adapts for hidden-representation editing.","marker":"(Zou et al. 2023)"},{"why":"Defines the GLUE benchmark used for the NLU evaluation.","marker":"(Wang et al. 2019)"},{"why":"Provides the LLaMA-7B backbone used for the commonsense and mathematical reasoning experiments.","marker":"(Touvron et al. 2023)"}],"fun_headline_variants":["Task-aware low-rank tuning: smaller, and stronger","Filtered low-rank adaptation beats baselines with 33% fewer params","LoRATRF: Task-relevant masks enhance low-rank fine-tuning","Editing hidden states with task filters improves low-rank LLM tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if a single learned task vector, combined with cosine-similarity reweighting and a low-rank transformation, reliably identifies which features matter for each task in every layer of the network.","fun_headline_variants_meta":{"raw":{"variants":["Task-aware low-rank tuning: smaller, and stronger","Filtered low-rank adaptation beats baselines with 33% fewer params","LoRATRF: Task-relevant masks enhance low-rank fine-tuning","Editing hidden states with task filters improves low-rank LLM tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1763,"prompt_tokens":865,"completion_tokens":898,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":825}},"tokens_in":481,"tokens_out":898,"duration_ms":9199,"temperature":1.0,"reasoning_tokens":825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:40:28.414229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the HellaSwag experiment and check whether the accuracy is 85.6 or 82.4, since the paper's text claims 85.6 but Table 2 reports 82.4; if the true value is 82.4, the claimed gain over DoRA disappears. Also freeze the task vector $t_\\xi$ to a random vector and repeat the GLUE runs: if accuracy stays the same, the learned direction is not the cause of the gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GLUE benchmark used for the NLU evaluation."}],"review_version":1}