{"id":"9cafc719-ee7e-4c9c-822b-118443fb1e53","arxiv_id":"2607.13425","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.","lead":"A new fine-tuning method for LLMs learns just one number per attention head, letting models adapt to text-classification tasks from as few as 10 labeled examples. It beats standard low-rank fine-tuning (LoRA) in the low-data regime while modifying fewer than one-millionth of the model's parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison to LoRA is confounded by fixed r=1 baseline not tuned by the paper's own CV protocol; re-tuning rank/alpha and testing significance is needed.","rationale":"The paper presents a simple, plausible PEFT method with broad experimental support for the narrow claim that AHR beats LoRA/AdaLoRA/IA3 when training data is very small. The central claim is defensible, but the paper should address the overclaim of 'consistently outperforms' (Emotion counterexample in Table 2), integrate the ICL baseline into the main comparison (it often wins on larger models), and provide released code with a commit hash. These are condition-worthy issues rather than fatal flaws. My additional concern is that the comparison to LoRA is not fully fair: LoRA's rank and alpha are fixed a priori, while AHR's hyperparameters are tuned via CV. Because rank controls capacity, this asymmetry directly affects the reported margin. The reader's weakest assumption (functional specialization / expressivity) is explicitly conceded in Section 6.1 and does not invalidate the empirical claim; it only bounds its scope. Therefore I agree with the conditional verdict but for a different, more empirically load-bearing reason. The proposed re-tuning experiment is feasible and would settle whether the headline advantage holds against a properly tuned baseline.","tokens_in":22742,"tokens_out":11146,"duration_ms":109439,"concrete_test":"Re-run the low-data comparison (|D_train| = 10, 15, 20, 30) across the 6 datasets and 4 models, using the paper's 5-fold CV to select LoRA rank (1,2,4,8) and alpha (1,2,4) in addition to lr/steps/reg strength. Then compute paired differences (AHR minus best-tuned baseline) over the 10 seeds × 4 models and test with Wilcoxon signed-rank or bootstrap CIs per dataset. If AHR is not significantly positive in the majority of low-data settings, the claim should be downgraded from 'outperforms' to 'is competitive with a well-tuned low-rank baseline.' If it remains significant, the original claim is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 fixes LoRA to r=1 and alpha=1 based on 'increased training stability' rather than including these capacity hyperparameters in the stated 5-fold CV selection. The paper's protocol says all hyperparameters are selected via CV, so this is asymmetric: AHR's learning rate, steps, and regularization strength are tuned per seed, while LoRA's rank — the primary capacity control in low-data settings — is set a priori. A rank-1 LoRA may be too constrained to represent a fair 'standard baseline,' or too unstable at D=10 (e.g., Appendix B: Web/GPT2-XL LoRA 41.91 ± 10.90). The central claim ('outperforms LoRA with 200-1000x fewer parameters') therefore conflates the effect of per-head scalar reweighting with the effect of an untuned, possibly over-regularized baseline. If LoRA rank and alpha were selected under the same CV protocol, the reported 2-4% margin might shrink or reverse. The reader's expressivity concern is real but explicitly acknowledged in Section 6.1 and does not threaten the narrow empirical claim; the baseline-tuning asymmetry does.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Attention Head Reweighting (AHR), a parameter-efficient adaptation method that learns one scalar per attention head, multiplying that head's residual-stream contribution. On six text-classification datasets and four LLMs (GPT2-XL, Llama-3.2-1B/3B, Qwen3-8B), it reports that AHR outperforms PEFT baselines (LoRA, AdaLoRA, IA3) in low-data regimes (|D_train| ≤ 30) by 2–4% absolute accuracy, while using 200–1000× fewer trainable parameters. Training is done with in-context finetuning and L1/L2 regularization, with hyperparameters selected by per-seed 5-fold CV. The paper also presents interpretability analyses of the learned head weights, including a case study where a few task-specific and general ICL heads are identified and manually manipulated.","tokens_in":23063,"tokens_out":3918,"duration_ms":41215,"significance":"If the empirical claims hold, AHR is a conceptually clean and practically attractive method: it uses an extremely small parameter budget, can be merged into W_O for zero inference overhead, and provides an interpretable interface to model adaptation. The evaluation is broad (4 models, 6 datasets, 10 seeds) and the CV-within-training-set protocol is a reasonable attempt to honor data scarcity. The interpretability analyses (Section 5, Appendix A.2) are a genuine strength: they connect the method to functional specialization and test causal hypotheses about specific heads. However, the central comparative claim—'AHR outperforms standard baselines like LoRA'—is currently weakened by a baseline-configuration asymmetry and by overstatement relative to the paper's own per-model tables. The method is worth publishing after the claims are calibrated and the LoRA baseline is made fair.","major_comments":[{"comment":"The comparison to LoRA is confounded: LoRA is fixed to r=1 and α=1 a priori ('which we found to increase training stability'), while AHR's learning rate, steps, and regularization strength are selected per seed by 5-fold CV. Rank is LoRA's primary capacity control in low-data settings; a rank-1 LoRA may be too constrained or too unstable (e.g., Web/GPT2-XL at D=10: 41.91±10.90, Table B.13). To support 'outperforms standard baselines like LoRA', LoRA's rank/α must be included in the same CV protocol, or a sensitivity analysis over r and α must show the reported margin is robust.","section":"§4.1, LoRA configuration"},{"comment":"The claim that AHR 'consistently outperforms existing PEFT methods' is contradicted by the paper's own per-model tables. Examples: Table B.7 (Llama-3.2-3B, AG News, D=10): AHR 84.55 vs AdaLoRA 86.59 and LoRA 85.69; Table B.10 (Llama-3.2-1B, Emotion, D=10): AHR 55.95 vs AdaLoRA 57.22; Table B.22 (Llama-3.2-1B, Jailbreak, D=30): AHR 86.41 vs ICL 91.07. The averaged Table 2 omits the ICL baseline, which is however invoked in §4.2 as the reference for overfitting. The headline should be reworded to reflect the per-model/dataset picture and the exact baseline set.","section":"§4.2 and Table 2 vs Appendix B"},{"comment":"The paper reports 10-seed means with standard errors but no significance tests. Many of the claimed improvements are within overlapping error bars (e.g., Web Llama-3.2-3B D=10: AHR 86.97±1.05 vs ICL 87.32±1.04; SST2 Llama-3.2-1B D=15: AHR 90.31±1.20 vs LoRA 91.20±0.96). Given that the central claim is '2–4% absolute improvement', paired or per-seed significance testing (e.g., bootstrapped paired differences) is needed to establish whether the effect is real and not seed noise.","section":"Statistical significance"}],"minor_comments":[{"comment":"'~0.0001% of the model's parameters' is accurate for some models but should be stated per model (e.g., Qwen3-8B: 1152/8.19B ≈ 0.000014%).","section":"Abstract/§1"},{"comment":"The notation h(z)=Az W_V W_O is slightly compressed; a parenthetical clarifying that A is computed from W_Q, W_K would improve readability.","section":"§3, Eq. (1)"},{"comment":"The plot showing accuracy vs number of modified heads would benefit from error bars or shaded confidence intervals, as the text reports a 10-seed average.","section":"Fig. 3b"},{"comment":"The manual head-manipulation experiment (Figure A.3) is a nice causal check, but the heads were selected from the same datasets used for evaluation; this potential circularity should be acknowledged or mitigated by selecting heads on a held-out split.","section":"§5.3 / Appendix A.2"},{"comment":"The term 'best baseline' is used ambiguously: in Table 2 the PEFT baselines are averaged, while in the text ICL is also called a baseline (§4.2). Define the baseline set explicitly in the experimental setup.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core idea and a broad evaluation, but the main empirical claim needs work before acceptance: the LoRA baseline must be tuned fairly, the 'consistently outperforms' language must be reconciled with the appendix tables, and significance testing should be added. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, simple PEFT paper with good low-data results, but the paper's own headline is too strong. The method — one scalar per attention head, merged into W_O — is the obvious next step after (IA)3 and attention steering, and the evaluation is careful: four models, six datasets, ten seeds, error bars. I believe the narrow claim: with ≤30 training examples, AHR beats LoRA, AdaLoRA, and IA3 on average, with 200-1000x fewer trainable params. The IC-FT ablation is useful, and the head-level analysis (L15H14/L15H3 as general ICL heads, L14H11 as phishing-specific) is a genuine bonus, with the manual intervention in Appendix A.2 giving real causal evidence.\n\nThe soft spots are real but not fatal. First, the stress-test note about LoRA is largely on target. Section 4.1 fixes LoRA to r=1, α=1 'for training stability' rather than putting rank/alpha in the stated 5-fold CV protocol. That asymmetry means the 'outperforms LoRA' claim is really 'outperforms this particular rank-1 LoRA.' The margin might shrink with a tuned rank, though I doubt it reverses — the D=10 margin on Web and Jailbreak is huge. Still, it needs an acknowledgment or a sensitivity check.\n\nSecond, the 'consistently outperforms' language doesn't survive contact with the appendix. ICL is omitted from the averaged Table 2, but in the per-model tables ICL is frequently the best method on Llama-3.2-3B and Qwen3-8B, especially at D≥20. On Emotion D=100, AdaLoRA beats AHR by 2.4 points. The caveat in Section 6.1 about ≥300 examples is honest, but the abstract and Section 4.2 overstate.\n\nThird, code is promised but not yet released; for a method this simple, a commit hash would be worth requiring.\n\nNet: the paper is a useful contribution to the low-data PEFT space and the interpretability story. It deserves peer review, not desk rejection. I'd ask the authors to tune LoRA's rank under the same protocol, move ICL into the main comparison, and soften 'consistently outperforms' to match the data. I would cite this in my own work on security classification.","headline":"Solid low-data PEFT paper with real results, but the 'consistently outperforms' headline is too strong and the LoRA baseline is not tuned on equal terms.","tokens_in":23529,"tokens_out":2653,"would_cite":true,"duration_ms":27615,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adapting an LLM to a text-classification task can be reduced to learning one scalar per attention head, and in the few-shot regime this beats parameter-efficient baselines like LoRA despite training 200–1000x fewer parameters.","keywords":["attention head reweighting","parameter-efficient finetuning","few-shot text classification","functional specialization of attention heads","in-context finetuning","interpretability","LoRA comparison","LLM adaptation"],"falsifier":"Run AHR on a classification task deliberately constructed so the label is a nonlinear combination of many dispersed features, using 10–30 training examples; if AHR fails to beat the in-context-learning baseline while LoRA succeeds, the functional-specialization premise is refuted. A cheaper check: ablate all heads AHR does not upweight and show whether the task still solves, or verify there exists a task where no single-head scaling captures the decision.","tokens_in":22672,"feed_emoji":"⚖️","tokens_out":4583,"duration_ms":41855,"temperature":0.7,"pith_summary":"This paper argues that a large language model can be adapted to a new text-classification task by learning just one number per attention head — a scalar that scales how much that head's output contributes to the model's residual stream — while everything else stays frozen. The central claim is that in the few-shot regime (roughly 10 to 30 labeled examples), this 'attention head reweighting' (AHR) consistently beats parameter-efficient baselines such as low-rank adaptation (LoRA), despite training 200–1000 times fewer parameters (around 0.0001% of the model). The authors further claim the learned weights are interpretable: the method upweights a handful of task-specific heads plus a few general in-context-learning heads, and the paper analyzes which heads matter. If true, this would give security-style applications — phishing URL detection, jailbreak detection — a way to specialize a large model almost for free, with transparent edits and no inference overhead.","feed_headline":"A scalar per attention head beats LoRA on 10 samples","feed_subtitle":"Reweighting existing heads — not rewriting them — yields 2-4% accuracy gains and zero inference overhead.","key_machinery":"The central object is the attention-head scalar β_h, one per head in every layer. Applying (1+β_h) to head h's output path — equivalently scaling head h's contribution to the residual stream — changes the model's behavior without touching any weights inside the head. Because β_h can be merged into the output projection W_O at inference, the method adds zero inference cost. It is the pairing of (i) an extremely low-dimensional parameter space and (ii) the functional-specialization premise that carries the argument: if task-relevant behavior is concentrated in a few heads, a few scalars suffice to amplify or suppress it.","core_discovery":"The paper's discovery is a parameter-efficient adaptation mechanism: AHR introduces a scalar β_h per attention head, multiplies that head's contribution to the residual stream by (1+β_h), and learns only these scalars by cross-entropy on a few-shot prompt while keeping all original weights frozen. At test time the β_h are folded into the head's output projection W_O, so inference costs nothing extra. Across six text-classification datasets and four models, the method reports a 2–4 point absolute accuracy improvement over the best baseline when training data is ≤30 examples, with larger gains (6–7 points at 10 examples) on phishing-URL and jailbreak detection. The authors attribute this to fu","pith_inferences":["Editorial extension: the near-zero parameter cost makes it cheap to cache many per-task AHR scalar sets and switch between them at inference time, enabling a lightweight model library that the paper mentions only as future work.","Editorial extension: AHR's advantage should grow with model scale if larger models pack more specialized heads per task; the four-model sweep is consistent with but does not isolate that trend, so a scaling-focused experiment would be a natural test.","Editorial extension: the same reweighting logic could apply to interpretable subcomponents other than attention heads, such as sparse-autoencoder latents or circuit nodes, which the paper lists as future work and which would increase expressive power while keeping edits transparent.","Editorial extension: if the shared heads really are general in-context-learning heads, AHR weights trained on one classification task could be transferred as a cheap generic ICL boost to other tasks; the paper shows supporting evidence (heads upweighted on every dataset) but does not test transfer directly."],"forward_implications":["In low-data settings (≤30 examples), AHR claims 2–4% absolute accuracy gains over LoRA/AdaLoRA/IA3 averaged across six datasets and four models, with the largest gains on phishing and jailbreak classification.","Because the learned changes live in ~0.0001% of model parameters and are mergeable into the output projection, deployment adds no inference latency and very little storage.","AHR's edits are less destructive than larger PEFT updates: finetuning on a phishing-URL task slightly improved performance on other classification tasks rather than degrading it, a result the paper attributes to avoiding overfitting.","Analyzing the learned β weights identifies a small set of heads shared across tasks (suggesting general in-context-learning heads) and task-specific heads; manually doubling or disabling two such heads shifts average accuracy by roughly ±2.5–3.9 points, supporting the mechanistic story.","With more data (roughly ≥300 examples), the paper concedes other finetuning methods overtake AHR, so the claimed advantage is specifically the data-scarce regime."],"fun_headline_variants":["One scalar per attention head: 1000x fewer params, beats LoRA on few-shot","Reweight attention heads with one scalar, outlearn LoRA on 10 samples","AHR: 200-1000x fewer trainable params, better few-shot accuracy","Attention head reweighting: single scalar per head, big few-shot gains","One scalar per attention head: zero inference overhead, better than LoRA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method's effectiveness hinges on the assumption that the target task's behavior is concentrated enough in existing attention heads that multiplying some of those heads by a scalar can express the needed decision; if the relevant computation is spread thinly across many heads or requires nonlinear changes inside heads, AHR cannot express the task.","fun_headline_variants_meta":{"raw":{"variants":["One scalar per attention head: 1000x fewer params, beats LoRA on few-shot","Reweight attention heads with one scalar, outlearn LoRA on 10 samples","AHR: 200-1000x fewer trainable params, better few-shot accuracy","Attention head reweighting: single scalar per head, big few-shot gains","One scalar per attention head: zero inference overhead, better than LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2489,"prompt_tokens":713,"completion_tokens":1776,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1668}},"tokens_in":457,"tokens_out":1776,"duration_ms":12805,"temperature":1.0,"reasoning_tokens":1668,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:11:51.119933+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AHR on a classification task deliberately constructed so the label is a nonlinear combination of many dispersed features, using 10–30 training examples; if AHR fails to beat the in-context-learning baseline while LoRA succeeds, the functional-specialization premise is refuted. A cheaper check: ablate all heads AHR does not upweight and show whether the task still solves, or verify there exists a task where no single-head scaling captures the decision.","supporting_citations":[],"review_version":1}