{"id":"90cc96f5-d321-4f77-b759-a48381f81aa0","arxiv_id":"2607.18413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding a residual depthwise convolution (kernel 3) to QKV projections before attention raises average downstream accuracy in Qwen3-1.7B/4B by 1.6-3.8 points with negligible parameter cost.","lead":"A small sliding-window filter placed right before attention improved average benchmark scores in Qwen3 language models by roughly 1.5 to 3.8 points. The gains were consistent in direction but came from single runs, so they need confirming with repeated experiments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"At 4B/30B the entire headline gain rests on WSC, the most volatile task (baseline 50.00→36.54 across budgets); excluding WSC that row is negative. Single-run, seed-free evals cannot support 'improves average accuracy.'","rationale":"The reader's weakest assumption is correct and is the load-bearing point: with a single run per cell, the headline average gains cannot be distinguished from evaluation noise. The paper itself admits at the end of §5 that 'additional training budgets and repeated runs would be needed to establish a general scaling trend,' and §4 notes that an aggregate analysis would be needed to establish a general mechanism. These self-imposed limitations align with my concern. I would not move to REJECT because the direction of average gain is positive in all four settings, and even excluding WSC three of four rows remain positive; the design may well have a real effect. But the 4B/30B row, the one most independent of the 1.7B/100B selection setting, depends entirely on WSC, which shows the largest unexplained swings. Repeated seeds and confidence intervals are the minimal check that would settle whether the claim, as stated, should be accepted. The parameter-count inconsistency is real but secondary: if the accuracy claim survives repeated seeds, the abstract still needs correction from '<0.01%' to roughly 0.03%.","tokens_in":8764,"tokens_out":5909,"duration_ms":54799,"concrete_test":"Rerun Table 8 at least 3 seeds per condition (minimum: Qwen3-4B/30B plus Qwen3-4B/100B) with identical evaluation; report mean±std of each benchmark and of the 7-task average, plus the WSC-excluded average. Pre-register the aggregate statistic. If the 4B/30B mean gain is not positive, or the 95% CI of the pooled gain crosses zero, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim requires the observed average gains to be signal. Table 8 is a single run for every cell, and the decisive row is Qwen3-4B at 30B tokens: Conv1D improves average by 1.57 points, but if WSC is excluded the averages are 58.32 (baseline) vs 57.91 (Conv1D), i.e., a 0.41-point deficit. WSC is exactly the task with the least stable scores: the baseline drops from 50.00 at 30B to 36.54 at 100B, a 13.5-point swing at fixed architecture, and Conv1D itself moves 63.46→54.81. With no seeds or confidence intervals, this one task can determine whether an entire row supports the claim. The other three rows remain positive without WSC (+1.04, +0.79, +1.34), so the design is not obviously null, but the central claim is 'improves the average accuracy' across all reported settings, and the evidence at 4B/30B is not robust. A secondary internal inconsistency independently weakens the 'less than 0.01% parameters' assertion: Table 2/Table 1 parameters imply 0.027–0.034%, not <0.01%.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes inserting a residual depthwise Conv1D (kernel size 3, random initialization, no normalization/activation) on the concatenated QKV projection outputs before attention in a Qwen3 Transformer block. It presents a macro-level placement study (17 candidate locations), micro-level module design ablations, and downstream evaluations on seven benchmarks for Qwen3-1.7B at 30B tokens and Qwen3-4B at 30B/60B/100B tokens. The authors claim the selected module improves average benchmark accuracy while adding less than 0.01% parameters, and include a WSC case study suggesting that repeated token IDs become more context-sensitive.","tokens_in":9116,"tokens_out":7976,"duration_ms":62944,"significance":"If the average improvements are real, the contribution is a simple, low-cost architectural addition that could transfer across Transformer LLMs. The controlled placement and design ablations are a strength: the module was selected using training loss and WikiText-103 perplexity, not downstream benchmark outcomes, which mitigates circularity. However, the headline empirical claim currently rests on single-run evaluations whose decisive row is driven by a volatile benchmark, and the parameter-overhead claim is inconsistent with the paper's own tables. These issues are fixable but require revision before the claim can be accepted.","major_comments":[{"comment":"The central claim that Conv1D 'improves the average accuracy on seven downstream benchmarks' rests entirely on single-run evaluations. For Qwen3-4B at 30B, the +1.57 average gain is driven by WSC (50.00→63.46); excluding WSC, the averages are 58.32 (baseline) vs. 57.91 (Conv1D), a 0.41-point deficit. WSC is also the least stable task in the table: the baseline drops from 50.00 at 30B to 36.54 at 100B, and the Conv1D column moves 63.46→54.81. With no seeds or confidence intervals, the observed 4B/30B gain cannot be distinguished from noise, and the abstract/§1 claim 'across ... several pre-training data budgets' is not established for all four settings.","section":"§5.1, Table 8"},{"comment":"The 'less than 0.01% parameters' claim is contradicted by the paper's own counts. Baseline Qwen3-1.7B is 1720.57M; Conv+Shortcut is 1721.03M, an increase of 0.46M (0.027%). The P5 configuration without shortcut (Table 1) is 1721.15M, an increase of 0.58M (0.034%). The §3.1 back-of-the-envelope figure of 0.006% uses k=2 depthwise conv without the shortcut/bias and does not describe the selected module. Please correct the parameter-overhead statement in the abstract, Figure 1, and §1.","section":"Abstract, §3.1, Table 2"}],"minor_comments":[{"comment":"The location ablation says models are trained on FineWeb-100B, but it is not stated how many tokens are used (the dataset name vs. the 100B budget in Table 8). Please clarify the training budget for Table 1.","section":"§3.2"},{"comment":"No random seeds or evaluation variance are reported. Even for the main results, a single run cannot distinguish a 1-2 point average improvement from noise; consider reporting repeated-seed means and standard deviations.","section":"§5.1"},{"comment":"The parameter counts are not obvious: 'Convolution' (1721.15M) has more parameters than 'Conv + Shortcut' (1721.03M), although the shortcut adds no parameters. Please specify how biases, kernel widths (H_q+2H_kv), and the residual path are counted.","section":"§3.3, Table 2"},{"comment":"The case study is explicitly anecdotal. The sentence 'this case shows that the convolution changes QKV representations according to nearby context' is acceptable, but the abstract's 'further suggests' might be strengthened or softened depending on the aggregate analysis promised.","section":"§4, Fig. 4"},{"comment":"The phrase 'less than 0.01% parameters' appears in the abstract, contributions, and Figure 1; once corrected, ensure consistency across all occurrences.","section":"§1 and Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The WSC sensitivity and the parameter-count error are likely to be the main concerns for other readers. The single-run issue is fixable by more seeds or a more cautious claim; the parameter count is a straightforward correction. I would support publication after these are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a clean empirical ablation: 17 conv insertion points in a Qwen3 block, with a careful micro-study that lands on a residual depthwise Conv1D, kernel 3, no activation, on the concatenated QKV projections. The placement sweep is the genuinely new bit; CvT did convolutional projections, but nobody has systematically compared post-QKV vs. post-attention vs. FFN locations in a modern LLM block in one controlled setting. The causal conv with a sliding cache that keeps decoding cost flat is a nice detail. The work is worth engaging with.\n\nThe main claim — \"improves average accuracy on seven benchmarks while adding less than 0.01% parameters\" — has two problems. First, the parameter number is wrong. Table 2 shows ~0.46M added parameters on the 1.7B model, which is about 0.027%, and the P5 row in Table 1 implies the same order; the \"less than 0.01%\" figure in the abstract and figure caption does not match the paper's own tables. Second, every benchmark cell in Table 8 is a single run. WSC, which drives much of the gain, is extremely unstable: the baseline drops 50.00 → 36.54 between 30B and 100B, and the Conv1D model moves 63.46 → 54.81. Exclude WSC and the 4B/30B row goes negative (58.32 vs 57.91). Other rows stay positive without WSC, so the design is not obviously null, but \"improves average accuracy\" as a universal statement is not supported by these numbers.\n\nThere is also a mild selection issue: the module was chosen on Qwen3-1.7B validation loss/perplexity, and the headline 1.7B/30B result is that same setting. The 4B rows at 60B/100B are more independent, and they look better, but still noisy.\n\nWhat's good: the ablation is systematic, the reporting of negative results (activation, reparameterization, second conv) is honest, and the limitations section acknowledges single-run caveats in the scaling trend. The paper does not overclaim the mechanism; the one-WSC representation example is presented as illustrative.\n\nWho should read: people thinking about architectural inductive biases in Transformers, and anyone considering cheap retraining tweaks. It deserves a serious referee: the placement study is a real contribution, and with repeated seeds, error bars, artifact release, and a corrected parameter claim, the central result could become solid. I'd send it out rather than desk-reject, but I wouldn't cite the average-gain claim as is.\n\nRecommendation: conditional accept after major revision; require seeds and error bars.","headline":"Systematic placement study and a plausible lightweight conv module, but the headline average gain rests on single-run evaluations and WSC noise, and the parameter claim doesn't match the paper's own tables.","tokens_in":9600,"tokens_out":2099,"would_cite":false,"duration_ms":23220,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Inserting a 3-wide depthwise convolution before self-attention improves average LLM benchmark accuracy by up to 3.76 points while adding less than 0.01% parameters.","keywords":["depthwise convolution","local inductive bias","self-attention","large language models","QKV projection","Qwen3","Winograd Schema Challenge","parameter efficiency"],"falsifier":"Rerun the Qwen3-4B 100B comparison with, say, ten fresh seeds per condition and recompute average accuracy with and without WSC. If the 3.76-point gap collapses to within run-to-run variance, or if removing WSC makes the 30B and 60B gains vanish, the paper's central claim would not stand.","tokens_in":8672,"feed_emoji":"📈","tokens_out":5297,"duration_ms":43242,"temperature":0.7,"pith_summary":"Large language models built on self-attention have no built-in preference for nearby tokens, even though much of language meaning lives in short local units. This paper asks whether a single lightweight depthwise convolution can supply that local bias almost for free. It compares 17 insertion points inside a Qwen3 Transformer block and finds that convolving the projected queries, keys, and values just before attention works best. Across Qwen3-1.7B and Qwen3-4B at 30B, 60B, and 100B training tokens, the final design — a residual kernel-3 depthwise conv with no normalization or activation — improves average downstream accuracy by 1.57 to 3.76 points while adding under 0.01% parameters. A single case study further suggests that the conv makes repeated token IDs more sensitive to their immediate contexts.","feed_headline":"A 3-token conv before attention lifts LLM accuracy by up to 3.76 points","feed_subtitle":"The operator feeds self-attention two extra tokens of local context for less than 0.01% added parameters.","key_machinery":"The load-bearing object is the P5 module: a depthwise 1D convolution (one scalar weight per channel, so parameters scale as O(k*d_model)) placed after the QKV linear projections and before grouped-query attention, combined with a residual shortcut, kernel size 3, and neither normalization nor activation. Causal left-padding and a sliding-window cache keep per-token decoding cost constant. The conv supplies each of the three attention inputs with information from the two preceding tokens, which is what the ablation attributes the gains to.","core_discovery":"The central claim is that locality can be restored to a Transformer at negligible cost by placing a causal depthwise convolution with kernel size 3 on the concatenated QKV projection outputs, before attention mixes tokens globally. The paper shows this placement (P5) beats 16 other tested locations, and that the best module is a residual shortcut around the conv with random initialization and no added normalization or activation. Reported results show average accuracy on seven benchmarks improving in every evaluated setting: by 1.99 points for Qwen3-1.7B and by 1.57, 1.91, and 3.76 points for Qwen3-4B at 30B, 60B, and 100B tokens, with parameter overhead below 0.01%. The authors interpret th","pith_inferences":["Editorial inference: because the conv sits on QKV projections just before attention, the same insertion could plausibly transfer to other grouped-query-attention LLM families without changing the attention or FFN formulas.","Editorial inference: kernel size 3 beating both 2 and 4 suggests adjacent-token composition dominates the benefit; task-specific tuning (for example, kernel size 5 for longer phrasal units) is a cheap next experiment.","Editorial inference: the single-example cosine-similarity result is suggestive but not a mechanism proof; an aggregate analysis across many Winograd-style items would be a natural follow-up.","Editorial inference: since the module adds only about 0.1M parameters, including it in large-scale training runs is cheap enough that even a small consistent average gain would pay for itself many times over."],"forward_implications":["Adding this module to Qwen3-1.7B raises average accuracy from 52.71 to 54.70 at 30B tokens; Qwen3-4B goes from 57.13 to 58.70 at 30B, 58.78 to 60.69 at 60B, and 58.65 to 62.41 at 100B tokens.","The accuracy gain comes with less than 0.01% extra parameters (about 0.11M on Qwen3-1.7B versus a ~1.72B baseline), so it is not a capacity effect.","A causal kernel with a decoded-token cache means inference cost does not grow with context length.","The largest single-benchmark gains occur on WSC, where the conv changes repeated-token representations, consistent with a local-context mechanism.","No activation, normalization, second conv location, or multi-branch reparameterization improves on the chosen module."],"fun_headline_variants":["Kernel-3 conv before attention boosts LLM accuracy up to 3.76 pts","Depthwise conv at QKV adds <0.01% params, lifts accuracy by 3.76","Residual conv with k=3 before attention: +3.76 accuracy, <0.01% params","Local conv fixes LLM attention: +3.76 pts, <0.01% cost","QKV conv before self-attention: up to 3.76-point accuracy gain"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reported average accuracy differences are genuine improvements rather than run-to-run noise, because every number is a single run and the most volatile benchmark (WSC) drives much of the gain.","fun_headline_variants_meta":{"raw":{"variants":["Kernel-3 conv before attention boosts LLM accuracy up to 3.76 pts","Depthwise conv at QKV adds <0.01% params, lifts accuracy by 3.76","Residual conv with k=3 before attention: +3.76 accuracy, <0.01% params","Local conv fixes LLM attention: +3.76 pts, <0.01% cost","QKV conv before self-attention: up to 3.76-point accuracy gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3309,"prompt_tokens":713,"completion_tokens":2596,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2473}},"tokens_in":457,"tokens_out":2596,"duration_ms":15916,"temperature":1.0,"reasoning_tokens":2473,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:29:00.608321+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Qwen3-4B 100B comparison with, say, ten fresh seeds per condition and recompute average accuracy with and without WSC. If the 3.76-point gap collapses to within run-to-run variance, or if removing WSC makes the 30B and 60B gains vanish, the paper's central claim would not stand.","supporting_citations":[],"review_version":1}