Pith. sign in

REVIEW 2 major objections 5 minor 15 references

Inserting a 3-wide depthwise convolution before self-attention improves average LLM benchmark accuracy by up to 3.76 points while adding less than 0.01% parameters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:29 UTC pith:XAPRVNIA

load-bearing objection Systematic placement study and a plausible lightweight conv module, but the headline average gain rests on single-run evaluations and WSC noise, and the parameter claim doesn't match the paper's own tables. the 2 major comments →

arxiv 2607.18413 v1 pith:XAPRVNIA submitted 2026-07-20 cs.CL

Convolution for Large Language Models

Yuchuan Tian , Yingte Shu , Wei He , Shuo Zhang , Tianchen Zhao , Chao Xu , Xinghao Chen , Yunhe Wang
show 2 more authors
Hanting Chen Yu Wang
This is my paper
classification cs.CL
keywords depthwise convolutionlocal inductive biasself-attentionlarge language modelsQKV projectionQwen3Winograd Schema Challengeparameter efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models built on self-attention have no built-in preference for nearby tokens, even though much of language meaning lives in short local units. This paper asks whether a single lightweight depthwise convolution can supply that local bias almost for free. It compares 17 insertion points inside a Qwen3 Transformer block and finds that convolving the projected queries, keys, and values just before attention works best. Across Qwen3-1.7B and Qwen3-4B at 30B, 60B, and 100B training tokens, the final design — a residual kernel-3 depthwise conv with no normalization or activation — improves average downstream accuracy by 1.57 to 3.76 points while adding under 0.01% parameters. A single case study further suggests that the conv makes repeated token IDs more sensitive to their immediate contexts.

Core claim

The central claim is that locality can be restored to a Transformer at negligible cost by placing a causal depthwise convolution with kernel size 3 on the concatenated QKV projection outputs, before attention mixes tokens globally. The paper shows this placement (P5) beats 16 other tested locations, and that the best module is a residual shortcut around the conv with random initialization and no added normalization or activation. Reported results show average accuracy on seven benchmarks improving in every evaluated setting: by 1.99 points for Qwen3-1.7B and by 1.57, 1.91, and 3.76 points for Qwen3-4B at 30B, 60B, and 100B tokens, with parameter overhead below 0.01%. The authors interpret th

What carries the argument

The load-bearing object is the P5 module: a depthwise 1D convolution (one scalar weight per channel, so parameters scale as O(k*d_model)) placed after the QKV linear projections and before grouped-query attention, combined with a residual shortcut, kernel size 3, and neither normalization nor activation. Causal left-padding and a sliding-window cache keep per-token decoding cost constant. The conv supplies each of the three attention inputs with information from the two preceding tokens, which is what the ablation attributes the gains to.

Load-bearing premise

The load-bearing premise is that the reported average accuracy differences are genuine improvements rather than run-to-run noise, because every number is a single run and the most volatile benchmark (WSC) drives much of the gain.

What would settle it

Rerun the Qwen3-4B 100B comparison with, say, ten fresh seeds per condition and recompute average accuracy with and without WSC. If the 3.76-point gap collapses to within run-to-run variance, or if removing WSC makes the 30B and 60B gains vanish, the paper's central claim would not stand.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adding this module to Qwen3-1.7B raises average accuracy from 52.71 to 54.70 at 30B tokens; Qwen3-4B goes from 57.13 to 58.70 at 30B, 58.78 to 60.69 at 60B, and 58.65 to 62.41 at 100B tokens.
  • The accuracy gain comes with less than 0.01% extra parameters (about 0.11M on Qwen3-1.7B versus a ~1.72B baseline), so it is not a capacity effect.
  • A causal kernel with a decoded-token cache means inference cost does not grow with context length.
  • The largest single-benchmark gains occur on WSC, where the conv changes repeated-token representations, consistent with a local-context mechanism.
  • No activation, normalization, second conv location, or multi-branch reparameterization improves on the chosen module.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the conv sits on QKV projections just before attention, the same insertion could plausibly transfer to other grouped-query-attention LLM families without changing the attention or FFN formulas.
  • Editorial inference: kernel size 3 beating both 2 and 4 suggests adjacent-token composition dominates the benefit; task-specific tuning (for example, kernel size 5 for longer phrasal units) is a cheap next experiment.
  • Editorial inference: the single-example cosine-similarity result is suggestive but not a mechanism proof; an aggregate analysis across many Winograd-style items would be a natural follow-up.
  • Editorial inference: since the module adds only about 0.1M parameters, including it in large-scale training runs is cheap enough that even a small consistent average gain would pay for itself many times over.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes inserting a residual depthwise Conv1D (kernel size 3, random initialization, no normalization/activation) on the concatenated QKV projection outputs before attention in a Qwen3 Transformer block. It presents a macro-level placement study (17 candidate locations), micro-level module design ablations, and downstream evaluations on seven benchmarks for Qwen3-1.7B at 30B tokens and Qwen3-4B at 30B/60B/100B tokens. The authors claim the selected module improves average benchmark accuracy while adding less than 0.01% parameters, and include a WSC case study suggesting that repeated token IDs become more context-sensitive.

Significance. If the average improvements are real, the contribution is a simple, low-cost architectural addition that could transfer across Transformer LLMs. The controlled placement and design ablations are a strength: the module was selected using training loss and WikiText-103 perplexity, not downstream benchmark outcomes, which mitigates circularity. However, the headline empirical claim currently rests on single-run evaluations whose decisive row is driven by a volatile benchmark, and the parameter-overhead claim is inconsistent with the paper's own tables. These issues are fixable but require revision before the claim can be accepted.

major comments (2)
  1. [§5.1, Table 8] The central claim that Conv1D 'improves the average accuracy on seven downstream benchmarks' rests entirely on single-run evaluations. For Qwen3-4B at 30B, the +1.57 average gain is driven by WSC (50.00→63.46); excluding WSC, the averages are 58.32 (baseline) vs. 57.91 (Conv1D), a 0.41-point deficit. WSC is also the least stable task in the table: the baseline drops from 50.00 at 30B to 36.54 at 100B, and the Conv1D column moves 63.46→54.81. With no seeds or confidence intervals, the observed 4B/30B gain cannot be distinguished from noise, and the abstract/§1 claim 'across ... several pre-training data budgets' is not established for all four settings.
  2. [Abstract, §3.1, Table 2] The 'less than 0.01% parameters' claim is contradicted by the paper's own counts. Baseline Qwen3-1.7B is 1720.57M; Conv+Shortcut is 1721.03M, an increase of 0.46M (0.027%). The P5 configuration without shortcut (Table 1) is 1721.15M, an increase of 0.58M (0.034%). The §3.1 back-of-the-envelope figure of 0.006% uses k=2 depthwise conv without the shortcut/bias and does not describe the selected module. Please correct the parameter-overhead statement in the abstract, Figure 1, and §1.
minor comments (5)
  1. [§3.2] The location ablation says models are trained on FineWeb-100B, but it is not stated how many tokens are used (the dataset name vs. the 100B budget in Table 8). Please clarify the training budget for Table 1.
  2. [§5.1] No random seeds or evaluation variance are reported. Even for the main results, a single run cannot distinguish a 1-2 point average improvement from noise; consider reporting repeated-seed means and standard deviations.
  3. [§3.3, Table 2] The parameter counts are not obvious: 'Convolution' (1721.15M) has more parameters than 'Conv + Shortcut' (1721.03M), although the shortcut adds no parameters. Please specify how biases, kernel widths (H_q+2H_kv), and the residual path are counted.
  4. [§4, Fig. 4] The case study is explicitly anecdotal. The sentence 'this case shows that the convolution changes QKV representations according to nearby context' is acceptable, but the abstract's 'further suggests' might be strengthened or softened depending on the aggregate analysis promised.
  5. [§1 and Figure 1] The phrase 'less than 0.01% parameters' appears in the abstract, contributions, and Figure 1; once corrected, ensure consistency across all occurrences.

Circularity Check

0 steps flagged

No circularity: the module configuration is selected on training loss and perplexity, and the headline benchmark gains are independent evaluations, not fitted predictions.

full rationale

The paper's derivation chain is empirical and self-contained. The convolution location (P5, post-QKV) and module hyperparameters (residual shortcut, kernel size 3, random initialization, no activation or reparameterization) are chosen through controlled ablations using mean training loss and WikiText-103 perplexity on Qwen3-1.7B (Tables 1–7). The downstream benchmarks in Table 8 were not used to select or fit any parameter of the module, and no equation makes the benchmark outcome equal to an input. The Qwen3-4B rows at 30B/60B/100B tokens are held-out with respect to the 1.7B-based design selection and provide independent evidence for the central claim. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in by citation. The paper's own limitations support this reading: it states that 'additional training budgets and repeated runs would be needed to establish a general scaling trend' and that 'the evidence is limited to the evaluated Qwen3 configurations.' Concerns about single-run evaluation, WSC volatility, and the apparent inconsistency between the '<0.01% parameters' claim and the parameter deltas in Tables 1–2 are correctness and robustness issues, not circularity: they do not make the reported improvements an input to the design or the derivation. Therefore no significant circularity is present, and the score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper is empirical and has no theoretical derivation. The design choices were selected by validation, not derived. The main unstated load-bearing assumptions are single-run stability and the representativeness of the chosen benchmarks and backbones.

free parameters (1)
  • depthwise convolution kernel size k = 3
    Selected by validation on WikiText-103 perplexity (Table 3); the central claimed configuration depends on this choice.
axioms (4)
  • domain assumption Causal left-padding by k-1 preserves autoregressive generation and matches the decoding cache.
    Section 5.1 'Causal implementation and decoding'; if the cache does not exactly reproduce the padded training behavior, all downstream numbers are invalid.
  • domain assumption Single-run differences between baseline and Conv1D are attributable to the module, not to random seed or training variance.
    Section 5.1 and Table 8; no repeated runs are reported and WSC scores are highly volatile.
  • domain assumption The average of the seven benchmark accuracies is a meaningful summary despite large benchmark-specific declines.
    Table 8 shows BoolQ, HellaSwag, and WinoGrande declines in several rows; the central claim rests on the average rather than on individual benchmarks.
  • domain assumption Qwen3 backbones and FineWeb-100B pretraining are representative of LLM practice.
    Section 5.1; the paper itself limits the conclusion to the evaluated Qwen3 configurations in Section 6.

pith-pipeline@v1.3.0-alltime-deepseek · 8519 in / 19595 out tokens · 162006 ms · 2026-08-01T15:29:00.608321+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size. Our macro-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention. A subsequent micro-level study favors a residual depthwise convolution with kernel size $k=3$, without additional normalization or activation. Across Qwen3 models and several pre-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than $0.01\%$ parameters. A representation-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context. These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions.

Figures

Figures reproduced from arXiv: 2607.18413 by Chao Xu, Hanting Chen, Shuo Zhang, Tianchen Zhao, Wei He, Xinghao Chen, Yingte Shu, Yuchuan Tian, Yunhe Wang, Yu Wang.

Figure 1
Figure 1. Figure 1: Summary of the downstream evaluation. Left: average accuracy at the 30B-token training budget for the evaluated Qwen3 model sizes. Right: average accuracy of Qwen3-4B at three training-data budgets. The post-QKV depthwise convolution adds less than 0.01% parameters. We organize the study from macro-level placement to micro-level module design. At the macro level, we insert depthwise Conv1D at 17 locations … view at source ↗
Figure 2
Figure 2. Figure 2: A Qwen3 Transformer block and the 17 candidate locations for a Conv1D module. 3.2 Comparing Convolution Locations [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The four Conv1D module designs. From left to right: plain convolution, convolution with a shortcut, convolution with pre-normalization, and convolution with sandwich normalization. Module design. We first compare four internal configurations of the depthwise convolution module ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cosine-similarity heatmaps of QKV representations from the Qwen3 baseline and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 14 linked inside Pith

  1. [1]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. arXiv preprint arXiv:2312.04927,

    Pith/arXiv arXiv

  2. [7]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  3. [9]

    The winograd schema challenge.KR, 2012(13th):3,

    Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge.KR, 2012(13th):3,

    2012

  4. [11]

    RWKV: Reinventing RNNs for the transformer era.arXiv preprint arXiv:2305.13048,

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. RWKV: Reinventing RNNs for the transformer era.arXiv preprint arXiv:2305.13048,

    Pith/arXiv arXiv

  5. [12]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  6. [13]

    Gated delta networks: Improving Mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    Pith/arXiv arXiv

  7. [14]

    HellaSwag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,

    Pith/arXiv arXiv 1905

  8. [15]

    TiC: Exploring vision transformer in convolution.arXiv preprint arXiv:2310.04134,

    Song Zhang, Qingzhong Wang, Jiang Bian, and Haoyi Xiong. TiC: Exploring vision transformer in convolution.arXiv preprint arXiv:2310.04134,

    Pith/arXiv arXiv

  9. [2019]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    Pith/arXiv arXiv

  10. [2020]

    Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372,

    Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Yukun Li, et al. Conditional memory via scalable lookup: A new axis of sparsity for large language models.arXiv preprint arXiv:2601.07372,

    Pith/arXiv arXiv

  11. [2021]

    Multi-token attention

    Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Multi-token attention. arXiv preprint arXiv:2504.00927,

    Pith/arXiv arXiv

  12. [2023]

    Simple linear attention language models balance the recall–throughput tradeoff.arXiv preprint arXiv:2402.18668,

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall–throughput tradeoff.arXiv preprint arXiv:2402.18668,

    Pith/arXiv arXiv

  13. [2024]

    Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100,

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100,

    Pith/arXiv arXiv 2005

  14. [2025]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al

    URLhttps://arxiv.org/abs/2503.04715. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  15. [2026]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Pith/arXiv arXiv 1905