Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Block sparse attention matches dense quality if it keeps the blocks with the largest attention mass; a compressed second-order key covariance estimates that mass well enough to close most of the gap.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 00:41 UTC pith:M5YTSXUB

load-bearing objection Solid mechanism paper: mass-oracle selection is the right target, and a compressed second-order covariance selector closes most of the first-order gap under a controlled NSA setup. the 3 major comments →

arxiv 2607.09052 v1 pith:M5YTSXUB submitted 2026-07-10 cs.LG

COBS: Cumulant Order Block Sparse Attention

Alexander Tian , Aditya Ghai , Sanjit Neelam , Zaal Vasania , Akshay Mishra This is my paper
classification cs.LG
keywords block sparse attentionKV cacheattention masscumulant expansionNative Sparse Attentionlong-context retrievalsecond-order selectorCOBS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-context language models are limited by how much of the key–value cache they must read at every decode step. Block sparse attention is hardware-friendly, yet leading open models still prefer dense or fine-grained selection. This paper isolates the hard part—choosing which blocks to keep—and shows that the right criterion is ranking blocks by attention mass, the total softmax weight inside each block. An oracle that ranks by the true mass essentially recovers dense accuracy. Existing cacheable selectors only estimate that mass to first order in the query, so they miss within-block curvature. COBS stores a compressed second-order statistic (a low-rank, query-subspace, FP4 key covariance) per block and scores with the second-order cumulant term. On 32k RULER retrieval it lifts a first-order NSA baseline from 0.30 to 0.82, closing about 86% of the gap to dense while reading far less cache traffic, without harming short-context behavior.

Core claim

If a block selector retrieves the blocks with the largest attention mass, sparse attention can match dense quality. The remaining problem is estimating that mass from a compact, query-independent summary. A cumulant expansion of the mass shows why first-order summaries fail and why adding a compressed within-block key covariance restores the missing second-order term. COBS implements that selector and closes most of the measured gap to dense attention on long-context retrieval.

What carries the argument

Attention mass and its cumulant expansion: the log mass of a block equals a constant plus the cumulant generating function of its keys, whose second-order term is one-half the query quadratic form with the within-block key covariance. COBS caches a compressed form of that covariance and scores blocks with the second-order truncation.

Load-bearing premise

The argument that selection should rank by additive attention mass rests on relaxing the true reconstruction error into a value-agnostic, linearized bound; if those relaxations fail for real multi-head groups, the mass target itself is the wrong objective.

What would settle it

Train the same backbone with the same protocol, then replace COBS’s second-order scores with exact block masses (the mass oracle) versus a first-order summary on multi-key long-context retrieval; if the second-order selector does not close most of the oracle-to-baseline gap, or if the oracle itself falls far below dense, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Hardware-friendly large-block sparse attention can approach dense long-context quality without reading every past key.
  • Cacheable selectors should be judged by cumulant order: first-order summaries are fundamentally curvature-blind.
  • Storing a low-rank, subspace-projected, FP4 covariance per block is enough to recover most of the mass-oracle gain at a small multiple of mean-pool traffic.
  • Short-context reasoning and position-wise language-modeling loss need not degrade when the second-order selector is used.
  • Design work on block sparse methods can focus on better mass estimators rather than on inventing a new selection criterion.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mass criterion is correct, any future block selector—learned or analytic—should be evaluated first by how well its ranking correlates with true attention mass, not only by end-task accuracy.
  • The same second-order summary could compose with token-level indexers or long-span compression branches, since those methods address orthogonal cost axes.
  • High-rank covariance regression on multi-key tasks suggests a practical operating band for stored rank rather than “more eigenvectors always help.”
  • A value-aware or head-specific mass score might close the remaining oracle–dense gap if the value-agnostic bound is the residual bottleneck.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies block-sparse attention through DeepSeek’s NSA, isolating the selection branch. Under assumptions A1–A3 it reduces selection to ranking blocks by attention mass (Eq. 5), shows that an exact-mass oracle (OSA) nearly matches dense attention on an 11-task 32k RULER suite (0.9010 vs 0.9040), and argues via the cumulant expansion of the block mass that first-order cacheable selectors miss the second-order curvature term ½qᵀΣ_bq. COBS stores a compressed within-block key covariance (low-rank, optional query-subspace projection, FP4) and, under the paper’s controlled NSA protocol, raises mean RULER score from the NSA-MLP baseline 0.2999 to 0.8195 (≈86 % of the dense gap) at 1.21× the baseline’s KV-read traffic and 15.15× less traffic than dense, while preserving short-context parity and reporting lower position-wise NLL than dense. Ablations isolate NoPE, rank, subspace, quantization, and negative results on query-centered expansion and diagonal skew.

Significance. If the mass criterion and second-order estimator hold beyond the present scale and protocol, the work supplies both a clean diagnostic for why large-block sparse attention has lagged and a practical, hardware-friendly selector that recovers most of the dense quality at a small multiple of first-order traffic. Strengths include an explicit derivation of the selection objective, the OSA diagnostic that nearly saturates dense performance, systematic ablations that isolate the covariance term, and transparent traffic accounting. The cumulant-order framing is a reusable design axis for cacheable selectors. The main caveats are the ~1.2 B scale, RULER-style SFT on the same task family, and the NoPE confound with the dense baseline; these limit the strength of the external gap claim but do not erase the internal selector comparison.

major comments (3)
  1. Limitations §9 and §5.1 state that sparse variants (including COBS and OSA) strip RoPE from compression/selection (NoPE) while dense retains RoPE throughout. Table 1 shows NoPE alone lifts mean-pool from 0.4186 to 0.5554. The headline 86 % gap-to-dense (0.2999 → 0.8195 vs 0.9040) therefore mixes selector quality with a position-encoding difference that is never ablated on the dense side. A dense-NoPE control, or an explicit statement that the gap percentage is only relative under the paper’s protocol, is needed for the external claim.
  2. §C and Limitations §9: long-context signal is obtained by SFT on generated RULER-style data that uses the same templates as the evaluation suite (instances are stated to be disjoint). Combined with the NoPE confound, this makes the absolute dense-matching numbers and the 86 % figure protocol-dependent. The internal ranking (OSA ≈ dense ≫ COBS ≫ first-order) remains informative, but the paper should either add a non-RULER long-context check or clearly frame the gap percentage as relative selection quality under this SFT protocol.
  3. §3.3 Assumptions A1–A3 (common bound on centroid deviations, discarding unknown c^(g,h), linearizing τ/(1−τ)) reduce head-output reconstruction error to the additive mass score (Eq. 5). For G=1 the ranking is exact under the bound; for GQA the linearization is only leading-order. OSA’s near-match (0.9010) empirically supports the target under the paper’s GQA-4 setup, but a short sensitivity check (e.g., how often the top-k by exact mass differs from the top-k that minimize the true reconstruction error) would strengthen the claim that mass is the right oracle rather than a convenient surrogate.
minor comments (5)
  1. Figure 1 and Table 3: clarify that “summary keys” traffic for COBS includes the mean plus the compressed covariance factors, and that FP4 applies only to the scaled eigenvectors (mean stays bf16).
  2. §5.4: the adaptive s rule (1.25× rank capturing 90 % of query spectral energy) is clear in the text and Figure 6, but a one-line formula in the main text would help reproducibility.
  3. §7.1: NSA MLP is trained at ~1.7 B / ~28 B tokens while other variants are ~1.2 B / ~20 B; the text notes this, but a short remark that the MLP baseline is therefore a charitable first-order comparator would prevent misreading of Table 1.
  4. Typos / polish: “COBS:CUMULANTORDER…” title casing; occasional missing spaces after periods in the abstract; “bZ” notation in Eq. (12) is slightly hard to parse—consider Ẑ.
  5. Related work §6.1: CCQ is correctly distinguished; a sentence on whether any concurrent second-order block selectors exist would complete the landscape.

Circularity Check

0 steps flagged

No significant circularity: mass criterion, cumulant truncation, and COBS estimator are derived independently of the reported RULER scores; OSA is an empirical diagnostic, not a definitional identity.

full rationale

The paper's central chain is: (i) exact single-head reconstruction error (Eq. 3) under A1–A3 reduces to ranking by additive attention mass (Eq. 5 / Appendix A); (ii) OSA applies the exact masses and empirically nearly matches dense (0.9010 vs 0.9040); (iii) the cumulant expansion of log-mass (Eqs. 6–9) is the standard Taylor expansion of the CGF, truncated at second order to motivate storing a compressed covariance (Eq. 10); (iv) COBS is then evaluated as an estimator of that mass against held-out RULER instances and position-wise NLL. None of these steps is self-definitional: the mass ranking is not defined as the quantity that maximizes RULER score, the second-order term is not fitted to the evaluation data, and OSA is a diagnostic that re-reads all keys rather than a tautology. Assumptions A1–A3 are explicit relaxations whose validity is checked by the OSA experiment rather than assumed into the metric. The RULER-style SFT protocol and NoPE scheme are experimental design choices (Limitations §9, §C) that may confound absolute gap-to-dense numbers, but they do not make any equation reduce to its own input by construction. There are no load-bearing self-citations of uniqueness theorems, no fitted constants renamed as predictions, and no ansatz smuggled via prior author work. Design hyperparameters (rank r, adaptive s, FP4) are ablated openly and do not force the headline accuracy. The derivation is therefore self-contained against the paper's own equations and external benchmarks; score 0 is the correct outcome.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The central empirical claim rests on standard attention math plus three explicit selection relaxations, a cacheability constraint, and several design knobs (rank, subspace, quantization, NoPE, RULER-style SFT) chosen by ablation rather than derived. No new physical entity is postulated; the method invents a compressed second-order block descriptor.

free parameters (5)
  • covariance rank r = r=4 (deliverable); peak at r=8
    Chosen by rank sweep on 32k RULER; operating range r≤8, deliverable r=4; higher ranks regress on multi-key tasks.
  • query subspace dimension s = s≈85 average
    Adaptive per-layer rule: 1.25× rank capturing 90% of query spectral energy; average s≈85 chosen for traffic/quality tradeoff.
  • block size L and top-k = L=32, k=16
    Fixed protocol L=32 tokens, top-k=16 blocks, 256-token window; not derived from first principles.
  • FP4 E2M1 quantization of scaled eigenvectors = E2M1 FP4 + bf16 mean
    Chosen for traffic reduction; validated as nearly lossless on the same RULER suite used for the headline claim.
  • NoPE on compression/selection branches = NoPE for sparse branches
    Design choice that improves mean-pool and is applied to COBS/Quest/OSA but not to the dense baseline; confounds dense comparison.
axioms (6)
  • ad hoc to paper A1: within-head centroid deviations share a common magnitude bound across blocks
    §3.3; used to turn exact reconstruction error into a value-agnostic bound.
  • ad hoc to paper A2: ignore unknown per-head deviation coefficients c when ranking
    §3.3; required for a pure mass ranking under GQA.
  • ad hoc to paper A3: replace GQA penalty τ/(1−τ) by its linear leading term
    §3.3–3.4 and Appendix A; makes the score additive over blocks.
  • domain assumption Cacheable selectors must store only query-independent per-block summaries
    §2; forces the cumulant order of the stored summary to be the binding constraint.
  • standard math Multivariate cumulant expansion of the key MGF / CGF
    §4 and Appendix B; standard Taylor expansion of log M_X around q=0.
  • ad hoc to paper Second-order truncation of the CGF is a sufficient mass estimator for selection
    §5.2; higher-order terms (skew) are tried and largely rejected in §7.7.
invented entities (2)
  • COBS second-order block selector (compressed Σ_b descriptor) no independent evidence
    purpose: Approximate block attention mass from a cacheable mean + low-rank/subspace/FP4 covariance summary.
    Core proposed method; independent evidence is the empirical RULER/NLL comparison, not an external physical measurement.
  • Attention mass m_b as the formal selection criterion independent evidence
    purpose: Reduce block selection to ranking a single scalar per block under stated assumptions.
    Defined from existing softmax scores; justified by the reconstruction bound and OSA diagnostic rather than postulated as a new physical object.

pith-pipeline@v1.1.0-grok45 · 24384 in / 3612 out tokens · 46083 ms · 2026-07-13T00:41:34.104912+00:00 · methodology

0 comments
read the original abstract

Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs). However, it is not prevalent among leading open-weight LLMs, which rely instead on dense attention or fine-grained selection, thereby motivating our analysis. We study DeepSeek's Native Sparse Attention (NSA) as a representative method, whose three-branch design lets us isolate block selection, the most challenging and consequential stage. We formalize selection and reduce it to ranking blocks by a single quantity, the attention mass: the sum of a block's attention scores. We show that if selection retrieves the blocks with the largest attention mass, block sparse attention can match the quality of dense attention. However, computing the exact attention mass requires reading every key, so the problem of block selection ultimately reduces to approximating this mass from a compact summary instead of the full keys. Via a cumulant expansion, we show why existing methods falter: their selection strategies attempt to estimate the attention mass, but are confined to a first-order approximation. Therefore, we propose COBS (Cumulant Order Block Sparse Attention), an attention method that builds on NSA, incorporating a novel selector that stores a compressed second-order statistic per block. On the 32k RULER long-context retrieval benchmark, COBS raises the NSA baseline's mean score from 0.2999 to 0.8195, approaching dense attention at 0.9040 and closing about 86% of the gap, while using only 1.21x the KV cache read traffic of the NSA baseline and 15.15x less read traffic than dense. The same model preserves short-context behavior and attains lower position-wise negative log-likelihood (NLL) than dense attention in our comparison.

Figures

Figures reproduced from arXiv: 2607.09052 by Aditya Ghai, Akshay Mishra, Alexander Tian, Sanjit Neelam, Zaal Vasania.

Figure 1
Figure 1. Figure 1: COBS approaches dense accuracy at a fraction of the KV cache read traffic. Accuracy on 32k RULER versus per-decode-step KV cache read traffic (KiB per layer, log scale; see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: First-order block scores miss within-block curvature. Left (2D example): two blocks share the same mean key ¯kb but spread differently relative to the query direction q. Spread codirectional with q (from the blue block) raises the block’s softmax mass while spread orthogonal to q (from the orange block) does not, a difference invisible to the shared mean ¯kb. Right: for queries along a fixed direction qˆ, … view at source ↗
Figure 3
Figure 3. Figure 3: 32k RULER gap ladder (COBS variants in green). The main gain comes from the covariance summary; [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Position-wise next-token NLL on held-out natural LongCrawl64 at 32k (GQA-4, 32k YaRN SFT; [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: COBS full-space rank sweep on 32k RULER. Selection peaks at [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Adaptive per-layer query-subspace dimension [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 23 linked inside Pith

  1. [1]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv:2502.11089, 2025. DeepSeek-AI

    Pith/arXiv arXiv 2025

  2. [2]

    RULER: What’s the real context size of your long-context language models? arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? arXiv:2404.06654, 2024. COLM 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  4. [4]

    Big Bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Bird: Transformers for longer sequences. arXiv:2007.14062, 2020. NeurIPS 2020. 14 COBS: Cumulant Order Block Sparse Attention

    Pith/arXiv arXiv 2007

  5. [5]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv:2309.17453, 2023. ICLR 2024

    Pith/arXiv arXiv 2023

  6. [6]

    H2O: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. arXiv:2306.14048, 2023. NeurIPS 2023

    Pith/arXiv arXiv 2023

  7. [7]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. arXiv:2404.14469, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Loki: Low-rank keys for efficient sparse attention

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. arXiv:2406.02542, 2024. NeurIPS 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Quest: Query-aware sparsity for efficient long-context LLM inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context LLM inference. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024. arXiv:2406.10774

    Pith/arXiv arXiv 2024

  10. [10]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2305.13245

    Pith/arXiv arXiv 2023

  11. [11]

    Hy3: Tencent hunyuan large language model

    Tencent Hunyuan. Hy3: Tencent hunyuan large language model. https://github.com/Tencent-Hunyuan/Hy3, 2026

    2026

  12. [12]

    DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv:2405.04434, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Kimi K2.6.https://huggingface.co/moonshotai/Kimi-K2.6, 2026

    Moonshot AI. Kimi K2.6.https://huggingface.co/moonshotai/Kimi-K2.6, 2026

    2026

  14. [14]

    Gemma 4 technical report

    Gemma Team, Google DeepMind. Gemma 4 technical report. arXiv:2607.02770, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    DeepSeek-V3.2: Pushing the frontier of open large language models

    DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, et al. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026

    Zhipu AI. GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026

    2026

  17. [17]

    DeepSeek-V4: Towards highly efficient million-token context intelligence

    DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, et al. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv:2606.19348, 2026

    arXiv 2026

  18. [18]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017. arXiv:1706.03762

    Pith/arXiv arXiv 2017

  19. [19]

    Dover Publications, Mineola, NY , 2018

    Peter McCullagh.Tensor Methods in Statistics. Dover Publications, Mineola, NY , 2018. Republication of the 1987 Chapman & Hall edition

    2018

  20. [20]

    Don’t read everything: A curvature-conditioned query for linear attention

    Dong Le, Thong Nguyen, Cong-Duy Nguyen, and Anh Tuan Luu. Don’t read everything: A curvature-conditioned query for linear attention. arXiv:2606.01294, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    RoFormer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. arXiv:2104.09864, 2021

    Pith/arXiv arXiv 2021

  22. [22]

    LongCrawl64: A long-context natural-language dataset

    Jacob Buckman. LongCrawl64: A long-context natural-language dataset. Manifest AI, 2024. https://manifestai.com/articles/longcrawl64/

    2024

  23. [23]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

    Pith/arXiv arXiv 2022

  24. [24]

    YaRN: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. arXiv:2309.00071, 2023. ICLR 2024

    Pith/arXiv arXiv 2023

  25. [25]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. arXiv:1809.02789

    Pith/arXiv arXiv 2018

  26. [26]

    PIQA: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020. arXiv:1911.11641

    Pith/arXiv arXiv 2020

  27. [27]

    HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019. arXiv:1905.07830. 15 COBS: Cumulant Order Block Sparse Attention

    Pith/arXiv arXiv 2019

  28. [28]

    Think you have solved question answering? try ARC, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  29. [29]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. arXiv:1705.03551

    Pith/arXiv arXiv 2017

  30. [30]

    WinoGrande: An adversarial Winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI),

  31. [31]

    seqax = sequence modeling + JAX, 2024

    Reiner Pope, Vaclav Cvicek, Daniel Heinlein, Akshay Mishra, Mahdi Nazemi, and Sanjit Neelam. seqax = sequence modeling + JAX, 2024. URLhttps://github.com/MatX-inc/seqax. 16 COBS: Cumulant Order Block Sparse Attention A Derivation of the selection score We derive the additive GQA selection score(5) and its MHA special case from the per-KV-head objective(4)...

    2024