Dynamic Linear Attention

Boyuan Zheng; Hui Shen; Minkyoung Cho; Mi Zhang; Shen Yan; Xin Wang; Xueshen Liu; Zesen Zhao; Zhongwei Wan; Zhuoqing Mao

arxiv: 2606.10650 · v1 · pith:L5W7XFXJnew · submitted 2026-06-09 · 💻 cs.CL · cs.AI

Dynamic Linear Attention

Xin Wang , Hui Shen , Boyuan Zheng , Xueshen Liu , Minkyoung Cho , Zhongwei Wan , Zesen Zhao , Zhuoqing Mao

show 2 more authors

Shen Yan Mi Zhang

This is my paper

Pith reviewed 2026-06-27 13:16 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords dynamic linear attentionmulti-state linear attentionstate merginglong context modelinginformation-aware mergingcapacity-bounded memoryLLM efficiency

0 comments

The pith

Dynamic Linear Attention adapts state merging to token information variation to reduce error accumulation over long sequences.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that fixed state merging policies in multi-state linear attention irreversibly obscure critical tokens and accumulate errors in long contexts. DLA counters this with Information-Aware Dynamic State Merging that sets boundaries according to token-level variation and Capacity-Bounded Memory Modeling that keeps a fixed-size cache by merging low-information states. Pre-training on two linear attention backbones and testing across 16 datasets in three categories shows consistent gains over prior methods. A sympathetic reader would care because this directly targets the representation-capacity bottleneck that has limited linear attention's adoption for extended inputs.

Core claim

DLA addresses the limitation of fixed state merging policies in multi-state linear attention by introducing Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation while preserving high-resolution representations around semantic transitions, and Capacity-Bounded Memory Modeling, which maintains a fixed-size chronologically ordered state cache by selectively merging adjacent low-information states.

What carries the argument

Information-Aware Dynamic State Merging, which measures token-level information variation to set adaptive state boundaries and preserve critical tokens.

If this is right

High-resolution representations are preserved around semantic transitions while stable regions are aggressively summarized.
Memory growth is controlled with minimal information loss through selective merging of low-information states.
The approach scales to longer contexts without quadratic cost while maintaining a bounded state cache.
Pre-training DLA on existing linear attention models yields measurable gains across language, reasoning, and retrieval tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same adaptive boundary logic could be tested on other sub-quadratic attention variants that already use multi-state memory.
If the information-variation metric proves stable across domains, it might generalize to non-language sequence models such as time-series forecasting.
Capacity-bounded merging may interact with existing KV-cache eviction heuristics, suggesting a hybrid implementation for production inference engines.

Load-bearing premise

Token-level information variation can be measured reliably enough to set adaptive state boundaries that preserve critical tokens without the measurement itself depending on the fixed policies being replaced or introducing new error.

What would settle it

A controlled experiment on long sequences where DLA produces higher cumulative error or requires more compute than fixed merging baselines while showing no accuracy gain on the 16 evaluation datasets.

read the original abstract

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

DLA adds info-aware dynamic merging to multi-state linear attention to fix fixed-policy error buildup, but the abstract supplies no numbers or baselines so the gains stay unverified.

read the letter

The core move is replacing fixed state merging with boundaries that shift based on measured token-level information variation, plus a capacity-bounded cache that merges low-info states to stay fixed-size. That directly targets the irreversible loss of critical tokens the abstract flags in prior multi-state linear attention work.

The paper does the obvious next step: it pre-trains the new mechanism on two different linear attention backbones and runs it across 16 datasets in three categories. If the dynamic rule actually preserves semantic transitions without adding its own error, that would be useful for anyone scaling linear attention to longer contexts.

The soft spot is the complete absence of numbers. No perplexity deltas, no baseline tables, no error bars, no dataset sizes. Without those, the claim of superiority over state-of-the-art cannot be checked. The stress-test worry about circularity also needs a look: if the token variation metric is itself computed from representations shaped by the old fixed merging, the adaptivity may not be independent. The full paper should show the exact computation and confirm it does not inherit the same policy.

This is for people already working on efficient attention variants who need a concrete way to handle varying token importance. It is coherent on its own terms and engages the right limitation, so it deserves referee time even though the current write-up leaves the empirical case thin.

Referee Report

1 major / 1 minor

Summary. The paper proposes Dynamic Linear Attention (DLA), a framework for multi-state linear attention that replaces fixed state merging policies with (i) Information-Aware Dynamic State Merging, which adaptively sets state boundaries using token-level information variation to preserve high-resolution representations at semantic transitions, and (ii) Capacity-Bounded Memory Modeling, which enforces a fixed-size chronologically ordered state cache by merging low-information states. The authors pre-train DLA on two linear attention backbones and report superiority over state-of-the-art methods on 16 datasets across three categories.

Significance. If the claimed gains are reproducible and the dynamic component is shown to be independent of the fixed policies it replaces, the work could meaningfully advance efficient long-context modeling by reducing irreversible error accumulation in linear attention while controlling memory growth.

major comments (1)

[Information-Aware Dynamic State Merging] The description of Information-Aware Dynamic State Merging does not establish that the token-level information variation metric is computed independently of the fixed state merging policies being replaced. If the metric relies on representations produced under those same fixed policies, the adaptivity cannot guarantee preservation of critical tokens or reduced error accumulation, and apparent gains may be attributable to the underlying fixed policy rather than the dynamic mechanism (see also Capacity-Bounded Memory Modeling).

minor comments (1)

[Abstract] The abstract asserts quantitative superiority but supplies no numerical results, baselines, dataset names, error bars, or statistical tests; these must be presented with full details in the experimental section for the superiority claim to be evaluable.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful review and constructive feedback. We address the single major comment below.

read point-by-point responses

Referee: [Information-Aware Dynamic State Merging] The description of Information-Aware Dynamic State Merging does not establish that the token-level information variation metric is computed independently of the fixed state merging policies being replaced. If the metric relies on representations produced under those same fixed policies, the adaptivity cannot guarantee preservation of critical tokens or reduced error accumulation, and apparent gains may be attributable to the underlying fixed policy rather than the dynamic mechanism (see also Capacity-Bounded Memory Modeling).

Authors: We acknowledge that the current manuscript description does not explicitly establish the independence of the token-level information variation metric from the fixed state merging policies. We will revise the relevant sections (including the method description and any associated analysis) to provide a precise definition of the metric and demonstrate that it is computed directly from input token features without reference to merged states or fixed policies. revision: yes

Circularity Check

0 steps flagged

No circularity in derivation chain

full rationale

The abstract and description present DLA as a proposed framework with two components (Information-Aware Dynamic State Merging based on token-level information variation, and Capacity-Bounded Memory Modeling) but contain no equations, derivations, fitted parameters, or predictions. No self-citations, uniqueness theorems, or ansatzes are referenced. The central claims rest on experimental evaluation rather than any reduction of outputs to inputs by construction. This is the expected self-contained case with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only input supplies no equations, training details, or modeling choices, so no free parameters, axioms, or invented entities can be identified.

pith-pipeline@v0.9.1-grok · 5744 in / 1109 out tokens · 52734 ms · 2026-06-27T13:16:43.592855+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

28 extracted references · 10 canonical work pages · 2 internal anchors

[1]

Effi- cient large language models: A survey.arXiv preprint arXiv:2312.03863,

Zhongwei Wan, Xin Wang, et al. Efficient large language models: A survey.arXiv preprint arXiv:2312.03863, 2023

work page arXiv 2023
[2]

Iot in the era of generative ai: Vision and challenges.arXiv preprint arXiv:2401.01923, 2024

Xin Wang, Zhongwei Wan, Arvin Hekmati, Mingyu Zong, Samiul Alam, Mi Zhang, and Bhaskar Krishnamachari. Iot in the era of generative ai: Vision and challenges.arXiv preprint arXiv:2401.01923, 2024

work page arXiv 2024
[3]

D2O: dynamic discriminative operations for efficient long-context inference of large language models

Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. D2O: dynamic discriminative operations for efficient long-context inference of large language models. InICLR. OpenReview.net, 2025

2025
[4]

Parallelizing linear transformers with the delta rule over sequence length

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. InNeurIPS, 2024

2024
[5]

Gated delta networks: Improving mamba2 with delta rule

Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In ICLR. OpenReview.net, 2025

2025
[6]

Xing, Tri Dao, and Yoon Kim

Han Guo, Songlin Yang, Tarushii Goel, Eric P. Xing, Tri Dao, and Yoon Kim. Log-linear attention.CoRR, abs/2506.04761, 2025

work page arXiv 2025
[7]

RAT: bridging RNN efficiency and attention accuracy in language modeling.CoRR, abs/2507.04416, 2025

Xiuying Wei, Anunay Yadav, Razvan Pascanu, and Caglar Gulcehre. RAT: bridging RNN efficiency and attention accuracy in language modeling.CoRR, abs/2507.04416, 2025

work page arXiv 2025
[8]

Generalizing the cauchy-schwarz inequality: Hadamard powers and tensor products.arXiv preprint arXiv:2507.10327, 2025

Nathaniel Johnston, Sarah Plosker, Charles Torrance, and Luis Varona. Generalizing the cauchy-schwarz inequality: Hadamard powers and tensor products.arXiv preprint arXiv:2507.10327, 2025

work page arXiv 2025
[9]

Triangle inequality variations in the internet

Cristian Lumezanu, Randy Baden, Neil Spring, and Bobby Bhattacharjee. Triangle inequality variations in the internet. InProceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, IMC ’09, page 177–183, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605587714. doi: 10.1145/ 1644893.1644914. URL https://doi.org/10.1145/16...

work page doi:10.1145/1644893.1644914 2009
[10]

Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. InICML. OpenReview.net, 2024

2024
[11]

The LAMBADA dataset: Word prediction requiring a broad discourse context

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. InACL (1). The Association for Computer Linguistics, 2016

2016
[12]

PIQA: reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. InAAAI, pages 7432–7439. AAAI Press, 2020

2020
[13]

Hellaswag: Can a machine really finish your sentence? InACL (1), pages 4791–4800

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InACL (1), pages 4791–4800. Association for Computational Linguistics, 2019

2019
[14]

Winogrande: an adversarial winograd schema challenge at scale.Commun

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale.Commun. ACM, 64(9):99–106, 2021. 11

2021
[15]

Can a suit of armor conduct electricity? A new dataset for open book question answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. InEMNLP, pages 2381–2391. Association for Computational Linguistics, 2018

2018
[16]

Commonsenseqa: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InNAACL-HLT(1), pages 4149–4158. Association for Computational Linguistics, 2019

2019
[17]

arXiv preprint arXiv:2102.03315 , year =

Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge.CoRR, abs/2102.03315, 2021

work page arXiv 2021
[18]

Openceres: When open information extraction meets the semi-structured web

Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extraction meets the semi-structured web. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Volume1 (Long and Short Papers), pages 3047–3056, 2019

2019
[19]

Know what you don’t know: Unanswerable questions for squad

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, 2018

2018
[20]

Language models enable simple systems for generating structured views of heterogeneous data lakes

Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. Proceedings of the VLDB Endowment, 17(2):92–105, 2023

2023
[21]

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers), pages 1601–1611, 2017

2017
[22]

Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Volume1 (Long and Short Papers), ...

2019
[23]

Natural questions: a benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019

2019
[24]

RULER: What's the Real Context Size of Your Long-Context Language Models?

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: what’s the real context size of your long-context language models?CoRR, abs/2404.06654, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024
[25]

Longbench: A bilingual, multitask benchmark for long context understanding

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. InACL (1), pages 3119–3137. Association for Computational Linguistics, 2024

2024
[26]

The language model evaluation harness, 07 2024

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

work page arXiv 2024
[27]

Simple linear attention language models balance the recall-throughput tradeoff

Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. InProceedings of the 41st International Conference on Machine Learning, pages 1763–1840, 2024

2024
[28]

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752, 2023. 12

work page internal anchor Pith review Pith/arXiv arXiv 2023

[1] [1]

Effi- cient large language models: A survey.arXiv preprint arXiv:2312.03863,

Zhongwei Wan, Xin Wang, et al. Efficient large language models: A survey.arXiv preprint arXiv:2312.03863, 2023

work page arXiv 2023

[2] [2]

Iot in the era of generative ai: Vision and challenges.arXiv preprint arXiv:2401.01923, 2024

Xin Wang, Zhongwei Wan, Arvin Hekmati, Mingyu Zong, Samiul Alam, Mi Zhang, and Bhaskar Krishnamachari. Iot in the era of generative ai: Vision and challenges.arXiv preprint arXiv:2401.01923, 2024

work page arXiv 2024

[3] [3]

D2O: dynamic discriminative operations for efficient long-context inference of large language models

Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. D2O: dynamic discriminative operations for efficient long-context inference of large language models. InICLR. OpenReview.net, 2025

2025

[4] [4]

Parallelizing linear transformers with the delta rule over sequence length

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. InNeurIPS, 2024

2024

[5] [5]

Gated delta networks: Improving mamba2 with delta rule

Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In ICLR. OpenReview.net, 2025

2025

[6] [6]

Xing, Tri Dao, and Yoon Kim

Han Guo, Songlin Yang, Tarushii Goel, Eric P. Xing, Tri Dao, and Yoon Kim. Log-linear attention.CoRR, abs/2506.04761, 2025

work page arXiv 2025

[7] [7]

RAT: bridging RNN efficiency and attention accuracy in language modeling.CoRR, abs/2507.04416, 2025

Xiuying Wei, Anunay Yadav, Razvan Pascanu, and Caglar Gulcehre. RAT: bridging RNN efficiency and attention accuracy in language modeling.CoRR, abs/2507.04416, 2025

work page arXiv 2025

[8] [8]

Generalizing the cauchy-schwarz inequality: Hadamard powers and tensor products.arXiv preprint arXiv:2507.10327, 2025

Nathaniel Johnston, Sarah Plosker, Charles Torrance, and Luis Varona. Generalizing the cauchy-schwarz inequality: Hadamard powers and tensor products.arXiv preprint arXiv:2507.10327, 2025

work page arXiv 2025

[9] [9]

Triangle inequality variations in the internet

Cristian Lumezanu, Randy Baden, Neil Spring, and Bobby Bhattacharjee. Triangle inequality variations in the internet. InProceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, IMC ’09, page 177–183, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605587714. doi: 10.1145/ 1644893.1644914. URL https://doi.org/10.1145/16...

work page doi:10.1145/1644893.1644914 2009

[10] [10]

Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. InICML. OpenReview.net, 2024

2024

[11] [11]

The LAMBADA dataset: Word prediction requiring a broad discourse context

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. InACL (1). The Association for Computer Linguistics, 2016

2016

[12] [12]

PIQA: reasoning about physical commonsense in natural language

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. InAAAI, pages 7432–7439. AAAI Press, 2020

2020

[13] [13]

Hellaswag: Can a machine really finish your sentence? InACL (1), pages 4791–4800

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InACL (1), pages 4791–4800. Association for Computational Linguistics, 2019

2019

[14] [14]

Winogrande: an adversarial winograd schema challenge at scale.Commun

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale.Commun. ACM, 64(9):99–106, 2021. 11

2021

[15] [15]

Can a suit of armor conduct electricity? A new dataset for open book question answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. InEMNLP, pages 2381–2391. Association for Computational Linguistics, 2018

2018

[16] [16]

Commonsenseqa: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InNAACL-HLT(1), pages 4149–4158. Association for Computational Linguistics, 2019

2019

[17] [17]

arXiv preprint arXiv:2102.03315 , year =

Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge.CoRR, abs/2102.03315, 2021

work page arXiv 2021

[18] [18]

Openceres: When open information extraction meets the semi-structured web

Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extraction meets the semi-structured web. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Volume1 (Long and Short Papers), pages 3047–3056, 2019

2019

[19] [19]

Know what you don’t know: Unanswerable questions for squad

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, 2018

2018

[20] [20]

Language models enable simple systems for generating structured views of heterogeneous data lakes

Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. Proceedings of the VLDB Endowment, 17(2):92–105, 2023

2023

[21] [21]

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers), pages 1601–1611, 2017

2017

[22] [22]

Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Volume1 (Long and Short Papers), ...

2019

[23] [23]

Natural questions: a benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019

2019

[24] [24]

RULER: What's the Real Context Size of Your Long-Context Language Models?

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: what’s the real context size of your long-context language models?CoRR, abs/2404.06654, 2024

work page internal anchor Pith review Pith/arXiv arXiv 2024

[25] [25]

Longbench: A bilingual, multitask benchmark for long context understanding

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. InACL (1), pages 3119–3137. Association for Computational Linguistics, 2024

2024

[26] [26]

The language model evaluation harness, 07 2024

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

work page arXiv 2024

[27] [27]

Simple linear attention language models balance the recall-throughput tradeoff

Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. InProceedings of the 41st International Conference on Machine Learning, pages 1763–1840, 2024

2024

[28] [28]

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752, 2023. 12

work page internal anchor Pith review Pith/arXiv arXiv 2023