DiffLoRA: Differential Low-Rank Adapters for Large Language Models

Alexandre Misrahi; Maxime Louis; Nadezhda Chirkova; Vassilina Nikoulina

REVIEW 3 major objections 4 minor 15 references

DiffLoRA: Differential Low-Rank Adapters for Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DiffLoRA grafts differential attention onto a pretrained LLM with low-rank adapters, but on most tasks it falls short of LoRA while gaining 11 points on HumanEval.

desk verdict An honest, mostly negative empirical test of a natural LoRA+differential-attention combination; the positive islands are not yet trustworthy due to a baseline confound and missing error bars. read the letter →

arxiv 2507.23588 v1 pith:PUYDOFPK submitted 2025-07-31 cs.CL

Alexandre Misrahi , Nadezhda Chirkova , Maxime Louis , Vassilina Nikoulina This is my paper

classification cs.CL

keywords differentialattentionlow-rankadaptersparameter-efficientfine-tuningLoRAsinksin-contextlearningretrieval-augmentedgenerationlong-contextevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether the noise-cancelling attention of the Differential Transformer can be transplanted into an already-trained LLM using only small low-rank adapters, keeping parameter counts at LoRA levels. The proposed DiffLoRA adds trainable adapters to both the positive and the negative attention terms of the differential attention formula at every layer. Across general benchmarks, many-shot in-context learning, retrieval-augmented generation, and long-context needle tests, DiffLoRA mostly trails plain LoRA, but it beats LoRA by 11 points on HumanEval and on multi-value needle retrieval. Attention-mass analysis shows the adapter-trained models shift only slightly away from the base model's patterns, suggesting the denoising behavior of differential attention did not fully emerge.

What carries the argument

The load-bearing object is the differential attention score with one positive and one negative softmax term, $\mathrm{DiffAttn}(X)=(\mathrm{softmax}(Q_1K_1^\top/\sqrt d)-\lambda\,\mathrm{softmax}(Q_2K_2^\top/\sqrt d))V$, where $Q_1$, $K_1$ are produced by frozen pretrained projections plus optional adapters and $Q_2$, $K_2$ are produced entirely by low-rank adapters, $Q_2 = X(B_{Q2}A_{Q2})$, $K_2 = X(B_{K2}A_{K2})$. The two-sided low-rank parameterization is what carries the argument: it is meant to give the pretrained model a learnable denoising side while reusing the existing attention projections. The paper also treats $\lambda$ as either learned or frozen to $0.1$, and compares a group-normalized variant, finding that the normalization collapses training.

What would settle it

Build a from-scratch Differential Transformer and a DiffLoRA model on the same data, then compare their attention maps on prompts with attention sinks: if DiffLoRA's attention, even after longer training, does not suppress the BOS sink and concentrate mass on the relevant context the way the from-scratch model does, the claim that low-rank adapters can induce the denoising mechanism is refuted.

Watch

Extended reading notes

Core claim

The central claim, read in good faith, is that differential attention's denoising can be realized in a pretrained transformer by learning $Q_2 = X(B_{Q2}A_{Q2})$ and $K_2 = X(B_{K2}A_{K2})$ for the negative term, with optional low-rank adapters on the frozen positive projections, so that the attention weight becomes $\mathrm{softmax}(Q_1K_1^\top/\sqrt d)-\lambda\,\mathrm{softmax}(Q_2K_2^\top/\sqrt d)$. The authors show this is trainable at LoRA-level parameter counts, but against LoRA baselines it improves only on coding (HumanEval, +11 points) and on multi-value needle retrieval, while falling short on most other general, in-context-learning, RAG, and long-context tasks. They further show that attention mass after fine-tuning resembles the LoRA model more than the Differential Transformer, and that training on a larger mixture (Tulu-3) strengthens denoising without changing the overall pattern. The conclusion is that more data or further architectural support is needed to make the differential mechanism work through adapters.

Load-bearing premise

The method's bet is that a pretrained Transformer can learn differential attention's noise cancellation from small trainable add-ons on its query and key projections alone, keeping the base model frozen and not adding the head-wise normalization that the original Differential Transformer uses.

Editorial extensions

If this is right

A pretrained LLM can be given a differential-attention denoiser with roughly LoRA parameter counts, without full fine-tuning, so the mechanism is not restricted to models trained from scratch.
In this low-rank form, differential attention does not generalize its advantage: most benchmarks stay within the base model's range or fall below LoRA, so the gains are task-specific.
HumanEval (+11 points over LoRA) and multi-value needle retrieval are the two places where the denoiser adapters help, pointing to settings with many competing tokens or repeated retrieval.
Attention patterns after DiffLoRA training resemble LoRA's, so much of the observed shift may come from the instruction-tuning data rather than from the differential mechanism.
Scaling the tuning mixture from Tulu-2 to Tulu-3 strengthens denoising but not enough to change the hierarchy, implying data volume is a bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the transfer assumption is the bottleneck, then full-precision fine-tuning of the denoiser projections or adding the head-wise normalization used by Differential Transformer would be a more direct test of whether differential attention needs to be learned from initialization.
The HumanEval and multi-value retrieval wins suggest a sharper hypothesis: the denoiser helps mainly when the prompt contains distractors, and a dedicated benchmark varying the number and ratio of distractors could reveal where the adapter port is actually doing work.
One can interpret the negative results as setting an upper bound on what pure low-rank parameter-efficient tuning can buy on top of an already-trained attention mechanism: the adapter cannot recreate a substantially different attention geometry.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is an honest, modest empirical paper that tests a natural combination of LoRA and differential attention. The headline result is negative — DiffLoRA mostly underperforms LoRA on a 1B model — and that is worth having. The two positive exceptions (HumanEval +11, multi-value retrieval) are not yet trustworthy, and the comparison setup has a confound that the paper doesn't fully address.

The new configuration is low-rank adapters on both the positive and negative attention terms, with the pre-trained Q1/K1 kept fixed in the default variant. That's a reasonable thing to try, and I don't know of a prior parameter-matched evaluation of it. The paper does something useful after the experiments: it looks at the attention maps and shows DiffLoRA does not recover the denoising behavior of the original Differential Transformer, with patterns close to LoRA. That is a good sanity check and it supports the idea that the transfer assumption is the weak link.

What I'd want fixed before relying on the positive claims. First, the claimed parameter match between LoRA and DiffLoRA doesn't obviously hold. DiffLoRA-64 right-only trains two rank-64 adapter pairs on Q2/K2 only; the LoRA baseline trains rank-8 adapters on Q, K, V, O, and MLP. Per layer that is roughly 2x less trainable capacity for LoRA, depending on dimensions. So the comparison conflates the negative attention term with where the capacity goes. A LoRA control with the same Q/K-only placement is needed. The stress-test note is right about this. Second, there are no error bars anywhere; the +11 HumanEval gain is a single run. Third, the paper's own generation examples show degeneration; that explains the RAG drops, but it also suggests the adapters are doing something unstable.

I agree with the reader's conditional verdict: the negative result is consistent, the positive exceptions are suggestive but not established. The paper acknowledges its limitations in the discussion. It's a useful data point for anyone trying to port architectural innovations to pre-trained models via PEFT.

For a reading group it's a moderate pick: it will spark discussion about matching baselines and why a method fails to transfer. I'd cite it only if I were writing specifically about PEFT for differential attention. It should be refereed: the empirical work is honest, the confound is fixable, and the negative result deserves a place in the literature.

Referee Report

3 major / 4 minor

Summary. The paper proposes DiffLoRA, a parameter-efficient adaptation of the differential attention mechanism in which low-rank adapters are trained on both the positive and negative attention terms (Q1/K1 and Q2/K2) of a frozen pre-trained LLM. The authors fine-tune Llama-3.2-1B-Instruct on Tulu-2 and evaluate on general benchmarks (TruthfulQA, PopQA, BoolQ, HumanEval, DROP, GSM8K, BBH, ARC-C), many-shot in-context learning, needle-in-a-haystack retrieval, and RAG. Their central finding is that DiffLoRA mostly underperforms standard LoRA, with notable exceptions on HumanEval (+11 points over LoRA) and multi-value needle retrieval, and they present an attention-mass analysis suggesting that the differential denoising behavior is only weakly learned. The paper is transparent about these largely negative results and releases code.

Significance. If the parameter-matched comparison were properly validated, this paper would provide a useful negative result: it would show that low-rank adapters on Q/K alone do not readily induce differential attention in a pre-trained transformer, and it would document where the approach does or does not help. The authors should be credited for exploring a plausible adaptation and for reporting failures as well as successes. However, the current evidence is weakened by an unverified parameter-matching claim, the absence of a location-matched LoRA control, and single-run evaluations without error bars. These issues directly affect the interpretation of the positive exceptions and the central efficiency claim, so the paper needs substantive revision before the empirical conclusions can be trusted.

major comments (3)

[§3.3 and Appendix B] The paper claims that the LoRA rank is set 'to match the number of trainable parameters of DiffLoRA models', but no parameter counts are reported. For Llama-3.2-1B with d_model=2048 and intermediate size 8192, LoRA r=8 on Q, K, V, O and the three MLP matrices uses roughly 350k trainable parameters per layer. DiffLoRA-64 with adapters on Q2 and K2 (each with matrices 2048x64 and 64x2048) uses roughly 520k parameters per layer, and the both-terms variant uses more. Even accounting for GQA reducing the K dimension, the stated matching is not self-evident. Please provide exact per-layer and full-model parameter counts for every configuration, and add a LoRA baseline with the same parameter budget and location (e.g., LoRA on Q/K only with the same rank) to make the comparison in Fig. 1 and Fig. 3 meaningful.
[§4, Fig. 1 and Fig. 3] The positive exceptions (HumanEval +11 points for DiffLoRA-64, and the MV retrieval advantage) are reported from single training runs without error bars. On HumanEval's 164 problems, an 11-point difference is roughly 18 problems and can easily fall within seed-to-seed variance. The MV curves in Fig. 3 also show no uncertainty bands. Please run at least three seeds for the key comparisons and report means and standard deviations, or otherwise supply evidence that these differences are reproducible and not noise.
[§5, Fig. 4] The attention analysis cannot isolate the causal role of the differential negative term because the comparison is not location- and parameter-matched (see the first major comment). The paper notes that DiffLoRA's attention patterns remain close to LoRA's, which is consistent with the alternative explanation that any observed differences come from the additional Q/K capacity rather than from differential denoising. A Q/K-only LoRA control should be included in both the quantitative evaluations and the attention analysis to support the claim that the behavioral changes are attributable to the negative attention term.

minor comments (4)

[Throughout] There are several typos and formatting issues: 'FullLora' should be 'LoRA', 'DiiffLoRA' in Table 1/3 is misspelled, 'fix =0.1' should be 'fixed lambda=0.1', and 'Differentaion' in the Fig. 5 caption should be 'Differential'.
[§3.2] The in-context learning evaluation is described as 'zero-shot', but the models are given many demonstrations; the wording should be changed to reflect that this is many-shot ICL.
[§5, Fig. 4/5] Please clarify how the attention mass is aggregated over heads and layers and over the BOS/CONTEXT/MAGIC NUMBER/QUERY segments; without this methodological detail the qualitative claim about denoising is hard to verify. The 'Magic Number' is not defined in the main text, so it should be introduced or referenced to the NIH setup.
[Appendix B] The hyperparameter table does not list the trainable parameter counts or any measure of computational cost (FLOPs, wall-clock time). Since the abstract claims LoRA-level efficiency, these numbers should be reported alongside the hyperparameters.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DiffLoRA's results are empirical evaluations on external benchmarks, and its hyperparameters (rank, lambda) are reported choices, not fitted outputs.

full rationale

DiffLoRA is defined as low-rank adapters on both the positive and negative attention terms of differential attention, with pre-trained projections fixed and adapter matrices trained. The paper's conclusions are based on evaluations on fixed external benchmarks (OLMES tasks, HELMET ICL and needle tests, and BERGEN RAG settings) after training on Tulu-2 or Tulu-3. No target benchmark result is used to fit a parameter, constant, or design choice in the method; the rank values (32 and 64), LoRA rank 8, and lambda settings (fixed 0.1 or learnable) are hyperparameters that are explicitly varied and reported, and the paper reports both successes and failures across tasks. The central comparison between DiffLoRA and LoRA is empirical, and the paper itself notes that DiffLoRA attention patterns remain close to LoRA and do not reproduce the strong denoising behavior of the original Differential Transformer. The skeptic's concern that DiffLoRA and LoRA differ in parameter location (Q/K-only adapters versus LoRA on Q/K/V/O/MLP) is a valid experimental confound and a correctness or interpretation risk, but it is not a circularity: no predicted quantity is defined in terms of an input, and no derivation reduces to the method's own output. The self-citations in the paper (for example, the BERGEN framework by Rau et al.) are tooling and evaluation infrastructure, not load-bearing evidence for the method's core claim, and no uniqueness theorem is imported from the authors' prior work to force a choice. Therefore, the paper does not exhibit circular reasoning, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central claim rests mainly on the transfer assumption and on a modest set of hyperparameters. The method introduces no new physical entities. Free parameters are limited to the learnable lambda scale and LoRA rank choices.

free parameters (2)

lambda (differential attention scaling) = learned; a fixed value of 0.1 is used in some variants
Controls the subtraction strength between positive and negative attention terms; in the original Differential Transformer it is learned, and here some runs freeze it to 0.1.
LoRA rank r and alpha = r=64/32 for DiffLoRA, r=8 for matched LoRA; alpha=128/64
Hyperparameters chosen to match parameter counts, not fitted; still they are choices that affect the comparison.

assumptions (3)

ad hoc to paper Pre-trained attention projections can serve as the positive stream while low-rank adapters learn the negative stream.
Section 2 constructs Q1,K1 from pretrained weights and only trains adapters for Q2,K2; this is the core transfer premise.
domain assumption One epoch over Tulu-2 (max length 4096) is enough to compare adapter methods.
Section 3 says all models are trained for a single epoch; if the methods need more data, the comparison may be unfair.
domain assumption LLM-as-a-judge with SOLAR-10.7B gives reliable RAG answer quality scores.
Appendix A states this metric correlates with GPT-4, but it remains a proxy and might affect RAG conclusions.

invented entities (1)

DiffLoRA adapter set (additional low-rank matrices for Q2/K2 and optionally Q1/K1) independent evidence
purpose: Enables a parameter-efficient approximation of differential attention in pre-trained LLMs
The adapters have no separate physical or mathematical existence beyond the method; evidence is the external evaluations. They are a model component, not a new physical entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DiffLoRA: Differential Low-Rank Adapters for Large Language Models." pith.science (2026). https://pith.science/paper/PUYDOFPK

@misc{pith2026250723588,
  author       = {Pith},
  title        = {Pith review of: DiffLoRA: Differential Low-Rank Adapters for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUYDOFPK}},
  note         = {Machine review of arXiv:2507.23588}
}

read the original abstract

Differential Transformer has recently been proposed to improve performance in Transformer models by canceling out noise through a denoiser attention mechanism. In this work, we introduce DiffLoRA, a parameter-efficient adaptation of the differential attention mechanism, with low-rank adapters on both positive and negative attention terms. This approach retains the efficiency of LoRA while aiming to benefit from the performance gains of differential attention. We evaluate DiffLoRA across a broad range of NLP tasks, including general benchmarks, many-shot in-context learning, RAG, and long-context tests. We observe that, although DiffLoRA falls short of other parameter-efficient fine-tuning methods in most evaluation tasks, it shows interesting results in certain domains (+11 pts on LoRA for HumanEval). We analyze the attention patterns post-finetuning to identify the reasons for this behavior.

Figures

Figures reproduced from arXiv: 2507.23588 by the authors.

**Figure 3.** Needle-in-Haystack tests with variants Mul [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Change in attention pattern distribution in different models. For DiffLoRA variants we plot attention [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Disttribution of attention mass depending on the position of learnable parameters in Differentaion attention [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 4 canonical work pages

[2]

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V

Preprint, arXiv:2311.10702. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Da...

arXiv
[4]

Preprint, arXiv:2106.09685

Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Belt- agy, and Hannaneh Hajishirzi

arXiv
[6]

An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics. Carlos Lassance,...

work page 2019
[7]

arXiv preprint arXiv:2403.06789

Splade-v3: New baselines for splade. arXiv preprint arXiv:2403.06789. Xin Li and Dan Roth

arXiv
[10]

Springer Nature Switzerland

Overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedi- cal Semantic Indexing and Question Answering, page 227–250. Springer Nature Switzerland. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäsc...

work page 2023
[11]

Preprint, arXiv:2009.02252

Kilt: a benchmark for knowledge in- tensive language tasks. Preprint, arXiv:2009.02252. 6 David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant

arXiv 2009
[12]

Preprint, arXiv:2407.01102

Bergen: A benchmarking library for retrieval-augmented generation. Preprint, arXiv:2407.01102. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

arXiv
[13]

Preprint, arXiv:2309.17453

Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453. Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei

arXiv

Show all 15 references

[14]

Preprint, arXiv:2410.05258

Differential trans- former. Preprint, arXiv:2410.05258. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen

arXiv
[15]

These documents are further reranked using DeBERTa-v3 (He et al., 2021), a cross-encoder computing relevance score for each document relative to the query

retriever to identify a first set of relevant documents from Wikipedia collection. These documents are further reranked using DeBERTa-v3 (He et al., 2021), a cross-encoder computing relevance score for each document relative to the query. For generation, we use instruction-tun...

2021
[2002]

In COLING 2002: The 19th International Conference on Computational Linguistics

Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi

2002
[2019]

Preprint, arXiv:1911.02984

The techqa dataset. Preprint, arXiv:1911.02984. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, ...

1911 arXiv
[2021]

Preprint, arXiv:2111.09543

Debertav3: Improving deberta using electra-style pre- training with gradient-disentangled embedding shar- ing. Preprint, arXiv:2111.09543. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

arXiv
[2023]

Preprint, arXiv:2212.10511

When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. Preprint, arXiv:2212.10511. Anastasios Nentidis, Georgios Katsimpras, Anasta- sia Krithara, Salvador Lima López, Eulália Farré- Maduell, Luis Gasco, Martin Krallinger, a...

arXiv
[2024]

Preprint, arXiv:2407.21783

The llama 3 herd of models. Preprint, arXiv:2407.21783. Pengcheng He, Jianfeng Gao, and Weizhu Chen

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [2]

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V

Preprint, arXiv:2311.10702. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Da...

arXiv

[2] [4]

Preprint, arXiv:2106.09685

Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Belt- agy, and Hannaneh Hajishirzi

arXiv

[3] [6]

An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics. Carlos Lassance,...

work page 2019

[4] [7]

arXiv preprint arXiv:2403.06789

Splade-v3: New baselines for splade. arXiv preprint arXiv:2403.06789. Xin Li and Dan Roth

arXiv

[5] [10]

Springer Nature Switzerland

Overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedi- cal Semantic Indexing and Question Answering, page 227–250. Springer Nature Switzerland. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäsc...

work page 2023

[6] [11]

Preprint, arXiv:2009.02252

Kilt: a benchmark for knowledge in- tensive language tasks. Preprint, arXiv:2009.02252. 6 David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Vassilina Nikoulina, and Stéphane Clinchant

arXiv 2009

[7] [12]

Preprint, arXiv:2407.01102

Bergen: A benchmarking library for retrieval-augmented generation. Preprint, arXiv:2407.01102. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

arXiv

[8] [13]

Preprint, arXiv:2309.17453

Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453. Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei

arXiv