Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

Mohsen Arjmandi

REVIEW 2 major objections 4 minor 13 references

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Middle-removal benchmarks measure how often the answer survives, not context-window effects.

desk verdict The paper's warning about naive truncation is right and the BABILong experiment is clean, but the equal-token comparison is unverified on GraphWalks and MRCR, so the headline overclaims. read the letter →

arxiv 2608.03297 v1 pith:B73VGNQL submitted 2026-08-04 cs.AI cs.CL

Mohsen Arjmandi This is my paper · ORCID

classification cs.AIcs.CL

keywords long-contextbenchmarksdistractor-awaretruncationsignallossmiddle-removalprotocolcontext-lengtheffectsretrieval-augmentedgenerationpairedWilcoxontestBABILong

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard way of testing whether shorter context helps language models, truncating long prompts by deleting the middle, cannot distinguish two opposite effects: removing irrelevant distraction versus deleting the very content needed to answer. It proposes a distractor-aware protocol that first defines, for each sample, the minimal task-relevant signal and deletes only the rest. Measured this way, on BABILong and GraphWalks across three Claude model sizes and GPT-5.5, the same nominal truncation no longer hurts: the smaller models improve significantly at 25% retention, and the larger models stay at their full-context ceiling. The naive protocol's collapse coincides with a signal-preservation rate below 1% for BABILong and 26% for GraphWalks at 25% retention. The paper's conclusion is that context-length studies must specify how signal and distractor are separated, or their reported direction is ambiguous.

What carries the argument

The load-bearing object is the sample-level signal set: a minimal subset of the prompt sufficient to determine the gold label, with the distractor defined as everything else. The paper instantiates this per benchmark: bAbI fact sentences in BABILong matched by a constrained regex, the depth-bounded BFS-reachable subgraph in GraphWalks recomputed from the query, and the needle query-response pair in MRCR v2 located through the dataset's desired_msg_index field, and verifies each against the dataset's own ground truth. Distractor-aware truncation keeps this set verbatim and compresses or drops only the remainder, so the answer is preserved by construction. The paired statistical design, Wilcoxon signed-rank tests on sample IDs with Holm correction plus paired-bootstrap confidence intervals, is what makes the protocol comparison interpretable as a within-sample effect.

What would settle it

Audit the released per-sample records for a counterexample: a BABILong sample at 25% distractor-aware retention whose score drops below full context, together with a sentence dropped by the distractor-aware truncation that is required to derive the gold answer, would falsify the 'preserved by construction' guarantee; a GraphWalks sample whose kept reachable subgraph fails to derive a gold answer node would do the same.

Watch

Extended reading notes

Core claim

The central claim is that 'shorter context hurts' and 'shorter context helps' can both be true of the same samples at the same nominal context fraction, depending only on what the truncation deletes. Naive middle-removal is a measurement of how often the answer-bearing content survives the cut; distractor-aware truncation, which preserves the task-sufficient subset by construction, isolates the effect of distractor reduction. On the primary benchmarks, all eight benchmark-by-model cells show significant harm under naive truncation at 25% retention, while distractor-aware truncation produces no harm and, for Haiku and Sonnet on BABILong, statistically significant gains of +0.083 and +0.104. The pattern replicates on GPT-5.5, ruling out a single-provider artifact, and the negative control (Oolong, where every item is signal) shows no protocol difference. The paper frames this as a correction: prior middle-removal results are ambiguous between two opposite hypotheses unless they separate signal from distractor.

Load-bearing premise

The load-bearing premise is that each benchmark's signal definition captures every piece of content the model could need to answer correctly; if it misses any answer-bearing fact, edge, or turn, the distractor-aware protocol deletes it and the 'preserved by construction' guarantee fails.

Editorial extensions

If this is right

Any future context-length study that uses middle-removal must report how often the answer-bearing content survives truncation; otherwise a 'shorter is worse' result cannot be distinguished from signal loss.
A perfect retriever, which removes only distractors, gives a controlled upper bound on retrieval-augmented and memory-augmented system headroom; on BABILong this bound is modest in absolute terms and depends on baseline model strength.
At 25% retention, distractor-aware truncation significantly improves Haiku and Sonnet on BABILong, while Opus and GPT-5.5 remain at their full-context ceiling, suggesting saturation rather than context-window failure.
For GraphWalks, naive truncation asks the model the wrong question in 74% of samples at 25% retention, so the naive curve is not about reasoning degradation.
Oolong, where no distractor exists, shows no protocol difference, confirming that the distractor-aware advantage comes from identifying signal, not from shorter prompts per se.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the signal sets are incomplete for other splits or task families, the 'preserved by construction' guarantee silently becomes 'preserved only on known signal'; the paper's hand-verified definitions should be re-checked on every new task before the protocol is adopted.
The same distractor/signal separation could be extended to natural-language and multilingual long-context tasks, where signal definition would need to be learned or annotated rather than matched by regex or reachability.
The upper-bound framing suggests a testable scaling prediction: retrieval headroom should shrink as base-model context utilization improves, so 'less is more' claims should be stratified by model capability.
Published naive-truncation results could be re-analyzed post hoc by recomputing signal-preservation rates on the released samples; if preservation is low, those results are consistent with signal loss rather than context-window effects.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's a paper that deserves referee time, but with a clear caveat.

What it does well: it formalizes a real confound. Naive middle-removal truncation doesn't measure context-length effects because it deletes the answer in most samples. The diagnostics are convincing: at 25% naive retention, bAbI facts survive in under 1% of BABILong samples, and the BFS gold in 26% of GraphWalks. On BABILong, where the distractor-aware condition actually compresses filler to the target alpha*L while preserving facts, the paired comparison is clean: naive collapses (Holm-significant harm in all cells), while distractor-aware is flat or improves, with significant gains for the two smaller Claude models. That is a real, useful result. The statistical design — paired Wilcoxon, Holm correction, split-half stability, and the Oolong negative control — is better than most of what gets published in this space. The code and data release is a plus.

The soft spot is load-bearing. The stress-test note checks out: for GraphWalks, the signal is the full reachable subgraph, and truncation drops only edges whose source is unreachable. There is no mechanism that reduces the retained graph to alpha*L. For MRCR, they drop middle turns but never state or enforce a token-budget target. No realized token counts by protocol and alpha are reported. So the 'same nominal truncation' comparison is only true on BABILong. The flat GraphWalks curves, for instance, are likely just full-context curves wearing a '25% retention' label. This doesn't undo the central warning — BABILong alone is enough to show the naive protocol is broken — but the abstract and conclusion claim a controlled contrast on all four benchmarks, and that's not established.

Other limitations (temperature heterogeneity for Opus and GPT-5.5, n=50 on GraphWalks/MRCR, hand-crafted signal definitions) are acknowledged honestly and are minor in comparison. The fix is straightforward: report token counts, or actually truncate the distractor-aware prompts to alpha*L for GraphWalks and MRCR, e.g., by discarding non-gold paths after retaining a witness path for each answer node.

My take: bring it to the reading group. It's a good case study in how a well-intentioned protocol can quietly change what you're measuring. I wouldn't cite it until the token-budget issue is fixed, but I'd send it to a serious referee. The methodological message is important and the execution on BABILong is credible.

Referee Report

2 major / 4 minor

Summary. The paper argues that the standard 'middle-removal' truncation protocol used in many long-context LLM studies confounds two effects: removal of task-irrelevant distractor content (which can help) and removal of answer-bearing signal (which hurts). To separate these, the authors propose a 'distractor-aware' truncation protocol that identifies and preserves a per-sample signal set defined from each benchmark's structure (bAbI fact sentences for BABILong, the depth-bounded reachable subgraph for GraphWalks, needle query/response pairs for MRCR v2, and a recomputed subsample gold for Oolong). They evaluate three Claude models and GPT-5.5 on two primary benchmarks (BABILong, GraphWalks) at four retention fractions (100%, 75%, 50%, 25%) under both protocols, plus MRCR v2 and Oolong as additional/control benchmarks. The headline finding is that naive truncation causes large, statistically significant score drops in all eight primary eval-by-model cells, while distractor-aware truncation at the same nominal retention either preserves performance or, for smaller Claude models on BABILong, significantly improves it. The authors conclude that naive truncation is not a valid measurement of context-length effects and that future studies must distinguish signal from distractor.

Significance. If the central comparison holds, the paper provides an important methodological correction for long-context evaluation. Its strengths are substantial: a clean paired design with Wilcoxon signed-rank and Holm correction; a negative control (Oolong) that behaves as predicted; split-half stability analysis bounding seed sensitivity; explicit per-sample signal definitions verified against dataset ground truth; and a fully released pipeline with raw JSONL records, provenance hashes, and verification scripts. The empirical demonstration of a protocol-induced sign flip (harm under naive, help/neutral under signal-aware) is a valuable caution for the field. The paper also makes a concrete, falsifiable prediction: that a perfect retriever, modeled by distractor-aware truncation, yields only modest gains and mostly helps smaller models, not ceiling-saturating large ones.

major comments (2)

[§3.2 and Tables 1-2] The central contrast is presented as 'same nominal truncation' at alpha in {1.0, 0.75, 0.5, 0.25} under both protocols, but the token budget is enforced only for BABILong, where filler is compressed 'to reach the target alpha L'. For GraphWalks the retained signal is the full depth-bounded reachable subgraph, with no mechanism to compress it to alpha L; for MRCR the protocol drops turns but sets no token budget. No realized token counts by (protocol, alpha) are reported (Section 5.5 gives only full-context medians). If the retained reachable subgraph or needle-turn set occupies substantially more than alpha L tokens, then the flat or improved distractor-aware curves reflect a longer effective context rather than '25% context with signal preserved'. Because the abstract and conclusion explicitly claim equality of nominal truncation, this missing budget check is load-bearing. The released JSONL records contain provider-reported token counts, so the check is feasible without new API calls; the authors should either report realized token counts and show they match alpha L, or re-run the comparison with enforced token budgets.
[§3.2, BABILong signal definition] The completeness of the signal set is asserted rather than exhaustively verified. The paper states that the constrained-vocabulary regex achieves '100% recall on the no-filler 0k splits' and 'stable fact counts across all higher-filler splits', but it does not demonstrate that every answer-bearing fact is captured when facts are embedded in PG-19 filler, e.g., if filler is inserted inside a fact sentence. If any supporting fact is missed, the distractor-aware protocol would delete it and the 'preserved by construction' guarantee would fail, invalidating the recovery result on affected samples. The authors should either verify against per-sample supporting-fact annotations if available in BABILong, or perform a reconstructive check: confirm for every distractor-aware truncated prompt that the gold answer is still derivable from the retained facts. The verification scripts currently in the release should be extended to make this check explicit and exhaustive for all splits used.

minor comments (4)

[§4.6 and Abstract/Conclusion] The abstract and conclusion state that distractor-aware truncation produces 'no harm' across cells, but Table 1 reports a statistically significant harm for Sonnet on MRCR v2 signal-aware (delta=-0.112, p_adj=0.034). The paper acknowledges this residual harm in §4.6, but the summary claims should be qualified to avoid overstatement.
[Table 3] In the 100% column, the GraphWalks and MRCR rows show '(100%)' with parentheses while other cells are plain percentages; please make the notation consistent and clarify whether the parenthesized values are computed on the same basis as the others.
[§3.2, Oolong] The sentence 'the n_needles user-query needle pairs in MRCR v2' appears to be a typo for 'the n_needles user-query needle pairs' or 'the needle user-query pairs'; please correct the phrasing.
[§5.5] The realized input medians (~7k BABILong, ~50k MRCR, ~68k GraphWalks tokens) are reported only for full context; reporting the realized medians for each protocol and retention level would directly address the token-budget concern raised in the major comments and strengthen the paper's transparency.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild, acknowledged self-definitional element: distractor-aware truncation preserves the answer by construction, but the paper's core critique and main empirical contrasts are independent.

self definitional [Section 3.2 ('Two truncation protocols'), echoed in the abstract and Section 4.1/Table 1.]
"Distractor-aware truncation drops only content in x\S and is by construction answer-preserving. ... Under the distractor-aware protocol — which preserves the signal by construction — performance is preserved or improves."

The protocol is defined by first taking S to be a 'minimal subset S⊆x sufficient to determine y' and then deleting only x\S; keeping S is therefore not a discovered condition but the definition of the truncation. The paper's headline statement that under this protocol 'performance is preserved or improves' is, for the 'preserved' half, largely baked into the construction: the answer-bearing content is guaranteed to remain in the prompt, so the score cannot collapse for the reason that the naive protocol collapses.

full rationale

The paper's central methodological claim—that naive middle-removal confounds signal loss with distractor reduction and therefore cannot measure context-window effects—stands independently of the construction. The signal-survival diagnostics (Table 3) are measured properties of the datasets and the truncation procedure, not assumptions. The distractor-aware protocol's answer preservation is definitional in the sense that the retained set S is chosen to be sufficient for the gold label, but the paper states this openly and does not present it as an empirical discovery. No fitted parameters are used, no load-bearing self-citations appear, and the benchmarks and models are external. The observed improvements on smaller Claude models, the flat curves for larger models, and the Oolong negative control are empirical and not tautological. The skeptic concern about unverified equal-token budgets on GraphWalks and MRCR is a correctness/threat-to-validity issue, not a circularity issue. Overall the derivation is self-contained, with only the acknowledged construction-dependent element noted above.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted to data. The paper's empirical claims rest on four hand-crafted signal definitions, each verified against dataset ground truth, plus standard statistical assumptions. The only invented conceptual object is the 'distractor-aware truncation' protocol itself, which is a procedure, not an entity.

assumptions (5)

domain assumption The BABILong constrained-vocabulary regex has 100% recall on pure bAbI text and stable fact counts on 2k/8k/32k filler splits.
Section 3.2 BABILong instantiation; the distractor-aware protocol preserves all fact sentences, assuming the regex finds all of them on every split.
domain assumption For GraphWalks, the depth-bounded reachable subgraph from start nodes is the minimal sufficient subset for the BFS answer.
Section 3.2 GraphWalks instantiation; the protocol keeps exactly those edges and drops all others, relying on reachability being the complete task signal.
domain assumption For MRCR v2, the needle user-query and immediately-following assistant response, plus the preamble and final query, are sufficient for the correct answer.
Section 3.2 MRCR instantiation; the residual harm for Sonnet shows this definition is not fully sufficient, as the paper itself acknowledges.
domain assumption For Oolong, recomputing the correct aggregate over the subset of items the model sees is the proper gold standard for truncated prompts.
Section 3.2 Oolong; this defines the negative control and assumes no distractor exists in aggregation tasks.
standard math Paired Wilcoxon signed-rank tests on per-sample differences are valid for comparing truncation levels within the same samples.
Section 3.5; the within-subject design justifies the nonparametric paired test and the Holm correction controls the family-wise error rate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks." pith.science (2026). https://pith.science/paper/B73VGNQL

@misc{pith2026260803297,
  author       = {Pith},
  title        = {Pith review of: Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B73VGNQL}},
  note         = {Machine review of arXiv:2608.03297}
}

read the original abstract

A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved. We test this claim by running every sample of two long-context benchmarks -- BABILong and GraphWalks (BFS) -- at four context-retention fractions (100%, 75%, 50%, 25%) under two truncation protocols. The first is the naive protocol implicitly used in much prior work: drop content from the middle of the prompt. The second is distractor-aware: identify the task-relevant content for each sample and drop only the rest. We evaluate three sizes of the Claude family (Haiku 4.5, Sonnet 4.6, Opus 4.7) and, to test cross-provider generality, GPT-5.5 from a different provider; we apply the same protocol to two further benchmarks (MRCR v2, Oolong). Under naive truncation, score collapses monotonically (paired Wilcoxon, Holm-corrected p_adj < 0.05 in all eight BABILong and GraphWalks cells). Under the distractor-aware protocol -- which preserves the signal by construction -- performance is preserved or improves: the two smaller Claude models show statistically significant gains on BABILong, while the larger models (Opus 4.7 and GPT-5.5) sit at their full-context ceiling. The naive collapse and its distractor-aware recovery replicate on GPT-5.5, ruling out a single-provider artifact. The mechanism is direct: under the naive protocol the answer-bearing content survives in fewer than 1% of samples at 25% retention; under the distractor-aware protocol it is preserved by construction. The naive protocol is therefore not a measurement of context-window effects; it is a measurement of how often middle-removal happens to spare the answer. We conclude that future studies of context-length effects must specify how they distinguish signal from distractor, or they are at best ambiguous between two opposite hypotheses.

Figures

Figures reproduced from arXiv: 2608.03297 by the authors.

**Figure 1.** BABILong: score vs. context retention by protocol and model. Each panel is one truncation protocol (left: naive middle-removal; right: distractor-aware). Each line is one model; shaded bands are paired-bootstrap 95% CIs over n = 240 samples per cell (5,000 bootstrap resamples, seed 42). The protocol-induced sign flip is the central finding: under naive truncation every model shows monotone collapse; under distractor… view at source ↗

**Figure 2.** GraphWalks (BFS): set-F1 vs. context retention by protocol and model. Same axes and legend as [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages

[1]

Oolong: Evaluating long context reasoning and aggregation capabilities.arXiv preprint arXiv:2511.02817, 2025

Amanda Bertsch, Mirac Suzgun, et al. Oolong: Evaluating long context reasoning and aggregation capabilities.arXiv preprint arXiv:2511.02817, 2025

arXiv 2025
[2]

RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

arXiv 2024
[3]

Needle in a haystack: Pressure testing LLMs.https://github.com/gkamradt/ LLMTest_NeedleInAHaystack, 2023

Greg Kamradt. Needle in a haystack: Pressure testing LLMs.https://github.com/gkamradt/ LLMTest_NeedleInAHaystack, 2023

work page 2023
[4]

BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024

work page 2024
[5]

Same task, more tokens: The impact of input length on the reasoning performance of large language models.arXiv preprint arXiv:2402.14848, 2024

Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: The impact of input length on the reasoning performance of large language models.arXiv preprint arXiv:2402.14848, 2024. 13

arXiv 2024
[6]

Retrieval-augmented generation for knowledge-intensive NLP tasks

Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems, 2020

work page 2020
[7]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 2023

work page 2023
[8]

MRCR dataset.https://huggingface.co/datasets/openai/mrcr, 2025

OpenAI. MRCR dataset.https://huggingface.co/datasets/openai/mrcr, 2025

work page 2025

Show all 13 references

[9]

GraphWalks dataset

OpenAI. GraphWalks dataset. https://huggingface.co/datasets/openai/graphwalks, 2026

2026
[10]

Patil, Ion Stoica, and Joseph E

Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems.arXiv preprint arXiv:2310.08560, 2023

2023 arXiv
[11]

Rae, Anna Potapenko, Siddhant M

Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling.arXiv preprint arXiv:1911.05507, 2019

1911 arXiv
[12]

Michelangelo: Long context evaluations beyond haystacks via latent structure queries.arXiv preprint arXiv:2409.12640, 2024

Kiran Vodrahalli, Santiago Ontañón, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries.arXiv preprint arXiv:2409.12640, 2024

2024 arXiv
[13]

Towards AI-complete question answering: A set of prerequisite toy tasks.arXiv preprint arXiv:1502.05698, 2015

Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks.arXiv preprint arXiv:1502.05698, 2015. 14

2015 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Oolong: Evaluating long context reasoning and aggregation capabilities.arXiv preprint arXiv:2511.02817, 2025

Amanda Bertsch, Mirac Suzgun, et al. Oolong: Evaluating long context reasoning and aggregation capabilities.arXiv preprint arXiv:2511.02817, 2025

arXiv 2025

[2] [2]

RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

arXiv 2024

[3] [3]

Needle in a haystack: Pressure testing LLMs.https://github.com/gkamradt/ LLMTest_NeedleInAHaystack, 2023

Greg Kamradt. Needle in a haystack: Pressure testing LLMs.https://github.com/gkamradt/ LLMTest_NeedleInAHaystack, 2023

work page 2023

[4] [4]

BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024

work page 2024

[5] [5]

Same task, more tokens: The impact of input length on the reasoning performance of large language models.arXiv preprint arXiv:2402.14848, 2024

Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: The impact of input length on the reasoning performance of large language models.arXiv preprint arXiv:2402.14848, 2024. 13

arXiv 2024

[6] [6]

Retrieval-augmented generation for knowledge-intensive NLP tasks

Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems, 2020

work page 2020

[7] [7]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 2023

work page 2023

[8] [8]

MRCR dataset.https://huggingface.co/datasets/openai/mrcr, 2025

OpenAI. MRCR dataset.https://huggingface.co/datasets/openai/mrcr, 2025

work page 2025