Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Shapley values, computed by a small learned subset scorer, rank and prune RAG context sentences so QA accuracy holds at half the token budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:26 UTC pith:KETWXBVX

load-bearing objection A solid engineering contribution whose formal Shapley guarantee is conditional on an unverified pointwise approximation assumption; the empirical ranking results are competitive and the limitations are honestly stated. the 4 major comments →

arxiv 2607.16209 v1 pith:KETWXBVX submitted 2026-05-10 cs.AI

Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning

Yanqiao Chen , Dongsheng Hou , Yuhan Rui , Zhen Cao , Yepang Liu This is my paper
classification cs.AI
keywords Shapley valuecontext pruningretrieval-augmented generationDeep Setscooperative game theoryMonte-Carlo samplingpermutation-invariant value functionsentence reranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes Shapley Context Pruning (SCP), which models a retrieved context as a cooperative game where each sentence is a player and any subset of sentences has a value. A lightweight Deep Sets network approximates that value function, trained only with pairwise margin labels that put supporting sentences above distractors. Each sentence is then scored by its Shapley value, the average marginal contribution over all orderings, which is estimated by Monte-Carlo sampling. The paper argues this yields an interpretable, redundancy-robust, scalable reranker: at around 50% token compression it preserves downstream QA accuracy on multi-hop benchmarks. It also proves a high-probability error bound separating value-function approximation error from sampling error.

Core claim

Context reranking can be cast as cooperative game theory: sentences are players, a subset's value is its usefulness for answering the query, and each sentence's importance is its Shapley value. Because the true value function is unknown and expensive, the paper learns it as a permutation-invariant Deep Sets network vθ(S)=ρ(Σ_{ci∈S} ψ(ci)), trained with a pairwise margin ranking loss. At inference, Monte-Carlo permutation sampling efficiently approximates Shapley values. The approach is shown to keep supporting-sentence recall high and downstream LLM QA accuracy close to the unpruned context on HotpotQA and 2WikiMH at around 50% compression, while outperforming lexical and prompt-compression

What carries the argument

The core object is the Shapley value of a sentence under a learned value function. The value function is a Deep Sets scorer vθ(S)=ρ(Σ_{ci∈S} ψ(ci)) with ψ embedding each sentence with the query and ρ aggregating the sum to a scalar; permutation invariance lets it evaluate arbitrary coalitions. Monte-Carlo sampling over random sentence permutations estimates each sentence's Shapley value, and a pairwise margin ranking loss trains the scorer to put supporting sentences ahead of distractors. The central identity is Shapley additivity, used to decompose total attribution error into sampling error plus 2ε_approx, yielding the paper's main bound and a top-K sample-complexity corollary.

Load-bearing premise

The theorem relies on the trained subset scorer being pointwise close to the true coalition-value function, but the pairwise ranking loss used in training does not enforce or measure that pointwise closeness, so the claimed bound is not instantiated in practice.

What would settle it

On a small context, exhaustively evaluate all coalitions with a downstream QA system to obtain an empirical true value function v*, then compute max_S |vθ(S)-v*(S)|; if this exceeds the ε_approx used in the bound, Theorem 3.1's guarantee does not apply. Alternatively, run the top-K sample-complexity formula with a chosen M on a test set and check whether the estimated top-K set matches the exhaustive true top-K set at the claimed probability; systematic mismatch would refute the ranking-preservation guarantee.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At roughly 50% token compression, SCP matches or beats lexical and cross-encoder pruning on several multi-hop QA benchmarks, sitting within about 1-2 F1 points of the unpruned context.
  • Supporting-sentence recall improves with more Monte-Carlo permutations; M=50 is a usable default but larger M approaches exact leave-one-out recall on needle-in-a-haystack tests.
  • Shapley attribution avoids the leave-one-out failure of zeroing duplicated supporting sentences, keeping positive credit for redundant evidence.
  • The error bound is additive, so increasing permutation samples reduces ranking error even if the learned value function is imperfect.
  • The attribution scorer is modular: it runs on top of any sentence embedding model and contributes only about 3M parameters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If a value function could be made supermodular, the paper's 'landscape of context' hierarchy would enable incremental attribution updates as new retrieved sentences arrive, avoiding full recomputation.
  • The same learned-Shapley pipeline could apply to feature selection or test-case prioritization whenever a subset's value is learnable from pairwise preference labels.
  • The set-based ranking aligns with the paper's stated hypothesis that sentence order is less important than token order in long RAG contexts, suggesting document-level set encoders as a research direction.
  • A testable extension is to fine-tune the value function on downstream LLM feedback, turning the coarse reranker into a task-optimized pruner.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Shapley Context Pruning (SCP), a sentence-level context reranking/pruning method for RAG. A Deep Sets value function v_θ maps subsets of sentences to scalar utilities, trained with a pairwise margin ranking loss over supporting vs. distractor sentences; Monte-Carlo permutation sampling then estimates Shapley values, and top-K pruning is applied before LLM generation. The paper claims scalability, interpretability, and redundancy robustness from this game-theoretic formulation, and includes a formal error bound (Theorem 3.1) and a sample-complexity guarantee for top-K preservation (Theorem A.3). Experiments cover reranking AUC, downstream QA, NIAH, cross-model robustness, and ablations, with an emphasis on a lightweight 3.03M-parameter value network. The authors are candid about limitations, including no repeated LLM QA runs and the gap between theoretical structure and empirical performance.

Significance. If the empirical claims hold, SCP would be a useful, lightweight and modular contribution to RAG context pruning, with a plausible connection between cooperative game theory and set-based reranking. The paper has notable strengths: it explicitly decouples attribution from the embedder, includes a controlled comparison of Shapley vs. LOO attribution on a matched value function, reports NIAH and downstream QA evaluations that are independent of the pairwise training objective, and provides careful ablation of embedding quality and Monte-Carlo budget. The theoretical results, however, are currently conditional on an unverified pointwise approximation bound, which limits the force of the 'without sacrificing mathematical rigor' claim in the abstract. The empirical results are competitive rather than uniformly superior, and the absence of confidence intervals for LLM QA results makes the headline margins hard to assess.

major comments (4)
  1. [§3.3, Theorem 3.1 and Appendix A.6, Theorem A.3] The load-bearing formal claim is the error bound |φ̂_i(v_θ) - φ_i(v*)| ≤ B√(2ln(2n/δ)/M) + 2ε_approx, which assumes |v*(S)-v_θ(S)| ≤ ε_approx for every coalition S. The training loss L(θ) in §3.2 only enforces pairwise Shapley-value margins on labeled positives/negatives; it does not regress v_θ(S) toward v*(S), and ε_approx is never measured or bounded. As the paper itself notes in Appendix D.4, top-K pruning depends only on relative Shapley ordering, not pointwise set values. Many value functions can produce identical Shapley rankings while differing from v* pointwise by an arbitrary amount. Thus Theorem A.3's condition Δ(K)>4ε_approx cannot be instantiated with the given training procedure. The theoretical guarantee should either be explicitly relabeled as conditional on a separately verified ε_approx, or supported by an empirical estimate/bound on the pointwise error.
  2. [§5.1 and Table 4] The paper states it did not implement multiple runs in LLM QA tasks due to API budget constraints. The headline downstream QA results (e.g., HotpotQA F1 0.830 vs. Not Pruned 0.845; 2WikiMH F1 0.864 vs. Not Pruned 0.870) are single-run numbers with no confidence intervals or variance estimates. Differences of 1–2 points are within typical run-to-run noise for LLM evaluation. I am not treating this as fatal, because the AUC and NIAH results provide corroborating evidence, but the main competitive claim needs either repeated runs or bootstrapped confidence intervals to support the reported margins.
  3. [§4.1 vs. Appendix C.1] There is a direct inconsistency about which trained model is used. §4.1 says 'By default, models are trained on the corresponding dataset (80% for training...)' and lists training on MS MARCO, HotpotQA, 2WikiMultiHop, MuSiQue, and FEVER. Appendix C.1 says 'We use the SCP model trained on MS MARCO by default for inference.' Table 20 reports an MS-MARCO-trained SCP on HotpotQA with AUC 0.750 at MC=50, whereas Table 3 reports SCP(MC=50) on HotpotQA with AUC 0.779. The two numbers are not reconciled. This matters for interpreting the AUC comparisons in Table 3 and for reproducibility; the paper should state per-table which training set was used.
  4. [Table 3 and Conclusion] The conclusion states that SCP 'outperforms baselines on multi-hop benchmarks', but Table 3 shows that SCP is not the best method on HotpotQA AUC (Provence 0.872 vs. SCP 0.779) or MS MARCO AUC (CrossEncoder 0.881 vs. SCP 0.773). The abstract's 'competitive' wording is fair, but the conclusion overstates the AUC comparison. This is a presentation issue, but it should be corrected to match the reported table.
minor comments (5)
  1. [§3.3] The sentence 'See 6 for a visualization' lacks a figure or appendix number; it should reference Figure 6.
  2. [Appendix A.3] Typo: 'hieararchy' should be 'hierarchy'.
  3. [Appendix C.4.3] Typo: 'Spped here only refers' should be 'Speed here only refers'.
  4. [Appendix C.5] The phrase 'CL-Bench reuses the dataset-releasedmessagesverbatim' is missing spacing: 'dataset-released messages verbatim'.
  5. [General] No code or model release is mentioned. Given the paper's engineering emphasis and reproducibility claims, providing the training/inference code or a public implementation would strengthen the contribution.

Circularity Check

1 steps flagged

In-domain Pairwise AUC is the training objective of L(θ), so it is partly a fit diagnostic; the ε_approx-dependent formal guarantee is an unverified premise (rigor gap, not circularity), while downstream QA is independent.

specific steps
  1. fitted input called prediction [Section 3.2 (pairwise margin ranking loss) and Section 4.1 / Table 3 (Pairwise AUC results)]
    "This objective only forces the network to correctly rank optimal evidence passages above distractors, rather than memorize exact performance figures: L(θ) = 1/|P| · |N| ∑_{i∈P}∑_{j∈N} max(0, ϵ−(ϕ_i(vθ)−ϕ_j(vθ))) ... The Pairwise AUC(Area Under Curve) metric, or Pairwise Ranking Accuracy, evaluates the performance of ranking the supporting sentences before the distractors."

    L(θ) is a margin penalty on exactly the pairwise comparison ϕ_i(vθ) > ϕ_j(vθ) for gold-positive vs gold-negative sentences, and AUC is the empirical fraction of such positive/negative pairs ordered correctly. Reporting in-domain AUC (HotpotQA 0.779, MuSiQue 0.946, FEVER 0.927 in Table 3) therefore measures the same objective used to fit vθ, on the same human supporting-sentence labels, so the high ranking accuracy is expected from the training objective rather than an independent external prediction. Downstream QA (EM/F1) and NIAH are not optimized by L(θ) and provide independent evidence.

full rationale

The central derivation—modeling context as a cooperative game, learning a Deep Sets value function vθ, and ranking sentences by MC-Shapley—is not circular. The Shapley definition and the Hoeffding/additivity proof in Appendix A.6 are standard, and the MC estimator is unbiased with respect to vθ; no load-bearing self-citation chains appear in the references. The downstream QA results (HotpotQA F1 0.830, 2WikiMH F1 0.864 at 50% compression), NIAH recall, and cross-domain transfer experiments evaluate quantities not used in L(θ), giving independent support. The one genuine circular element is the in-domain Pairwise AUC: since L(θ) directly optimizes the same pairwise Shapley margin that AUC measures, Table 3 is partly a fit diagnostic rather than an external validation. I also flag the formal guarantee in Theorem 3.1/A.3: it is conditional on an unmeasured pointwise bound |v*(S)−vθ(S)|≤ε_approx, and the pairwise-margin training does not instantiate this bound (Shapley values are a low-dimensional projection of vθ, so many value functions share the learned ranking while differing pointwise). This is a correctness/rigor gap in the 'formal guarantees' claim, not a circular reduction, so it does not raise the circularity score beyond 4. The empirical ranking may still be useful, but the theorem's premise remains unverified.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

Central claim is conditional on the existence and approximation of a true subset-value function v* that is not operationalized; the training signal is binary supporting labels, not utility values. The MC sampling theory is standard, but the ε_approx premise is assumed, not measured. Several hyperparameters (M, ε, keep ratio, architecture) are chosen by hand, and the 'Landscape of Context' is a conceptual add-on rather than a validated component.

free parameters (4)
  • pairwise margin ε = 0.15
    Hand-set hyperparameter in the ranking loss L(θ) (§3.2); no sensitivity analysis is reported.
  • Monte-Carlo permutation count M = 50 (default); ablations up to 1000
    Sampling budget chosen for efficiency; Theorem A.3 gives a bound but requires unknown Δ(K) and ε_approx, so M is an empirical choice (§3.3, §4.3.3).
  • Top-K keep ratio = 50% main experiments; also 70%, 30%, 7%, 3%
    Pruning criterion K is set by compression target; no adaptive threshold is proposed (§5.1).
  • Deep Sets architecture dimensions = hidden=1024, latent=512 (~3.03M params)
    Architectural hyperparameters chosen by hand; model size is presented as a feature, but no architecture search is shown.
axioms (6)
  • domain assumption A true subset-performance value function v* exists and v*(S) measures the utility of a coalition of sentences.
    Definition A.1; never operationally measured; training uses binary supporting labels instead of utility values.
  • ad hoc to paper Pointwise approximation error is bounded: |v*(S)-vθ(S)| ≤ ε_approx for all S.
    Assumption in Theorem 3.1; no bound or estimate is provided, and pairwise ranking loss does not imply pointwise accuracy.
  • domain assumption vθ and v* are uniformly bounded by B (Assumption 3.1).
    vθ can be enforced with sigmoid/clipping; boundedness of v* is assumed without justification.
  • standard math Deep Sets can represent the true value function.
    Deep Sets universal approximation theorem (Zaheer et al. 2018), used in Definition 3.2.
  • standard math Uniform random permutations yield unbiased i.i.d. Shapley marginals.
    Castro et al. 2009; used in Algorithm 1 and Theorem 3.1.
  • domain assumption Binary supporting-sentence labels are a sufficient training signal for the value function.
    Pairwise margin loss uses human annotations; MuSiQue paragraph-level labels are acknowledged noisy (§C.3.3), so the assumption is only partially met.
invented entities (1)
  • Landscape of Context no independent evidence
    purpose: Theoretical forest/laminar hierarchy of context coalitions, intended to organize long contexts as trees.
    Appendix A.4; explicitly conceptual and not implemented; no falsifiable prediction outside the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 31328 in / 15912 out tokens · 159734 ms · 2026-08-02T14:26:20.418013+00:00 · methodology

0 comments
read the original abstract

Context reranking and pruning have become essential for improving the efficiency of modern Retrieval-Augmented Generation (RAG) systems, yet an interpretable and unified framework remains underexplored. Previous work has primarily emphasized lexical retrieval, cross-encoder architectures, model distillation, and Low-Rank Adaptation (LoRA), mostly relying on heuristic loss functions and empirical attribution. This paper presents Shapley Context Pruning (SCP), a novel framework for context reranking that establishes a cooperative-game-theory perspective for importance attribution by modeling the context as a cooperative game. Balancing the trade-off between fine-grained and coarse-grained representations, we employ a Deep Sets architecture to approximate a permutation-invariant value function at the sentence level, utilizing pre-trained language models as sentence embedders and optimizing via a pairwise margin ranking loss. To ensure practical scalability without sacrificing mathematical rigor, we leverage Monte-Carlo sampling for efficient training and inference, providing formal theoretical error bounds and sample complexity guarantees for preserving Top-K subset rankings. Furthermore, we conduct comprehensive experiments-spanning supporting-sentence recall, Needle-in-the-Haystack (NIAH) evaluations, long-context QA, and multi-hop reasoning-alongside rigorous ablation studies on embedding quality and attribution strategies. The model achieves competitive downstream QA performance against robust baselines.

Figures

Figures reproduced from arXiv: 2607.16209 by Dongsheng Hou, Yanqiao Chen, Yepang Liu, Yuhan Rui, Zhen Cao.

Figure 1
Figure 1. Figure 1: An overview of concept evolution and how SCP different from the previous methods. SCP [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture Overview. The SCP framework consists of three main components: (i) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of SCP empirical results. (a) Reranking AUC (Pairwise Ranking Accuracy) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation studies. (a) Embedding mechanism comparison; the default SentenceTransformer [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Landscape of Context: What Will This Look Like? In this figure, we show an example of a [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The error bound for pruning attribution can be decoupled into two independent sources: [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: CL-Bench overall results (k=0.5). Judged by GPT-5.4. GPT-5.4 performs not idealy on [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 2 canonical work pages

  1. [1]

    MS MARCO: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    Pith/arXiv arXiv 2016

  2. [2]

    Blei, Andrew Y

    David M. Blei, Andrew Y . Ng, and Michael I. Jordan. Latent dirichlet allocation.J. Mach. Learn. Res., 3(null):993–1022, March 2003. ISSN 1532-4435

    2003

  3. [3]

    Learning to rank: from pairwise approach to listwise approach

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. InProceedings of the 24th International Conference on Machine Learning, ICML ’07, page 129–136, New York, NY , USA, 2007. Association for Computing Machinery. ISBN 9781595937933. doi: 10.1145/1273496.1273513. URL https://doi.org/10.11...

    arXiv 2007

  4. [4]

    Polynomial calculation of the Shapley value based on sampling.Computers and Operations Research, 36(1):303–314, 2009

    Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the Shapley value based on sampling.Computers and Operations Research, 36(1):303–314, 2009

    2009

  5. [5]

    Provence: Efficient and robust context pruning for retrieval-augmented generation

    Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, and Stéphane Clinchant. Provence: Efficient and robust context pruning for retrieval-augmented generation. InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2501.16214

    Pith/arXiv arXiv 2025

  6. [6]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention. InProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019. arXiv:1906.04341

    Pith/arXiv arXiv 2019

  7. [7]

    Subset ranking using regression

    David Cossock and Tong Zhang. Subset ranking using regression. In Gábor Lugosi and Hans Ulrich Simon, editors,Learning Theory, pages 605–619, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-35296-9

    2006

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/ abs/1810.04805

    Pith/arXiv arXiv 2019

  9. [9]

    Loocomp: Leverage leave-one-out strategy to encoder-only transformer for efficient query-aware context compres- sion, 2026

    Thao Do, Dinh Phu Tran, An V o, Seon Kwon Kim, and Daeyoung Kim. Loocomp: Leverage leave-one-out strategy to encoder-only transformer for efficient query-aware context compres- sion, 2026. URLhttps://arxiv.org/abs/2603.09222

    arXiv 2026

  10. [10]

    CL-bench: A benchmark for context learning.arXiv preprint arXiv:2602.03587, 2026

    Shihan Dou et al. CL-bench: A benchmark for context learning.arXiv preprint arXiv:2602.03587, 2026

    arXiv 2026

  11. [11]

    Data Shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. InInternational Conference on Machine Learning (ICML), 2019. arXiv:1904.02868

    Pith/arXiv arXiv 2019

  12. [12]

    Tokenshap: Interpreting large language models with monte carlo shapley value estimation, 2024

    Roni Goldshmidt and Miriam Horovicz. Tokenshap: Interpreting large language models with monte carlo shapley value estimation, 2024. URLhttps://arxiv.org/abs/2407.10114

    Pith/arXiv arXiv 2024

  13. [13]

    Gemini 3

    Google DeepMind. Gemini 3. https://deepmind.google, 2026. The experiment corre- sponding to Gemini 3 Pro/Flash is implemented within 27 March 2026 to 15 April 2026

    2026

  14. [14]

    Daya Guo, Dejian Yang, and Haowei et al. Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10. 1038/s41586-025-09422-z. URLhttp://dx.doi.org/10.1038/s41586-025-09422-z. 10

    doi:10.1038/s41586-025-09422-z 2025

  15. [15]

    Harsanyi.A Simplified Bargaining Model for the n-Person Cooperative Game, pages 44–70

    John C. Harsanyi.A Simplified Bargaining Model for the n-Person Cooperative Game, pages 44–70. Springer Netherlands, Dordrecht, 1982. ISBN 978-94-017-2527-9. doi: 10.1007/ 978-94-017-2527-9_3. URLhttps://doi.org/10.1007/978-94-017-2527-9_3

    work page doi:10.1007/978-94-017-2527-9_3 1982

  16. [16]

    Herbrich, T

    R. Herbrich, T. Graepel, and K. Obermayer. Support vector learning for ordinal regression. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 1, pages 97–102 vol.1, 1999. doi: 10.1049/cp:19991091

    work page doi:10.1049/cp:19991091 1999

  17. [17]

    Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics (COLING), 2020. arXiv:2011.01060

    Pith/arXiv arXiv 2020

  18. [18]

    Ruler: What’s the real context size of your long-context language models?, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024. URLhttps://arxiv.org/abs/2404.06654

    Pith/arXiv arXiv 2024

  19. [19]

    Leveraging passage retrieval with generative models for open domain question answering

    Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. InInternational Conference on Learning Representations (ICLR), 2021. arXiv:2007.01282

    Pith/arXiv arXiv 2021

  20. [20]

    LLMLingua: Com- pressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2310.05736

    Pith/arXiv arXiv 2023

  21. [21]

    Optiset: Unified optimizing set selection and ranking for retrieval-augmented generation, 2026

    Yi Jiang, Sendong Zhao, Jianbo Li, Bairui Hu, Yanrui Du, Haochun Wang, and Bing Qin. Optiset: Unified optimizing set selection and ranking for retrieval-augmented generation, 2026. URLhttps://arxiv.org/abs/2601.05027

    arXiv 2026

  22. [22]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin et al. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. arXiv:2004.04906

    Pith/arXiv arXiv 2020

  23. [23]

    Kosiorek, Seungjin Choi, and Yee Whye Teh

    Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks, 2019. URLhttps://arxiv.org/abs/1810.00825

    Pith/arXiv arXiv 2019

  24. [24]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://arxiv.org/abs/2005.11401

    Pith/arXiv arXiv 2021

  25. [25]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2310.06201

    Pith/arXiv arXiv 2023

  26. [26]

    A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    Jiaheng Liu et al. A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    arXiv 2025

  27. [27]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics (TACL), 12:157–173, 2024. arXiv:2307.03172

    Pith/arXiv arXiv 2024

  28. [28]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https: //arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  29. [29]

    A unified approach to interpreting model predictions

    Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv:1705.07874

    Pith/arXiv arXiv 2017

  30. [30]

    A survey of context engineering for large language models, 2025

    Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, and Shenghua Liu. A survey of context engineering for large language models, 2025. URL https://arxiv.org/abs/2507.13334. 11

    Pith/arXiv arXiv 2025

  31. [31]

    Distributed representations of words and phrases and their compositionality, 2013

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality, 2013. URL https://arxiv. org/abs/1310.4546

    Pith/arXiv arXiv 2013

  32. [32]

    cross-encoder/ms-marco-MiniLM-L6-v2

    Nils Reimers, Tom Aarsen. cross-encoder/ms-marco-MiniLM-L6-v2. https://huggingface. co/cross-encoder/ms-marco-MiniLM-L6-v2, 2021. Accessed: 2026-03-27

    2021

  33. [33]

    Passage re-ranking with bert, 2020

    Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert, 2020. URL https: //arxiv.org/abs/1901.04085

    Pith/arXiv arXiv 2020

  34. [34]

    OpenAI. GPT-5.4. https://openai.com, 2026. The experiment corresponding to GPT-5.4 (Medium) is implemented within 27 March 2026 to 15 April 2026

    2026

  35. [35]

    LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, et al. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. InFindings of the Association for Computational Linguistics: ACL 2024, 2024

    2024

  36. [36]

    Sentence-bert: Sentence embeddings using siamese bert- networks, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019. URLhttps://arxiv.org/abs/1908.10084

    Pith/arXiv arXiv 2019

  37. [37]

    The probabilistic relevance framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 3(4):333–389, 2009

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends in Information Retrieval, 3(4):333–389, 2009

    2009

  38. [38]

    Term-weighting approaches in automatic text re- trieval.Inf

    Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text re- trieval.Inf. Process. Manage., 24(5):513–523, August 1988. ISSN 0306-4573. doi: 10.1016/ 0306-4573(88)90021-0. URLhttps://doi.org/10.1016/0306-4573(88)90021-0

    doi:10.1016/0306-4573(88)90021-0 1988

  39. [39]

    Springer Nature Switzerland, 2025

    Ferdinand Schlatt, Maik Fröbe, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Benno Stein, Martin Potthast, and Matthias Hagen.Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders, pages 1–19. Springer Nature Switzerland, 2025. ISBN 9783031887116. doi: 10.1007/978-3-031-88711-6_1. a...

    Pith/arXiv arXiv 2025

  40. [40]

    Lloyd S. Shapley. A value for n-person games. InContributions to the Theory of Games. Princeton University Press, 1953

    1953

  41. [41]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML), 2017

    2017

  42. [42]

    Openprovence: An open-source implementation of efficient and robust context pruning for retrieval-augmented generation, 2025

    Yuichi Tateno. Openprovence: An open-source implementation of efficient and robust context pruning for retrieval-augmented generation, 2025. URLhttps://github.com/hotchpotch/ open_provence

    2025

  43. [43]

    FEVER: a large-scale dataset for fact extraction and VERification

    James Thorne et al. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018. arXiv:1803.05355

    Pith/arXiv arXiv 2018

  44. [44]

    MuSiQue: Multihop questions via single-hop question composition

    Harsh Trivedi et al. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics (TACL), 10:569–585, 2022. arXiv:2108.00573

    Pith/arXiv arXiv 2022

  45. [45]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv. org/abs/1706.03762

    Pith/arXiv arXiv 2023

  46. [46]

    Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne

    Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne. On the limitations of representing functions on sets. InInternational Conference on Machine Learning (ICML), 2019. arXiv:1901.09006

    Pith/arXiv arXiv 2019

  47. [47]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020. URL https://arxiv.org/abs/2002.10957

    Pith/arXiv arXiv 2020

  48. [48]

    SWE-Pruner: Self-adaptive context pruning for coding agents.arXiv preprint arXiv:2601.16746, 2026

    Yuhang Wang et al. SWE-Pruner: Self-adaptive context pruning for coding agents.arXiv preprint arXiv:2601.16746, 2026. 12

    Pith/arXiv arXiv 2026

  49. [49]

    Recomp: Improving retrieval-augmented lms with compression and selective augmentation, 2023

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation, 2023. URL https://arxiv.org/abs/2310. 04408

    2023

  50. [50]

    Qwen3 technical report, 2025

    An Yang and Anfeng Li et al. Qwen3 technical report, 2025. URL https://arxiv.org/ abs/2505.09388

    Pith/arXiv arXiv 2025

  51. [51]

    HotpotQA: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang et al. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. arXiv:1809.09600

    Pith/arXiv arXiv 2018

  52. [52]

    Deep sets, 2018

    Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets, 2018. URLhttps://arxiv.org/abs/1703.06114

    Pith/arXiv arXiv 2018

  53. [53]

    Core" and

    Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. A setwise approach for effective and highly efficient zero-shot ranking with large language models. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, page 38–47. ACM, 2024. doi: 10.1145/3626772.3657813. URL http://...

    arXiv 2024