Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Encoding retrieved examples as compact soft prompts instead of raw text makes retrieval-augmented generation more accurate, cheaper, and invariant to example ordering.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:19 UTC pith:2YPZV2V6

load-bearing objection Real architecture, correct proof, but the 20-point gain is not yet established because hyperparameter selection is unreported and the numbers lack error bars. the 3 major comments →

arxiv 2510.05363 v2 pith:2YPZV2V6 submitted 2025-10-06 cs.AI

MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts

Abhinav Jain , Xinyu Yao , Thomas Reps , Christopher Jermaine This is my paper
classification cs.AI
keywords retrieval-augmented generationsoft promptsmulti-head attentionin-context learningorder invariancecontext compressionmolecular property predictionPubMedQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the standard way of doing retrieval-augmented generation—pasting retrieved examples into the prompt as text—is wasteful, brittle, and suboptimal. The authors propose instead to learn a compact soft prompt: a multi-head attention module reads the query and the retrieved examples, and outputs a small set of continuous vectors that are prepended to the model's input. On molecular property prediction and biomedical question answering across three language models, the method reports an average gain of about 20 effective-accuracy points over text-based RAG, a roughly 10× reduction in inference FLOPs, and exact invariance to the order of the retrieved examples. If the results hold, retrieval-augmented adaptation can be both cheaper and more accurate, and its outputs become reproducible regardless of how the retrieval list is ordered.

Core claim

The central claim is that representing in-context exemplars as text is not the best representation. MHA-RAG learns a function g(D_K|x) that maps the top-K retrieved examples, conditioned on the user query x, into a soft prompt Z of length H—one vector per attention head—where each head computes a scaled dot-product attention output over the exemplar embeddings. Because the attention output is a permutation-invariant weighted sum, permuting the exemplars leaves Z unchanged, eliminating the order-sensitivity that plagues text-based RAG. Training the head weights (and LoRA-adapting the sentence encoder) on the target domain lets the model extract the information it needs from a handful of vecto

What carries the argument

Multi-head scaled dot-product attention over exemplar embeddings. Each head i projects the query embedding (via W_Q) and each example embedding (via W_K, W_V) and computes z_i = softmax(q_i K_i^T / sqrt(d)) V_i; the H head outputs are concatenated into the soft prompt. This single mechanism carries the whole argument: it compresses K text exemplars into H vectors (compression ratio |D_K|/H), it is order-invariant because attention weights depend only on the set of exemplars, and the number of heads is a tunable hyperparameter that controls the prompt length and thus the accuracy/cost trade-off.

Load-bearing premise

The headline gains assume that the hyperparameter sweeps (the number of heads and the learning rate) were tuned on a held-out validation split rather than on the small test sets, but the experimental section describes only train/test splits and never mentions a validation set.

What would settle it

Run the full experimental protocol with a properly held-out validation set: tune H and the learning rate on validation, freeze them, then evaluate on the test set over 3 seeds. If the geometric-mean effective-accuracy gain over RAG falls below about 10 points (or turns negative), the reported 20-point improvement is a test-set-selection artifact. A second check: with identical hyperparameters, shuffle the exemplar order and confirm that MHA-RAG produces exactly the same output logits; any difference means the implemented system does not match the order-invariance proof.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval-augmented inference can be made roughly 10× cheaper in FLOPs while improving accuracy, because the soft prompt scales with the number of heads rather than the number or length of retrieved documents.
  • Exemplar-order variance disappears by construction: shuffling the retrieved list produces the identical soft prompt, so downstream answers are reproducible regardless of retrieval ordering.
  • MHA-RAG reaches context saturation earlier than RAG (peak accuracy around K=5 rather than K=10), meaning fewer retrieved examples are needed to answer questions well.
  • The number of attention heads acts as a practical control knob: more heads improve accuracy once enough context is available (K=5), but add no benefit when only one exemplar is retrieved.
  • A small number of text exemplars can be re-inserted alongside the soft prompt (up to budget c) to push accuracy even higher, giving a tunable accuracy-versus-cost knob beyond heads.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the multi-head encoder is trained on the target domain but operates on any fresh retrieval at inference time, the method should amortize its training cost across many incoming queries—aligning naturally with production RAG services that serve one model to many users.
  • The same encoding function could be ported to tasks with long or multi-hop documents; the compression viewpoint suggests MHA-RAG may sidestep the 'lost in the middle' problem by not feeding raw text at all, and the paper's stated future work anticipates this direction.
  • Order invariance holds for the soft prompt, but the full pipeline's reproducibility also depends on the retrieval step; a practical test is to shuffle the retrieved list at deployment and confirm that end-to-end outputs are bit-identical, not merely close.
  • Nothing in the design ties the encoder to the specific sentence embedders used here (ChemBERTa-2 and Qwen3-Embedding), so the same head-count dial could be reused with any domain embedder, making the method a generic retrofit for existing RAG pipelines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MHA-RAG, a method that converts retrieved in-context exemplars into a small set of soft-prompt vectors via multi-head scaled dot-product attention. The number of attention heads H controls the soft-prompt length m=H, and the attention aggregation is order-invariant by construction, with a proof in Appendix A.1. The authors benchmark the method on molecular-property-prediction tasks (BACE, BBBP, ClinTox) and PubMedQA across Qwen3-0.6B, Qwen3-4B, and Llama3.2-3B-Instruct, comparing against RAG, xRAG, xRAG-K, and several PEFT baselines. The headline claims are a roughly 20-point gain in 'effective accuracy' over standard RAG, a 10x reduction in inference FLOPs, and exact invariance to exemplar order.

Significance. If the empirical claims are substantiated, MHA-RAG would be a practically useful architecture: it is simple, order-invariant, and has a clear mechanism for trading soft-prompt capacity against inference cost. The order-invariance proof in Appendix A.1 is mathematically correct, and the architectural description in Section 3.2 is clear. The paper also includes useful ablations (varying K and H) and an additional benchmark in Appendix A.2. However, the central empirical claim is currently not supported with adequate statistical rigor, and the evaluation protocol as described leaves open a serious selection-on-the-test-set concern. Because these issues concern the main contribution rather than presentation, the manuscript needs substantive revision.

major comments (3)
  1. [§4.1–4.2, Table 1] The central claim of a 19.66-point geometric-mean gain over RAG depends on hyperparameters H and learning rate being selected appropriately, but the paper never describes a validation split for BACE, BBBP, ClinTox, or PubMedQA. Section 4.1 lists only train/test splits, while Table 1 reports 'hyperparameters are tuned via sweeps: MHA-RAG (H∈{1,2,4,8})' and learning rates are drawn from {1e-5,3e-5,5e-4}. Appendix A.2 uses a train/dev split for IneqMath, demonstrating awareness of the distinction. With test sets of only 100 examples, selecting H and the learning rate on the test set can make the reported improvement an artifact of test-set selection. For ClinTox, which has roughly 10 positive test examples, effective accuracy is a coarse staircase and a single luck choice can shift the reported value by several points. Please report a validation procedure, or show that chosen hyperparameter
  2. [§4.1, Tables 1, 3, 4] All results are reported as averages over 3 random seeds with no error bars, standard deviations, or per-seed values. The test sets contain 100 examples in the molecular benchmarks, so binomial noise alone gives standard errors of several effective-accuracy points. Several reported comparisons are within this noise range (e.g., BACE Qwen3-0.6B: 75.08 vs. 76.01; PubMedQA Qwen3-4B: 76.59 vs. 79.26). Without a measure of variance, the claim that MHA-RAG 'achieves the greatest improvement' in 'almost all configurations' is not statistically supported. Please report per-seed results, standard deviations, or confidence intervals, and preferably a paired test across seeds.
  3. [§4.5, Table 4] The fine-tuning comparison reports effective accuracy of 0.0 for LoRA on BACE, ClinTox, and several other molecular tasks, attributing this to collapse to a single class. If these baselines are degenerate, then the aggregated Δavg comparisons in Table 4 are dominated by a failure mode rather than by a meaningful accuracy comparison. The text should provide explicit evidence of the class collapse (e.g., predicted label distributions) and should report results in a way that does not treat 0.0 effective accuracy as a real operating point. The missing-validation issue from the previous comment applies equally to this table and to the hyperparameters of LoRA, PT, and IDPG.
minor comments (5)
  1. [Abstract, §1] Typo: 'examplar-order variance' should be 'exemplar-order variance'.
  2. [§3.2 / Appendix A.1] The attention equations write q_i K_i^T with q_i treated as a row vector, while the proof in Appendix A.1 uses scores q·k_k/√d without transpose notation. The notation should be made consistent for readability.
  3. [Table 4] Some entries use inconsistent arrows, e.g., '↑-7.42' and '↑-7.42' in the Qwen3-4B BACE row and '↑-7.42' for PubMedQA. A negative improvement should not be marked with an up arrow.
  4. [Figure 2] The y-axis label is not visible in the caption; please state whether FLOPs are measured for the full inference pipeline (encoder plus foundation model) and include axis labels in the figure.
  5. [General] No code or configuration files are provided. Given the central importance of the exact hyperparameter sweep and training details, releasing code would materially help reproducibility.

Circularity Check

0 steps flagged

No circular derivation; minor non-load-bearing self-citations and an evaluation-protocol caveat keep the score at 2.

full rationale

The paper's central claims are empirical and the methodological components are self-contained. The order-invariance claim is proved from the model's own definition in Appendix A.1 (attention weights α_k = e^{s_k}/Σ e^{s_j}, output z = Σ α_k v_k), so it is a direct mathematical consequence, not an imported uniqueness theorem or an ansatz justified by citation. The efficiency claim is token-count arithmetic (soft-prompt length H versus textual exemplars), and the benchmark comparisons are external (BACE, BBBP, ClinTox, PubMedQA, IneqMath). The self-citations (Jain et al. 2024a,b) appear only as related-work motivation and are not load-bearing for MHA-RAG's derivation. The only flagged concern is that Section 4.1 describes train/test splits without a validation split while hyperparameters are tuned via sweeps (H ∈ {1,2,4,8}; LRs {1e-5,3e-5,5e-4}), and Appendix A.2 uses a train/dev split for IneqMath. If H/LR were selected on the small 100-example test sets, the 19.66-point average gain in Table 1 would be partly a selection artifact. However, the paper does not state that selection was made on test labels, so this is a potential evaluation-contamination/correctness risk, not an established circular reduction of the kind required for a higher score. I therefore find no significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

MHA-RAG introduces no new physical or conceptual entities; its contributions are an architecture and a metric. The free-parameter count is modest and mostly reflects ordinary hyperparameters and learned weights. The larger epistemic burden comes from domain assumptions about embedding quality, retrieval quality, and lossless compression, plus the ad hoc effective-accuracy metric.

free parameters (5)
  • Number of attention heads H (soft-prompt length m=H) = swept over {1,2,4,8}; per-task best not reported in main text
    Tunable hyperparameter controlling the compression ratio |D_K|/H and representational capacity; results in Tables 1-4 depend on selecting H per benchmark.
  • Learning rate = selected from {1e-5, 3e-5, 5e-4}
    Hyperparameter swept per experiment; exact per-task assignments are not reported.
  • Training epochs = 10 for molecular benchmarks, 1 for PubMedQA/IneqMath
    Chosen by hand based on dataset scale; no sensitivity analysis.
  • LoRA rank for embedding-model fine-tuning = 64
    Fixed for ChemBERTa and Qwen embedding fine-tuning; no ablation.
  • Soft-prompt encoder weights (W_Q, W_K, W_V per head) = Learned; e.g., 4.75M trainable params for Llama3.2-3B at m=1 (Appendix A.5)
    These are the standard trained parameters of the encoding function g_phi; they are fitted to the training data and are the main learned component of the method.
axioms (6)
  • domain assumption A frozen base LM can be effectively conditioned by prepending soft-prompt vectors to its word embeddings.
    Used throughout Section 3: the soft prompt Z is prepended to x without updating theta; the LM is assumed to respond to such continuous prompts.
  • domain assumption The sentence-embedding model E (ChemBERTa-2-10M-MTR or Qwen3-Embedding-0.6B/8B) produces embeddings of exemplar pairs that retain task-relevant information.
    Keys and values are computed from E_{x_k ⊕ y_k} (Section 3.2). If this embedding discards chemical/medical signal, the attention cannot recover it.
  • domain assumption Retrieval functions (Tanimoto similarity for molecules, cosine similarity for PubMedQA) return the most useful exemplars for a query.
    The top-K exemplars are selected by these functions (Section 4.1); the paper does not analyze retrieval quality or miss-rate.
  • domain assumption Compressing K text exemplars into H vectors does not lose information essential for the task.
    The whole efficiency gain rests on this compression; Section 4.3 shows accuracy degrades at K=10, implying compression can be lossy but the assumption is that H=4/8 suffices at K=5.
  • standard math Scaled dot-product attention over a set is permutation invariant.
    Appendix A.1 uses the softmax sum over permuted indices; this is a textbook property of softmax-weighted sums.
  • ad hoc to paper The effective-accuracy metric (geometric mean of TPR/TNR) is a meaningful primary metric for these tasks.
    Introduced in Section 4.1 and used for all headline numbers; no comparisons are reported with standard metrics such as accuracy, AUC, or F1.

pith-pipeline@v1.3.0-alltime-deepseek · 14338 in / 15886 out tokens · 123230 ms · 2026-08-04T11:19:40.658457+00:00 · methodology

0 comments
read the original abstract

Adapting Foundation Models to new domains with limited training data is challenging and computationally expensive. While prior work has demonstrated the effectiveness of using domain-specific exemplars as in-context demonstrations, we investigate whether representing exemplars purely as text is the most efficient, effective, and stable approach. We explore an alternative: representing exemplars as soft prompts with an exemplar order invariant model architecture. To this end, we introduce Multi-Head Attention Retrieval-Augmented Generation (MHA-RAG), a framework with the number of attention heads serving as a simple hyperparameter to control soft prompt-generation across different tasks. Across multiple question-answering benchmarks and model scales, MHA-RAG achieves a 20-point performance gain over standard RAG, while cutting inference costs by a factor of 10X GFLOPs-delivering both higher accuracy and greater efficiency, invariant to exemplar order.

Figures

Figures reproduced from arXiv: 2510.05363 by Abhinav Jain, Christopher Jermaine, Thomas Reps, Xinyu Yao.

Figure 1
Figure 1. Figure 1: Comparison of domain-adaptation methods: (a) RAG uses retrieved exemplars directly as [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of inference compute in FLOPS [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Varying number of heads in MHA-RAG. Performance averaged across 3 seeded runs. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Total FLOPs for inference with encoder ChemBERTa-10M-MTR and foundation model [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Varying number of heads in MHA-RAG. Performance averaged across 3 seeded runs. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 35 linked inside Pith

  1. [1]

    Chemberta-2: Towards chemical foundation models.arXiv preprint arXiv:2209.01712,

    Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models.arXiv preprint arXiv:2209.01712,

    Pith/arXiv arXiv

  2. [3]

    Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663,

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663,

    Pith/arXiv arXiv

  3. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  4. [8]

    In-context autoencoder for context compression in a large language model.arXiv preprint arXiv:2307.06945,

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model.arXiv preprint arXiv:2307.06945,

    Pith/arXiv arXiv

  5. [9]

    A comprehensive survey of retrieval- augmented generation (rag): Evolution, current landscape and future directions.arXiv preprint arXiv:2410.12837,

    Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. A comprehensive survey of retrieval- augmented generation (rag): Evolution, current landscape and future directions.arXiv preprint arXiv:2410.12837,

    Pith/arXiv arXiv

  6. [12]

    Exit: Context-aware extractive compression for enhancing retrieval-augmented generation.arXiv preprint arXiv:2412.12559,

    Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C Park. Exit: Context-aware extractive compression for enhancing retrieval-augmented generation.arXiv preprint arXiv:2412.12559,

    Pith/arXiv arXiv

  7. [13]

    Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation.Advances in Neural Information Processing Systems, 37:47297–47316, 2024a

    Abhinav Jain, Swarat Chaudhuri, Thomas Reps, and Chris Jermaine. Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation.Advances in Neural Information Processing Systems, 37:47297–47316, 2024a. Abhinav Jain, Chris Jermaine, and Vaibhav Unhelkar. Rag-modulo: Solving sequential tasks using experience, critics, and language...

    Pith/arXiv arXiv

  8. [14]

    Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146,

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering.arXiv preprint arXiv:1909.06146,

    Pith/arXiv arXiv 1909

  9. [16]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691,

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691,

    Pith/arXiv arXiv

  10. [17]

    500xcompressor: Generalized prompt compression for large language models.arXiv preprint arXiv:2408.03094,

    Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models.arXiv preprint arXiv:2408.03094,

    Pith/arXiv arXiv

  11. [18]

    Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172,

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172,

    Pith/arXiv arXiv

  12. [19]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602,

    Pith/arXiv arXiv

  13. [20]

    Pisco: Pretty simple compression for retrieval-augmented generation.arXiv preprint arXiv:2501.16075,

    11 Preprint Maxime Louis, Herv ´e D ´ejean, and St ´ephane Clinchant. Pisco: Pretty simple compression for retrieval-augmented generation.arXiv preprint arXiv:2501.16075,

    Pith/arXiv arXiv

  14. [21]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.arXiv preprint arXiv:2212.10511,

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.arXiv preprint arXiv:2212.10511,

    Pith/arXiv arXiv

  15. [22]

    Rethinking the role of demonstrations: What makes in-context learning work?arXiv preprint arXiv:2202.12837,

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?arXiv preprint arXiv:2202.12837,

    Pith/arXiv arXiv

  16. [23]

    Few-shot fine- tuning vs

    Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine- tuning vs. in-context learning: A fair comparison and evaluation.arXiv preprint arXiv:2305.16938,

    Pith/arXiv arXiv

  17. [24]

    Large language models sensitivity to the order of options in multiple-choice questions.arXiv preprint arXiv:2308.11483,

    Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions.arXiv preprint arXiv:2308.11483,

    Pith/arXiv arXiv

  18. [25]

    Context embeddings for efficient answer generation in rag.arXiv preprint arXiv:2407.09252,

    David Rau, Shuai Wang, Herv´e D´ejean, and St´ephane Clinchant. Context embeddings for efficient answer generation in rag.arXiv preprint arXiv:2407.09252,

    Pith/arXiv arXiv

  19. [26]

    Solving inequality proofs with large language models.arXiv preprint arXiv:2506.07927,

    Jiayi Sheng, Luna Lyu, Jikai Jin, Tony Xia, Alex Gu, James Zou, and Pan Lu. Solving inequality proofs with large language models.arXiv preprint arXiv:2506.07927,

    arXiv

  20. [27]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Pith/arXiv arXiv

  21. [28]

    Dynamic parametric retrieval augmented generation for test-time knowledge enhancement.arXiv preprint arXiv:2503.23895,

    Yuqiao Tan, Shizhu He, Huanxuan Liao, Jun Zhao, and Kang Liu. Dynamic parametric retrieval augmented generation for test-time knowledge enhancement.arXiv preprint arXiv:2503.23895,

    Pith/arXiv arXiv

  22. [29]

    Perception compressor: A training-free prompt compression framework in long context scenarios

    Jiwei Tang, Jin Xu, Tingwei Lu, Zhicheng Zhang, Yiming Zhao, Lin Hai, and Hai-Tao Zheng. Perception compressor: A training-free prompt compression framework in long context scenarios. arXiv preprint arXiv:2409.19272,

    Pith/arXiv arXiv

  23. [31]

    Idpg: An instance-dependent prompt generation method.arXiv preprint arXiv:2204.04497,

    Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. Idpg: An instance-dependent prompt generation method.arXiv preprint arXiv:2204.04497,

    Pith/arXiv arXiv

  24. [32]

    An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    Pith/arXiv arXiv

  25. [33]

    Recomp: Improving retrieval-augmented lms with compression and selective augmentation.arXiv preprint arXiv:2310.04408,

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation.arXiv preprint arXiv:2310.04408,

    Pith/arXiv arXiv

  26. [34]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    12 Preprint An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  27. [35]

    Compact: Com- pressing retrieved documents actively for question answering.arXiv preprint arXiv:2407.09014,

    Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Com- pressing retrieved documents actively for question answering.arXiv preprint arXiv:2407.09014,

    Pith/arXiv arXiv

  28. [36]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176,

    Pith/arXiv arXiv

  29. [37]

    In-context exemplars as clues to retrieving from large associative memory.arXiv preprint arXiv:2311.03498,

    Jiachen Zhao. In-context exemplars as clues to retrieving from large associative memory.arXiv preprint arXiv:2311.03498,

    Pith/arXiv arXiv

  30. [38]

    Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882,

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors.arXiv preprint arXiv:2309.03882,

    Pith/arXiv arXiv

  31. [39]

    ,(kπ(K) ,v π(K) )

    Kand consider the permuted sequence (kπ(1),v π(1)), . . . ,(kπ(K) ,v π(K) ). For the permuted sequence, the scores and weights are s′ k = q·k π(k)√ d =s π(k) α′ k = es′ k PK j=1 es′ j = esπ(k) PK j=1 esπ(j) Because {sπ(j) :j= 1, . . . , K}is a reordering of {sj :j= 1, . . . , K}, we have PK j=1 esπ(j) =PK j=1 esj . Therefore,α ′ k =α π(k). Now, the head’s...

    2025

  32. [40]

    and rank candidates by cosine similarity. We report results primarily onLlama3.2-3B-Instruct, since models in the Qwen3 family already achieve high off-the-shelf accuracy (e.g.,Qwen3-4Bat 65%), where adding retrieved exemplars led to performance drops. This observation is consistent with Sheng et al. (2025), who found that only certain model families bene...

    2025

  33. [2017]

    On the influence of context size and model choice in retrieval- augmented generation systems.arXiv preprint arXiv:2502.14759,

    Juraj Vladika and Florian Matthes. On the influence of context size and model choice in retrieval- augmented generation systems.arXiv preprint arXiv:2502.14759,

    Pith/arXiv arXiv

  34. [2019]

    Sufficient context: A new lens on retrieval augmented generation systems.arXiv preprint arXiv:2411.06037,

    Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: A new lens on retrieval augmented generation systems.arXiv preprint arXiv:2411.06037,

    Pith/arXiv arXiv

  35. [2020]

    Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366,

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366,

    Pith/arXiv arXiv

  36. [2021]

    Found in the middle: Calibrat- ing positional attention bias improves long context utilization.arXiv preprint arXiv:2406.16008,

    Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. Found in the middle: Calibrat- ing positional attention bias improves long context utilization.arXiv preprint arXiv:2406.16008,

    Pith/arXiv arXiv

  37. [2022]

    Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts.arXiv preprint arXiv:2205.11961,

    Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts.arXiv preprint arXiv:2205.11961,

    Pith/arXiv arXiv

  38. [2023]

    From reading to compressing: Exploring the multi-document reader for prompt compression.arXiv preprint arXiv:2410.04139,

    Eunseong Choi, Sunkyung Lee, Minjin Choi, June Park, and Jongwuk Lee. From reading to compressing: Exploring the multi-document reader for prompt compression.arXiv preprint arXiv:2410.04139,

    Pith/arXiv arXiv

  39. [2024]

    Adapting language models to compress contexts.arXiv preprint arXiv:2305.14788,

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts.arXiv preprint arXiv:2305.14788,

    Pith/arXiv arXiv

  40. [2025]

    Attentionrag: Attention-guided context pruning in retrieval-augmented generation.arXiv preprint arXiv:2503.10720,

    Yixiong Fang, Tianran Sun, Yuling Shi, and Xiaodong Gu. Attentionrag: Attention-guided context pruning in retrieval-augmented generation.arXiv preprint arXiv:2503.10720,

    arXiv