Pith. sign in

REVIEW 2 major objections 4 minor 38 references

Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Swapping the causal mask for bi-directional attention reliably improves multi-hop question answering across decoder-only language models.

desk verdict The paper is worth your time, but don't take the bidirectional-mask result at face value: the prompt puts the question after the documents, so the prefix-mask gain is confounded with giving documents access to the question. read the letter →

arxiv 2505.11754 v1 pith:LH22QFPY submitted 2025-05-16 cs.CL

Wenyu Huang , Pavlos Vougiouklis , Mirella Lapata , Jeff Z. Pan This is my paper · ORCID
classification cs.CL
keywords multi-hopquestionansweringcausalmaskprefixbi-directionalattentioncontextpermutationInformationContributionscoreretrieval-augmentedgenerationMuSiQue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the causal mask that forces decoder-only language models to read contexts left-to-right is a genuine bottleneck for multi-hop question answering, where a model must combine evidence from several retrieved documents. The authors show that it is: fine-tuning the same models with a simple prefix mask, which lets earlier documents attend to later ones, improves accuracy on the MuSiQue benchmark for every model tested, while also making performance less sensitive to document order. They also find that when a model answers correctly, its attention on the final supporting document peaks higher, and they exploit that signal to pick the best of several shuffled contexts, lifting Qwen 7B accuracy from 28.6% to 33.7%. A sympathetic reader would take away a practical recipe for RAG systems: order documents along the reasoning chain, keep them close together, and consider replacing the causal mask with a prefix mask if fine-tuning is affordable.

What carries the argument

The central mechanism is the attention mask itself: a causal mask forbids earlier tokens from attending to later ones, and the paper replaces it with a prefix mask $\mathbf{M}_{i,j}=0$ if $i\ge j$ or if both $i,j$ are within the context, and $1$ otherwise, converting a decoder-only model into a prefix LM during fine-tuning. To analyse behaviour, the paper introduces grouped attention weights $\mathrm{GA}_{l,h}(X,Y)$ that aggregate attention between token blocks, and the Information Contribution score $\mathrm{IC}_l(d)$ that sums attention from answer tokens to a document across heads and layers. The peak IC score is used both to explain correctness and as a heuristic to select among context permutations.

What would settle it

Run the identical peak-IC reranking experiment on a held-out test set of MuSiQue or on a different multi-hop dataset (for instance, HotpotQA or 2WikiMultihopQA with fresh questions) and check whether the best-permutation accuracy still beats the average shuffled accuracy by the same margin; if the gap shrinks to near zero, the peak-IC signal does not generalise. A second check: train the prefix-masked models on a dataset where gold documents are explicitly in backward order and see whether the forward-order preference still emerges, which would test the claim that the bias is an emergent fine-tuning property.

Watch

Extended reading notes

Core claim

The paper's central claim is that the causal mask is a measurable and fixable limitation for multi-hop reasoning in decoder-only language models. Replacing the causal mask with a prefix mask (where every context token can attend to every other context token, while the question and answer region remain causally masked) and fine-tuning with LoRA yields consistent gains over ordinary fine-tuning: for example, Qwen2.5 7B rises from 58.05% to 62.96% and Llama3.1 8B from 63.51% to 65.48%, with smaller sensitivity to forward/backward document order. The paper further claims that encoder-decoder models such as Flan-T5 outperform similarly sized decoder-only models out of the box, that fine-tuned models prefer contexts where gold documents appear in reasoning-chain order, and that the distance between gold documents matters more than their order for non-fine-tuned models. Finally, the paper argues that the peak of the Information Contribution score -- the maximum attention a model gives to any single document while generating the answer -- is a signal of answer correctness, and that selecting the permutation with the highest peak score improves accuracy.

Load-bearing premise

The paper assumes that the peak Information Contribution score observed on the MuSiQue development set is a stable, transferable signal of answer correctness, and that the improvement from selecting the highest-peak permutation will hold on genuinely new data rather than being an artifact of the same 2,417 queries used to discover the correlation.

Editorial extensions

If this is right

  • RAG pipelines that use off-the-shelf causal decoder-only readers should place high-relevance documents near the end of the context and minimise the distance between supporting documents, since non-fine-tuned models are order-invariant but distance-sensitive.
  • When fine-tuning is an option, replacing the causal mask with a prefix mask is a cheap, LoRA-compatible change that improves multi-hop accuracy and robustness to document order across model families.
  • The peak Information Contribution score can serve as a confidence or selection signal: among multiple context orderings, the one with the highest peak IC tends to give the correct answer more often.
  • Fine-tuned models, including those with bi-directional attention, still mostly fail to refuse answering when the first-hop document is missing, so evidence attribution and completeness checks remain necessary in deployed QA systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The peak-IC selection heuristic resembles a self-contained reranking method, and if it generalises beyond MuSiQue, it could be used to choose between multiple retrievals or generation drafts without any additional training or external scorers.
  • The finding that fine-tuning instils a forward-order bias, even when training data has no such order, suggests that the bias is an emergent inductive bias of the fine-tuning objective rather than a property of the base model; testing this on other reasoning datasets would clarify whether it is a general phenomenon.
  • Because the bi-directional gain is demonstrated with LoRA fine-tuning, one natural extension is to test the same prefix-mask modification at larger scale or with full fine-tuning, where the gap between causal and bi-directional reading may widen or narrow.
  • The paper evaluates peak-IC selection on the same development set used to discover the signal; a genuinely held-out test would settle whether the heuristic overfits to MuSiQue's particular answerable questions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies how context permutation affects multi-hop QA in encoder-decoder (Flan-T5) and causal decoder-only (Qwen2.5, Llama 3.x) language models. It evaluates three manipulations: the order of gold documents (forward/backward/original), the distance between them, and their completeness (removing the first-hop document). The main empirical findings are that Flan-T5 models outperform similarly sized decoder-only models in zero-shot settings, fine-tuned decoder-only models prefer forward ordering, increasing inter-document distance hurts non-fine-tuned models, and replacing the causal mask with a prefix mask (Eq. 4) improves accuracy and robustness. The paper further analyzes attention via an Information Contribution (IC) score, finding that higher peak IC scores correlate with correct answers, and uses the peak IC to select among 20 random document permutations, improving Qwen 7B accuracy from 28.6% to 33.7%. Experiments are conducted on MuSiQue and supplemented by 2WikiMultihopQA results in the appendix.

Significance. If the findings hold, the paper provides practical guidance for retrieval-augmented generation: document order and inter-document distance matter, and a simple prefix-mask fine-tuning recipe can make decoder-only readers more accurate and robust. The paper has notable strengths: it covers three model families across a range of sizes, validates core trends on two datasets, reports a Spearman/Kendall check showing that training-set order is not spuriously correlated with the forward/backward distinction, and includes a parametric-knowledge probe in Section 5.3. The code is released. The main risks are the prompt-layout confound in the Finetuned+Bi comparison and the lack of uncertainty quantification for the peak-IC selection heuristic; both are addressable with additional experiments.

major comments (2)
  1. [§4, Eq. (4), and Appendix C] The comparison between Finetuned and Finetuned+Bi is potentially confounded by the prompt layout. The prompt places documents before the question (Appendix C), so under a causal mask document tokens cannot attend to the question, whereas under the prefix mask defined by Eq. (4) all context tokens—depending on the definition of c—attend bidirectionally, which would make document encodings question-aware. The paper attributes the gains to bidirectional document-document attention (Section 7.2, Figure 1), but it does not report a causal-mask baseline with the question placed before the documents, which would make document encodings question-aware without changing the mask. If that baseline matches Finetuned+Bi, the conclusion that 'the causal mask hinders multi-hop reasoning' would instead be a prompt-layout artifact; if the gap persists, the bidirectional-document account is confirmed. Please clarify the value of c in Eq. (4) (does the prefix include the question tokens?) and add the question-first control, or soften the mechanistic claim accordingly.
  2. [§6.1 and Figure 5] The peak-IC selection heuristic is discovered and evaluated on the same MuSiQue development set (2,417 queries), and the reported improvement (28.6% to 33.7% for Qwen 7B Answer Only) is a single point estimate with no confidence interval, bootstrap, or other significance assessment. Because the heuristic is arguably selected based on this very dataset, the improvement could reflect dev-set overfitting rather than a generalizable signal. The 2WikiMultihopQA results in Appendix H provide some cross-dataset transfer evidence, but they are also reported without variance. Please report paired bootstrap or per-question standard errors on MuSiQue (and ideally on 2Wiki), for example by computing the accuracy of the max-IC selection over repeated bootstrap samples of the 2,417 queries, and state whether the same heuristic was fixed before evaluating on MuSiQue or whether it was tuned there.
minor comments (4)
  1. [Table 1] The definitions of ΔB and ΔF are not explicitly stated; the text says they are 'performance differences between original documents and re-ordered backward and forward documents respectively,' but it is not clear whether the difference is new minus original or original minus new. Please state the sign convention explicitly (e.g., ΔB = Acc_backward − Acc_original).
  2. [§3.3, Eq. (3)] The set A is defined as 'the set of answer tokens in the prediction,' but when the prediction is not boxed (e.g., for CoT when models fail to follow the instruction), the definition of answer tokens is unclear. Please specify how answer tokens are identified in those cases.
  3. [Abstract and §5.1] The abstract states that encoder-decoder models 'generally outperform' causal decoder-only LMs despite being smaller, but this comparison is only for the zero-shot Answer Only and CoT setups; fine-tuned decoder-only models such as Qwen 2.5 14B reach higher accuracy than Flan-T5 xxl. Please clarify that the superiority claim is for off-the-shelf/zero-shot use, not for fine-tuned settings.
  4. [Figure 4] In panel (a), the legend contains a typo: 'Inorrect' should be 'Incorrect'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central claims are directly measured empirical comparisons, with the attention-based selection validated on an external dataset.

full rationale

No circular derivation chain is present. The paper's main claims (encoder-decoder advantage, forward-order sensitivity, and prefix-mask gains) are reported as measured accuracies under explicit experimental conditions, not as quantities derived from fitted parameters. Eq. (3) defines the Information Contribution score as an aggregation of attention weights; Eq. (4) is an intervention that changes the mask, and the resulting accuracy differences are directly observed. The peak-IC selection heuristic uses the model's own attention as an internal confidence signal to choose among shuffled-context answers, and the selection step does not use answer labels; the improvement is therefore not an input renamed as a prediction. The paper additionally tests the same heuristic and order-sensitivity findings on the 2WikiMultihopQA dataset in Appendix H, providing external validation beyond the MuSiQue development set. The only in-house citation (Huang et al. 2024 in Section 2.1) is used as an example of RAG-based KGQA applications and is not load-bearing for any of the paper's conclusions. A possible prompt-order confound exists because the question follows the documents (Appendix C), but a confound is an experimental-design concern, not a circular reduction of the claim to its inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger contains no invented theoretical entities. The central claims rest on standard transformer assumptions, dataset labels, and hyperparameters chosen by hand. The free parameters are LoRA settings and shuffle counts, none fitted to maximize any reported metric. The main hidden assumptions are that MuSiQue gold documents reflect the true reasoning chain and that grouped attention weights meaningfully measure document contribution.

free parameters (5)
  • LoRA rank r = 8
    Hyperparameter for PEFT finetuning; chosen by hand, not fitted to data. All finetuning and bi-attention results use this.
  • LoRA alpha = 16
    Hyperparameter for PEFT finetuning; chosen by hand, not fitted to data.
  • Learning rate = 2e-5
    Hyperparameter for finetuning; standard choice, not fitted to data.
  • Training epochs = 5
    Hyperparameter for finetuning; chosen by hand, not fitted to data.
  • Number of random shuffles for peak-IC selection = 20 (MuSiQue), 10 (2Wiki)
    The peak-IC reranking observes the model across these shuffles; the counts are arbitrary and not justified by a power analysis.
assumptions (4)
  • domain assumption MuSiQue gold documents and hop decomposition are correct and define the reasoning chain.
    All order and distance experiments rely on the dataset's gold documents and hop indices; if labels were noisy, the forward/backward distinction would be weakened.
  • domain assumption Attention weights can be aggregated across heads and tokens into a document-level IC score that reflects information used by the model.
    Eq. (1)-(3) define a normalization that sums to 1; the paper treats high IC as evidence of use, though attention weights are not strictly causal in model predictions.
  • standard math The prefix mask defined in Eq. (4) correctly converts a causal decoder-only LM into a bidirectional-context LM.
    The mask allows all context tokens to attend to each other while preserving causal generation; this is a standard prefix-LM construction.
  • domain assumption The atomic first-hop question probe reliably identifies whether the model holds the removed information in its parameters.
    Section 5.3 uses this probe to separate cases with and without parametric knowledge; an incorrect probe would confound the 'do LMs guess' interpretation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation." pith.science (2026). https://pith.science/paper/LH22QFPY

@misc{pith2026250511754,
  author       = {Pith},
  title        = {Pith review of: Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LH22QFPY}},
  note         = {Machine review of arXiv:2505.11754}
}
read the original abstract

Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging. When Language Models (LMs) are prompted with multiple search results, they are tasked not only with retrieving relevant information but also employing multi-hop reasoning across the information sources. Although LMs perform well on traditional question-answering tasks, the causal mask can hinder their capacity to reason across complex contexts. In this paper, we explore how LMs respond to multi-hop questions by permuting search results (retrieved documents) under various configurations. Our study reveals interesting findings as follows: 1) Encoder-decoder models, such as the ones in the Flan-T5 family, generally outperform causal decoder-only LMs in MHQA tasks, despite being significantly smaller in size; 2) altering the order of gold documents reveals distinct trends in both Flan T5 models and fine-tuned decoder-only models, with optimal performance observed when the document order aligns with the reasoning chain order; 3) enhancing causal decoder-only models with bi-directional attention by modifying the causal mask can effectively boost their end performance. In addition to the above, we conduct a thorough investigation of the distribution of LM attention weights in the context of MHQA. Our experiments reveal that attention weights tend to peak at higher values when the resulting answer is correct. We leverage this finding to heuristically improve LMs' performance on this task. Our code is publicly available at https://github.com/hwy9855/MultiHopQA-Reasoning.

Figures

Figures reproduced from arXiv: 2505.11754 by the authors.

Figure 1
Figure 1. Context attention distribution of Qwen 2.5 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Context permutation design. Take 3-hop ques [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Acc of Qwen2.5 7B across different gold doc [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: IC distribution across different layers of Qwen2.5 7B with different setups for 2-hop, 3-hop and 4-hop [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qwen 2.5 7B performance (Answer Only) peak IC ranking from higher peak IC to lower peak IC. A clear trend can be observed that higher peak IC among the 20 random shuffles brings higher accuracy. and Flan T5 models. Even when we ask models to only use the given context,…
Figure 6
Figure 6. Figure 6: Position of last hop document when the or [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Statistics of peak IC score of correct and [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: IC distribution across different layers of Qwen2.5 7B with different order of gold document. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: IC distribution across different layers of Qwen2.5 7B with different distances [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: IC distribution across different layers of Qwen2.5 7B with and without first hop document. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Distance results on 2WikiMultihopQA development set. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 6 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    George Arthur Baker, Ankush Raut, Sagi Shaier, Lawrence E Hunter, and Katharina von der Wense. 2024. https://arxiv.org/abs/2412.10079 Lost in the middle, and in-between: Enhancing language models' ability to reason over long contexts in multi-hop qa . Preprint, arXiv:2412.10079

    arXiv 2024

  4. [4]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. https://openreview.net/forum?id=IW1PR7vEBf LLM 2vec: Large language models are secretly powerful text encoders . In First Conference on Language Modeling

    2024

  5. [5]

    Chi, Xuezhi Wang, and Denny Zhou

    Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org

    work page 2024

  6. [6]

    Zhao, Yanping Huang, Andrew M

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro - Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanp...

    2024

  7. [7]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    arXiv 2024

  8. [8]

    Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. https://openreview.net/forum?id=hkujvAPVsg Hippo RAG : Neurobiologically inspired long-term memory for large language models . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

Show all 38 references
  1. [9]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Li...

    2020 doi

  2. [10]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and o...

    2025 doi

  3. [11]

    Wenyu Huang, Guancheng Zhou, Hongru Wang, Pavlos Vougiouklis, Mirella Lapata, and Jeff Z. Pan. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.927 Less is more: Making smaller language models competent subgraph retrievers for multi-hop KGQA . In Findings of the Associati...

    2024 doi

  4. [12]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. https://arxiv.org/abs/2307.10169 Challenges and applications of large language models . Preprint, arXiv:2307.10169

    2023 arXiv

  5. [13]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc...

    2020

  6. [14]

    Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. https://doi.org/10.18653/v1/2023.emnlp-industry.39 Are C hat GPT and GPT -4 general-purpose solvers for financial text analytics? a study on several typical tasks . In Proceeding...

    2023 doi

  7. [15]

    Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. https://aclanthology.org/2024.emnlp-industry.66 Retrieval augmented generation or long-context LLM s? a comprehensive study and hybrid approach . In Proceedings of the 2024 Conference on Empirical ...

    2024

  8. [16]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12...

    2024 doi

  9. [17]

    AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  10. [18]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference ...

    2023

  11. [19]

    Linhao Luo, Yuan-Fang Li, Reza Haf, and Shirui Pan. 2024. https://openreview.net/forum?id=ZGNWW7xZ6Q Reasoning on graphs: Faithful and interpretable large language model reasoning . In The Twelfth International Conference on Learning Representations

    2024

  12. [20]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  13. [21]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft

    2022

  14. [22]

    Fengran Mo, Kelong Mao, Ziliang Zhao, Hongjin Qian, Haonan Chen, Yiruo Cheng, Xiaoxi Li, Yutao Zhu, Zhicheng Dou, and Jian-Yun Nie. 2024. https://arxiv.org/abs/2410.15576 A survey of conversational search . Preprint, arXiv:2410.15576

    2024 arXiv

  15. [23]

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. https://arxiv.org/abs/2402.09906 Generative representational instruction tuning . Preprint, arXiv:2402.09906

    2024 arXiv

  16. [24]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 GPT -4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  17. [25]

    Muhammad Qorib, Geonsik Moon, and Hwee Tou Ng. 2024. https://doi.org/10.18653/v1/2024.findings-acl.967 Are decoder-only language models better than encoder-only language models in understanding word meaning? In Findings of the Association for Computational Linguistics: ACL 202...

    2024 doi

  18. [26]

    Qwen. 2025. https://arxiv.org/abs/2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115

    2025 arXiv

  19. [27]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. https://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res.,...

    2020

  20. [28]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  21. [29]

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2024. https://arxiv.org/abs/2402.15449 Repetition improves language model embeddings . Preprint, arXiv:2402.15449

    2024 arXiv

  22. [30]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2022 doi

  23. [31]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/v1/2023.acl-long.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  24. [32]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  25. [33]

    Pan, and Kam-Fai Wong

    Hongru Wang, Wenyu Huang, Yang Deng, Rui Wang, Zezhong Wang, Yufei Wang, Fei Mi, Jeff Z. Pan, and Kam-Fai Wong. 2024. https://arxiv.org/abs/2401.13256 Unims-rag: A unified multi-source retrieval-augmented generation for personalized dialogue systems . Preprint, arXiv:2401.13256

    2024 arXiv

  26. [34]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  27. [35]

    Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and Shuai Ma. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.871 Re-reading improves reasoning in large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natu...

    2024 doi

  28. [36]

    Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024. https://arxiv.org/abs/2402.18013 A survey on recent advances in llm-based multi-turn dialogue systems . Preprint, arXiv:2402.18013

    2024 arXiv

  29. [37]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. https://arxiv.org/abs/2309.01219 Siren's song in the ai ocean: A survey on hallucination in ...

    2023 arXiv

  30. [38]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/9cb2a7495900f8b602cb10159246a016-Paper-Datasets_and_Benchmarks.pdf Toolqa: A dataset for llm question answering with external tools . In Advances in ...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.