Pith. sign in

REVIEW 5 major objections 5 minor 38 references

SUGAR: Leveraging Contextual Confidence for Smarter Retrieval

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims semantic entropy can decide when retrieval is needed, improving open-domain QA accuracy and efficiency without extra training.

desk verdict Plausible adaptive-RAG idea with a useful within-model ablation, but the main comparison is confounded by generator mismatch and threshold fitting. read the letter →

arxiv 2501.04899 v1 pith:53FIGS3V submitted 2025-01-09 cs.CL cs.AI

Hanna Zubkova , Ji-Hoon Park , Seong-Whan Lee This is my paper · ORCID
classification cs.CLcs.AI
keywords semanticentropyadaptiveretrievalretrieval-augmentedgenerationquestionansweringuncertaintyestimationopen-domainQAselectivelargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a language model's semantic uncertainty, measured as entropy over meaning-equivalent answer clusters, is a reliable signal for deciding when retrieval-augmented generation should be triggered. It introduces SUGAR, a training-free adaptive strategy that skips retrieval when the model is confident, performs one retrieval round at intermediate uncertainty, and performs multiple rounds at high uncertainty. On five open-domain question answering datasets, SUGAR reports higher accuracy than naive retrieval and prior adaptive RAG baselines, while often using fewer retrieval steps. The paper's significance is that uniform retrieval is wasteful and can even hurt answers by injecting noisy context, so a cheap, meaning-aware uncertainty gate could make RAG both more accurate and more efficient.

What carries the argument

The central mechanism is semantic entropy, defined as $SE(x) \approx -|C|^{-1} \sum_{i=1}^{|C|} \log p(C_i \mid x)$, where $C$ is a set of clusters of generated answers that share the same meaning, grouped by bidirectional entailment. Unlike token-level predictive entropy, it treats surface-form variations as one meaning and therefore separates genuine knowledge uncertainty from lexical choice. In SUGAR, this entropy is computed from the generator's own high-temperature samples and compared against thresholds that select among no retrieval, single-step retrieval, and multi-step retrieval, making the retrieval policy a direct function of the model's estimated knowledge boundary.

What would settle it

Run SUGAR's decision rule on a fixed generator (the same model for no-retrieval, single-step, predictive-entropy, and semantic-entropy conditions) over SQuAD and TriviaQA and compare accuracy and retrieval steps; if semantic-entropy gating does not beat predictive-entropy gating at matched thresholds, the claim that meaning-aware uncertainty drives the gains is falsified.

Watch

Extended reading notes

Core claim

The central claim is that semantic entropy, computed by sampling candidate answers and clustering them by bidirectional entailment, can serve as a task-independent controller for adaptive retrieval in open-domain QA. SUGAR first generates answers from the model's parametric knowledge alone; if semantic entropy is below a threshold, it answers directly, if intermediate it triggers single-step retrieval, and if high it triggers multi-step retrieval. The paper reports that this gate outperforms no retrieval, uniform single-step retrieval, and adaptive strategies based on entity popularity, reflection tokens, and question-complexity labels on SQuAD, Natural Questions, TriviaQA, HotpotQA, and 2WikiMultiHopQA, while reducing retrieval steps relative to multi-step IRCoT. The ablation further shows that semantic-entropy-gated retrieval beats predictive-entropy-gated retrieval on TriviaQA and SQuAD, supporting the claim that meaning-aware uncertainty mitigates overconfidence and lexical-form confusions.

Load-bearing premise

The paper's headline comparison uses different base models for SUGAR and for the no-retrieval and single-step baselines; if the base model, not the retrieval decision, is what makes answers more accurate, the central claim collapses.

Editorial extensions

If this is right

  • Adaptive retrieval can be performed without training a classifier or relying on task-specific labels, since the gate is computed from the generator's own outputs.
  • Semantic-entropy gating improves accuracy over both no-retrieval and uniform single-step retrieval on single-hop and multi-hop QA benchmarks, as reported in the paper.
  • The approach reduces retrieval steps relative to uniform multi-step retrieval methods like IRCoT, yielding more efficient inference on multi-hop questions.
  • The gate generalizes across datasets with different complexity levels, because the entropy computation does not depend on annotated complexity labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond QA, the same semantic-entropy gate could be applied to other knowledge-intensive generation tasks such as fact-checking or dialogue grounding, where knowing when to consult external sources is equally important; the paper does not test these settings.
  • Because SUGAR requires sampling multiple answers per query, its per-query overhead could be reduced by reusing samples across related queries or by approximating semantic clusters with a trained entailment model; the paper acknowledges the time cost but does not explore such optimizations.
  • A natural extension is to make the entropy thresholds adaptive per query or per dataset rather than fixed by cross-validation, which could improve robustness when the test distribution shifts away from the calibration set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SUGAR, an adaptive retrieval-augmented generation strategy for open-domain QA that uses semantic entropy to decide whether to retrieve, and if so, whether to perform single-step or multi-step retrieval. The method is training-free and is evaluated on five QA benchmarks against no-retrieval, single-step retrieval, and several adaptive RAG baselines. The authors report accuracy gains on some datasets and a reduction in retrieval steps, while acknowledging an inference-time trade-off. The central empirical claim is that semantic-entropy-guided retrieval improves QA accuracy and efficiency relative to naive and adaptive retrieval baselines.

Significance. The idea of replacing predictive entropy with semantic entropy as the trigger for retrieval is well motivated and, to my knowledge, not previously applied in exactly this way. The within-model ablation in Table III provides suggestive evidence that semantic entropy is a better confidence signal than predictive entropy for the same generator. If the headline comparisons were properly controlled, the method would be a useful and simple addition to adaptive RAG. However, as presented, the main tables do not support the stated claims because the generator is not held constant and the thresholds are tuned on the same benchmarks used for evaluation.

major comments (5)
  1. [III-B and III-C] The main comparison is confounded: Section III-B states that the No Retrieval and Single-step baselines use FLAN-T5-XL, while Section III-C states that SUGAR uses Llama-2-chat (7B) as the generator. Tables I and II therefore vary both the retrieval policy and the base model, so the observed improvements over No Retrieval and Single-step retrieval cannot be attributed to SUGAR's decision rule. The authors should rerun all baselines with Llama-2-chat (7B) or run SUGAR with FLAN-T5-XL, and ideally also report the adaptive baselines on the same generator.
  2. [III-C] The semantic entropy thresholds are determined by cross-validation on the same datasets that are later used for reporting the main results in Tables I and II. This makes the headline numbers fitted values, not predictions, and contradicts the claim in Section I that the approach is "not task- or data-dependent." The paper should report thresholds selected on a held-out validation set or on separate datasets, and show sensitivity to the threshold choice.
  3. [Table II] The claim that SUGAR outperforms naive single-step retrieval is not supported on 2WikiMultiHopQA: SUGAR achieves 39.75 accuracy while Single-step retrieval achieves 42.80, and IRCoT achieves 55.40. On HotpotQA, SUGAR's EM and F1 (38.77 and 49.85) are below those of the Multi-step IRCoT baseline (44.60 and 56.54), so the accuracy-only framing overstates the comparison. The multi-hop results need to be reported and discussed with the same metrics as the baselines.
  4. [III-C] The efficiency claim is not supported by the reported wall-clock times. On SQuAD, SUGAR takes 4.43 relative time versus 1.00 for single-step retrieval, and on TriviaQA it takes 3.13 versus 1.00, despite using fewer retrieval steps on TriviaQA. The paper's own text admits that "inference for SUGAR takes longer than other adaptive methods for single-hop datasets." The efficiency advantage should be stated only in terms of retrieval steps, or the time measurements should include the cost of semantic entropy estimation and be compared on a common basis.
  5. [III-D and Table III] The ablation study is a useful partial control, but it does not repair the main comparison: it covers only TriviaQA and SQuAD, uses thresholds tuned on those datasets, and does not compare SUGAR against Adaptive Retrieval, Self-RAG, or Adaptive-RAG with the same generator. The claim that SUGAR "helps mitigate overconfidence" is not directly tested; the table reports accuracy and steps, not calibration or confidence. The ablation should be extended to all datasets and to the adaptive baselines, and should report significance or variance.
minor comments (5)
  1. [Table I] The row labels in Table I are confusing: "Single-hop Simple No Retrieval" and the "Data Types" column are not explained, and the multi-hop Table II has rows that run together visually.
  2. [II-B, Eq. (1)] Equation (1) uses the notation |C| and C_i without defining whether C is the set of clusters, the number of clusters, or a specific cluster; this should be clarified.
  3. [II-C, References [18]--[22]] References [18]--[22] concern robotics, EEG classification, and computer vision, and appear unrelated to the discussion of LLMs being distracted by noisy retrieved documents; these citations should be removed or replaced with relevant RAG-robustness literature.
  4. [III-C, Figure 2] The cross-validation procedure is not described: the number of folds, the search range for tau, and the exact effectiveness metric used for selection are missing, which prevents reproduction.
  5. [III-A] The "Step" and "Time" metrics are said to be relative to single-step retrieval, but no formula or unit definition is provided; please specify how these values were computed.

Circularity Check

1 steps flagged · score 6.0 of 10

SUGAR's headline results are partly fitted values: thresholds are tuned on the same benchmarks used for evaluation, so the main comparison is not an independent prediction.

  1. fitted input called prediction [Section III-C (Results), Tables I and II]
    "To set the semantic entropy thresholds, we first performed a case study on the datasets to see what levels of semantic entropy the model normally demonstrates when generating answers. We then used cross-validation to determine the thresholds that yield the best performance in terms of effectiveness metrics, we report the case study results in Figure 2."

    SUGAR's entire retrieval policy is determined by semantic entropy thresholds: below the threshold the model answers from parametric knowledge, above it the retriever is triggered, and a higher interval triggers multi-step retrieval. The paper states that these thresholds were set by cross-validation on the datasets to maximize 'effectiveness metrics', and then Tables I and II report SUGAR's accuracy, EM, and F1 on those exact datasets (SQuAD, Natural Questions, TriviaQA, HotpotQA, 2WikiMultiHopQA). The reported 'improvements' are therefore selected on the evaluation benchmarks rather than predicted on held-out data.

full rationale

The paper's main circularity is the threshold-fitting issue: the adaptive retrieval decisions that define SUGAR are controlled by thresholds tuned via cross-validation on the same QA benchmarks on which the headline results are reported. Because there is no stated held-out split or nested evaluation procedure, the 'improvement' over baselines is partly a post-fit optimum. This corresponds to the fitted-input-called-prediction pattern. I did not find a self-citation chain or imported uniqueness theorem: the semantic-entropy estimator is adopted from the cited work of Kuhn et al., and the paper's own citations to the authors' prior robotics/EEG papers are unrelated and not load-bearing. The generator mismatch between SUGAR (Llama-2-chat) and the no-retrieval/single-step baselines (FLAN-T5-XL) is a serious experimental confound, but it is not itself a circularity: it is a failure to hold the generator fixed, not a reduction of the claim to its own inputs. Similarly, the absence of an external benchmark for the thresholds is a statistical validity concern rather than a definitional equivalence. On balance, the central empirical claim is partially circular because the main results depend on parameters tuned on the evaluation sets, so the score is 6 rather than 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on several premises the paper does not independently establish: semantic entropy reliably reflects knowledge gaps, the entailment grouping is accurate, and the cross-validated thresholds transfer to the evaluation datasets. The free parameters are mainly the per-dataset thresholds and the sampling budget.

free parameters (3)
  • Semantic entropy threshold tau (per dataset) = e.g., tau=0.4 for TriviaQA, tau=0.9 for SQuAD (ablation); main results use per-dataset cross-validated thresholds
    The threshold determines when retrieval is triggered and how many steps are used. It is chosen via a case study and cross-validation on the evaluation datasets, so reported results are fitted rather than predicted.
  • Number of sampled answers for semantic entropy = 10
    Semantic entropy is approximated by generating 10 answers, following Kuhn et al. [29]; this hyperparameter is not varied and its sensitivity is not tested.
  • Maximum retrieval steps for multi-step branch = not specified
    The multi-step retrieval branch does not define a maximum step count; the reported average steps (e.g., 2.31 on HotpotQA) depend on an unspecified stopping criterion.
assumptions (3)
  • domain assumption Semantic entropy computed from sampled generations is a reliable indicator of the model's knowledge boundaries.
    The entire method relies on semantic entropy correlating with answer correctness; the paper provides only indirect evidence via the ablation.
  • domain assumption Bidirectional entailment correctly groups answers that share meaning.
    The semantic entropy calculation depends on the entailment model's grouping being accurate; this is assumed from Kuhn et al. without re-validation here.
  • ad hoc to paper Cross-validation on benchmark datasets yields thresholds that generalize to those test sets.
    The paper selects thresholds to maximize effectiveness on the same datasets it evaluates on, which is a fitting procedure rather than a predictive one.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SUGAR: Leveraging Contextual Confidence for Smarter Retrieval." pith.science (2026). https://pith.science/paper/53FIGS3V

@misc{pith2026250104899,
  author       = {Pith},
  title        = {Pith review of: SUGAR: Leveraging Contextual Confidence for Smarter Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53FIGS3V}},
  note         = {Machine review of arXiv:2501.04899}
}
read the original abstract

Bearing in mind the limited parametric knowledge of Large Language Models (LLMs), retrieval-augmented generation (RAG) which supplies them with the relevant external knowledge has served as an approach to mitigate the issue of hallucinations to a certain extent. However, uniformly retrieving supporting context makes response generation source-inefficient, as triggering the retriever is not always necessary, or even inaccurate, when a model gets distracted by noisy retrieved content and produces an unhelpful answer. Motivated by these issues, we introduce Semantic Uncertainty Guided Adaptive Retrieval (SUGAR), where we leverage context-based entropy to actively decide whether to retrieve and to further determine between single-step and multi-step retrieval. Our empirical results show that selective retrieval guided by semantic uncertainty estimation improves the performance across diverse question answering tasks, as well as achieves a more efficient inference.

Figures

Figures reproduced from arXiv: 2501.04899 by the authors.

Figure 1
Figure 1. Overview of the proposed retrieval strategy. Semantic entropy is used to measure how confident the model is to answer the question based on its [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Semantic entropy levels and corresponding accuracy. Gradient [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 29 canonical work pages

  1. [1]

    Language models are few-shot learners,

    T. B. Brown et al. , “Language models are few-shot learners,” in Advances in neural information processing systems , 2020

    work page 2020

  2. [2]

    Gpt-4 technical report,

    OpenAI et al. , “Gpt-4 technical report,” in arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Training language models to follow instructions with human feedback,

    L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Neural Information Processing Systems , 2022

    work page 2022

  4. [4]

    Large language models: A survey,

    S. Minaee et al., “Large language models: A survey,” in arXiv preprint arXiv:2402.06196, 2024

    arXiv 2024

  5. [5]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories,

    A. Mallen, A. Asai, V . Zhong, R. Das, D. Khashabi, and H. Hajishirzi, “When not to trust language models: Investigating effectiveness of parametric and non-parametric memories,” in Annual Meetings of the Association for Computational Linguistics , 2023

    work page 2023

  6. [6]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao et al. , “Retrieval-augmented generation for large language models: A survey,” in arXiv preprint arXiv:2312.10997 , 2023

    arXiv 2023

  7. [7]

    Unsupervised dense information retrieval with con- trastive learning,

    G. Izacard et al. , “Unsupervised dense information retrieval with con- trastive learning,” in Transactions on Machine Learning Research, 2022

    work page 2022

  8. [8]

    Realm: retrieval-augmented language model pre-training,

    K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang, “Realm: retrieval-augmented language model pre-training,” in International Con- ference on Machine Learning , 2020

    work page 2020

Show all 38 references
  1. [9]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” in Conference on Neural Information Processing Systems , 2020

    2020

  2. [10]

    Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity,

    S. Jeong, J. Baek, S. Cho, S. J. Hwang, and J. C. Park, “Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity,” in Annual Conference of the North American Chapter of the Association for Computational Linguistics , 2024

    2024

  3. [11]

    Large language models can be easily distracted by irrelevant context,

    F. Shi et al. , “Large language models can be easily distracted by irrelevant context,” in International Conference on Machine Learning , 2023

    2023

  4. [12]

    Interpretable unified language checking,

    T. Zhang et al. , “Interpretable unified language checking,” in arXiv preprint arXiv:2304.03728, 2023

    2023 arXiv

  5. [13]

    Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence,

    K. Wu, E. Wu, and J. Zou, “Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence,” in arXiv preprint arXiv:2404.10198, 2024

    2024 arXiv

  6. [14]

    Evaluating the external and parametric knowledge fusion of large language models,

    H. Zhang et al. , “Evaluating the external and parametric knowledge fusion of large language models,” in arXiv preprint arXiv:2405.19010 , 2024

    2024 arXiv

  7. [15]

    From rags to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries,

    H. Wadhwa et al., “From rags to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries,” in arXiv preprint arXiv:2406.12824 , 2024

    2024 arXiv

  8. [16]

    Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise,

    G. Hong, J. Kim, J. Kang, S. Myaeng, and J. J. Whang, “Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise,” in North American Chapter of the Association for Computational Linguistics, 2024

    2024

  9. [17]

    Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi- llm collaboration,

    S. Feng, W. Shi, Y . Wang, W. Ding, V . Balachandran, and Y . Tsvetkov, “Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi- llm collaboration,” in Annual Meetings of the Association for Computa- tional Linguistics, 2024

    2024

  10. [18]

    An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions,

    D. Won, K.-R. M ¨uller, and S. Lee, “An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions,” in Science Robotics, 2020

    2020

  11. [19]

    Text extraction in mpeg compressed video for context-based indexing,

    Y . Lim, S. Choi, and S. Lee, “Text extraction in mpeg compressed video for context-based indexing,” in International Conference on Pattern Recognition, 2000

    2000

  12. [20]

    A multi-view cnn with novel variance layer for motor imagery brain computer interface,

    R. Mane, “A multi-view cnn with novel variance layer for motor imagery brain computer interface,” in International Conference of the IEEE Engineering in Medicine and Biology Society , 2020

    2020

  13. [21]

    Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional lstm network using electroencephalography signals,

    J. Jeong, B. Yu, D. Lee, and S. Lee, “Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional lstm network using electroencephalography signals,” in Brain Sciences, 2019

    2019

  14. [22]

    Reconstruction of 3d human body pose from stereo image sequences based on top-down learning,

    H. Yang and S. Lee, “Reconstruction of 3d human body pose from stereo image sequences based on top-down learning,” in Pattern Recognition, 2007

    2007

  15. [23]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection,

    A. Asai, Z. Wu, Y . Wang, A. Sil, and H. Hajishirzi, “Self-rag: Learning to retrieve, generate, and critique through self-reflection,” in Interna- tional Conference on Learning Representations , 2024

    2024

  16. [24]

    The web can be your oyster for improving large language models,

    J. Li, T. Tang, W. X. Zhao, J. Wang, J. Nie, and J. Wen, “The web can be your oyster for improving large language models,” in Annual Meetings of the Association for Computational Linguistics , 2023

    2023

  17. [25]

    Active retrieval augmented generation,

    Z. Jiang et al. , “Active retrieval augmented generation,” in Empirical Methods in Natural Language Processing , 2023

    2023

  18. [26]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi- step questions,

    H. Trivedi, N. Balasubramanian, T. Khot, and Sabharwal, “Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi- step questions,” in Annual Meetings of the Association for Computa- tional Linguistics, 2023

    2023

  19. [27]

    A semantic similarity approach to paraphrase detection,

    S. Fernando and M. Stevenson, “A semantic similarity approach to paraphrase detection,” in Annual Research Colloquium of the UK Special Interest Group for Computational Linguistics , 2008

    2008

  20. [28]

    Surface form competition: Why the highest probability answer isn’t always right,

    A. Holtzman, P. West, V . Schwartz, Y . Choi, and L. Zettlemoyer, “Surface form competition: Why the highest probability answer isn’t always right,” in Empirical Methods in Natural Language Processing , 2021

    2021

  21. [29]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,

    L. Kuhn, Y . Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,” in International Conference on Learning Representations , 2023

    2023

  22. [30]

    Self-knowledge guided retrieval augmentation for large language models,

    Y . Wang, P. Li, M. Sun, and Y . Liu, “Self-knowledge guided retrieval augmentation for large language models,” in Empirical Methods in Natural Language Processing, 2023

    2023

  23. [31]

    Adaptation with self-evaluation to improve selective prediction in llms,

    J. Chen, J. Yoon, S. Ebrahimi, S. O. Arik, T. Pfister, and S. Jha, “Adaptation with self-evaluation to improve selective prediction in llms,” in Empirical Methods in Natural Language Processing , 2023

    2023

  24. [32]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” in Empirical Methods in Natural Language Processing, 2016

    2016

  25. [33]

    Natural questions: A benchmark for question an- swering research,

    T. Kwiatkowski et al., “Natural questions: A benchmark for question an- swering research,” in Transactions of the Association for Computational Linguistics, 2019

    2019

  26. [34]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,” in Annual Meeting of the Association for Computational Linguistics , 2017

    2017

  27. [35]

    Hotpotqa: A dataset for diverse, explainable multi- hop question answering,

    Z. Yang et al. , “Hotpotqa: A dataset for diverse, explainable multi- hop question answering,” in Empirical Methods in Natural Language Processing, 2018

    2018

  28. [36]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps,

    X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa, “Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps,” in International Conference on Computational Linguistics , 2020

    2020

  29. [37]

    Scaling instruction-finetuned language models,

    H. W. Chung et al., “Scaling instruction-finetuned language models,” in arXiv preprint arXiv:2210.11416 , 2022

    2022 arXiv

  30. [38]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron et al. , “Llama 2: Open foundation and fine-tuned chat models,” in arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.