Pith. sign in

REVIEW 3 major objections 4 minor 55 references

SGIC: A Self-Guided Iterative Calibration Framework for RAG

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Uncertainty scores from a model's own token probabilities, fed back into the prompt over multiple rounds, let retrieval-augmented language models calibrate their answers and improve exact-match accuracy on HotpotQA, Natural Questions, and…

desk verdict Solid RAG calibration framework with a likely sign typo in the answer-uncertainty equation that, as written, makes the final-answer selection self-defeating. read the letter →

arxiv 2506.16172 v1 pith:ILUAO4CS submitted 2025-06-19 cs.CL

Guanhua Chen , Yutong Yao , Lidia S. Chao , Xuebo Liu , Derek F. Wong This is my paper
classification cs.CL
keywords retrieval-augmentedgenerationself-calibrationuncertaintyestimationiterativepromptingquestionansweringlargelanguagemodelsin-contextreasoningsupervisedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a retrieval-augmented language model can calibrate its own answers by being shown, in the prompt, how confident it was in each retrieved document and in each of its previous answers. The authors build a self-guided iterative calibration loop: generate an answer, score it with the product of per-token maximum probabilities, re-ask the question with those scores attached, and repeat up to five rounds, finally choosing the answer with the lowest uncertainty. They report consistent gains over plain repeated sampling on HotpotQA and Natural Questions for both closed-source and open-weight models, and show that fine-tuning open-weight models on a self-generated calibration set transfers the ability across datasets and to GSM8K math word problems. The practical stake is that RAG accuracy can be improved without external rerankers or retrieval-specific training.

What carries the argument

The load-bearing object is the uncertainty score, defined as the product of the per-token maximum softmax probabilities of a generated sequence, $s_{\mathrm{ans}} = \prod_{i=1}^m p_i$, accompanied by per-dataset normalization (subtracting the dataset mean and rescaling to a 0–100 range, Eqs. 2–3). The same construction, with the sign flipped, scores each retrieved document by asking the model to answer the question with only that document and measuring how confident it is. These two scores do triple duty: they tag documents and previous answers inside the calibration prompt, they select which of K rounds' answers to return as final, and they label the self-generated training samples used for supervised fine-tuning. The iterative prompt format, which accumulates “Previous Generated Answer” entries with their scores, is the mechanism that lets the model reason about its own past outputs.

What would settle it

Before running the calibration loop on a new QA dataset, check whether the model's self-reported confidence score separates correct from incorrect answers at better than chance; if it does not, SGIC's gains over K-times sampling should vanish. A simpler check: compare the minimal-uncertainty round's answer with a randomly chosen round's answer; if their accuracy is equal, the selection mechanism is doing no work.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that the gap between uncertainty scores of correct and incorrect answers and of relevant and irrelevant documents is large enough to serve as a usable calibration signal. Concretely, the score $s_{\mathrm{ans}} = \prod_{i=1}^m p_i$ (the product of maximum token probabilities, normalized by dataset mean and min/max) ranks answer correctness, and a companion document score ranks which retrieved passages matter. When these scores are appended to the prompt, each document tagged with its relevance score and each prior answer tagged with its confidence, the model's in-context reasoning revises a wrong or partial answer toward the correct one, and the revision becomes more reliable over rounds. The authors select the round with the minimal uncertainty score as the final answer, and they report that this selection rule is the difference between their method and K-times sampling. The paper also claims that fine-tuning on an iterative self-calibration training set, built by keeping samples that calibrate to the correct answer within k rounds, teaches open-weight models to exploit the uncertainty cues, and that the resulting calibration skill transfers across datasets and to a math-reasoning benchmark.

Load-bearing premise

The whole method rests on the assumption that the model's self-reported confidence—how sure it sounds, token by token—correctly ranks which documents matter and which answers are right; if that ranking fails, the prompt cues and the final-answer selection both lose their value.

Editorial extensions

If this is right

  • Adding document and answer uncertainty scores to the prompt improves answer accuracy more than K-times sampling with the same number of generations.
  • Selecting the response with minimal uncertainty across calibration rounds is what turns iterative recalibration into a gain; the ablation shows that calibration alone, calibration with answer uncertainty, and calibration with document uncertainty each add to the final score.
  • Fine-tuning on the self-generated iterative calibration set lets open-weight models (Phi-3.5-mini and Llama2-7B-Chat) outperform their tuned baselines on HotpotQA and NQ, and the effect transfers cross-dataset.
  • The same recipe lifts exact-match accuracy on GSM8K when reasoning steps are treated as pseudo-documents, indicating the mechanism is not RAG-specific.
  • Oracle-perfect uncertainty scores raise HotpotQA EM from 77.2 to 85.7, so the framework's ceiling is set by the uncertainty estimator.
  • # The uncertainly*
  • # The uncertainly*

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: because the oracle-uncertainty run reaches 85.7 EM versus 77.2 for the real scores, the uncertainty estimator is the binding constraint, so any better-calibrated confidence measure—semantic entropy, ensembles, or a learned verifier—should directly extend SGIC's gains.
  • A testable extension: the minimal-uncertainty selection rule is a cheap alternative to self-consistency voting, and the two could be combined by running multiple calibration chains and choosing the lowest-uncertainty answer across all chains.
  • A connection worth probing: since the document scores are computed with the same generator that answers, SGIC couples retrieval guidance to the generator's own competence; a natural stress test is to apply it with a deliberately weak retriever and check whether the uncertainty tags still rescue accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SGIC, a Self-Guided Iterative Calibration framework for retrieval-augmented generation (RAG). It computes uncertainty scores for retrieved documents and for LLM-generated answers from token-level maximum softmax probabilities, then iteratively feeds these scores, together with previously generated answers, back into the model for up to K rounds of calibration. A fine-tuning strategy is also introduced, in which the training inputs are reformatted to include document and answer uncertainty scores. The method is evaluated on HotpotQA, Natural Questions, and GSM8K using GPT-4o, GPT-4o-mini, Phi-3.5-mini, and Llama2-7B-Chat, reporting consistent improvements over baselines in EM and F1, along with ablations, an oracle study, transfer experiments, and attention analyses.

Significance. If validated, SGIC would offer a simple, model-agnostic method for improving RAG accuracy through prompt-level uncertainty cues and iterative self-calibration, with potential applicability to open-weight models via fine-tuning. The paper's strengths include the oracle experiment in Table 6, which demonstrates clear headroom for the approach, the ablation study in Table 5, and the cross-dataset transfer results in Table 8. The method does not require an external verifier or additional agents, which is practically appealing. However, the central uncertainty-score definition is internally inconsistent as written, and the experimental protocol has several underspecified aspects; these issues currently prevent the reported gains from being interpreted unambiguously.

major comments (3)
  1. [§3.1 (Eq. 1–2), §4.1] The answer uncertainty score in Eq. (1) is the product of the maximum token probabilities, which is a standard confidence score: higher values indicate a more confident model. Eq. (2) is meant to normalize this score, but as written its piecewise condition is `if sans < sans` (a self-referential, always-false statement) and both branches are identical, so the expression reduces to `(sans - sans)/(1 - sans) = 0` and the normalized score is constant. Even if one interprets the intended formula as `(sans - mean)/(1 - mean)`, that transform is monotone increasing in the raw product, so minimizing the normalized score is equivalent to minimizing the confidence product, i.e., selecting the least confident answer. This contradicts the reported improvements and the authors' own discussion in §5.3, where correct answers are associated with lower uncertainty. The definition must be corrected—likely by using `1 - product` for answers, as done for documents in Eq. (3)—and the exact formula used in the experiments must be stated. This is load-bearing because the score serves both as the prompt cue in §3.2 and as the final-answer selection rule in §4.1.
  2. [§3.1 (Eq. 2–3)] The normalization in Eqs. (2)–(3) uses the average uncertainty score (and the min/max for documents) computed "seperately" on the training, validation, or test set. Using the test set's own aggregate statistics during evaluation is a transductive form of test-set leakage. While a purely monotone min-max transform preserves ranking, the intended transform in Eq. (2) is not clearly monotone, and the paper does not indicate whether the comparison baselines are given equivalent access to test-set statistics. Please clarify the protocol and either compute normalization constants from the training set only or justify why test-set statistics do not inflate the reported gains.
  3. [§4.1 (Table 2, Figure 3)] The baseline "sampling K times" is not fully specified: the paper does not state how the final answer is selected among the K samples (e.g., majority vote, last sample, or minimal uncertainty). The comparison in Table 2 and Figure 3 is therefore ambiguous, as a different aggregation rule could account for part of the observed gap. Please define the baseline's selection rule explicitly and, for the closed-source models, describe the prompting and decoding settings used for sampling.
minor comments (4)
  1. [Throughout] There are several typos and stylistic issues: "seperately" should be "separately", "close-source" should be "closed-source", and "docuemtns" appears in Table 11. The caption of Table 2 is ungrammatical ("which sampling K times following the iterative calibration process").
  2. [Figure 3] The legend labels are concatenated without separation, making it difficult to associate curves with models; please reformat the legend.
  3. [§5.6] The sentence "In the baseline, error samples with low uncertainty are typically more likely to be calibrated successfully" appears to refer to the proposed model rather than the baseline, since the baseline does not use uncertainty in the described setting; please clarify.
  4. [Table 10] The metric R10@k is defined only in the text; please make the definition explicit in the caption, and clarify whether it is computed over the two gold documents.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SGIC is validated on held-out benchmarks, and its uncertainty signal is ablated against external and oracle scorers rather than being equivalent to its own output.

full rationale

SGIC's central claim is not a derivation from its own inputs. The reported EM/F1 gains on HotpotQA, NQ, and GSM8K are measured on held-out dev/test splits against independently trained or API-based baselines, so the headline result is externally falsifiable rather than constructed. The uncertainty score is used both as a prompt cue and as a final-answer selector, but that coupling is an empirical design choice: the model must still generate improved answers, and the selector only ranks those candidates. The paper also tests this dependency by replacing document uncertainty with an external relevance scorer and with oracle scores in Table 6, which shows that the mechanism is not vacuously forced. The self-citations present (Liu et al. 2024b as inspiration for uncertainty-based prompting; Chen et al. 2025 for LoRA in the appendix) are prior work and implementation details, not load-bearing uniqueness arguments or unverified premises. The training-set filtering in Section 3.3 is ordinary self-training: correct-answer status is used to construct training data, while evaluation remains on unseen samples. The Limitation section appropriately notes dependence on uncertainty-estimation precision; that is a robustness caveat, not a circular step. I note, without scoring it as circularity, that Eq. (2) has identical branches and that the product-of-maximum-token-probabilities is a confidence measure, so the literal 'minimal uncertainty' selection rule is internally inconsistent as written, and the normalization in Eq. (2) uses the evaluation set's own mean. These are correctness/leakage concerns, not reductions of the claimed predictions to their inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework's load-bearing premises are the reliability of the uncertainty score, the legitimacy of dataset-level normalization, and the transfer of filtered self-calibration training to inference. There are no fitted parameters in the classical sense, but K, truncation length, and training hyperparameters are chosen by hand, and the training set is filtered using ground-truth correctness. No new physical or conceptual entities are introduced.

free parameters (3)
  • Maximum calibration rounds K (training round limit k) = 5 for QA experiments; one calibration round for GSM8K result
    Chosen by hand. Figure 3 shows EM rises with K, so the magnitude of the reported gains depends on this choice.
  • Document truncation length = 200 tokens
    Open-weight experiments truncate each document to 200 tokens (Appendix A.1); this limits the evidence available and was not swept.
  • Fine-tuning settings (learning rate, epochs, batch size) = 5e-5, 3 epochs, batch size 16
    Reported in Appendix A.1 as fixed settings for all fine-tuned models; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Product-of-maximum-probabilities uncertainty is a valid ranking signal for document relevance and answer correctness.
    The framework's prompt cues and final-answer selection depend on it. Validated only on 2,000 HotpotQA samples (Figure 1) and one AUROC comparison (Appendix A.5, 68.4 vs 65.5).
  • ad hoc to paper Using evaluation-set statistics to normalize uncertainty scores is acceptable.
    Equations 2 and 3 normalize with mean and min/max from the training, validation, or test set separately. At test time this injects distributional information from the test set and is not justified.
  • domain assumption Fine-tuning on successfully calibrated, correctness-filtered samples teaches general calibration behavior.
    Section 3.3 removes samples whose answers remain incorrect after k rounds; this assumes the remaining subset does not bias the model toward easy cases.
  • domain assumption The Probability Ranking Principle (Robertson 1977) applies to uncertainty-guided scoring in this setting.
    Appendix A.4 invokes PRP as the theoretical basis, but the paper does not derive that uncertainty scores satisfy PRP conditions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SGIC: A Self-Guided Iterative Calibration Framework for RAG." pith.science (2026). https://pith.science/paper/ILUAO4CS

@misc{pith2026250616172,
  author       = {Pith},
  title        = {Pith review of: SGIC: A Self-Guided Iterative Calibration Framework for RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ILUAO4CS}},
  note         = {Machine review of arXiv:2506.16172}
}
read the original abstract

Recent research in retrieval-augmented generation (RAG) has concentrated on retrieving useful information from candidate documents. However, numerous methodologies frequently neglect the calibration capabilities of large language models (LLMs), which capitalize on their robust in-context reasoning prowess. This work illustrates that providing LLMs with specific cues substantially improves their calibration efficacy, especially in multi-round calibrations. We present a new SGIC: Self-Guided Iterative Calibration Framework that employs uncertainty scores as a tool. Initially, this framework calculates uncertainty scores to determine both the relevance of each document to the query and the confidence level in the responses produced by the LLMs. Subsequently, it reevaluates these scores iteratively, amalgamating them with prior responses to refine calibration. Furthermore, we introduce an innovative approach for constructing an iterative self-calibration training set, which optimizes LLMs to efficiently harness uncertainty scores for capturing critical information and enhancing response accuracy. Our proposed framework significantly improves performance on both closed-source and open-weight LLMs.

Figures

Figures reproduced from arXiv: 2506.16172 by the authors.

Figure 1
Figure 1. The uncertainty score of the relevant/irrelevant documents and correct/wrong generated answers on 2,000 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of our framework. or utilizing LLMs with extremely large parameters, such as ChatGPT, for many rounds of interaction. In contrast, our approach requires fine-tuning the LLMs only once, enabling them to perform com￾prehensive reasoning and self-calibration based on retrieved documents, previously generated answers, and their associated uncertainty scores. 3 Method [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The EM scores of the Llama2-7B-Chat fine [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The uncertainty scores of each round of cali [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The answer correctness and uncertainty scores [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The visualization of the attention distribution [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The experimental results of using the “<KEY>” tag to guide LLM on the RAG task at differ￾ent order of documents. The solid line is the result of adding the “<KEY>” tag, while the dashed line is the baseline result. with full-parameters mode because of the limitation of…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 12 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    u rek, Ekin Aky \

    Afra Feyza Aky \"u rek, Ekin Aky \"u rek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.08844

    arXiv 2023

  3. [3]

    Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2022. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260

    arXiv 2022

  4. [4]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511

    arXiv 2023

  5. [5]

    Chao, Feng Wan, and Derek F

    Guanhua Chen, Yutong Yao, Ci - Jun Gao, Lidia S. Chao, Feng Wan, and Derek F. Wong. 2025. https://doi.org/10.48550/ARXIV.2503.23360 Not all lora parameters are essential: Insights on inference necessity . CoRR, abs/2503.23360

    arXiv doi:10.48550/arxiv.2503.23360 2025

  6. [6]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754--17762

    work page 2024

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495

    arXiv 2023

Show all 55 references
  1. [9]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325

    2023 arXiv

  2. [10]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2022

  3. [11]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:2307.01379

    2023 arXiv

  4. [12]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,...

    2024

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  6. [14]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representations

    2024

  7. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  8. [16]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv preprint arXiv:2403.14403

    2024 arXiv

  9. [17]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983

    2023 arXiv

  10. [18]

    Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. Few-shot reranking for multi-hop qa via language model prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15882--15897

    2023

  11. [19]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  12. [20]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  13. [21]

    Ming Li, Jiuhai Chen, Lichang Chen, and Tianyi Zhou. 2024 a . Can llms speak for diverse people? tuning llms via debate to generate controllable controversial statements. arXiv preprint arXiv:2402.10614

    2024 arXiv

  14. [22]

    Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024 b . Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. arXiv preprint arXiv:2403.09972

    2024 arXiv

  15. [23]

    Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. Deep reinforcement learning with distributional semantic rewards for abstractive summarization. arXiv preprint arXiv:1909.00141

    2019 arXiv

  16. [24]

    Yuanzhi Li, S \'e bastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463

    2023 arXiv

  17. [25]

    Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuanjing Huang, and Zhongyu Wei. 2024. Debatrix: Multi-dimensinal debate judge with iterative chronological analysis based on llm. arXiv preprint arXiv:2403.08010

    2024 arXiv

  18. [26]

    Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang

    Liangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang. 2024 a . Selectit: Selective instruction tuning for llms via uncertainty-aware self-reflection. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Infor...

    2024

  19. [27]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172

    2023 arXiv

  20. [28]

    Wong, Lidia S

    Shudong Liu, Zhaocong Li, Xuebo Liu, Runzhe Zhan, Derek F. Wong, Lidia S. Chao, and Min Zhang. 2024 b . https://aclanthology.org/2024.emnlp-main.1205 Can llms learn uncertainty on their own? expressing uncertainty effectively in A self-training manner . In Proceedings of the 2...

    2024

  21. [29]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023 a . Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283

    2023 arXiv

  22. [30]

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023 b . Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156

    2023 arXiv

  23. [31]

    Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023 c . Large language model is not a good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08559

    2023 arXiv

  24. [32]

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188

    2023 arXiv

  25. [33]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813

    2023 arXiv

  26. [34]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  27. [35]

    Stephen E Robertson. 1977. The probability ranking principle in ir. Journal of documentation, 33(4):294--304

    1977

  28. [36]

    Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366

    2023 arXiv

  29. [37]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542

    2023 arXiv

  30. [38]

    Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2023. Found in the middle: Permutation self-consistency improves listwise ranking in large language models. arXiv preprint arXiv:2310.07712

    2023 arXiv

  31. [39]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  32. [40]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509

    2022 arXiv

  33. [41]

    Inigo Jauregi Unanue, Jacob Parnell, and Massimo Piccardi. 2021. Berttune: Fine-tuning neural machine translation with bertscore. arXiv preprint arXiv:2106.02208

    2021 arXiv

  34. [42]

    Jinyuan Wang, Junlong Li, and Hai Zhao. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.179 Self-prompted chain-of-thought on large language models for open-domain multi-hop reasoning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 27...

    2023 doi

  35. [43]

    Liang Wang, Nan Yang, and Furu Wei. 2023 b . Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678

    2023 arXiv

  36. [44]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022 a . Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  37. [45]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022 b . Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  38. [46]

    Qingyang Wu, Lei Li, and Zhou Yu. 2021. Textgail: Generative adversarial imitation learning for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14067--14075

    2021

  39. [47]

    Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I Wang, Wen-tau Yih, and Ziyu Yao. 2023. Learning to simulate natural language feedback for interactive semantic parsing. arXiv preprint arXiv:2305.08195

    2023 arXiv

  40. [48]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  41. [49]

    Shuyang Yu, Runxue Bao, Parminder Bhatia, Taha Kass-Hout, Jiayu Zhou, and Cao Xiao. 2024. Dynamic uncertainty ranking: Enhancing in-context learning for long-tail knowledge in llms. arXiv preprint arXiv:2410.23605

    2024 arXiv

  42. [50]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476--15488

    2022

  43. [51]

    Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. 2024. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131

    2024 arXiv

  44. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685

    2023 arXiv

  45. [53]

    Noah Ziems, Wenhao Yu, Zhihan Zhang, and Meng Jiang. 2023. Large language models are built-in autoregressive search engines. arXiv preprint arXiv:2305.09612

    2023 arXiv

  46. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.