Pith. sign in

REVIEW 6 major objections 6 minor 40 references

MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness

T0 review · 6 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Separating answer learning from confidence learning during fine-tuning gives large language models more reliable per-answer certainty in multi-problem prompts.

desk verdict A plausible two-stage fine-tuning recipe for multi-problem confidence, undermined by missing error bars, an inconsistent headline claim, and an untested train/serve mismatch. read the letter →

arxiv 2504.21773 v4 pith:Z4Y3T3MP submitted 2025-04-30 cs.CL cs.AI

Junsheng Huang , Zhitao He , Yucheng Huang , Sandeep Polisetty , Qingyun Wang , Yi.R Fung This is my paper · ORCID
classification cs.CLcs.AI
keywords multi-problemsettingconfidencecalibrationknowledgeboundarytwo-stagefine-tuninghallucinationmitigationaverageprecisioninstructiontuninglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets the multi-problem setting, in which a single prompt asks an LLM to answer several distinct sub-questions at once. Its central claim is that confidence calibration in this setting improves when answer learning and confidence learning are separated into two fine-tuning stages rather than taught together. The proposed MAC-Tuning first fine-tunes on multi-question answer pairs only, then fine-tunes again to output a per-answer "I am sure" or "I am unsure" judgment, with confidence labels derived from whether the base model could answer each question correctly before tuning. Across six datasets and several base models, the method reports better average precision (how well the model's confidence ranks correct answers above wrong ones) and lower expected calibration error than answer-only, joint, and single-problem baselines, with gains up to 25% in AP and a human-verified gap between sure and unsure answers. A reader should care because batched or multi-query prompting is increasingly common for efficiency and cost, and per-answer confidence is what would let users trust or discard individual responses.

What carries the argument

The load-bearing object is the two-stage training schedule named Multiple Answers and Confidence Stepwise Tuning (MAC-Tuning). In stage one the model is fine-tuned on multi-question input/output pairs ($D_{\mathrm{MultiQA}}$), maximizing $\log P(A \mid Q; \Theta_0)$; in stage two it is fine-tuned on confidence pairs ($D_{\mathrm{MultiQA,C}}$) in which the input already contains the question set and the answers, maximizing $\log P(C \mid Q, A; \Theta_1)$. The confidence labels are generated before training by running the base model on the composed multi-problem prompts and comparing each answer to the ground truth, which the paper treats as locating the model's knowledge boundary. This separation is what carries the claim: the answer stage teaches the model to produce content in the multi-problem format, and the confidence stage teaches it to map a question-answer pair to a calibrated certainty judgment.

What would settle it

Take a MAC-Tuning checkpoint and evaluate confidence twice on the same multi-question test prompts: once with the ground-truth answers inserted into the confidence prompt template and once with the model's own sampled answers. If the expected calibration error is substantially higher, or the ranking of "sure" versus "unsure" answers is much closer to random, in the generated-answer condition, the method's confidence mapping does not carry over to the actual inference distribution.

Watch

Extended reading notes

Core claim

On the paper's own terms, MAC-Tuning establishes that a language model can learn to solve multiple problems in one prompt and, separately, learn to say whether each answer is trustworthy, and that the separation is what makes the confidence signal work. The method is a two-step supervised fine-tuning recipe: step one maximizes the probability of the multi-answer sequence given the multi-question input; step two, with the step-one model as a starting point, maximizes the probability of the per-answer confidence sequence given the same questions and the already-generated answers. Confidence labels are assigned automatically by comparing the base model's outputs with ground-truth answers before any fine-tuning, so "I am sure" marks questions the model could already answer and "I am unsure" marks questions it could not. The paper reports that this recipe beats answer-only fine-tuning, joint answer-and-confidence fine-tuning, and single-problem fine-tuning on all tested datasets, with the largest advantage over the jointly trained baseline, and that the effect persists across different backbone models and model sizes.

Load-bearing premise

MAC-Tuning assumes that confidence judgments learned on prompts that include the ground-truth answer, with labels based on whether the base model answered correctly before fine-tuning, transfer to prompts filled with the model's own generated answers at inference.

Editorial extensions

If this is right

  • On all six datasets (CoQA, ParaRel, GSM, MMLU, MTI-Bench, SQA), MAC-Tuning reports the best average precision and the lowest ECE among the tested baselines, with up to a 25% AP gain over the joint-training baseline and an average 23.7% accuracy gain over the base model.
  • Separate stages beat joint training: the Merge-AC baseline that learns answers and confidence in one pass is consistently worse, so the central claim is not merely that confidence language helps but that the stepwise separation matters.
  • A model fine-tuned on three-question prompts retains or improves single-question performance on easier datasets and transfers to five-question prompts, suggesting the multi-problem behavior is not just memorization of a fixed input length.
  • The confidence signal survives out-of-domain transfer: a model fine-tuned on SQA still improves AP and ECE over the base model on CoQA, ParaRel, MMLU, and MTI-Bench.
  • Human judges who rated outputs without seeing confidence labels confirmed the signal is real: answers marked "I am sure" were factually accurate 89.2% of the time versus 41.2% for "I am unsure", a 48-point gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The confidence stage is trained on prompts that already contain the correct answer, while inference fills the same template with the model's own generated answers; a direct extension would retrain the confidence stage on generated answers, or a mix of gold and generated answers, and measure whether the calibration gain survives that shift.
  • Because the confidence labels come from the base model's own correctness before fine-tuning, MAC-Tuning can be read as teaching the model to reproduce an external judge's verdict about itself; the same two-stage recipe could be adapted to other self-assessment tasks, such as abstention decisions or retrieval confidence, by changing what the second stage predicts.
  • The out-of-domain transfer results hint that part of what is learned is format-level, namely knowing when to emit per-answer "sure" or "unsure" tokens in a multi-answer response, rather than dataset-specific content. A testable extension would train only on easy independent questions and check whether the calibration transfers to hard sequential questions with no additional data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces MAC-Tuning, a two-stage supervised fine-tuning method for confidence estimation in multi-problem QA, where a single prompt contains several questions. Stage 1 fine-tunes the base model on multi-question answer pairs; stage 2 fine-tunes the resulting model to output per-question "I am sure"/"I am unsure" labels, with labels derived from whether the base model answered each question correctly. Experiments compare MAC-Tuning with QA-Only, Single-QA, and Merge-AC baselines on CoQA, ParaRel, GSM, MMLU, MTI-Bench, and SQA using LLaMA3-8B, Qwen2-7B, Llama-3.2-3B, and Phi-3.5-mini, reporting AP, ECE, and accuracy. The authors report large AP gains and low ECE, plus a human evaluation on 100 ParaRel examples.

Significance. If the reported gains hold, the method is a simple and practical recipe for obtaining per-question confidence signals in multi-problem prompts, a setting of growing practical importance. The paper is among the first to address confidence calibration specifically in multi-problem reasoning, and the breadth of base models and datasets is a strength. The human evaluation is a valuable complement to automatic metrics. However, the central quantitative claims currently rest on several unverified assumptions and missing supporting tables; the core idea is plausible and testable, so the paper warrants major revision rather than rejection. The release of code is promised but not yet accessible.

major comments (6)
  1. [Section 2.1, Eq. (2), Appendix A.3] The confidence stage is trained on prompts in which the Answer field is the ground-truth answer (Appendix A.3) and labels are assigned from the base model's correctness (Section 2.1, Step 2), but at inference the same prompt is filled with the step-1 model's own generated answers, and AP/ECE are computed against those generations. The paper reports accuracy gains of up to 45.8% after step 1, so the base model's knowledge boundary is not the boundary of the model being evaluated. Because neither the distribution shift from gold to generated answers nor the shift in the underlying correctness boundary is tested, the reported calibration numbers may reflect the training distribution rather than the deployed model's confidence. Please add a control experiment that either (a) trains the confidence stage on generated answers with labels re-derived from the step-1 model, or (b) evaluates the current model on gold-answer prompts, and report AP/ECE for both conditions.
  2. [Abstract vs. Section 3.4] The abstract claims 'up to 25%' AP gain over baselines, but Section 3.4 first states 'up to a 15% improvement' and later states 'up to 25% ... compared with Merge-AC'. These are different claims: 25% is relative only to the Merge-AC ablation, not to the best baseline, and 15% appears to be an absolute percentage-point gain over the base model on CoQA. Please state precisely which comparison supports the headline and ensure the abstract and main text are consistent.
  3. [Section 3.4 and Section 3.6] Accuracy is claimed to improve by up to 45.8% (average 23.7%) in Section 3.4 and by at least 10.0% (average 26.8%) in Section 3.6, but no table reports accuracy for the main experiments; Figure 3 shows only selected n-sweep results. Additionally, no standard errors or multiple seeds are reported anywhere, so it is impossible to tell whether the AP/ECE differences in Table 1 (e.g., CoQA AP 69.8 vs. 67.4) are significant. Please add an accuracy table and variance estimates.
  4. [Table 3] The out-of-domain claim that 'MAC-Tuning still outperforms the base model' is not supported by the table, which contains only MAC-Tuning rows. Please include the vanilla model's AP/ECE/accuracy on the same OOD evaluation, or state explicitly that the comparison is against the in-domain baselines in Table 1.
  5. [Table 7] On SQA, Phi-3.5-mini's AP drops from 96.6 (Vanilla) to 52.3 (MAC-Tuning), a large regression that contradicts the text's claim of consistent AP gains and is not discussed. Please explain this result or qualify the claim.
  6. [Appendix A.5] The computation of AP and ECE depends on a 'certain prediction probability' that is described only as 'the weighted average of certain prediction probability and uncertain prediction probability.' Without a precise procedure for converting the textual 'I am sure'/'I am unsure' outputs into probabilities, the headline metrics are not reproducible. Please specify token-level probability extraction or an equivalent operationalization.
minor comments (6)
  1. [Table 6] In Table 6, MAC-Tuning's ECE is identically 33.5 for all four datasets; this is likely a copy/paste error and should be corrected.
  2. [Section 3.10] The human evaluation uses 100 examples and a single annotator; please report the selection procedure, confidence intervals, and ideally inter-annotator agreement, and frame the result as supporting rather than definitive.
  3. [Appendix A.6] The implementation section does not state whether the two stages use separate training epochs or whether the same hyperparameters are used for all datasets; please clarify.
  4. [References] Cheng et al. (2023a) and (2023b) appear to be the same technical report; please merge or disambiguate the references.
  5. [General] The phrase 'we release our code and resource at MAC-Tuning' does not include a URL; please provide an accessible repository link.
  6. [Section 2.1] The term 'knowledge boundary' is used for a behavioral correctness proxy; please define it explicitly to avoid conflating parametric knowledge with supervised accuracy on the training set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the method is supervised fine-tuning against external ground-truth labels with held-out evaluation.

full rationale

MAC-Tuning is a supervised fine-tuning recipe, not a derivation. Confidence labels are assigned in Section 2.1 by comparing base-model outputs to ground-truth answers ('I am sure' if the output aligns with ground-truth answer; 'I am unsure' elsewise), which is an external behavioral signal. The two objectives (Eq. 1 and Eq. 2) are standard maximum-likelihood training losses on answer and confidence data; neither objective is defined in terms of the evaluation metric. Evaluation (Section 3.2) compares post-fine-tuning generations to held-out ground truth and computes AP and ECE, so the reported gains are not forced by construction. The only related concern is a train/serve distribution shift: confidence training prompts use ground-truth answers while deployment uses generated answers, and confidence labels describe the base model's boundary before answer fine-tuning. That is a validity and robustness limitation, not a circular reduction: no equation equals its own input and no fitted parameter is renamed as a prediction. Citations to prior work (Zhang et al. 2024; He et al. 2025c) are motivational or dataset sourcing, not load-bearing uniqueness claims. Therefore no circularity step meets the quoted-reduction bar.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper's empirical claim rests on behavioral correctness labels, on the transfer of confidence learned from ground-truth answers to model-generated answers, and on LoRA fine-tuning stability. No new mathematical entities or fitted constants are introduced.

assumptions (2)
  • domain assumption Correctness of the base model on training instances is a valid operational definition of the knowledge boundary.
    Section 2.1 Step 2 labels "I am sure" when the output matches ground truth and "I am unsure" otherwise; no internal model signal is measured, so the method assumes behavior-based labels capture knowledge boundaries.
  • domain assumption Confidence learned on prompts containing the ground-truth answer transfers to prompts containing the model's generated answer.
    The training template (Appendix A.3) inserts the correct answer, while inference inserts the model-generated answer; the paper provides no experiment checking this transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness." pith.science (2026). https://pith.science/paper/Z4Y3T3MP

@misc{pith2026250421773,
  author       = {Pith},
  title        = {Pith review of: MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4Y3T3MP}},
  note         = {Machine review of arXiv:2504.21773}
}
read the original abstract

The hallucination of non-existent facts by LLMs is an important problem given its widespread adoption across various applications. Previous research addresses this problem by analyzing the internal parameterized knowledge boundaries to estimate confidence. However, these studies focus on the single-problem setting and have not explored the more challenging multi-problem setting, which requires accurately answering multiple questions simultaneously. We introduce a novel method for the multi-problem setting, Multiple Answers and Confidence Stepwise Tuning (MAC-Tuning), that separates the learning of answer prediction and confidence estimation during fine-tuning on instruction data. Extensive experiments demonstrate that our method outperforms baselines by up to 25\% in average precision.

Figures

Figures reproduced from arXiv: 2504.21773 by the authors.

Figure 1
Figure 1. An illustration of the multi-problem setting. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We first construct the Multi-Problem dataset, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Accuracy for combining different number ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The full case of examples in introduction in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Specific cases for MAC-Tuning under the multiple problem setting with question number [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: A specific case to show how baseline methods are doing the fine-tuning. The answers are derived from the [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Certainty distribution of the training set under multi-problem setting with [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 4 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    Lida Chen, Zujie Liang, Xintao Wang, Jiaqing Liang, Yanghua Xiao, Feng Wei, Jinglei Chen, Zhenghong Hao, Bing Han, and Wei Wang. 2024. http://arxiv.org/abs/2406.10881 Teaching large language models to express knowledge boundary from their own signals

    arXiv 2024

  3. [3]

    Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. 2023. https://doi.org/10.18653/v1/2023.acl-long.75 A close look into the calibration of pre-trained language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1343--1367, Toronto, Canada. Association for Computatio...

    doi:10.18653/v1/2023.acl-long.75 2023

  4. [4]

    Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-industry.74 Batch prompting: Efficient inference with large language model API s . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 792--810, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-industry.74 2023

  5. [5]

    Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023 b . http://arxiv.org/abs/2301.08721 Batch prompting: Efficient inference with large language model apis

    arXiv 2023

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  7. [7]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. http://arxiv.org/abs/2305.14325 Improving factuality and reasoning in language models through multiagent debate

    arXiv 2023

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

Show all 40 references
  1. [9]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Ed Hovy, Hinrich Schutze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. ArXiv, abs/2102.01017

    2021 arXiv

  2. [10]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. http://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey

    2024 arXiv

  3. [11]

    Yu Gu, Xiang Deng, and Yu Su. 2023. https://doi.org/10.18653/v1/2023.acl-long.270 Don`t generate, discriminate: A proposal for grounding language models to real-world environments . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol...

    2023 doi

  4. [12]

    Zhitao He, Pengfei Cao, Yubo Chen, Kang Liu, Ruopeng Li, Mengshu Sun, and Jun Zhao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.613 LEGO : A multi-agent collaborative framework with role-playing and iterative feedback for causality explanation generation . In Finding...

    2023 doi

  5. [13]

    Zhitao He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024. http://arxiv.org/abs/2403.02959 Agentscourt: Building judicial decision-making agents with court debate simulation and legal knowledge augmentation

    2024 arXiv

  6. [14]

    Zhitao He, Zijun Liu, Peng Li, Yi R Fung, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. 2025 a . http://arxiv.org/abs/2502.14496 Advancing language multi-agent learning with credit re-assignment for interactive environment generalization

    2025 arXiv

  7. [15]

    Zhitao He, Zongwei Lyu, Dazhong Chen, Dadi Guo, and Yi R. Fung. 2025 b . http://arxiv.org/abs/2506.06034 Matp-bench: Can mllm be a good automated theorem prover for multimodal problems?

    2025 arXiv

  8. [16]

    Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, and Yi R. Fung. 2025 c . http://arxiv.org/abs/2505.23224 Mmboundary: Advancing mllm knowledge boundary awareness through reasoning step confidence calibration

    2025 arXiv

  9. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  10. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. http://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models

    2021 arXiv

  11. [19]

    Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. https://doi.org/10.18653/v1/2023.acl-long.309 Won`t get fooled again: Answering questions with false premises . In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023 doi

  12. [20]

    Yuchen Huang, Zhiyuan Fan, Zhitao He, Sandeep Polisetty, Wenyan Li, and Yi R. Fung. 2025. http://arxiv.org/abs/2507.06210 Cultureclip: Empowering clip with cultural awareness through synthetic images and contextualized captions

    2025 arXiv

  13. [21]

    Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. https://doi.org/10.18653/v1/P17-1167 Search-based neural structured learning for sequential question answering . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2017 doi

  14. [22]

    Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. http://arxiv.org/abs/2406.10890 Rwku: Benchmarking real-world knowledge unlearning for large language models

    2024 arXiv

  15. [23]

    Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. 2024. http://arxiv.org/abs/2405.13326 Mosaic-it: Free compositional data augmentation improves instruction tuning

    2024 arXiv

  16. [24]

    Qiuyu Liang, Weihua Wang, Feilong Bao, and Guanglai Gao. 2024 a . https://aclanthology.org/2024.lrec-main.873/ L 2 GC :lorentzian linear graph convolutional networks for node classification . In Proceedings of the 2024 Joint International Conference on Computational Linguistic...

    2024

  17. [25]

    Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. 2024 b . https://doi.org/10.18653/v1/2024.knowledgenlp-1.4 Learning to trust your feelings: Leveraging self-awareness in LLM s for hallucination mitigation . In Proceedings of the 3rd Workshop on Knowledge Augmented Meth...

    2024 doi

  18. [26]

    Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. 2024. http://arxiv.org/abs/2309.00384 Batchprompt: Accomplish more with less

    2024 arXiv

  19. [27]

    Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. 2023. https://api.semanticscholar.org/CorpusID:263891105 A first look at llm-powered generative news recommendation . ArXiv, abs/2305.06566

    2023 arXiv

  20. [28]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft

    2022

  21. [29]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1...

    2020 doi

  22. [30]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813

    2023 arXiv

  23. [31]

    Fung, Weizhu Chen, Minhao Cheng, and Furu Wei

    Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. 2025. http://arxiv.org/abs/2503.19551 Scaling laws of synthetic data for language models

    2025

  24. [32]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. https://doi.org/10.1162/tacl_a_00266 C o QA : A conversational question answering challenge . Transactions of the Association for Computational Linguistics, 7:249--266

    2019 doi

  25. [33]

    Ruiyang Ren, Wayne Xin Zhao, Jing Liu, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023. http://arxiv.org/abs/2305.11161 Tome: A two-stage approach for model-based retrieval

    2023 arXiv

  26. [34]

    Guijin Son, Sangwon Baek, Sangdae Nam, Ilgyun Jeong, and Seungone Kim. 2024. http://arxiv.org/abs/2402.11597 Multi-task inference: Can large language models follow multiple instructions at once?

    2024 arXiv

  27. [35]

    Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. http://arxiv.org/abs/2305.03047 Principle-driven self-alignment of language models from scratch with minimal human supervision

    2023 arXiv

  28. [36]

    Zhengxiang Wang, Jordan Kodner, and Owen Rambow. 2024. http://arxiv.org/abs/2406.10786 Exploring the zero-shot capabilities of llms handling multiple problems at once

    2024 arXiv

  29. [37]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  30. [38]

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.394 R -tuning: Instructing large language models to say I don`t know' . In Proceedings of the 2024 Conference of the ...

    2024 doi

  31. [39]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.