Pith. sign in

REVIEW 2 major objections 3 minor 36 references

UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions

T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces UAQFact, a bilingual benchmark of 6,985 unanswerable questions and 6,985 answerable ones built from factual triples, and uses it to show that LLMs often know the relevant facts yet still answer instead of refusing.

desk verdict A solid bilingual UAQ benchmark grounded in Wikidata, but Task 3's CoT clues embed the label rule, so the external-knowledge-utilization claim needs a control or a disclaimer. read the letter →

arxiv 2505.23461 v1 pith:BWN5XYRI submitted 2025-05-29 cs.CL

Chuanyuan Tan , Wenbiao Shao , Hao Xiong , Tong Zhu , Zhenhua Liu , Kai Shi , Wenliang Chen This is my paper
classification cs.CL
keywords UAQFactunanswerablequestionsfactualknowledgeutilizationknowledge-awarerefusalratebilingualbenchmarkgraphchain-of-thoughtLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces UAQFact, a bilingual English-Chinese benchmark of 6,985 unanswerable questions (UAQ) paired with 6,985 answerable ones, each question carrying auxiliary factual knowledge. It defines three tasks: distinguishing UAQ from answerable questions (ABQ), probing whether LLMs use internal factual knowledge, and probing whether they use external factual knowledge supplied as a reasoning clue. Across open-source and closed-source LLMs, the paper finds a knowledge-utilization gap: models pass factual-knowledge tests 68–83% of the time, yet their knowledge-aware refusal rates stay at 54–69%. Adding the relevant facts as a chain-of-thought clue raises the refusal gap substantially, but the best English score still reaches only about 75%, so the paper concludes LLMs do not fully use factual knowledge when deciding whether a question has an answer. The contribution is the dataset, the two new evaluation tasks, and the knowledge-aware refusal-rate metric that make this gap measurable.

What carries the argument

The load-bearing object is UAQFact, a template-generated bilingual dataset in which every UAQ is built from a contradiction among real factual triples, via three question types: set intersection, time constraint, and false-dilemma choice. On top of it, two new evaluation tasks and the knowledge-aware refusal rate (KRR) metric carry the argument: Task 2 converts each UAQ into multiple-choice factual probes and scores refusal only on cases where the model passes the knowledge test, while Task 3 embeds the same facts in a chain-of-thought clue and measures how much the refusal gap improves. KRR is what lets the paper compare models with different knowledge stores on a single utilization axis.

What would settle it

Run Task 3 with the same factual answers but without the explicit refusal rule, stating the sub-answers but omitting the instruction that an empty intersection means no answer; if the refusal gap drops sharply, the Task 3 gains are largely instruction-following, not knowledge use.

Watch

Extended reading notes

Core claim

The central claim is that UAQFact measures whether LLMs can use factual knowledge—stored internally or supplied externally—when facing unanswerable questions, and that the measurements reveal a systematic gap between knowledge and refusal. The dataset is generated from real factual triples so that each UAQ has an empty answer set by construction: an intersection with no common element, a time window no fact falls in, or a dilemma whose listed candidates are all wrong. Task 2 finds that LLMs carry the relevant facts, with knowledge pass rates of 68–83%, but their knowledge-aware refusal rate remains low, meaning they refuse only 54–69% of UAQs that they demonstrably have knowledge about. Task 3 provides the same facts inside a step-by-step clue; this improves refusal discrimination but does not close the gap, and the paper reports the best English refusal-gap score at roughly 75%. The paper attributes this residual gap to limits in knowledge utilization rather than knowledge storage.

Load-bearing premise

The load-bearing premise is that Task 3 tests external knowledge use; the paper does not control for the fact that its reasoning clue embeds the exact decision rule that makes the question unanswerable, so the measured gains may reflect instruction-following rather than knowledge utilization.

Editorial extensions

If this is right

  • Since knowledge pass rates are high while knowledge-aware refusal rates are low, improving refusal behavior should target knowledge utilization, not knowledge storage.
  • Supplying external factual knowledge as a step-by-step clue raises refusal discrimination for every tested model, so knowledge access is a useful lever even when models already hold the facts.
  • The gap does not close with perfect facts: the best English refusal-gap score is about 75%, so LLMs still answer a meaningful fraction of unanswerable questions despite having the needed information.
  • Larger models show better refusal discrimination in the scaling experiments, but their refusal rates on UAQ and ABQ fluctuate independently, indicating that scale alone does not calibrate refusal behavior.
  • The knowledge-aware refusal rate allows comparisons across models with different internal knowledge levels, making it a usable metric for future UAQ evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the Task 3 clue states the refusal rule outright, the reported external-knowledge gains likely overstate knowledge use; a controlled variant that hides the rule would disentangle the two.
  • Editorial inference: if the utilization gap is real, fine-tuning on tasks that reward refusing when two fact sets have empty intersection should improve KRR without adding new facts; this is directly testable.
  • Editorial inference: the KRR formula couples refusal discrimination with knowledge pass rate, so a low KRR can also come from over-refusing answerable questions; future work could decompose KRR into knowledge-conditional false-refusal and false-answer components.
  • Editorial inference: extending the same knowledge-graph contradiction construction to longer multi-hop chains of facts would show whether the utilization gap grows with reasoning depth, not just factual overlap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces UAQFact, a bilingual (English/Chinese) dataset of 6,985 unanswerable questions (UAQ) and an equal number of answerable questions (ABQ), generated from Wikidata triples and accompanied by auxiliary factual knowledge and reasoning clues. The authors propose three evaluation tasks: (1) basic UAQ/ABQ discrimination, (2) assessing utilization of internal knowledge via multiple-choice probes and a new composite metric (knowledge-aware refusal rate, KRR), and (3) assessing utilization of external knowledge via chain-of-thought clues. Experiments across open-source and black-box LLMs show that models pass knowledge probes at high rates (KPR 68-83%) but achieve lower knowledge-aware refusal rates, and that providing external clues improves refusal discrimination but does not fully close the gap.

Significance. If its evaluation methodology is sound, UAQFact is a potentially valuable resource: it is large, bilingual, and built from a knowledge graph with transparent and manually inspected construction (99.2% quality on a 900-item check), and it makes a concrete step toward measuring whether LLMs can apply factual knowledge to recognize unanswerability. The paper also ships the dataset and code, which supports reproducibility. However, the two new tasks that carry the central knowledge-utilization claim currently have important methodological confounds, so the significance of the empirical findings depends on whether those confounds can be addressed.

major comments (2)
  1. [§4.4, Table 17, Table 13] Task 3's CoT clue does more than supply external facts: it embeds the exact decision rule that determines the UAQ label (e.g., 'If there is no intersection of the answers to the sub-questions, then there is no answer to the question') along with the answer sets themselves. An LLM can therefore output the correct refusal by applying a direct procedural instruction, without exercising the kind of knowledge-based judgment the task claims to measure. The preliminary comparison in Table 13 is revealing: EKnow (Triple) provides the factual sets without the rule, and for Llama3 the R∆ is 49.06, whereas EKnow (CoT) with the rule reaches 74.54; the paper chooses CoT because it yields higher R∆, thereby amplifying the confound. To support the claim that external factual knowledge utilization is being measured, the authors should include a control that provides the answer sets without the decision sentence, or a rule that does not directly entail the UAQ label, and show that the R∆ gain persists beyond instruction-following.
  2. [§4.1, Eq. (1)] The definition of the knowledge-aware refusal rate is not mathematically well-formed as typeset: 'KRR = (1 + e^{−R∆·KPR−1})^{-1}' is ambiguous, and under the most natural readings (e.g., a sigmoid of R∆·KPR) the reported values are inconsistent. For instance, Llama3 in English has R∆ = 20.89 and KPR = 73.11, so R∆·KPR is about 1527; a sigmoid of that argument would saturate at 1, not 57.10. Because KRR is the core metric behind the Task 2 conclusion that knowledge utilization lags knowledge storage, the authors must provide a clear, correct formula and verify that the reported numbers follow from it.
minor comments (3)
  1. [Table 17 (Time example)] The English Time reasoning clue mixes entities: the main question is about Erfurt's twinned city, but the auxiliary question asks 'When did Estadio GEBA start participating in association football the first time?', which is unrelated and appears to be a template-filling artifact. This should be corrected, and the clue-generation pipeline should be checked for similar mismatches.
  2. [§3.1 and Appendix D] The manual inspection reports that 7 out of 900 sampled questions (0.8%) are mislabeled due to Wikidata knowledge errors, but the paper does not state whether these questions were removed or corrected in the released dataset. If they remain, the benchmark contains known label noise; the authors should clarify and ideally filter or repair these instances.
  3. [§4.4 and Table 13] The preliminary EKnow (Triple) results show a large improvement in ABQ accuracy (e.g., Qwen2.5 from 47.73 to 95.00), yet the paper only reports R∆ and does not discuss this accuracy trade-off. A fuller presentation of the triple condition would help readers assess the relative roles of factual content and rule-based scaffolding.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity overall; Task 3's reasoning clue embeds the UAQ decision rule and answer sets, so its R∆ gains partly measure instruction following rather than knowledge utilization.

  1. self definitional [Section 3.2 Task 3; Table 17 (Inter clue); Section 4.4 and Table 12]
    "Combine the answers to q1 and q2 to make a judgment. If there is an intersection of the answers to the sub-questions, then the answer to the question is this intersection. If there is no intersection of the answers to the sub-questions, then there is no answer to the question. In summary, my answer is:"

    The Task 3 reasoning clue is constructed from the same Wikidata factual triples that define the UAQ/ABQ label, and it explicitly states the rule that generates the label: UAQ is precisely the case where the two sub-answer sets have no intersection. The prompt (Table 6) already instructs models to return ##None## when there is no suitable answer, so the final sentence of the clue restates the ground-truth label. A model can therefore output the correct refusal by following the decision rule without performing knowledge-based comparison or retrieval.

full rationale

UAQFact's labels and factual triples are obtained from external Wikidata SPARQL queries, so the dataset construction is not circular and is grounded outside the paper's own fitted values. Task 1 is a direct refusal-rate benchmark using human-validated lexical matching, and this part of the evaluation is independent of the paper's claims. Task 2's KPR is a standard multiple-choice knowledge probe, and KRR is a fixed monotone transform of Task 1's R∆ and KPR; it involves no fitted parameters, although it is a heuristic combination rather than a direct measure of utilization. The only load-bearing concern is Task 3, where the reasoning clue states the exact UAQ labeling rule and supplies the answer sets that determine the intersection, making the correct response directly readable from the clue. The paper also reports in Appendix F that CoT-form knowledge was preferred over triple-form knowledge partly because CoT gives higher R∆, which further indicates that the selected condition embeds the decision rule. This is a genuine but partial confound: it weakens the claim that Task 3 measures external knowledge utilization, yet it does not make the empirical outcomes tautological, since models still fail to follow the clue in many cases. There is no load-bearing self-citation chain and no fitted input renamed as a prediction. Overall, the paper is self-contained against external knowledge-graph ground truth, and the circularity score is low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its load-bearing assumptions are about the validity of KG-grounded labels, the interpretation of the Task 3 clue as external knowledge, and the reliability of lexical refusal and exact-match accuracy metrics.

assumptions (4)
  • domain assumption Unanswerability is defined by the absence of a query result in Wikidata, not by real-world verification.
    Section 3.1: the UAQ planned answer 'None' comes from empty intersection, time window, or candidate-set queries; this equates KG absence with unanswerability.
  • domain assumption A chain-of-thought clue that supplies sub-question answers and the 'if no intersection, no answer' rule is a valid measure of external factual knowledge utilization.
    Section 3.2 Task 3: the clue embeds the decision rule, so the task conflates knowledge provision with reasoning instruction.
  • domain assumption Lexical matching of refusal keywords identifies refusals with human-level accuracy.
    Section G.1: Cohen's Kappa of 94.90 on 3,600 sampled responses is reported, but the metric can still miss indirect refusals.
  • domain assumption GPT-3.5-generated templates, after manual inspection, are fluent and semantically aligned with their Wikidata properties.
    Section 3.1 and Appendix D: 395 of 1,259 templates were discarded, 229 revised, and a 900-question check passed 99.2%.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions." pith.science (2026). https://pith.science/paper/BWN5XYRI

@misc{pith2026250523461,
  author       = {Pith},
  title        = {Pith review of: UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWN5XYRI}},
  note         = {Machine review of arXiv:2505.23461}
}
read the original abstract

Handling unanswerable questions (UAQ) is crucial for LLMs, as it helps prevent misleading responses in complex situations. While previous studies have built several datasets to assess LLMs' performance on UAQ, these datasets lack factual knowledge support, which limits the evaluation of LLMs' ability to utilize their factual knowledge when handling UAQ. To address the limitation, we introduce a new unanswerable question dataset UAQFact, a bilingual dataset with auxiliary factual knowledge created from a Knowledge Graph. Based on UAQFact, we further define two new tasks to measure LLMs' ability to utilize internal and external factual knowledge, respectively. Our experimental results across multiple LLM series show that UAQFact presents significant challenges, as LLMs do not consistently perform well even when they have factual knowledge stored. Additionally, we find that incorporating external knowledge may enhance performance, but LLMs still cannot make full use of the knowledge which may result in incorrect responses.

Figures

Figures reproduced from arXiv: 2505.23461 by the authors.

Figure 1
Figure 1. Dataset Construction Process (QType Inter in English as an example) for unanswerable question (UAQ) and answerable question (ABQ): (1) Define the question type. (2) Sample factual triples from Wikidata as knowledge. (3) Generate questions by filling in the templates generated by LLM. (4) Define three tasks and compose unique inputs with factual knowledge from the preceding steps as references. QType Description Exam… view at source ↗
Figure 2
Figure 2. Refusal rate and Acc evaluated in Task 1 of Qwen2.5 series with parameters scaling from 0.5B to 72B. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. R∆ Comparison Between Base and EKnow. EN and ZH are abbreviations for English and Chinese, respectively. Detailed results are shown in Appendix F. Among open-source LLMs, Mistral0.2 excels in KPR for English, whereas Qwen2.5 excels in KPR for Chinese. Although the English KPR of Mistral0.2 is 1.65 points lower than Qwen2.5, it achieves a higher KRR, suggesting it applies its knowledge more efficiently despite having… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 5 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and...

    arXiv 2024

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang. 2023. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. arXiv preprint arXiv:2305.13712

    arXiv 2023

  5. [5]

    Anthropic. 2024. https://assets.anthropic.com/m/61e7d27f8c8f5919/original/Claude-3-Model-Card.pdf The claude 3 model family: Opus, sonnet, haiku

    work page 2024

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  7. [7]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  8. [8]

    Q Cheng, Tand Sun, X Liu, et al. 2024. Can ai assistants know what they don't know? arXiv preprint arXiv:2401.13275

    arXiv 2024

Show all 36 references
  1. [9]

    DeepSeek-AI. 2024. https://github.com/deepseek-ai/DeepSeek-LLM Deepseek llm: Scaling open-source language models with longtermism . arXiv preprint arXiv:2401.02954

    2024 arXiv

  2. [10]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  3. [11]

    Google Gemini Team. 2024. https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    2024

  4. [12]

    Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477--3488

    2021

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  6. [14]

    Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. https://doi.org/10.18653/v1/2023.acl-long.309 Won ' t get fooled again: Answering questions with false premises . In Proceedings of the 61st Annual Meeting of the Association for Computati...

    2023 doi

  7. [15]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36

    2024

  8. [16]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  9. [17]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  10. [18]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  11. [19]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. https://arxiv.org/abs/2306.09212 Cmmlu: Measuring massive multitask language understanding in chinese . Preprint, arXiv:2306.09212

    2023 arXiv

  12. [20]

    Siheng Li, Yang Cheng, Wu Taiqiang, Shi Chufan, Zhang Yuji, Zhu Xinyu, Cheng Zesen, Cai Deng, Yu Mo, Liu Lemao, Zhou Jie, Yang Yujiu, Wong Ngai, Wu Xixin, and Lam Wai. 2024. A survey on the honesty of large language models. arXiv preprint arXiv:2409.18786

    2024 arXiv

  13. [21]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

    2021 arXiv

  14. [22]

    Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. 2023. Prudent silence or foolish babble? examining large language models' responses to the unknown. arXiv preprint arXiv:2311.09731

    2023 arXiv

  15. [23]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124

    2020 arXiv

  16. [24]

    Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Bing Liu, Haonan Lu, and Wenliang Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.86 Probing language models for pre-training data detection . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics...

    2024 doi

  17. [25]

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. 2016. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031

    2016 arXiv

  18. [26]

    Thomas Pellissier Tanon, Denny Vrande c i \'c , Sebastian Schaffert, Thomas Steiner, and Lydia Pintscher. 2016. From freebase to wikidata: The great migration. In Proceedings of the 25th international conference on world wide web, pages 1419--1428

    2016

  19. [27]

    Kartik Shenoy, Filip Ilievski, Daniel Garijo, Daniel Schwabe, and Pedro Szekely. 2022. A study of the quality of wikidata. Journal of Web Semantics, 72:100679

    2022

  20. [28]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  21. [29]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

    2022

  22. [30]

    Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. 2024. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 372--384. Springer

    2024

  23. [31]

    Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers...

    2016

  24. [32]

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.551 Do large language models know what they don ' t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--86...

    2023 doi

  25. [33]

    Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Crepe: Open-domain question answering with false presuppositions. arXiv preprint arXiv:2211.17257

    2022 arXiv

  26. [34]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414

    2022 arXiv

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.