Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Replacing uncertain claims with vaguer ones makes long-form LLM output more reliable without sacrificing much information.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:55 UTC pith:3TRSHRY6

load-bearing objection Solid framework, plausible mechanism, but the risk oracle likely flatters vague abstractions, so read the 27.73% as an upper bound until a stratified human check. the 3 major comments →

arxiv 2602.11908 v3 pith:3TRSHRY6 submitted 2026-02-12 cs.AI cs.CLcs.LG

When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation

Shani Goren , Ido Galil , Ran El-Yaniv This is my paper
classification cs.AI cs.CLcs.LG
keywords selective abstractionlong-form generationfactual correctnessatomic claimsuncertainty estimationrisk-coverage trade-offhallucination mitigationinformation content
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models are fluent but unreliable in long-form writing, where a single misplaced detail can sink a biography or a legal summary. This paper argues that instead of refusing to answer when confidence is low, or chopping out whole sentences, a model should rewrite only the uncertain parts into vaguer, safer versions: 'born in Atchison, Kansas' becomes 'born in Kansas.' Across six open-source models and two long-form factuality benchmarks, this claim-level 'selective abstraction' consistently beats deletion-based baselines on the risk-coverage trade-off, improving the area under the curve by up to 27.73%. If the result holds, reliability and informativeness are not opposites but endpoints of a dial a user can turn.

Core claim

The central discovery is that uncertainty can be managed at the level of individual atomic claims, not whole responses or sentences. The paper defines Selective Abstraction, formalized through selective risk and coverage, and instantiates it as Atom-wise SA: decompose the generation into atomic claims (short, self-contained factual statements), elicit a verbalized confidence per atom, generate a chain of increasingly general rewrites (the abstraction sequence), and pick the most specific rewrite whose confidence clears a threshold; reconstruct text from the selected atoms. The risk of an output is the fraction of claims a fact-checking agent cannot support using an open encyclopedia; coverag

What carries the argument

The core mechanism is the abstraction sequence: for each low-confidence atom, the model is prompted to produce a chain a^(0)=a, a^(1), ..., ⊤ where each step is a minimal generalization of the previous one (e.g., a precise birth date → month → year → decade), and confidence scores are elicited for the entire chain conditioned on the atom's own confidence. The selector then chooses the most specific chain element whose confidence exceeds a threshold θ; if none does, the atom is dropped. The risk-coverage curve is the evaluation instrument that demonstrates the mechanism's effect: risk falls smoothly as θ rises, and retained information is quantified via the entropy reduction of the set of ent

Load-bearing premise

The results stand on the assumption that the fact-checking agent's supported/unsupported labels are an unbiased measure of correctness—in particular, that vague abstractions are not systematically easier to label as supported simply because they are vague; only 102 claims were checked against human judgment.

What would settle it

Run the same six models and benchmarks, but have independent human annotators (or a second fact-checking method grounded in a different knowledge source) label every atom in the original and abstracted outputs. If the area-under-the-risk-coverage advantage of atom-wise SA over deletion shrinks or vanishes under human labels, the claimed risk reduction was an artifact of the checker; if it persists, the effect is real.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A user can dial reliability: raising the confidence threshold monotonically lowers factual risk while retaining a large fraction of the original information.
  • The method works with any confidence function; the paper shows verbalized self-confidence outperforms log-probability and P(True) baselines, suggesting better uncertainty models would further improve the trade-off.
  • Because atomization and reconstruction are prompt-based, the pipeline applies to any long-form domain, including non-factual generation such as forecasting, advice, and counterfactual reasoning, as the appendix demonstrates.
  • The conformal-style threshold selection lets a practitioner specify a target risk and receive a confidence threshold that achieves it with high probability on new prompts.
  • The observed cross-model consistency implies the benefit is not tied to one model family; hedging helps as long as the model produces confidence scores with some ranking ability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next test: apply the same abstraction idea in a retrieval-augmented setting, where the choice is not only 'how vague' but also 'what evidence to condition on'; the abstraction chain could be grounded in retrieved passages instead of self-confidence.
  • The information measure, which assumes a uniform prior over entities, likely underweights how surprising a rare-entity claim is; a frequency-weighted version would change coverage values and could alter which method wins at specific operating points.
  • If the fact-checking agent implicitly rewards vagueness, part of the reported risk reduction may be metric artifact; a human-annotation study on the abstracted outputs across all six models would settle this.
  • The abstraction chain itself is an audit trail: a reader can see exactly which details the model was unsure about and how it hedged, which could support human oversight in high-stakes writing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Selective Abstraction (SA), a framework for long-form generation that trades specificity for reliability by replacing low-confidence atomic claims with higher-confidence, less specific abstractions. It formalizes SA through selective risk and coverage, proposes atom-wise SA (atomize, elicit confidences, generate abstraction sequences, select the most specific abstraction above a threshold, reconstruct), and evaluates it on FactScore and LongFact-Objects with six open-source LLMs. Risk is measured by an LLM-based Wikipedia fact-checking agent; coverage is measured by an information-theoretic quantity computed from Wikidata counts via SPINACH. The main empirical claim is that atom-wise SA consistently improves the risk-coverage trade-off, with AURC improvements up to 27.73% over claim redaction. The paper also presents a conformal-inspired algorithm for selecting a confidence threshold to meet a target risk.

Significance. If the empirical claims hold, this is a useful and timely contribution: it gives a principled middle ground between full generation and binary abstention, at a finer granularity than prior selective generation work. The framing is clear, the pipeline is fully described with prompts, and the evaluation covers six models and two benchmarks while building on open resources. The comparison to redaction is fair in design, and the consistency of the direction of the effect across models is a genuine strength. The main risk to significance is external validity: the AURC gains are computed from labels produced by an LLM fact-checker and an approximate Wikidata-based information measure, and the validation of those proxies is currently thin. If the authors can close that gap, the paper would make a solid empirical contribution.

major comments (3)
  1. [Section 4.1, Appendix D.1] The risk labels are produced by a gpt-oss-120b agent instructed to mark SUPPORTED when Wikipedia contains clear supporting evidence and UNSUPPORTED otherwise. The reported human validation (F1=0.93, 102 claims, footnote 6) samples uniformly across the agent's own labels, but it is not stratified by abstraction level or specificity. Since SA inserts deliberately vague abstractions, and broad claims are easier to support on Wikipedia, label leniency toward vagueness would differentially lower SA's measured risk relative to redaction and inflate the AURC gains in Table 1. Please add human validation stratified by (i) original atoms vs abstractions and (ii) abstraction depth, reporting agreement separately for specific vs vague claims. This is the key external-validity check for the headline claim.
  2. [Table 1, Section 5] The main AURC comparison is reported as single numbers without variance. The evaluation uses only 36 FactScore and 76 LongFact prompts, and the improvements range from 5.41% to 27.73%. Without bootstrap confidence intervals over prompts or a paired significance test across thresholds, it is difficult to know whether, e.g., the Llama-3.3-70B-I gain of 6.48% is distinguishable from noise. Please report per-prompt AURC distributions, or at least standard errors or bootstrapped CIs, for the main table.
  3. [Section 4.2] Coverage, and therefore AURC, depends on the information measure I(c)=1−log|E_c|/log|E|, estimated by SPINACH queries over Wikidata. The paper acknowledges Wikidata sparsity and the uniform-prior assumption, but no validation of the natural-language-to-SPARQL conversion is reported. If the conversion systematically under- or over-counts |E_c| for abstracted predicates, the coverage values for SA relative to redaction shift and the AURC improvements change. Please include an error analysis of SPINACH on a sample of the predicates used, together with a sensitivity analysis of AURC to moderate perturbations of I(c).
minor comments (4)
  1. [Table 2] The header 'FACTSCORE∆RISK(%)' is missing spacing, and the table reports only averages. Reporting medians and per-model variability would help assess whether the matched-coverage risk gaps are consistent across models.
  2. [Appendix H] The theorem statement says P(|R(θ̂)−α|≤ϵ)≥1−δ, but ϵ is not explicitly defined in the statement; it is 'computed as a function' in the algorithm. Please spell out ϵ and clarify that R(θ̂)=P(θ_{n+1}>θ̂|D_cal) is a conditional probability over a single test claim, not the empirical risk plotted in Figure 4.
  3. [Section 2, References] There are minor typographical issues: '2In contrast' appears mid-sentence in the Related Work discussion, and reference [45] renders 'V olodya'. A final proofread would fix these.
  4. [Section 4.3] The anchor point at φ=0 with risk equal to the risk at the minimum observed coverage is a strong convention. Please justify it or show that the main conclusions are robust to alternative treatments of the low-coverage interval, such as linear interpolation from the highest-threshold operating point.

Circularity Check

0 steps flagged

No significant circularity: the central AURC result is an empirical benchmark comparison, not a reduction to fitted inputs or a load-bearing self-citation chain.

full rationale

The paper's central claim—that atom-wise Selective Abstraction improves the risk-coverage trade-off—is supported by an end-to-end empirical pipeline evaluated on external benchmarks (FactScore, LongFact). Risk labels come from a Wikipedia-based fact-checking agent (Section 4.1), and coverage comes from a Wikidata-derived information measure (Section 4.2); neither is fitted to the test predictions, and the AURC numbers are measured, not derived from a fitted parameter. The abstraction-confidence conditioning described in Section 3 and Appendix C is a design choice that anchors abstraction confidences to the original atom confidence, but the paper does not use that conditioning to prove the risk reduction; it evaluates the resulting trade-off empirically. The self-citations to the authors' prior work on selective prediction and hierarchical selective classification ([7], [14]) are used for framing and conceptual alignment, not as load-bearing justification for the empirical results. The main validity concern—that the fact-checker may be systematically lenient toward vague abstractions—is a measurement-bias / external-validity risk, not a definitional circularity: it cannot be exhibited as an equation-level reduction of the paper's own derivation. Therefore, no circular step meeting the required evidentiary standard is present, and the honest finding is a low circularity score.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No numeric free parameters are fitted to the empirical results; the method is prompt-based and threshold is swept. The principal un-checked assumptions concern the reliability of the LLM-based risk and coverage measurements rather than invented physical or mathematical entities.

axioms (6)
  • domain assumption Verbalized confidence scores reflect factual correctness of atoms and abstractions.
    The selection step assumes that κ_atom and κ_abs rank correct content above incorrect content. Table 4 reports AUROC 0.74 for verbal confidence on gpt-oss-120b/FactScore, so this is partially supported but not guaranteed.
  • domain assumption Agentic Wikipedia fact-checking labels are trustworthy ground truth for atom correctness.
    Risk is computed from SUPPORTED/UNSUPPORTED labels produced by a gpt-oss-120b agent searching Wikipedia (Section 4.1, Appendix D.1). Only 102 claims were human-validated, with F1 0.93; large-scale label bias is not ruled out.
  • domain assumption Uniform prior over entities in the information measure.
    I(c) = 1 − log|E_c|/log|E| assumes a uniform distribution over entities, acknowledged in Section 4.2 as a simplifying assumption with known limitations.
  • domain assumption Wikidata is a best-available proxy for real-world entity set sizes.
    Coverage is computed from Wikidata counts via SPINACH; the paper acknowledges Wikidata is sparse, so coverage numbers are approximate (Section 4.2).
  • domain assumption Calibration atoms are exchangeable with test atoms.
    Appendix H's conformal guarantee assumes exchangeability of A_cal and the test atom. Atoms drawn from the same prompt are not independent, and dependencies are ignored.
  • domain assumption Generated abstractions are logically implied by their source atoms.
    The abstraction prompt asks for generalizations that are true if the previous statement is true, but there is no verification that the model obeys this logical constraint (Appendix B).

pith-pipeline@v1.3.0-alltime-deepseek · 29492 in / 12040 out tokens · 117703 ms · 2026-08-02T23:55:52.725763+00:00 · methodology

0 comments
read the original abstract

LLMs are widely used, yet they remain prone to factual errors that erode user trust and limit adoption in high-risk settings. One approach to mitigate this risk is to equip models with uncertainty estimation mechanisms that abstain when confidence is low. However, this binary "all-or-nothing" approach is excessively restrictive in long-form settings, often discarding valuable information. We introduce Selective Abstraction (SA), a framework that enables LLMs to trade specificity for reliability by selectively reducing the detail of uncertain content. We first formalize SA through the lenses of selective risk and coverage. We then propose Atom-wise Selective Abstraction, a claim-level instantiation that decomposes responses into atomic claims (short, self-contained statements each expressing a single fact) and replaces uncertain atoms with higher confidence, less specific abstractions. To evaluate this framework, we develop a novel end-to-end pipeline for open-ended generation that instantiates risk as factual correctness and measures coverage using an information-theoretic measure of retained information. Across six open-source models on the FactScore and LongFact-Objects benchmarks, atom-wise SA consistently outperforms existing baselines, improving the area under the risk-coverage curve (AURC) by up to 27.73% over claim removal, demonstrating that reducing specificity can boost accuracy and reliability while preserving most of their original meaning.

Figures

Figures reproduced from arXiv: 2602.11908 by Ido Galil, Ran El-Yaniv, Shani Goren.

Figure 1
Figure 1. Figure 1: Left: Example abstraction sequence for atom￾wise selective abstraction (SA). Increasing the confidence threshold replaces low-confidence atoms with less spe￾cific, more reliable abstractions. Right: Example risk￾coverage curve comparing atom-wise SA to baselines. Model: gpt-oss-120b, dataset: FactScore. eration tasks [3, 13, 6]. Despite these advances, LLMs remain prone to factual errors, generating conten… view at source ↗
Figure 2
Figure 2. Figure 2: An overview of the Selective Abstraction framework. The generated text is decomposed into atoms, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative example of atom-wise Selective Abstraction at two confidence thresholds: a higher threshold [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Risk-guided threshold selection on FactScore [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Risk–coverage curves obtained by instantiating [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Risk–coverage curves on FactScore with gpt-oss-120b, comparing verbalized confidence, [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Risk-coverage curves for atom-wise selective abstraction (ours) compared to redaction, Inline, and Self [PITH_FULL_IMAGE:figures/full_fig_p030_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Risk-coverage curves for atom-wise selective abstraction (ours) compared to redaction, Inline, and Self [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Risk-coverage curves on 36 FactScore entities comparing atom-wise selective abstraction (ours) compared to redaction, Inline, and Self-Revision for gpt-oss-120b (left) and gpt-oss-20b (right). We also report thresholded Inline, and Self-Revision evaluated at three thresholds, θ ∈ {10, 50, 90} (shown in transparent markers, size increasing with θ). Introducing θ into these baselines does not yield a meaning… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems

    cs.CL 2026-04 unverdicted novelty 7.0

    Compositional selective specificity (CSS) decomposes generated answers into claims and emits each at the most specific level supported by evidence, raising overcommitment-aware utility from 0.846 to 0.913 on LongFact ...

  2. Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems

    cs.CL 2026-04 unverdicted novelty 6.0

    Compositional selective specificity (CSS) improves overcommitment-aware utility from 0.846 to 0.913 on LongFact while retaining 0.938 specificity by calibrating claim-level backoffs in agentic AI responses.

Reference graph

Works this paper leans on

102 extracted references · 2 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V

    Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V . Le. Towards a human- like open-domain chatbot, 2020. URL https: //arxiv.org/abs/2001.09977

    Pith/arXiv arXiv 2020

  2. [2]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.CoRR, abs/2107.07511, 2021. URL https://arxiv. org/abs/2107.07511

    Pith/arXiv arXiv 2021

  3. [3]

    Introducing the claude 3 model family

    Anthropic. Introducing the claude 3 model family. https://www.anthropic.com/news/ claude-3-family, 2024

    2024

  4. [4]

    Semantic information.British Journal for the Philosophy of Science, 4(14):147–157, 1953

    Yehoshua Bar-Hillel and Rudolf Carnap. Semantic information.British Journal for the Philosophy of Science, 4(14):147–157, 1953. doi: 10.1093/bjps/iv. 14.147

    work page doi:10.1093/bjps/iv 1953

  5. [5]

    Synthese Library

    Rudolf Carnap.An Outline of a Theory of Semantic Information. Synthese Library. Springer, Dordrecht, 1964

    1964

  6. [6]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. URLhttps://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv 2025

  7. [7]

    On the founda- tions of noise-free selective classification.J

    Ran El-Yaniv and Yair Wiener. On the founda- tions of noise-free selective classification.J. Mach. Learn. Res., 11:1605–1641, 2010. doi: 10.5555/ 1756006.1859904. URL https://dl.acm.org/ doi/10.5555/1756006.1859904

    arXiv 2010

  8. [8]

    Lm-polygraph: Un- certainty estimation for language models

    Ekaterina Fadeeva, Roman Vashurin, Akim Tsvi- gun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Bald- win, and Artem Shelmanov. Lm-polygraph: Un- certainty estimation for language models. In Yan- song Feng and Els Lefever, editors,Proceedings of the 2023 Conference on E...

    2023

  9. [9]

    Fact-checking the out- put of large language models via token-level uncer- tainty quantification

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the out- put of large language models via token-level uncer- tainty quantification. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar, ed...

    2024

  10. [10]

    Don’t hallucinate, abstain: Identifying LLM knowl- edge gaps via multi-llm collaboration

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying LLM knowl- edge gaps via multi-llm collaboration. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers)...

    2024

  11. [11]

    What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? InICLR, 2023

    Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? InICLR, 2023

    2023

  12. [12]

    Bias-reduced uncertainty estimation for deep neu- ral classifiers, 2019

    Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neu- ral classifiers, 2019. URL https://arxiv.org/ abs/1805.08206

    Pith/arXiv arXiv 2019

  13. [13]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/2507.06261

    Pith/arXiv arXiv 2025

  14. [14]

    Hierar- chical selective classification, 2025

    Shani Goren, Ido Galil, and Ran El-Yaniv. Hierar- chical selective classification, 2025. URL https: //arxiv.org/abs/2405.11533

    Pith/arXiv arXiv 2025

  15. [15]

    Can language models be spe- cific? how?, 2023

    Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen mei Hwu. Can language models be spe- cific? how?, 2023. URL https://arxiv.org/ abs/2210.05159. 9

    Pith/arXiv arXiv 2023

  16. [16]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Trans

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Trans. Inf. Syst., 43(2): 42:1–42:55, 2025. doi: 10.1145/3703155. URL https://doi.org/10.1145/3703155

    doi:10.1145/3703155 2025

  17. [17]

    Optimized batch prompt- ing for cost-effective llms.Proceedings of the VLDB Endowment, 18(7):2172–2184, 2025

    Zhaoxuan Ji, Xinlu Wang, Zhaojing Luo, Zhongle Xie, and Meihui Zhang. Optimized batch prompt- ing for cost-effective llms.Proceedings of the VLDB Endowment, 18(7):2172–2184, 2025. doi: 10.14778/3734839.3734853

    arXiv 2025

  18. [18]

    Survey of hallucination in natural language generation.ACM Comput

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation.ACM Comput. Surv., 55(12):248:1–248:38, 2023. doi: 10.1145/3571730. URLhttps://doi.org/10.1145/3571730

    doi:10.1145/3571730 2023

  19. [19]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading compre- hension. In Regina Barzilay and Min-Yen Kan, editors,Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Pap...

    doi:10.18653/v1/p17-1147 2017

  20. [20]

    Language models (mostly) know what they know, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec,...

    Pith/arXiv arXiv 2022

  21. [21]

    LLMs cannot (yet) match the specificity and simplicity of online commu- nities in long form question answering

    Kris-Fillip Kahl, Tolga Buz, Russa Biswas, and Gerard De Melo. LLMs cannot (yet) match the specificity and simplicity of online commu- nities in long form question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2028–2053, Miami, Florida, USA, November

    2024

  22. [22]

    Vem- pala, and Edwin Zhang

    Adam Tauman Kalai, Ofir Nachum, Santosh S. Vem- pala, and Edwin Zhang. Why language models hal- lucinate, 2025. URL https://arxiv.org/abs/ 2509.04664

    Pith/arXiv arXiv 2025

  23. [24]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language gen- eration

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language gen- eration. InThe Eleventh International Confer- ence on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net,

    2023

  24. [25]

    Generating with confidence: Uncertainty quantifi- cation for black-box large language models.Trans

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models.Trans. Mach. Learn. Res., 2024, 2024. URL https: //openreview.net/forum?id=DWkJCSxKU5

    2024

  25. [26]

    Semnani, Harold Tried- man, Jialiang Xu, Isaac Dan Zhao, and Monica S

    Shicheng Liu, Sina J. Semnani, Harold Tried- man, Jialiang Xu, Isaac Dan Zhao, and Monica S. Lam. SPINACH: sparql-based information nav- igation for challenging real-world questions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024...

    2024

  26. [27]

    Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev

    Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation, 2023. URL https://arxiv. org/abs/2212.07981

    Pith/arXiv arXiv 2023

  27. [28]

    URL https://openreview.net/forum? id=VD-AYtP0dve

  28. [29]

    Passonneau

    Ani Nenkova and Rebecca J. Passonneau. Evaluat- ing content selection in summarization: The pyra- mid method. In Julia Hirschberg, Susan T. Du- mais, Daniel Marcu, and Salim Roukos, editors, Human Language Technology Conference of the North American Chapter of the Association for Com- putational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May...

    2004

  29. [30]

    gpt-oss-120b & gpt-oss-20b model card,

    OpenAI. gpt-oss-120b & gpt-oss-20b model card,

  30. [31]

    findings-emnlp.938

    URL https://aclanthology.org/2024. findings-emnlp.938

    2024

  31. [32]

    Qwen3 technical report, 2025

    Qwen. Qwen3 technical report, 2025. URL https: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  32. [33]

    Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, edi- tors,Proceedings of the 2023 Conference on Em- pirical Methods in Natural Lan...

    2023

  33. [34]

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J. Liu. Out-of-distribution detection and selec- tive generation for conditional language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: //openreview.ne...

    2023

  34. [35]

    Using information content to eval- uate semantic similarity in a taxonomy.CoRR, abs/cmp-lg/9511007, 1995

    Philip Resnik. Using information content to eval- uate semantic similarity in a taxonomy.CoRR, abs/cmp-lg/9511007, 1995. URL http://arxiv. org/abs/cmp-lg/9511007

    Pith/arXiv arXiv 1995

  35. [36]

    Crowdsourcing lightweight pyramids for manual summary evalua- tion

    Ori Shapira, David Gabay, Yang Gao, Hadar Ro- nen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. Crowdsourcing lightweight pyramids for manual summary evalua- tion. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Confer- ence of the North American Chapter of the Associa- tion for Computational Ling...

    2019

  36. [37]

    Can knowledge graphs make large language models more trustworthy? an empirical study over open- ended question answering, 2025

    Yuan Sui, Yufei He, Zifeng Ding, and Bryan Hooi. Can knowledge graphs make large language models more trustworthy? an empirical study over open- ended question answering, 2025. URL https:// arxiv.org/abs/2410.08085

    Pith/arXiv arXiv 2025

  37. [38]

    Fact-checking complex claims with program-guided reasoning

    Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. Fact-checking complex claims with program-guided reasoning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, edi- tors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, T...

    doi:10.18653/v1/ 2023

  38. [39]

    Lamda: Lan- guage models for dialog applications, 2022

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, De- hao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vince...

    Pith/arXiv arXiv 2022

  39. [41]

    A stitch in time saves nine: Detecting and mitigating hallucina- tions of llms by validating low-confidence gener- ation.CoRR, abs/2307.03987, 2023

    Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucina- tions of llms by validating low-confidence gener- ation.CoRR, abs/2307.03987, 2023. doi: 10.48550/ ARXIV .2307.03987. URLhttps://doi.org/10. 48550/arXiv.2307.03987

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.03987 2023

  40. [42]

    Benchmarking uncertainty quantification methods 11 for large language models with lm-polygraph.Trans

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Ab- delrahman Boda Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking uncertainty quantification methods 11 for large language models with lm-polygra...

    doi:10.1162/tacl 2025

  41. [43]

    Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models

    Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. Token- level density-based uncertainty quantification meth- ods for eliciting truthfulness of large language mod- els.CoRR, abs/2502.14427, 2025. doi: 10.48550/ ARXIV .2502.14427. URLhttps://doi.org/10. 48550/arXiv.2502.14427

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.14427 2025

  42. [44]

    Conditional validity of inductive conformal predictors.CoRR, abs/1209.2673, 2012

    Vladimir V ovk. Conditional validity of inductive conformal predictors.CoRR, abs/1209.2673, 2012

    Pith/arXiv arXiv 2012

  43. [45]

    The llama 3 herd of models.CoRR, abs/2407.21783, 2024

    Llama Team. The llama 3 herd of models.CoRR, abs/2407.21783, 2024. doi: 10.48550/ARXIV . 2407.21783. URL https://doi.org/10.48550/ arXiv.2407.21783

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  44. [46]

    Wikidata: a free collaborative knowledgebase.Commun

    Denny Vrandecic and Markus Kr¨otzsch. Wikidata: a free collaborative knowledgebase.Commun. ACM, 57(10):78–85, 2014. doi: 10.1145/2629489. URL https://doi.org/10.1145/2629489

    doi:10.1145/2629489 2014

  45. [47]

    S. M. Towhidul Islam Tonmoy, S. M. Mehedi Za- man, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive sur- vey of hallucination mitigation techniques in large language models.CoRR, abs/2401.01313, 2024. doi: 10.48550/ARXIV .2401.01313. URL https: //doi.org/10.48550/arXiv.2401.01313

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.01313 2024

  46. [48]

    Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu

    Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification. In Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Processing Systems 38: Annual Confer...

    2024

  47. [49]

    Nar- rowing the knowledge evaluation gap: Open- domain question answering with multi-granularity answers, 2024

    Gal Yona, Roee Aharoni, and Mor Geva. Nar- rowing the knowledge evaluation gap: Open- domain question answering with multi-granularity answers, 2024. URL https://arxiv.org/abs/ 2401.04695

    Pith/arXiv arXiv 2024

  48. [50]

    LUQ: long-text uncertainty quan- tification for llms

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. LUQ: long-text uncertainty quan- tification for llms. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Mi- ami, FL, USA, November 12-16, 2024, pages 5244–

    2024

  49. [51]

    Finding a bal- anced degree of automation for summary evaluation

    Shiyue Zhang and Mohit Bansal. Finding a bal- anced degree of automation for summary evaluation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceed- ings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6617– 6632, Online and Punta Cana, Dominican Repub- lic, November 2021. Asso...

    doi:10.18653/v1/2021.emnlp-main 2021

  50. [52]

    Machine-learning applications of algo- rithmic randomness

    V olodya V ovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algo- rithmic randomness. In Ivan Bratko and Saso Dze- roski, editors,Proceedings of the Sixteenth Inter- national Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30, 1999, pages 444–453. Morgan Kaufmann, 1999

    1999

  51. [54]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V . Le. Long-form factuality in large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ul- rich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Process...

    2024

  52. [59]

    emnlp-main.299

    URL https://aclanthology.org/2024. emnlp-main.299

    2024

  53. [62]

    [Entity] [Fact]. ... # Examples ## Example 1 Input Text:Jennifer Coolidge is an American actress. Known for her work in the comedy genre, Coolidge is the recipient of several accolades, including a Golden Globe Award and two Primetime Emmy Awards. Atoms:

  54. [63]

    [Jennifer Coolidge] [is an American]

  55. [64]

    [Jennifer Coolidge] [is an actress]

  56. [65]

    [Jennifer Coolidge] [is known for her work in the comedy genre]

  57. [66]

    [Jennifer Coolidge] [has received several accolades]

  58. [67]

    [Jennifer Coolidge] [has received a Golden Globe Award]

  59. [68]

    ## Example 2 Input Text:Albert Einstein was a German-born theoretical physicist who is best known for developing the theory of relativity

    [Jennifer Coolidge] [has received two Primetime Emmy Awards]. ## Example 2 Input Text:Albert Einstein was a German-born theoretical physicist who is best known for developing the theory of relativity. Einstein also made important contributions to quantum mechanics. Atoms:

  60. [69]

    [Albert Einstein] [was born in Germany]

  61. [70]

    [Albert Einstein] [was a theoretical physicist]

  62. [71]

    [Albert Einstein] [is best known for developing the theory of relativity]

  63. [72]

    ## Example 3

    [Albert Einstein] [made important contributions to quantum mechanics]. ## Example 3 ... (more examples truncated) 18 Atoms Confidence Prompt # Role and Objective You are a careful and critical evaluator. Your task is to assess the factual correctness of each statement and provide a confidence score that reflects how likely it is that the statement is true...

  64. [73]

    - The confidence score must be a number between 0 and 100

    [Statement] Reasoning: [your reasoning here] Confidence: [score] - Use the same index numbers as in the list you are given. - The confidence score must be a number between 0 and 100. - Keep reasoning concise but informative — 1 to 3 sentences. - Do not repeat the statement in the reasoning. # Confidence Scoring Guidelines - Reasoning should explain why th...

  65. [78]

    # Review and Guidance Think step by step

    Assign a confidence score based on your belief that the statement is true. # Review and Guidance Think step by step. For each statement, explain your reasoning clearly and concisely before giving your final confidence score. 19 Abstraction Prompt # Role and Objective You are an Abstraction Assistant. Your task is to take a factual statement about an entit...

  66. [79]

    Reasoning: (your reasoning here)

    [ENTITY] [FACT]. Reasoning: (your reasoning here)

  67. [80]

    Reasoning: (your reasoning here)

    [ENTITY] [GENERALIZED FACT]. Reasoning: (your reasoning here) ... K. STOP. Reasoning: (why no further generalization is meaningful) # Reasoning Steps

  68. [81]

    Start with the original sentence

  69. [82]

    Do not change the ENTITY

    Work only inside the FACT (the second bracket). Do not change the ENTITY

  70. [83]

    If the FACT contains multiple details, choose the one you are least confident about

  71. [84]

    Make the smallest logical generalization at each step

    Generalize that detail to a broader term. Make the smallest logical generalization at each step

  72. [85]

    The new statement must be logically implied by the previous one — that is, if the previous statement is true, the new one must also be true

  73. [86]

    The rest of the sentence must remain unchanged

    Modify only one detail per step. The rest of the sentence must remain unchanged

  74. [87]

    Keep the sentence format fixed: [ENTITY] [FACT]

  75. [88]

    For each abstraction, write 1–3 sentences of reasoning explaining why you chose that part to generalize, and why the generalization is valid

  76. [89]

    In that case, output ‘STOP‘ and explain why

    Continue until further abstraction would make the sentence trivial or uninformative. In that case, output ‘STOP‘ and explain why. # Generalization Examples - Date: ”March 3, 1920”→”March 1920”→”1920”→”1920s”→”20th century” - Location: ”Paris”→” ˆIle-de-France”→”France”→”Europe” - Nationality: ”Texan”→”Southern American”→”American” - ... (more examples tru...

    1920

  77. [90]

    Reasoning: I’m less confident about the exact city than the country

    [Marie Curie] [was born in Warsaw, Poland]. Reasoning: I’m less confident about the exact city than the country

  78. [91]

    Reasoning: Warsaw is a city in Poland, so this is a valid generalization

    [Marie Curie] [was born in Poland]. Reasoning: Warsaw is a city in Poland, so this is a valid generalization

  79. [92]

    Reasoning: Poland is a European country

    [Marie Curie] [was born in Europe]. Reasoning: Poland is a European country

  80. [93]

    Reasoning: Further generalization would be too vague to retain meaning

    STOP. Reasoning: Further generalization would be too vague to retain meaning. ## Example 2 ... (more examples truncated) # Review and Guidance Think step by step. Each abstraction should be a clean, single-step generalization from the previous one, guided by your uncertainty. Focus on generalizing the part you are least confident about, and reflect that i...

Showing first 80 references.