Pith. sign in

REVIEW 3 major objections 6 minor 36 references

A lightweight probe can mark which claim spans in an LLM answer are uncertain, matching multi-sample quality from a single forward pass.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 03:03 UTC pith:IJPYSYRS

load-bearing objection Solid engineering paper that cleanly fills the span-level gap; label-target fidelity is the real caveat, not a collapse of the results. the 3 major comments →

arxiv 2607.05721 v1 pith:IJPYSYRS submitted 2026-07-07 cs.CL

SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

Yimeng Zhang , Yingying Zhuang , Ziyi Wang , Yuxuan Lu , Pei Chen , Aman Gupta , Zhe Su , Ming Tan
show 7 more authors
Zhilin Zhang Qun Liu Manikandarajan Ramanathan Rajashekar Maragoud Edward Vul Jing Huang Dakuo Wang
This is my paper
classification cs.CL
keywords span-level uncertaintyLLM generationhallucination localizationuncertainty quantificationDETR span detectionmixture of Betahidden-state probeself-refinement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Token-level uncertainty is hard to read as a fact, and sequence-level scores cannot show which part of an answer is wrong. This paper treats the natural unit as a span: a short, coherent stretch that carries one assessable claim. It formalizes span-level uncertainty estimation, builds SPANUQ-BENCH with continuous soft labels from multi-sample claim checks, and trains SPANUQ—a ~25M-parameter probe that reads frozen LLM hidden states, detects spans with a DETR-style decoder, and scores each span with a mixture of Beta distributions. Across five backbones the probe leads sampling and probe baselines on span discrimination and error while running 10–20× faster, and its detector reaches 0.910 F1. If that holds, systems can localize unreliable claims without paying for repeated generation.

Core claim

Continuous uncertainty over semantically coherent spans can be recovered from one forward pass of a frozen LLM by a small jointly trained probe that detects variable-length spans and models each span’s uncertainty as a mixture of Beta distributions, after distilling multi-sample claim-verification soft labels; on five LLMs this yields AUROC 0.908–0.944 and MAE 0.110–0.129, with DETR detection F1 0.910, while remaining an order of magnitude faster than sampling methods.

What carries the argument

SPANUQ: learnable DETR-style span queries over fused mid-layer hidden states, soft-mask content enrichment, Mixture-of-Beta heads trained with Beta NLL plus contrastive ranking, and optional Uncertainty-Conditioned Iterative Refinement (UCIR).

Load-bearing premise

Soft labels from twenty temperature-1 samples checked against Wikipedia by an LLM judge truly track the model’s epistemic uncertainty, and three mid-layer hidden states hold enough of that signal for a small probe to recover it.

What would settle it

Replace the automatic multi-sample soft labels on a held-out set of reasoning-heavy or non-Wikipedia answers with independent human span uncertainty ratings; if probe AUROC and ranking collapse toward chance while multi-sample agreement stays high, the distilled target is not the uncertainty that matters.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Unreliable claims can be highlighted without regenerating the answer many times.
  • Sequence-level uncertainty is partly recoverable as an importance-weighted sum of span scores (ρ_seq ≈ 0.84), so span methods can subsume sequence methods.
  • Selective verification or self-refinement can focus only on high-uncertainty spans.
  • The same probe pattern generalizes across dense and MoE models from 4B to 30B.
  • Binary hallucination detection emerges as a byproduct with high AUROC.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Span scores could power UIs that highlight only doubtful fragments for human review instead of flagging whole answers.
  • Labels built on Wikipedia-plus-judge may understate uncertainty for reasoning or time-sensitive claims; domain-specific verifiers would test label dependence.
  • Freezing the LLM and attaching a tiny probe is a practical path to ship uncertainty as a side channel without retraining the generator.
  • Compositional errors remain a failure mode, so joint span–relation uncertainty is a natural next unit of analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper formalizes Span-Level Uncertainty Estimation (SLUE) as the task of jointly detecting semantically coherent spans and assigning continuous uncertainty scores to each, bridging token-level and sequence-level methods. It introduces SPANUQ, a ~25M-parameter probe on frozen LLM hidden states that combines multi-layer fusion, a DETR-style set-prediction span decoder, span-feature enrichment, Mixture-of-Beta (MoB) uncertainty heads, and Uncertainty-Conditioned Iterative Refinement (UCIR). Soft training/test labels are obtained by multi-sample generation (S=20), LLM-judge claim decomposition, Wikipedia verification, and cross-sample support rates (Eq. 14). The authors release SPANUQ-BENCH (~20K prompts, ~293K spans) and report, across five backbones (Qwen3 4B–30B-A3B and Mistral-7B), span-level AUROC 0.908–0.944 and MAE 0.110–0.129, DETR detection F1 0.910, 10–20× speedups over sampling baselines, and importance-weighted span-to-sequence correlation ρ_seq ≈ 0.839.

Significance. If the reported numbers and the fidelity of the soft labels hold, the work is a clear advance for trustworthy LLM generation: it supplies a practical single-pass probe at the natural claim granularity, a first continuous span-level benchmark, and evidence that sequence-level uncertainty is partially decomposable into span components. Strengths that should be credited include thorough multi-backbone evaluation (dense and MoE), systematic ablations (enrichment gate, MoB K, UCIR, regression vs pointer, layer clustering), end-to-end detection comparisons against heuristics and BIO, and explicit efficiency accounting. These make the engineering contribution reproducible and useful even if the epistemic interpretation of the labels is refined later.

major comments (3)
  1. [App. C.3 / Sec. 4.1 / Eq. 14] App. C.3 and Sec. 4.1: The central quality claims (AUROC 0.908–0.944, MAE 0.110–0.129 in Table 1) treat the multi-sample soft labels u*_k (Eq. 14) as ground truth. The only human check is a 10% stratified review summarized as “reliable,” with no reported agreement rates, Cohen’s κ, or error breakdown by domain/uncertainty bin. Without quantified residual judge/Wikipedia noise (and without a fully human-annotated held-out subset scored separately), it is impossible to know how much of the reported AUROC is recovery of the labeling pipeline versus recoverable epistemic signal. Please add quantitative audit statistics and, ideally, a small fully human-labeled test slice with metrics recomputed against it.
  2. [Sec. 4.1 / Eq. 14 / App. C.2] Sec. 4.1, Eq. 14 and App. C.2: The target u*_k is the fraction of temperature-1 samples whose aligned claims fail Wikipedia+LLM-judge support. This mixes sampling variability, retrieval/judge errors, temporal drift, and true knowledge gaps; ambiguous claims are scored 0.5 by fiat. The manuscript claims to estimate epistemic uncertainty from mid-layer hidden states, yet does not test whether the probe is primarily recovering the multi-sample support process (construct mismatch). A load-bearing check would be correlation of SPANUQ scores with (i) human confidence ratings or (ii) an independent verification source not used in label construction, plus an explicit discussion of aleatoric vs epistemic content of the labels. Without this, the “best span-level uncertainty quality” claim is benchmark-relative rather than scientifically settled.
  3. [Table 1 / Sec. 4.2] Table 1 and Sec. 4.2: Sequence- and claim-level baselines are evaluated by broadcasting a single score to every span. While the paper notes the granularity mismatch, this design systematically disadvantages those methods on span-level Spearman/AUROC and inflates the apparent gap. A fairer secondary protocol—e.g., pairing SelfCheckGPT/FActScore with the same DETR or oracle spans, or reporting only sequence-level metrics for sequence methods—should be added so that the 13-point AUROC lift over SelfCheckGPT-NLI is not partly an artifact of score broadcasting.
minor comments (6)
  1. [Fig. 1 / App. B.4] Fig. 1 and the Marie Curie example are clear; ensure the same example’s predicted spans appear in App. B.4 so readers can compare qualitative behavior to the schematic.
  2. [Eq. 8 / Sec. 4.3] Eq. (8) and the claim ρ_seq = 0.839: clarify whether importance weights are learned only on training data and frozen at test time, and report sensitivity of ρ_seq to uniform vs learned pooling.
  3. [App. C.2] App. C.2 Stage 2–3: name the exact LLM judge model/version and release the decomposition/verification prompts for reproducibility of SPANUQ-BENCH.
  4. [Table 3] Table 3: the slight drop in ρ_seq when adding Enrichment Gate and UCIR is noted but not explained; a short sentence on score redistribution would help.
  5. [Sec. 5 / App. B.5] Limitations correctly flag white-box access and English factual focus; add a sentence on dependence on Wikipedia coverage for long-tail entities (cf. App. B.5).
  6. [Eq. 14 / Tables] Typos/notation: “spans k” in Eq. 14 text; consistent use of ρ_s vs ρ_span; arXiv-style “Preprint” header can be cleaned for journal submission.

Circularity Check

0 steps flagged

No significant circularity: SPANUQ is standard supervised distillation of externally constructed multi-sample labels into a probe; reported metrics are held-out evaluation, not inputs renamed as predictions.

full rationale

The paper’s load-bearing chain is: (i) define soft span labels u*_k via multi-sample generation + claim decomposition + Wikipedia/LLM-judge verification (Eq. 14, App. C.2); (ii) train a DETR-style probe on frozen LLM hidden states to regress those labels (MoB NLL, ranking, detection losses); (iii) evaluate AUROC/MAE/ρ on a held-out test set built with the same pipeline, and compare to baselines. That is ordinary supervised distillation, not a self-definitional loop: the target is fixed before training and is not a function of the probe’s parameters. Sequence-level aggregation (Eq. 8) and the consistency loss Lcon are secondary; ground-truth sequence uncertainty is the mean of span labels, so good span predictions will tend to compose—but the paper reports this as an empirical correlation (ρ_seq ≈ 0.839) against dedicated sequence baselines, not as a first-principles derivation that “proves” subsumption by construction. Architecture choices (DETR, Beta mixture, UCIR) are justified by external citations and ablations, not by uniqueness theorems or self-citation chains from the same authors. Concerns about whether multi-sample Wikipedia support rates equal true epistemic uncertainty are construct-validity / correctness issues, not circularity. No step reduces a claimed prediction to its own fitted input or definition.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on the multi-sample soft-label construction, the choice of three mid-layers, the DETR query budget, MoB mixture size, and a collection of loss-weight and UCIR hyperparameters. No new physical or mathematical entities are postulated; the invented objects are engineering modules and a dataset.

free parameters (6)
  • number of DETR span queries N = 32
    Set to 32 by hand to cover the 95th-percentile span count; ablation shows N=16 already close.
  • selected LLM layers L = model-dependent triple
    Chosen by single-layer probing sweep per backbone (e.g., 22/24/26 for Qwen3-14B); free architectural choice.
  • MoB mixture components J and α_min = J=3, α_min=0.5
    J=3 and α_min=0.5 selected after ablation; control expressivity of the uncertainty distribution.
  • UCIR mixing weight α = 0.7
    Fixed at 0.7 for Round-2 vs Round-1 combination.
  • loss weights λ_reg, λ_val, λ_uq, λ_con, λ_rank, λ_aux = see App. D.1
    Hand-set defaults (5.0, 2.0, 4.0, 1.0, 0.5, 0.4) that balance detection vs uncertainty objectives.
  • soft-mask sharpness τ = 10
    Controls boundary differentiability; set to 10.
axioms (4)
  • domain assumption A contiguous text span that expresses a single verifiable assertion is the natural atomic unit for uncertainty estimation.
    Stated in the introduction and used to justify SLUE; not derived.
  • domain assumption The fraction of temperature-1 samples in which a claim fails Wikipedia+LLM-judge verification is a valid continuous proxy for the model’s epistemic uncertainty.
    Eq. 14 and Sec. 4.1; the entire distillation target rests on this.
  • domain assumption Mid-to-late Transformer layers encode recoverable uncertainty signals that a small probe can read without fine-tuning the LLM.
    Supported by layer-probing study (App. B.6) but still an empirical premise.
  • standard math Hungarian matching with L1+GIoU + uncertainty + validity cost yields a correct one-to-one assignment for overlapping spans.
    Standard DETR technique; invoked in Sec. 3.1.
invented entities (3)
  • SPANUQ probe (DETR decoder + MoB + UCIR) no independent evidence
    purpose: Single-pass joint span detection and continuous uncertainty estimation from frozen LLM hidden states.
    Core technical contribution; no independent existence outside the paper.
  • SPANUQ-BENCH no independent evidence
    purpose: First large-scale continuous soft-label span-level uncertainty benchmark (20 K prompts, ~293 K spans).
    Constructed for this work; labels derived from the multi-sample pipeline.
  • Span-Level Uncertainty Estimation (SLUE) task no independent evidence
    purpose: Formal problem statement that sits between token- and sequence-level uncertainty.
    Named and defined here; prior work touched related ideas but did not formalize the joint detection+scoring objective.

pith-pipeline@v1.1.0-grok45 · 33524 in / 3319 out tokens · 32048 ms · 2026-07-11T03:03:17.323216+00:00 · methodology

0 comments
read the original abstract

Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refinement in LLM generation. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors. We formalize Span-Level Uncertainty Estimation (SLUE), a new task that targets the natural granularity for uncertainty: semantically coherent text spans, each conveying a single assessable unit of meaning. To address this task, we introduce SPANUQ, a lightweight probe that distills the uncertainty knowledge from expensive multi-sample inference into a single forward pass over LLM hidden states. SPANUQ employs a DETR-style span decoder to simultaneously detect spans and estimate their uncertainty via a Mixture of Beta distribution, trained with a principled combination of Beta NLL regression and contrastive ranking objectives. We construct SPANUQ-BENCH, the first span-level uncertainty benchmark comprising 20K prompts, 293K annotated spans, and continuous soft labels derived from multi-sample claim verification. Experiments on five LLM backbones show that SPANUQ consistently achieves the best span-level uncertainty quality, outperforming the strongest probe baseline and all sampling-based methods while being 10-20x faster. Its DETR-based span detector attains 0.910 F1, surpassing the best heuristic by 39.4%, enabling precise error localization that sequence-level methods cannot provide. The framework generalizes across five LLMs spanning two model families.

Figures

Figures reproduced from arXiv: 2607.05721 by Aman Gupta, Dakuo Wang, Edward Vul, Jing Huang, Manikandarajan Ramanathan, Ming Tan, Pei Chen, Qun Liu, Rajashekar Maragoud, Yimeng Zhang, Yingying Zhuang, Yuxuan Lu, Zhe Su, Zhilin Zhang, Ziyi Wang.

Figure 1
Figure 1. Figure 1: Comparison of uncertainty estimation granularities for an [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the SPANUQ architecture. Given a frozen LLM’s hidden states from three selected layers, the probe fuses, projects, and encodes token representations, then uses a DETR-style decoder with learnable span queries to simultaneously detect spans and estimate their uncertainty via Mixture of Beta (MoB) heads. Uncertainty-Conditioned Iterative Refinement (UCIR, dashed) feeds Round-1 estimates back thro… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 14 internal anchors

  1. [1]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,”ACM computing surveys, vol. 55, no. 12, pp. 1–38, 2023

    work page 2023

  2. [2]

    Language Models (Mostly) Know What They Know

    S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield- Dodds, N. DasSarma, E. Tran-Johnsonet al., “Language models (mostly) know what they know,”arXiv preprint arXiv:2207.05221, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    Fact-checking the output of large language models via token-level uncertainty quantification,

    E. Fadeeva, A. Rubashevskii, A. Shelmanov, S. Petrakov, H. Li, H. Mubarak, E. Tsymbalov, G. Kuzmin, A. Panchenko, T. Baldwinet al., “Fact-checking the output of large language models via token-level uncertainty quantification,” inFindings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 9367–9385

    work page 2024

  4. [4]

    TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning

    T. Zhang, H. Shi, Y . Wang, H. Wang, X. He, Z. Li, H. Chen, L. Han, K. Xu, H. Zhanget al., “Tokur: Token-level uncertainty estimation for large language model reasoning,”arXiv preprint arXiv:2505.11737, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    L. Kuhn, Y . Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,”arXiv preprint arXiv:2302.09664, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space,

    X. Qiu and R. Miikkulainen, “Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space,”Advances in neural information processing systems, vol. 37, pp. 134 507–134 533, 2024

    work page 2024

  7. [7]

    Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

    M. Xiong, Z. Hu, X. Lu, Y . Li, J. Fu, J. He, and B. Hooi, “Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms,”arXiv preprint arXiv:2306.13063, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Factscore: Fine-grained atomic evaluation of factual precision in long form text generation,

    S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine-grained atomic evaluation of factual precision in long form text generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 12 076–12 100

    work page 2023

  9. [9]

    Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models,

    A. Vazhentsev, L. Rvanova, I. Lazichny, A. Panchenko, M. Panov, T. Baldwin, and A. Shelmanov, “Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techno...

    work page 2025

  10. [10]

    Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities,

    A. Nikitin, J. Kossen, Y . Gal, and P. Marttinen, “Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities,”Advances in Neural Information Processing Systems, vol. 37, pp. 8901–8929, 2024

    work page 2024

  11. [11]

    To Believe or Not to Believe Your LLM

    Y . A. Yadkori, I. Kuzborskij, A. György, and C. Szepesvári, “To believe or not to believe your llm,”arXiv preprint arXiv:2406.02543, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentation

    L. Da, T. Chen, L. Cheng, and H. Wei, “Llm uncertainty quantification through directional entailment graph and claim level response augmentation,”arXiv preprint arXiv:2407.00994, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

    C. Chen, K. Liu, Z. Chen, Y . Gu, Y . Wu, M. Tao, Z. Fu, and J. Ye, “Inside: Llms’ internal states retain the power of hallucination detection,”arXiv preprint arXiv:2402.03744, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation

    Z. Li, J. Xiong, F. Ye, C. Zheng, X. Wu, J. Lu, Z. Wan, X. Liang, C. Li, Z. Sunet al., “Un- certaintyrag: Span-level uncertainty enhanced long-context modeling for retrieval-augmented generation,”arXiv preprint arXiv:2410.02719, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    The internal state of an llm knows when it’s lying,

    A. Azaria and T. Mitchell, “The internal state of an llm knows when it’s lying,” inFindings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 967–976

    work page 2023

  16. [16]

    LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

    H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y . Belinkov, “Llms know more than they show: On the intrinsic representation of llm hallucinations,”arXiv preprint arXiv:2410.02707, 2024. 10

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

    J. Kossen, J. Han, M. Razzak, L. Schut, S. Malik, and Y . Gal, “Semantic entropy probes: Robust and cheap hallucination detection in llms,”arXiv preprint arXiv:2406.15927, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Long-form factuality in large language models,

    J. Wei, C. Yang, X. Song, Y . Lu, N. Hu, J. Huang, D. Tran, D. Peng, R. Liu, D. Huanget al., “Long-form factuality in large language models,”Advances in Neural Information Processing Systems, vol. 37, pp. 80 756–80 827, 2024

    work page 2024

  19. [19]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” inProceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 9004–9017

    work page 2023

  20. [20]

    Fine-grained Hallucination Detection and Editing for Language Models

    A. Mishra, A. Asai, V . Balachandran, Y . Wang, G. Neubig, Y . Tsvetkov, and H. Ha- jishirzi, “Fine-grained hallucination detection and editing for language models,”arXiv preprint arXiv:2401.06855, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Semeval-2025 task 3: Mu-shroom, the multilingual shared-task on hallucinations and related observable overgeneration mistakes,

    R. Vázquez, T. Mickus, E. Zosa, T. Vahtola, J. Tiedemann, A. Sinha, V . Segonne, F. Sánchez- Vega, A. Raganato, J. Libovick`yet al., “Semeval-2025 task 3: Mu-shroom, the multilingual shared-task on hallucinations and related observable overgeneration mistakes,” inProceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), 2025, p...

    work page 2025

  22. [22]

    Neural architectures for named entity recognition,

    G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for named entity recognition,” inProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, 2016, pp. 260–270

    work page 2016

  23. [23]

    End-to-end neural coreference resolution,

    K. Lee, L. He, M. Lewis, and L. Zettlemoyer, “End-to-end neural coreference resolution,” in Proceedings of the 2017 conference on empirical methods in natural language processing, 2017, pp. 188–197

    work page 2017

  24. [24]

    Entity, relation, and event extraction with contextualized span representations,

    D. Wadden, U. Wennberg, Y . Luan, and H. Hajishirzi, “Entity, relation, and event extraction with contextualized span representations,” inProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), 2019, pp. 5784–5789

    work page 2019

  25. [25]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” inEuropean conference on computer vision. Springer, 2020, pp. 213–229

    work page 2020

  26. [26]

    CLUE: Concept-Level Uncertainty Estimation for Large Language Models

    Y .-H. Wang, A. Bai, C.-P. Tsai, and C.-J. Hsieh, “Clue: Concept-level uncertainty estimation for large language models,”arXiv preprint arXiv:2409.03021, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answering

    C. Tong, Q. Zhang, J. Gao, L. Jiang, Y . Liu, and N. Sun, “Halunet: Multi-granular uncer- tainty modeling for efficient hallucination detection in llm question answering,”arXiv preprint arXiv:2512.24562, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [28]

    The hungarian method for the assignment problem,

    H. W. Kuhn, “The hungarian method for the assignment problem,”Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955

    work page 1955

  29. [29]

    Generalized inter- section over union: A metric and a loss for bounding box regression,

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized inter- section over union: A metric and a loss for bounding box regression,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 658–666

    work page 2019

  30. [30]

    Natural questions: a benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Leeet al., “Natural questions: a benchmark for question answering research,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019

    work page 2019

  31. [31]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1601–1611

    work page 2017

  32. [32]

    Felm: Benchmarking factuality evaluation of large language models,

    Y . Zhao, J. Zhang, I. Chern, S. Gao, P. Liu, J. Heet al., “Felm: Benchmarking factuality evaluation of large language models,”Advances in Neural Information Processing Systems, vol. 36, pp. 44 502–44 523, 2023. 11

    work page 2023

  33. [33]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” inProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), 2022, pp. 3214–3252

    work page 2022

  34. [34]

    Halueval: A large-scale hallucination evaluation benchmark for large language models,

    J. Li, X. Cheng, X. Zhao, J.-Y . Nie, and J.-R. Wen, “Halueval: A large-scale hallucination evaluation benchmark for large language models,” inProceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 6449–6464

    work page 2023

  35. [35]

    Qwen3 Technical Report

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025. 12 Appendix A Architectural Design Details of SPANUQ Fig. A1illustrates the end-to-end span-level uncertainty estimation process in SPANUQ, from initial- ization to final uncertainty output. This append...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    why” (38%) and “how

    Reddit dataset. These are dominated by “why” (38%) and “how” (21%) questions requiring long-form reasoning. Responses average 349 words (the longest across domains) and hallucinations tend to be subtle reasoning errors rather than factual fabrications, testing the model’s ability to detect distributed uncertainty. Biography (2,000 prompts, 10%).Following ...

    work page 2048