Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

GENUINE claims that structuring LLM outputs as dependency graphs and pooling uncertainty through them improves confidence ranking by up to 29% AUROC over semantic entropy, with the largest gains on long-form generation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

GENUINE uses dependency parse trees and learnable graph pooling to produce uncertainty scores for LLM outputs, claiming AUROC gains of up to 29% over semantic entropy baselines.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Headline AUROC gains are confounded with supervision; the graph-specific contribution is weaker than claimed, but the method and evaluation breadth merit a serious referee. the 4 major comments →

arxiv 2509.07925 v1 pith:AXZZX3WR submitted 2025-09-09 cs.CL cs.AIcs.LG

GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Models

Tuo Wang , Adithya Kulkarni , Tyler Cody , Peter A. Beling , Yujun Yan , Dawei Zhou This is my paper
classification cs.CL cs.AIcs.LG MSC 68T5068T07
keywords uncertainty estimationlarge language modelsdependency parse treesgraph poolingsemantic entropycalibrationhierarchical graph neural networkssupervised uncertainty quantification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that uncertainty in LLM-generated text is better measured on a dependency parse tree than token by token. It builds GENUINE, a supervised framework that turns each answer into a paragraph-level graph by connecting sentence root tokens, then applies hierarchical graph pooling to propagate and aggregate token-level probability, entropy, and hidden-state signals. The claim is that this structure-aware aggregation beats semantic entropy, self-assessment, and supervised token-level baselines, with up to 29% AUROC improvement on long-form tasks (translation, finance QA, summarization) and more than 15% reduction in calibration error. If correct, it means that where a token sits in the grammatical hierarchy is a usable signal for whether the whole response is trustworthy, especially when responses are long.

Core claim

On its own terms, GENUINE is a claim that uncertainty should be aggregated over a graph whose edges are syntactic dependencies, not over a flat token sequence. Each generated sentence becomes a dependency tree; sentence roots are connected to form a document-level graph. A learned hierarchical pooling scheme, modeled on differentiable graph pooling, coarsens this graph while a learned fusion module combines a structural pooling path (probability/entropy features) with a semantic pooling path (hidden embeddings). The supervision signal is the correctness of the response against ground truth. GENUINE reports AUROC gains of up to 29% over semantic entropy and calibration-error reductions above

What carries the argument

The central object is the dependency parse tree turned into a paragraph-level graph by connecting sentence root nodes, with tokens as nodes and syntactic relations as edges. On top of it sits hierarchical graph pooling with learned soft-assignment matrices, derived from differentiable pooling, that coarsen the graph layer by layer so pivotal tokens keep more influence. A fusion module merges the assignment matrices produced by the grey-box (probability/entropy) and white-box (embedding) pooling paths rather than merging raw features, which the paper argues avoids dimensionality imbalance and preserves each feature type's role.

Load-bearing premise

GENUINE assumes that a dependency parse tree, with sentence roots connected across sentences, points at the tokens that actually decide whether an answer is right; when the parser picks the wrong head or the pivotal connection crosses sentence boundaries in ways roots do not capture, the graph weighting is misaligned and the reported gains would shrink.

What would settle it

Take the same outputs and replace the dependency graph with a random tree of identical size and density, keeping all features and training identical. If AUROC stays at the reported level, the syntactic structure is not the carrier. A milder check: evaluate on a corpus with ungrammatical or code-switched text, where parser accuracy is low; if GENUINE's advantage over next-token graphs disappears, the assumption about parse-tree reliability is the load-bearing part.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If GENUINE is right, long-form outputs—translation, finance answers, news summaries—can be assigned confidence scores that rank correctness substantially better than semantic entropy, because errors in individual tokens are smoothed through the syntactic hierarchy.
  • The method's grey-box variant (probabilities and entropy only) is claimed to retain most of the benefit, so black-box LLM APIs that expose token probabilities can use it without hidden states.
  • Dependency parse trees outperform next-token adjacency graphs on the reported tasks, implying the syntactic relation, not mere word order, carries the uncertainty signal.
  • Calibration improvements on long-form tasks indicate the confidence scores are not just better ranked but better aligned with empirical correctness.
  • The supervised setup means correctness labels are required for training; the paper shows AUROC degrades gracefully with 20–50% training data and only 0.1% label noise.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if dependency structure is truly the carrier, the method should transfer to other syntactic formalisms (constituency trees, semantic role graphs) and to languages with richer morphology; the paper does not test that.
  • Editorial inference: one testable extension is flagging individual low-confidence subtrees rather than whole responses; the hierarchical pooling assignments could localize the pivotal token the way the 'legal/illegal' example suggests, but the paper stops at response-level scores.
  • Editorial inference: the 29% headline is relative AUROC improvement over semantic entropy on selected datasets and LLMs; on short-answer datasets the paper's own ablation shows smaller gains, so the practical payoff is conditional on response length.
  • Editorial inference: because the graph is built from an external parser, parser errors are an unmeasured upper bound; adversarial or informal text that breaks parsing could invert the claimed gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GENUINE, a supervised framework that estimates the uncertainty of LLM-generated text by converting dependency parse trees into document-level graphs (connecting sentence root nodes) and applying hierarchical graph pooling to token-level features. Grey-box features (token probabilities/entropies) and white-box features (hidden embeddings) are pooled separately, and the resulting assignment matrices are fused by a learned module. The model is trained on correctness labels derived from task-specific scoring functions (Rouge/BLEU) and evaluated on QA (CoQA, TriviaQA, Finance), machine translation (WMT14), and summarization (CNN) across four open LLMs. The headline claims are up to 29% higher AUROC than semantic-entropy baselines and calibration-error reductions over 15%, with the largest gains on long-form outputs.

Significance. If the claimed gains hold, GENUINE would be a useful contribution to LLM uncertainty estimation: it is one of few methods that move beyond token-level aggregation and explicitly use syntactic dependency structure, and it addresses long-form generation where token-level entropy is known to degrade. The paper is also commendable for its breadth: five datasets, several LLM families, seven baselines, ablations, scalability experiments, parameter sensitivity, robustness to noisy labels, and a public code release. However, the significance is currently weakened by an experimental-design gap: GENUINE is supervised, yet its headline comparisons are to unsupervised methods, and the paper does not directly isolate the contribution of graph pooling from the contribution of supervised training. The self-reported appendix also concedes that the calibration advantage is not consistent across tasks. These issues are fixable within the manuscript's scope and do not invalidate the overall direction.

major comments (4)
  1. [§5.1, Table 1 (Ablation Study)] The central claim—that dependency-aware graph pooling improves uncertainty estimation—is not isolated from the effect of supervision. GENUINE is trained on correctness labels, while the headline comparison in Fig. 4 is against unsupervised semantic entropy. The ablation 'GENUINE w/o fusion & graph' is the natural supervised control, but the paper never identifies it with the Sup baseline (Liu et al., 2024b) or reports a direct GENUINE-vs-Sup comparison using identical features, splits, and training. Without this, the reported AUROC gains could be attributable to supervised learning rather than to graph structure. Please report the direct comparison and a statistical test.
  2. [§5.6, Table 5] The DPT-vs-NTG comparison, which is the paper's main evidence for the structural component, is not statistically supported. For Llama3-8B, DPT achieves 0.894±0.032 versus NTG 0.885±0.048; the grey-only variant is 0.903±0.025 versus 0.897±0.039. These differences are well within the reported error bars. The phrase 'consistently outperform' (Section 5.6) is too strong without paired significance tests or effect sizes. The same issue affects several other comparisons in Fig. 4 and the appendix tables.
  3. [Appendix B.1 / Abstract] The abstract and Section 5.2 claim that GENUINE reduces calibration errors by over 15% and 'achieves better calibration,' but Appendix B.1 explicitly states that 'GENUINE does not consistently achieve the lowest calibration error in TriviaQA and CoQA.' This is a load-bearing inconsistency for the calibration claim. Please qualify the claim in the abstract/main text, report the ECE numbers for all five datasets in the main body, and specify exactly where the 15% reduction is computed.
  4. [General experimental reporting] Across Fig. 4, Tables 1, 2, 5, 6, and the appendix, the paper reports many AUROC/ECE/NLL/Brier cells but never reports paired statistical tests, confidence intervals beyond mean±std, or multiple-comparison control. Given that several key differences (e.g., DPT vs NTG, GENUINE vs Sup, and the 0.903 vs 0.897 grey-feature comparison) are close, the absence of significance testing is not a presentation nit but a support issue for the central claims.
minor comments (5)
  1. [Abstract / Fig. 4] The 'up to 29% higher AUROC' figure is not tied to a specific dataset/model/baseline in the main text. Please state explicitly where this maximum occurs.
  2. [§5, contribution list] The list of experiments says results on dimensions 4, 5, 6, and 7 are in Appendix B, but Section 5.6 ('Graph Structure and Uncertainty Estimation') actually appears in the main text. This is internally inconsistent.
  3. [Table 2] In the label-noise experiment, increases in noise ratio improve ECE (0.246→0.232), NLL (0.362→0.414 at 0.02 then mixed), and Brier (0.094→0.117), while AUROC degrades. A reader would expect all metrics to worsen. Please explain this counterintuitive pattern.
  4. [References] The A4C baseline is cited as (Tian et al., 2023b), but the same title also appears as (Tian et al., 2023a); please align the reference entries and in-text citations.
  5. [§3.2, §4.1] The paper repeatedly calls dependency parse trees 'semantic' structures, but dependency syntax is a syntactic representation. Either justify the terminology or consistently say 'syntactic dependency structure used as a proxy for semantic importance.'

Circularity Check

0 steps flagged

No significant circularity: GENUINE is evaluated on held-out data and the graph-structure contribution is tested with a same-supervision ablation.

full rationale

I find no circular step in the paper's derivation chain. GENUINE is a supervised model: it is trained on correctness labels derived from task metrics (e.g., ROUGE, BLEU) and then evaluated on held-out test data (Section 5.1). This is ordinary supervised prediction, not circular, because the reported AUROC is computed on data not used for fitting. The graph-structure contribution is isolated by the ablation 'GENUINE w/o fusion & graph' versus 'GENUINE w/o fusion' versus GENUINE (Table 1), which keeps the supervised setting constant while adding graph structure and fusion; this directly tests the claimed mechanism rather than assuming it. The comparison to unsupervised semantic entropy baselines in the abstract and Figure 4 conflates supervision with graph structure, but that is a methodological comparison concern, not a circularity: the superior AUROC is an empirical result, not forced by construction. There are no self-citations used as load-bearing evidence, and no uniqueness theorem or ansatz is imported from the authors' prior work. The Limitations section concedes dependence on token logits/embeddings and labeled data, but those are ordinary assumptions, not circular reductions. The central dependency-parse-tree assumption is an empirical modeling choice tested by DPT-vs-NTG comparisons, not a definitional identity. Therefore the paper is self-contained against external benchmarks and its core claims do not reduce to their inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim depends on the arbitrary correctness thresholds, the choice to link sentence roots, and the assumption that dependency trees encode uncertainty-relevant semantics. The neural network parameters are learned rather than hand-set, so the only explicit free parameters are the thresholds and the pooling hyperparameters.

free parameters (5)
  • Correctness threshold for QA tasks (Rouge-1 >= 0.3) = 0.3
    Chosen by hand in Appendix A.1 to define whether a response is correct. This threshold directly determines the training labels and the AUROC ground truth.
  • Correctness threshold for MT (BLEU >= 0.3) = 0.3
    Chosen by hand in Appendix A.1 to binarize translation quality.
  • Correctness threshold for summarization (Rouge-L >= 0.35) = 0.35
    Chosen by hand in Appendix A.1 following prior work; affects label quality for CNN summarization.
  • Number of pooling layers = not reported in main text
    Tuned in Appendix B.4 over 1 to 4 layers; the exact value used for the main results is not stated in the main paper.
  • Remaining node ratio per pooling layer = not reported in main text
    A hyperparameter controlling how many nodes remain after each pooling step; sensitivity is analyzed but the chosen value is not given.
axioms (4)
  • domain assumption Dependency parse trees accurately reflect semantic importance of tokens for uncertainty.
    Used throughout Sections 3.2 and 4.2 to justify giving higher-level tree nodes more influence. If this relation does not hold, the graph weighting is not meaningful.
  • ad hoc to paper Connecting sentence root nodes creates meaningful cross-sentence semantic relationships.
    Introduced in Section 4.1 to build a paragraph-level graph. The paper does not validate that root-node connections capture inter-sentence semantics better than, say, coreference or discourse relations.
  • domain assumption Automatic metrics with chosen thresholds produce reliable binary correctness labels.
    Appendix A.1 defines correctness by ROUGE/BLEU thresholds. The noise in these labels directly affects training and evaluation; the paper's robustness experiments only add synthetic label noise, not realistic metric noise.
  • standard math Graph neural networks with softmax assignment pooling are learnable and differentiable as described.
    Equations 3-8 follow standard DiffPool-style hierarchical pooling, which is an established technique.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Models." pith.science (2026). https://pith.science/paper/AXZZX3WR

@misc{pith2026250907925,
  author       = {Pith},
  title        = {Pith review of: GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXZZX3WR}},
  note         = {Machine review of arXiv:2509.07925}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Uncertainty estimation is essential for enhancing the reliability of Large Language Models (LLMs), particularly in high-stakes applications. Existing methods often overlook semantic dependencies, relying on token-level probability measures that fail to capture structural relationships within the generated text. We propose GENUINE: Graph ENhanced mUlti-level uncertaINty Estimation for Large Language Models, a structure-aware framework that leverages dependency parse trees and hierarchical graph pooling to refine uncertainty quantification. By incorporating supervised learning, GENUINE effectively models semantic and structural relationships, improving confidence assessments. Extensive experiments across NLP tasks show that GENUINE achieves up to 29% higher AUROC than semantic entropy-based approaches and reduces calibration errors by over 15%, demonstrating the effectiveness of graph-based uncertainty modeling. The code is available at https://github.com/ODYSSEYWT/GUQ.

Figures

Figures reproduced from arXiv: 2509.07925 by Adithya Kulkarni, Dawei Zhou, Peter A. Beling, Tuo Wang, Tyler Cody, Yujun Yan.

Figure 1
Figure 1. Figure 1: An example highlighting the role of graph [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dependency parse tree example. Each tree [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The Overview of GENUINE, composed of three modules: (1) pooling based on grey-box features, (2) pooling based on white-box features, and (3) a learnable fusion process integrating both modules. Both the grey-box pooling process and white-box pooling process share the same graph structure, but differ in features, which leads to different fusion matrices. The structure fusion process helps better integrate v… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of AUROC on five datasets, four LLMs, and seven baselines. Error bars denote variance [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Scalability test on the node number and edge [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of ECE on five datasets, four LLMs, and seven baselines. Error bars denote variance over five [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of NLL on five datasets, four LLMs, and seven baselines. Error bars denote variance over [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of Brier scores on five datasets, four LLMs, and seven baselines. Error bars denote variance [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Experimental results on five datasets and seven baseline models on Llama2-13B model. Error bars denote [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Parameter analysis test on number of pooling [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bi-NAS: Towards Effective and Personalized Explanation for Recommender Systems via Bi-Level Neural Architecture Search

    cs.IR 2026-07 unverdicted novelty 3.0

    Bi-NAS applies bi-level NAS to search explanation architectures and LLMs for text generation, reporting gains in both recommendation accuracy and explanation effectiveness across four real-world datasets.

Reference graph

Works this paper leans on

59 extracted references · 39 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  2. [2]

    Kaikai An, Shuzheng Si, Helan Hu, Haozhe Zhao, Yuchi Wang, Qingyan Guo, and Baobao Chang. 2025. https://doi.org/10.18653/v1/2025.acl-short.79 Rethinking semantic parsing for large language models: Enhancing LLM performance with semantic hints . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pape...

    work page doi:10.18653/v1/2025.acl-short.79 2025

  3. [3]

    Filippo Maria Bianchi and Veronica Lachi. 2023. The expressive power of pooling in graph neural networks. Advances in neural information processing systems, 36:71603--71618

    work page 2023

  4. [4]

    Jari Bj \"o rne, Juho Heimonen, Filip Ginter, Antti Airola, Tapio Pahikkala, and Tapio Salakoski. 2009. https://www.aclweb.org/anthology/W09-1402 Extracting complex biological events with rich graph-based feature sets . In Proceedings of the B io NLP 2009 Workshop Companion Volume for Shared Task

    work page 2009

  5. [5]

    Ond r ej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12--58

    work page 2014

  6. [6]

    Xuelu Chen, Muhao Chen, Weijia Shi, Yizhou Sun, and Carlo Zaniolo. 2019. Embedding uncertain knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3363--3370

    work page 2019

  7. [7]

    Julius Cheng and Andreas Vlachos. 2024. https://aclanthology.org/2024.eacl-long.129/ Measuring uncertainty in neural machine translation with similarity-sensitive entropy . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2115--2128, St. Julian ' s, Malta. Associa...

    work page 2024

  8. [8]

    Inyoung Cheong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. 2024. (a) i am not a lawyer, but...: Engaging legal experts towards responsible llm policies for legal advice. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2454--2469

    work page 2024

  9. [9]

    I de Zarz \`a , J de Curt \`o , Gemma Roig, and Carlos T Calafate. 2023. Optimized financial planning: integrating individual and cooperative budgeting models with llm recommendations. AI, 5(1):91--114

    work page 2023

  10. [10]

    Micha \"e l Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29

    work page 2016

  11. [11]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050--5063

    work page 2024

  12. [12]

    David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael G\' o mez-Bombarelli, Timothy Hirzel, Al\' a n Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, page 2224–2232, C...

    work page 2015

  13. [13]

    Katrin Fundel, Robert Küffner, and Ralf Zimmer. 2006. https://doi.org/10.1093/bioinformatics/btl616 RelEx -- Relation extraction using dependency parse trees . Bioinformatics

    work page doi:10.1093/bioinformatics/btl616 2006

  14. [14]

    Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    doi:10.34740/kaggle/m/3301 2024

  15. [15]

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2001. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA

    work page 2001

  16. [16]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28

    work page 2015

  17. [17]

    Jos \'e Hern \'a ndez-Orallo, Peter A Flach, and C \`e sar Ferri Ramirez. 2011. Brier curves: a new cost-based visualisation of classifier performance. In Icml, pages 585--592

    work page 2011

  18. [18]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions . ACM Trans. Inf. Syst., 43(2)

    doi:10.1145/3703155 2025

  19. [19]

    Zhanming Jie, Aldrian Obaja Muis, and Wei Lu. 2017. Efficient dependency-guided named entity recognition. In Thirty-First AAAI Conference on Artificial Intelligence

    work page 2017

  20. [20]

    Zhijing Jin, Yuen Chen, Fernando Gonzalez, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Sch \"o lkopf, and Mona Diab. 2024. Analyzing the role of semantic representations in the era of large language models. arXiv preprint arXiv:2405.01502

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601--1611, Vancouver, Canada. Assoc...

    doi:10.18653/v1/p17-1147 2017

  22. [22]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. CoRR

    work page 2022

  23. [23]

    Amir Hosein Khasahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, and Quaid Morris. 2020. Memory-based graph networks. In International Conference on Learning Representations

    work page 2020

  24. [24]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations

    work page 2023

  25. [25]

    Chen Li, Yang Liu, Fei Liu, Lin Zhao, and Fuliang Weng. 2014. https://doi.org/10.3115/v1/D14-1076 Improving multi-documents summarization by sentence compression based on expanded constituent parse trees . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 691--701, Doha, Qatar. Association for Comp...

    work page doi:10.3115/v1/d14-1076 2014

  26. [26]

    Chin-Yew Lin and Franz Josef Och. 2004. https://doi.org/10.3115/1218955.1219032 Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics . In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics ( ACL -04) , pages 605--612, Barcelona, Spain

    work page arXiv 2004

  27. [27]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research

    work page 2022

  28. [28]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    Pith/arXiv arXiv 2024

  29. [29]

    Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. 2024 b . Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993

    Pith/arXiv arXiv 2024

  30. [30]

    Ning Liu, Songlei Jian, Dongsheng Li, and Hongzuo Xu. 2022. https://doi.org/10.1145/3511808.3557485 Unsupervised hierarchical graph pooling via substructure-sensitive mutual information maximization . In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, page 1299–1308, New York, NY, USA. Association for ...

    work page arXiv 2022

  31. [31]

    Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2024 c . https://arxiv.org/abs/2308.05374 Trustworthy llms: a survey and guideline for evaluating large language models' alignment . Preprint, arXiv:2308.05374

    Pith/arXiv arXiv 2024

  32. [32]

    Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations

    work page 2020

  33. [33]

    Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations

    work page 2021

  34. [34]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.557 S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004--9017, Singapore. Association for Computational...

    doi:10.18653/v1/2023.emnlp-main.557 2023

  35. [35]

    Diego Marcheggiani and Ivan Titov. 2017. https://doi.org/10.18653/v1/D17-1159 Encoding sentences with graph convolutional networks for semantic role labeling . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

    doi:10.18653/v1/d17-1159 2017

  36. [36]

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29

    work page 2015

  37. [37]

    Dimitrios P Panagoulias, Maria Virvou, and George A Tsihrintzis. 2024. Evaluating llm--generated multimodal diagnosis from medical images and symptom analysis. arXiv preprint arXiv:2402.01730

    Pith/arXiv arXiv 2024

  38. [38]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  39. [39]

    Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. 2024. Conformal language modeling. In The Twelfth International Conference on Learning Representations

    work page 2024

  40. [40]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. https://doi.org/10.1162/tacl_a_00266 C o QA : A conversational question answering challenge . Transactions of the Association for Computational Linguistics, 7:249--266

    doi:10.1162/tacl_a_00266 2019

  41. [41]

    Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. 2024. Llm-sr: Scientific equation discovery via programming with large language models. CoRR

    work page 2024

  42. [42]

    Zixing Song and Irwin King. 2022. Hierarchical heterogeneous graph attention network for syntax-aware summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11340--11348

    work page 2022

  43. [43]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  44. [44]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    Pith/arXiv arXiv 2023

  45. [45]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback . In Proceedings of the 2023 Conference on...

    doi:10.18653/v1/2023.emnlp-main.330 2023

  46. [46]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023 b . Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

    work page 2023

  47. [47]

    Francesco Tonolini, Nikolaos Aletras, Jordan Massiah, and Gabriella Kazai. 2024. https://doi.org/10.18653/v1/2024.findings-acl.728 B ayesian prompt ensembles: Model uncertainty estimation for black-box large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 12229--12272, Bangkok, Thailand. Association for Comp...

    doi:10.18653/v1/2024.findings-acl.728 2024

  48. [48]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    Pith/arXiv arXiv 2023

  49. [49]

    Boshi Wang, Xiang Yue, and Huan Sun. 2023. Can chatgpt defend its belief in truth? evaluating llm reasoning via debate. In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  50. [50]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling

    work page 2024

  51. [51]

    Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.538 Uncertainty quantification with pre-trained language models: A large-scale empirical analysis . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7273--7284, Ab...

    work page doi:10.18653/v1/2022.findings-emnlp.538 2022

  52. [52]

    Jiacheng Xu and Greg Durrett. 2019. https://doi.org/10.18653/v1/D19-1324 Neural extractive text summarization with syntactic compression . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3292--3303, Hong Kong, China. As...

    work page doi:10.18653/v1/d19-1324 2019

  53. [53]

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How powerful are graph neural networks? In International Conference on Learning Representations

    work page 2019

  54. [54]

    Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31

    work page 2018

  55. [55]

    Hao Yuan and Shuiwang Ji. 2020. Structpool: Structured graph pooling via conditional random fields. In Proceedings of the 8th international conference on learning representations

    work page 2020

  56. [56]

    Junyan Zhang, Yiming Huang, Shuliang Liu, Yubo Gao, and Xuming Hu. 2025. Do bert-like bidirectional models still perform better on text classification in the era of llms? arXiv preprint arXiv:2505.18215

    Pith/arXiv arXiv 2025

  57. [57]

    Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. 2020. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems, 33:12827--12836

    work page 2020

  58. [58]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  59. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.