Pith. sign in

REVIEW 4 major objections 4 minor 30 references

Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Measuring LLM uncertainty by aggregating pairwise similarities between sampled answers, rather than clustering them into semantic groups, detects hallucinations more accurately on long one-sentence outputs and provably generalizes…

desk verdict Promising, simple uncertainty score that improves on semantic entropy for longer outputs, but test-set hyperparameter selection and missing error bars mean the gains need a held-out check before being taken at face value. read the letter →

arxiv 2506.00245 v1 pith:LK4B6D5A submitted 2025-05-30 cs.LG cs.CL

Dang Nguyen , Ali Payani , Baharan Mirzasoleiman This is my paper
classification cs.LGcs.CL
keywords uncertaintyquantificationhallucinationdetectionsemanticentropynearestneighborpairwisesimilaritylog-sum-explargelanguagemodelsROUGE-L
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Semantic entropy detects LLM hallucinations by clustering reworded answers and measuring uncertainty over clusters, but it degrades as models produce longer one-sentence outputs, where each answer tends to land in its own cluster and the entropy becomes uninformative. The paper introduces Semantic Nearest Neighbor Entropy (SNNE), which replaces clustering with a log-sum-exp aggregation of pairwise similarities between sampled answers, thereby capturing both the spread within a semantic group and the distance between groups. It proves that with a specific similarity function SNNE reduces exactly to discrete semantic entropy, and its white-box variant reduces to semantic entropy, so the new estimator strictly generalizes the old one. Across question answering, summarization, and machine translation on two recent LLMs, SNNE and its white-box counterpart report higher AUROC and PRR scores than semantic entropy and other baselines for flagging hallucinated outputs.

What carries the argument

The central object is the Semantic Nearest Neighbor Entropy (SNNE) of Equation 3, $\mathrm{SNNE}(q) = -\frac{1}{n}\sum_{i=1}^{n}\log\sum_{j=1}^{n}\exp(f(a_i,a_j|q)/\tau)$, a nearest-neighbor entropy estimate in which the log-sum-exp acts as a smooth maximum so that outlier answers do not dominate the sum. The similarity function $f$ is plug-in: the experiments use ROUGE-L, with NLI entailment scores or sentence-embedding cosine similarity as alternatives, and the scale factor $\tau$ controls how sharply close answers count. Theorems 4.1 and 4.2 show that particular choices of $f$ reduce SNNE to discrete semantic entropy and its white-box variant to semantic entropy, which is what licenses the claim that pairwise-similarity aggregation subsumes cluster-based semantic entropy.

What would settle it

Construct a set of long one-sentence questions whose correct answers are phrased entirely differently from the model's sampled responses (high semantic equivalence, near-zero ROUGE-L); if SNNE then assigns high uncertainty to those correct answers, or its AUROC for hallucination detection falls below that of semantic entropy, the central claim fails.

Watch

Extended reading notes

Core claim

Hallucination in large language models can be flagged by estimating how uncertain the model is about its answer, and the standard uncertainty measure, semantic entropy, is built on a clustering step that groups semantically equivalent answers. As modern LLMs generate longer one-sentence responses, those answers increasingly fall into distinct clusters, so cluster-level entropy becomes nearly constant and loses its signal. The paper's central claim is that replacing clustering with pairwise similarity aggregation fixes this. Specifically, for each sampled answer SNNE computes the log-sum-exp of its similarity to all other sampled answers, averages over answers, and defines uncertainty as negative that average; WSNNE weights the same sum by normalized sequence probabilities in the white-box setting. Theorems show that when similarity is constant within a cluster and zero across clusters, SNNE equals discrete semantic entropy, and WSNNE equals semantic entropy, proving that the pairwise form strictly generalizes the cluster-based form. With ROUGE-L as the similarity function, the method reports higher AUROC on question answering and higher PRR on summarization and translation than semantic entropy and several baselines.

Load-bearing premise

The method assumes that ROUGE-L overlap between sampled answers faithfully reflects semantic similarity for long one-sentence outputs, so low overlap signals genuine uncertainty rather than mere paraphrase variation; if that proxy fails, the entropy estimate stops tracking hallucination.

Editorial extensions

If this is right

  • Hallucination detection on long one-sentence outputs such as summaries and translations improves without needing access to the model's internal probabilities, using only a handful of sampled answers per question.
  • Because SNNE reduces to discrete semantic entropy and WSNNE to semantic entropy under specific similarity choices, pipelines already built on semantic entropy can be upgraded without changing their operational meaning.
  • The plug-in similarity function and temperature allow the estimator to be adapted to other output formats, such as code or mathematical expressions, by swapping the similarity function $f$.
  • The log-sum-exp aggregation makes SNNE less sensitive to outlier answers than graph-based methods that sum pairwise similarities, which the paper argues improves reliability of the entropy estimate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A unification of uncertainty quantification methods may follow: if pairwise-similarity aggregation subsumes semantic entropy, then future refinements to clustering-based entropy could be recast as particular similarity choices, giving a common language for comparing methods.
  • The heavy reliance on a single similarity function suggests an ensemble of lexical and embedding-based similarities might further stabilize the estimate on paraphrastic outputs; the paper tests each similarity alone, not their combination.
  • For multi-sentence outputs, applying SNNE at the sentence level and aggregating, along the lines LUQ aggregates atomic scores, is a natural next experiment that would test whether the pairwise principle extends beyond one-sentence generations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SNNE, a black-box uncertainty quantification method for LLMs that computes, for each of n sampled model outputs, a log-sum-exp of pairwise semantic similarities with all other outputs, scaled by a temperature τ. A white-box variant WSNNE reweights the outer sum by normalized sequence probabilities. The authors argue that SE and DSE ignore intra- and inter-cluster similarity and therefore degrade on longer one-sentence generations, whereas SNNE captures both. They prove (Theorems 4.1–4.2) that DSE and SE are recovered as special cases under specially chosen similarity functions, and they report experiments on QA, summarization, and translation with several LLMs, comparing against a wide range of baselines and claiming consistent improvements.

Significance. If the empirical gains are robust, SNNE is a simple, intuitive, and computationally efficient (O(n^2)) alternative to SE for long-generation hallucination detection, and the white-box extension adds flexibility. The theoretical unification of SE/DSE as special cases is elegant, and the code is made available. However, the current evaluation does not yet establish the out-of-sample superiority claim because key hyperparameters are selected on the test sets and the main tables lack error bars. The conceptual theoretical results, while sound, do not speak to the specific choice of ROUGE-L as the similarity function.

major comments (4)
  1. [Appendix B, Appendix C.2, Tables 5–6] Test-set hyperparameter selection undermines the central 'consistently outperforms' claim. In Appendix B, the authors state 'We select the best scale factor τ in Eq 3 and 4 from {0.1,1,10,100}. For similarity function f, we use ROUGE-L score.' All main results (Tables 2–4, Figures 2–3) are computed on the same datasets used for this selection. Table 5 shows that the choice of f is consequential: on summarization, ROUGE-L gives PRR 0.26 versus 0.22 for entail and embed; on translation, 0.63 versus 0.60 and 0.62. Thus the reported gains could be partly an artifact of choosing the similarity function that best matches the test tasks. Please provide a held-out validation split for f and τ selection (or, failing that, report results for all candidate choices for every dataset) before the superiority claim can be accepted as an out-of-sample statement.
  2. [Tables 2–4] Main results are reported without error bars despite three repetitions. Appendix B notes 'We conduct each experiment three times using NVIDIA RTX A6000 GPUs,' but Tables 2–4 give only point estimates. Without standard deviations or confidence intervals, differences such as SNNE 0.84 vs. SE 0.80 on Phi-3 QA or SNNE 0.26 vs. SE 0.20 on summarization cannot be assessed for statistical significance. Please report mean ± std (or CIs) for all main tables, as is already done in Appendix C.2.
  3. [Table 4, Appendix B] The evaluation uses ROUGE-L both as the similarity function inside SNNE and as the correctness measure for the PRR results on summarization and translation. This dual use creates a potential positive bias for SNNE: its uncertainty score is directly tied to the same lexical-overlap measure that defines correctness, whereas baselines based on entailment or embeddings are not so aligned. While the paper does report BERTScore as a second correctness measure, the same concern applies to the choice of f. Please discuss this potential circularity and, ideally, add a correctness measure independent of the similarity function (e.g., human-annotated correctness or an entailment-based label) to confirm the gains are not an artifact of metric alignment.
  4. [Theorems 4.1–4.2, Section 4] The theoretical 'generalization' results do not support the empirical advantage claimed for ROUGE-L. Theorems 4.1 and 4.2 construct ad-hoc similarity functions (constant intra-cluster similarity with zero inter-cluster similarity, or a probability-weighted variant) under which SNNE/WSNNE algebraically reduce to DSE/SE. These functions are not the ROUGE-L similarity used in the experiments, so the theorems only demonstrate conceptual generality, not that ROUGE-L-based SNNE should outperform SE. The paper should avoid implying that the theorems explain the empirical gains; at most they position (W)SNNE as a flexible framework.
minor comments (4)
  1. [Appendix B] There is an apparent typo: 'sample 10 answers at a high temperature (T= 10)' should likely read (T= 1.0), since Figure 4 and the text in Section 5.1 state that the generation setting is temperature 1.0 adopted from Farquhar et al. (2024).
  2. [Appendix C.2] The citation for Qwen2-7B-instruct points to 'Towards general text embeddings with multi-stage contrastive learning' (Li et al., 2023), which appears to be the wrong reference; please verify the correct source.
  3. [Appendix C.2] In the sentence 'The superior of ROUGE-L also explains the good performance of LexSim', 'superior' should be 'superiority'.
  4. [Figure 1] The right panel would benefit from error bars or a statement about whether the curve is averaged over multiple seeds, since the main text emphasizes 'consistently'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claim is tested against external correctness labels, and the SE-recovery theorems are algebraic special cases rather than self-referential predictions.

full rationale

The paper's central claim is empirical: SNNE/WSNNE estimate hallucination risk from sampled outputs and are evaluated against external correctness labels (F1 for QA; ROUGE-L/BERTScore for TS/MT). The uncertainty scores are not defined from those labels, so the outperformance claim is not forced by construction. Theorems 4.1 and 4.2 are genuine algebraic reductions showing that, under specially constructed similarity functions f (constant intra-cluster similarity with zero inter-cluster similarity, or probability-weighted intra-cluster similarity), SNNE/WSNNE equal DSE/SE; these are existence results about the estimator family, not derivations of the empirical gains, and the f used in experiments (ROUGE-L) differs from the f in the theorems. There is no load-bearing self-citation: the cited baselines (SE, KLE, graph methods, LUQ) are external prior work, and the authors make no uniqueness claim resting on their own prior papers. The main validity concerns, namely selecting tau on the test tasks and using ROUGE-L both as the pairwise similarity and as a correctness metric for summarization and translation, are potential sources of optimistic bias but are not circular in the definitional sense, because the pairwise similarity input and the reference-based correctness label are distinct quantities, and tau selection does not make the reported AUROC or PRR equal to a fitted value by construction. These concerns belong to experimental methodology rather than to circularity of the derivation chain.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger is small: no new entities, one tuned continuous hyperparameter, one post-hoc discrete choice, and three domain assumptions about sampling, similarity proxies, and the entropy aggregation form. The central claim rests mostly on the empirical validity of ROUGE-L as a similarity proxy and on the representativeness of the sampled answers.

free parameters (2)
  • scale factor tau = 1 (default; tuned from {0.1,1,10,100})
    Hyperparameter controlling the softness of the LogSumExp aggregation; selected on the test datasets, so the reported numbers partly depend on test-set tuning.
  • similarity function f = ROUGE-L
    A discrete modeling choice, selected based on ablation on the same evaluation tasks; not a numeric fit but a post-hoc selection.
assumptions (4)
  • domain assumption Pairwise semantic similarities aggregated by LogSumExp yield a valid estimator of output uncertainty.
    Section 4 Eq 3 introduces SNNE with this principle; no theoretical justification is given beyond analogy to nearest-neighbor entropy.
  • domain assumption Sampling n=10 answers at temperature T=10 produces a representative sample of the model's semantic output distribution.
    Section 5.1 Generation setting; if the sample is not representative, the entropy estimates are biased.
  • domain assumption ROUGE-L lexical overlap approximates semantic similarity for long one-sentence outputs.
    Section 5.2 and Appendix C.2 use ROUGE-L as the default f; the paper's own ablation shows this is task-dependent (Table 5).
  • domain assumption Semantic clusters defined by bidirectional entailment (from SE) exist and are meaningful for the theoretical equivalence.
    Theorems 4.1 and 4.2 rely on the SE clustering; the equivalence is conditional on that clustering being well-defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity." pith.science (2026). https://pith.science/paper/LK4B6D5A

@misc{pith2026250600245,
  author       = {Pith},
  title        = {Pith review of: Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LK4B6D5A}},
  note         = {Machine review of arXiv:2506.00245}
}
read the original abstract

Hallucination in large language models (LLMs) can be detected by assessing the uncertainty of model outputs, typically measured using entropy. Semantic entropy (SE) enhances traditional entropy estimation by quantifying uncertainty at the semantic cluster level. However, as modern LLMs generate longer one-sentence responses, SE becomes less effective because it overlooks two crucial factors: intra-cluster similarity (the spread within a cluster) and inter-cluster similarity (the distance between clusters). To address these limitations, we propose a simple black-box uncertainty quantification method inspired by nearest neighbor estimates of entropy. Our approach can also be easily extended to white-box settings by incorporating token probabilities. Additionally, we provide theoretical results showing that our method generalizes semantic entropy. Extensive empirical results demonstrate its effectiveness compared to semantic entropy across two recent LLMs (Phi3 and Llama3) and three common text generation tasks: question answering, text summarization, and machine translation. Our code is available at https://github.com/BigML-CS-UCLA/SNNE.

Figures

Figures reproduced from arXiv: 2506.00245 by the authors.

Figure 1
Figure 1. Generated answers of Llama-3.1-8B on SQuAD. (left) The Spearman’s rank correlation coefficient between the average output length and the number of semantic clusters is 0.83, indicating a high correlation. (right) The AUROC difference between SNNE and SE when calculating on subsets of questions with different number of clusters. SNNE is consistently more distinctive than SE, especially when the number of clusters is … view at source ↗
Figure 2
Figure 2. Average AUROC score of Llama-3.1-8B and Phi-3-mini on 5 QA tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average PRR score of Phi-3-mini on 2 text summarization and 2 machine translation tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Effect of number of generated answers and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 10 canonical work pages

  1. [4]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu

    Inside: Llms’ internal states retain the power of hallu- cination detection.arXiv preprint arXiv:2402.03744. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu

    arXiv

  2. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    Shifting attention to relevance: To- wards the predictive uncertainty quantification of free-form large language models.arXiv preprint arXiv:2307.01379. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    arXiv

  3. [6]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov

    arXiv

  4. [7]

    InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 446–461, Singapore

    LM-polygraph: Uncer- tainty estimation for language models. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 446–461, Singapore. Association for Computational Linguistics. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal

    work page 2023

  5. [9]

    Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, San- woo Lee, and Yunfang Wu

    How good are gpt models at ma- chine translation? a comprehensive evaluation.arXiv preprint arXiv:2302.09210. Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, San- woo Lee, and Yunfang Wu

    arXiv

  6. [10]

    arXiv preprint arXiv:2410.15326

    A survey of un- certainty estimation in llms: Theory meets practice. arXiv preprint arXiv:2410.15326. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung

    arXiv

  7. [11]

    arXiv preprint arXiv:2310.06825

    Mistral 7b (2023). arXiv preprint arXiv:2310.06825. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer

    arXiv 2023

  8. [14]

    Chin-Yew Lin

    Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281. Chin-Yew Lin

    arXiv

Show all 30 references
  1. [15]

    Andrey Malinin, Anton Ragni, Kate Knill, and Mark Gales

    Uncertainty esti- mation in autoregressive structured prediction.arXiv preprint arXiv:2002.07650. Andrey Malinin, Anton Ragni, Kate Knill, and Mark Gales

    2002 arXiv

  2. [16]

    InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, On- line

    On faithfulness and factu- ality in abstractive summarization. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, On- line. Association for Computational Linguistics. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Ni...

    1906

  3. [17]

    Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar

    Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316. Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar

    arXiv

  4. [19]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal

    Kernel language en- tropy: Fine-grained uncertainty quantification for llms from semantic similarities.arXiv preprint arXiv:2405.20003. Arkil Patel, Satwik Bhattamishra, and Navin Goyal

    arXiv

  5. [21]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al

    Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024a. Gemini 1.5: Unlocking multimodal...

    arXiv

  6. [22]

    Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, et al

    Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, et al

    arXiv

  7. [23]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu

    Luq: Long-text uncertainty quantifica- tion for llms.arXiv preprint arXiv:2403.20279. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu

    arXiv

  8. [25]

    InInternational Conference on Learning Representations

    Bertscore: Eval- uating text generation with bert. InInternational Conference on Learning Representations. A Proofs A.1 Proof of Theorem 4.1 Proof.We define the similarity functionfas f(aaai, aaaj|qqq) = ( τlog( 1 n ),if∃ks.t.a aai, aaaj ∈C k −∞,otherwise (5) In other words, w...

    2023

  9. [26]

    and TriviaQA (Joshi et al., 2017)), general knowledge from Google search (NaturalQuestion (Kwiatkowski et al., 2019)), sim- ple math problems ( Svamp (Patel et al., 2021)), biology and medicine ( BioASQ (Krithara et al., 2023)). For text summarization, we adopt the ab- stracti...

    2017

  10. [27]

    Task Instruction QA Answer the following question as briefly as possible

    and black-box UQ methods including KLEheat (Nikitin et al., Table 1: Instruction prompts for different tasks. Task Instruction QA Answer the following question as briefly as possible. XSUM Here’s the text and it’s short one-sentence summary. AESLC Write a short subject line fo...

  11. [28]

    We use 5-shot in-context demonstrations for QA tasks and instruct the model to produce short answers following the brief prompt from SE

    to estimate uncer- tainty. We use 5-shot in-context demonstrations for QA tasks and instruct the model to produce short answers following the brief prompt from SE. For TS and MT tasks, we employ a 0-shot setting and adopt the instructions from LM-Polygraph. Table 1 provides a ...

    2024

  12. [30]

    Ta- ble 5 summarizes the PRR score on TS and MT tasks

    which ranked 1st in the MTEB benchmark (Muennighoff et al., 2022). Ta- ble 5 summarizes the PRR score on TS and MT tasks. ROUGE-L results in the best performance except for translation task with BERTScore in which ROUGE-L ties with embed. The superior of ROUGE-L also explains ...

    2022

  13. [2004]

    Hyperparameter setting.We select the best scale factor τ in Eq 3 and 4 from {0.1,1,10,100}

    and BERTScore (Zhang* et al., 2020), as in LM- Polygraph. Hyperparameter setting.We select the best scale factor τ in Eq 3 and 4 from {0.1,1,10,100} . For similarity function f, we use ROUGE-L score. Implementation details.For calculating ROUGE-L, we use the Python implementio...

    2020

  14. [2014]

    InProceedings of the ninth workshop on statistical machine translation, pages 12–58

    Findings of the 2014 workshop on statistical machine translation. InProceedings of the ninth workshop on statistical machine translation, pages 12–58. Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanz...

    2014

  15. [2017]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion.arXiv preprint arXiv:1705.03551. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al

    arXiv

  16. [2018]

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen

    Don’t give me the details, just the summary! topic-aware convolutional neural net- works for extreme summarization.arXiv preprint arXiv:1808.08745. Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen

    arXiv

  17. [2019]

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q

    This email could save your life: Introducing the task of email subject line generation.arXiv preprint arXiv:1906.03497. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi

    1906 arXiv

  18. [2020]

    Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla

    Deberta: Decoding-enhanced bert with disentangled attention.arXiv preprint arXiv:2006.03654. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla

    2006 arXiv

  19. [2021]

    Pranav Rajpurkar, Robin Jia, and Percy Liang

    Are nlp models really able to solve simple math word problems?arXiv preprint arXiv:2103.07191. Pranav Rajpurkar, Robin Jia, and Percy Liang

    arXiv

  20. [2022]

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras

    arXiv

  21. [2023]

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye

    Sparks of artificial general intelli- gence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye

    arXiv

  22. [2024]

    Jan Beirlant, Edward J Dudewicz, László Györfi, Ed- ward C Van der Meulen, et al

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219. Jan Beirlant, Edward J Dudewicz, László Györfi, Ed- ward C Van der Meulen, et al

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.