Pith. sign in

REVIEW 3 major objections 5 minor 47 references

FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Factuality of interpretive AI claims about contact center calls can be judged automatically with a 3D prompt, reaching mean F1 0.86 without fine-tuning.

desk verdict Useful, narrowly scoped benchmark for interpretive factuality in contact center transcripts, but the 0.82 human agreement was measured after removing ambiguous cases, so the headline 0.86 F1 rests on labels whose independent reliability is not established. read the letter →

arxiv 2508.00889 v1 pith:GQUZA7RJ submitted 2025-07-26 cs.CL cs.AIcs.LG

Hagyeong Shin , Binoy Robin Dalal , Iwona Bialynicka-Birula , Navjot Matharu , Ryan Muir , Xingwei Yang , Samuel W. K. Wong This is my paper
classification cs.CLcs.AIcs.LG
keywords LLM-as-a-Judgefactualityevaluationhallucinationdetectioncontactcenterconversationsinterpretiveclaimshumanannotationagreementtest-timecomputebenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contact center AI tools generate analytical claims—"the customer chose the plan for specific dentist coverage"—that cannot be fact-checked by finding an explicit evidence sentence in the transcript. The paper tries to make such judgments reliable by introducing the 3D paradigm (Decompose, Decouple, Detach): break the claim into minimal units, separate concrete words from subjective interpretations, and then verify each unit and the relations between units using explicit or implicit evidence in the conversation. This same guideline is used to label a new public benchmark, FECT, of 410 conversation-claim pairs, and then converted into a prompt for LLM judges. The paper reports that a reasoning model with the 3D prompt plus test-time compute reaches mean F1 0.86 for detecting non-factual claims, without fine-tuning or extensive prompt search. If the labels are trustworthy, this is a path to automated auditing of enterprise AI analyses that previously required slow, expensive human review.

What carries the argument

The load-bearing object is the 3D paradigm, a four-step evaluation routine: (1) Decompose the claim into minimal informational units; (2) Decouple those units into concrete words and subjective-interpretation words; (3) verify each unit against explicit mentions or minimal implicit evidence; (4) Detach the relation between words from their meanings and verify the relation—who did what to whom, and why—using explicit or implicit evidence. The same routine is encoded in a human annotation guideline and in the LLM-judge prompt, so that the human labels and the model judgments are aligned on the same granular steps rather than on an unstated notion of "factual." The paradigm is what converts an inherently subjective judgment into a checkable procedure, and it is the component that the paper varies (3D vs. BASIC prompts) to isolate the effect of structuring the evaluation process.

What would settle it

A second, independent annotation team could label all 410 conversation-claim pairs using the published 3D guideline without pre-filtering any items, then compare their majority labels and pairwise agreement to the published labels; if their agreement with the published labels is well below 0.82, or if their own agreement on the final 410 is low, the benchmark's ground truth—and the F1 scores derived from it—would not be reliable.

Watch

Extended reading notes

Core claim

The paper's central claim is that factuality of interpretive claims can be grounded in a shared, linguistically-informed evaluation routine, and that aligning both human annotators and LLM judges on that routine is enough to get usable automatic evaluation. The routine is the 3D paradigm: decompose a claim into atomic claims; decouple words with concrete meanings from words expressing subjective interpretation; detach the relation between words from their meanings; and verify each component with explicit mentions or, where needed, minimal implicit evidence. Applying the 3D guideline to human annotation raised pairwise inter-annotator agreement from 0.28 in unguided labeling to 0.58, and the final FECT benchmark of 410 pairs (345 factual, 65 non-factual) reports inter-annotator agreement of 0.82 after ambiguous pairs were excluded. On that benchmark, the best configuration—the o1 reasoning model with the 3D prompt that requests intermediate reasoning tokens—achieves mean F1 0.86 (std 0.02) over 10 runs, and the paper argues that no fine-tuning or extensive prompt optimization was needed to get there. The paper also finds that reasoning models generally outperform non-reasoning models, that adding test-time compute helps frontier non-reasoning models approach reasoning models, and that small models are often hurt by test-time compute.

Load-bearing premise

The load-bearing premise is that the reported 0.82 inter-annotator agreement, measured after removing conversation-claim pairs where raters disagreed, actually validates the factuality labels in the final 410-pair benchmark; if agreement on the excluded items was low and those items are the hard cases, the benchmark's ground truth is a survivorship-selected subset whose reliability remains unmeasured.

Editorial extensions

If this is right

  • If the 3D paradigm works as reported, an off-the-shelf reasoning LLM can audit contact-center AI claims at mean F1 0.86 with no fine-tuning, making automated factuality checks practical for enterprise deployment.
  • Structuring the prompt with granular 3D steps plus test-time compute lets frontier non-reasoning models such as GPT-4.1 and GPT-4o reach F1 0.84–0.85, nearly matching the best reasoning-model score.
  • The FECT benchmark deliberately contains only conversation-claim pairs whose factuality humans could agree on; model performance is therefore a measure of evidence-driven judgment, not of handling genuinely ambiguous interpretations.
  • Because the 3D prompt was format-optimized only for the o1 model, other reasoning models' scores (0.80–0.83 with BASIC prompts) may understate what the same recipe could achieve with model-specific formatting.
  • Factuality in this evaluation is groundedness of the claim in the referenced conversation alone; whether the claim actually answers the user's analysis task is treated as an orthogonal requirement outside the benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because ambiguous pairs are excluded from FECT, the benchmark measures agreement-selected tasks; an explicit report of pre-vote agreement on the final 410 pairs would directly quantify label reliability.
  • FECT uses synthetic conversations that mimic real contact-center patterns, so the headline F1 should be re-measured on real transcripts; the paper argues the distributions are similar but does not test that transfer.
  • The 3D recipe is not tied to contact centers: the same decompose–decouple–detach routine could be applied to interpretive claims in medical, legal, or financial summarization, where ground truth is likewise absent.
  • The smaller-model finding—test-time compute can hurt—is a caution for practitioners: adding reasoning tokens helps only if the model has capacity to do the multi-step verification, so the 3D prompt should be paired with TTC selectively.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FECT, a benchmark for evaluating the factuality of interpretive, single-sentence claims generated by LLMs about contact center conversation transcripts. To create reliable labels, the authors propose a 3D (Decompose, Decouple, Detach) annotation paradigm and apply the same paradigm in the prompts of LLM judges. They exclude conversation-claim pairs on which human annotators cannot agree, leaving 410 pairs (345 factual, 65 non-factual), and report an inter-annotator agreement of 0.82 after this exclusion. In the main experiment, 17 LLMs are evaluated under four prompt variants, each run 10 times; the best mean F1 is 0.86, achieved by o1 with the 3D_WITH_TTC prompt. The paper concludes that aligning LLM judges with the 3D paradigm yields strong factuality-detection performance without fine-tuning or extensive prompt optimization.

Significance. If the ground-truth labels on the final 410 pairs are reliable, this is a valuable contribution: it addresses an under-studied setting where claims are interpretive rather than directly extractable, and it provides a public benchmark for that setting. The experimental reporting is a notable strength: 10 runs per condition, means and standard deviations, and a prompt ablation are all clearly presented. The 3D rubric is concrete and linguistically motivated, and the paper honestly acknowledges that ambiguous cases are left out. The main unresolved risk is that the reliability evidence for the benchmark labels is incomplete, because the reported agreement is computed only on the subset that survived ambiguity exclusion, and no agreement is reported on the final 410 pairs themselves.

major comments (3)
  1. [Section 2.4, Table 1] The reliability of the benchmark's ground-truth labels is not established by the reported 0.82. Section 2.4 explicitly says this score was achieved after excluding ambiguous tasks, and Table 1 shows that 84 pairs were excluded (31 sentiment, 53 relation). The paper does not report any agreement statistic computed on the final 410 pairs, nor does it specify how many annotators labeled each final pair, whether the 0.82 is Cohen's kappa or percent agreement, or whether final labels were produced by voting, adjudication, or discussion. Because the headline F1 of 0.86 is measured against those very labels, a reader cannot tell whether FECT is a well-grounded benchmark or an easy residue of the original 510 pairs whose reliability was never measured. Please compute and report agreement on the final 410 pairs before any reconciliation, and describe the labeling protocol for those pairs in detail.
  2. [Section 2.3, Table 1, Section 4] The exclusion process removes precisely the subjective dimensions the 3D paradigm is designed to handle: 31 sentiment pairs and 53 relation pairs. The demonstrated agreement is therefore partly built into the construction of the benchmark, and the F1 result is scoped to the remaining unambiguous subset. This is acknowledged as future work, but the abstract and Section 1 present the 0.82 agreement and the 0.86 F1 without that caveat. Please add an explicit statement that both numbers apply only to the non-ambiguous subset, and report the LLM judges' predictions on the 84 excluded pairs as a diagnostic of where the 3D paradigm fails; otherwise the claim that the 3D paradigm grounds factuality labels overstates what is currently shown.
  3. [Section 3.1, Section 3.2] Because the 3D prompt was optimized in structure and formatting for the o1 model, the result that o1 achieves the best F1 is not an unbiased comparison across models. The paper discloses this, but the central claim that 'aligning LLM-judges on the 3D paradigm can achieve a mean F1 of 0.86 without extensive prompt optimizations' should be qualified with the fact that the prompt was specifically tuned for one model, and the other 16 models were evaluated with that same o1-optimized prompt. Please either report results with a prompt that is not tuned to any model, or explicitly frame the experiment as a transfer test of an o1-optimized prompt.
minor comments (5)
  1. [Abstract and Section 1] The abstract and the contributions list state the inter-annotator agreement of 0.82 without noting that it was computed after excluding ambiguous tasks; please add the exclusion caveat in both places, as it is essential for interpreting the number.
  2. [Table 1] The table header is confusing: the columns under 'Agreement achieved / Agreement not achieved' are labeled 'FECT', 'Sentiment', and 'Relation', but the numeric entries for the latter two are exclusion counts. Please use explicit column names such as 'Final benchmark', 'Excluded: sentiment', and 'Excluded: relation', and add a row for the total number of excluded pairs.
  3. [Section 2.1] The paper states that the synthetic conversations 'exhibit the same properties and challenges' as the real ones, but no quantitative evidence is given; please add a brief comparison of label distributions or claim characteristics between the synthetic and real sets.
  4. [Footnote 4] Footnote 4 contains a duplicated article: 'the the synthetic conversation dataset'; please correct the typo.
  5. [Section 4] The phrase 'c.f.' should be 'cf.' or 'see', and the references to 'Claude-Sonnet' are ambiguous because there are multiple Claude-Sonnet models; please specify which model or model family is meant.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the F1 scores are measured against human labels, and the disclosed post-hoc exclusions and prompt tuning are validity concerns, not input-output equivalences.

full rationale

The paper contains no derivation chain that reduces a claimed prediction to its own inputs. The headline F1 of 0.86 (Table 2) is an empirical measurement of 17 LLMs against 410 human-labeled conversation-claim pairs; the labels were produced by human annotators following a written guideline, and each LLM is evaluated by comparing its generated label with those human labels. The 3D prompt shares its decomposition rubric with the human annotation guideline, so the evaluation can be read as measuring how well an LLM follows the authors' rubric; however, this does not make the F1 score equal to the input by construction, because the model must still apply the steps to unseen conversations and can fail at any sub-step. The reported inter-annotator agreement of 0.82 is explicitly described as having been achieved after excluding ambiguous tasks (Section 2.4), which is a transparently reported selected-data statistic and a legitimate reliability concern, but not a logical circularity. Similarly, the statement that the 3D prompt was optimized for o1's structure and formatting (Section 3.1) indicates possible test-set selection or overfitting, yet the paper discloses this and does not relabel a fitted parameter as an independent prediction. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation steps were found. Therefore, under the hard rule requiring a specific reduction for a circularity finding, no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numerical free parameters are fitted in this paper; the method consists of prompt and dataset design choices. The load-bearing assumptions are the representativeness of synthetic data, the validity of the post-exclusion labels, and the applicability of the linguistic rubric. No new physical or mathematical entities are postulated.

assumptions (4)
  • domain assumption Synthetic conversations in FECT exhibit the same properties and challenges as real contact center conversations.
    Section 2.1 states the synthetic dataset was created to exhibit the same properties and challenges as real conversations, and Section 2.4 claims the label distribution is similar to real customer use cases. The transferability of all experimental conclusions to production systems rests on this unverified equivalence.
  • domain assumption Excluding ambiguous conversation-claim pairs yields a set of labels that can serve as ground truth for evaluating LLM-judges.
    Section 2.3 and Table 1 show that pairs on which humans did not agree were omitted from FECT, and Section 2.4 presents the remaining 410 pairs as the benchmark. This assumes the remaining labels are objectively determinable, though no agreement score is reported on the final 410 pairs.
  • domain assumption The 3D linguistic rubric (decompose, decouple, detach) is a valid and consistently applicable standard for factuality.
    Section 2.2 grounds the rubric in compositional semantics, but the operational definitions of "implicit evidence" and "inferring the reason behind an action" (Appendix B.1) leave room for judgment; both human annotators and LLMs must apply them consistently.
  • domain assumption The factuality of a claim can be determined solely from the conversation, independent of whether the claim answers the analysis task.
    Footnote 2 explicitly scopes factuality to the claim and referenced conversation, which is a reasonable scope but narrows the benchmark's connection to real task quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts." pith.science (2026). https://pith.science/paper/GQUZA7RJ

@misc{pith2026250800889,
  author       = {Pith},
  title        = {Pith review of: FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQUZA7RJ}},
  note         = {Machine review of arXiv:2508.00889}
}
read the original abstract

Large language models (LLMs) are known to hallucinate, producing natural language outputs that are not grounded in the input, reference materials, or real-world knowledge. In enterprise applications where AI features support business decisions, such hallucinations can be particularly detrimental. LLMs that analyze and summarize contact center conversations introduce a unique set of challenges for factuality evaluation, because ground-truth labels often do not exist for analytical interpretations about sentiments captured in the conversation and root causes of the business problems. To remedy this, we first introduce a \textbf{3D} -- \textbf{Decompose, Decouple, Detach} -- paradigm in the human annotation guideline and the LLM-judges' prompt to ground the factuality labels in linguistically-informed evaluation criteria. We then introduce \textbf{FECT}, a novel benchmark dataset for \textbf{F}actuality \textbf{E}valuation of Interpretive AI-Generated \textbf{C}laims in Contact Center Conversation \textbf{T}ranscripts, labeled under our 3D paradigm. Lastly, we report our findings from aligning LLM-judges on the 3D paradigm. Overall, our findings contribute a new approach for automatically evaluating the factuality of outputs generated by an AI system for analyzing contact center conversations.

Figures

Figures reproduced from arXiv: 2508.00889 by the authors.

Figure 1
Figure 1. Most factuality evaluation tasks require fact-checking evidence, as shown on the left [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our methodology to develop a reliable LLM-judge for factuality evaluations. We start with Phase 1 where [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The visualization of our 3D guideline used by human evaluators in annotating the factuality of claims. Annotators [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Claim 3 exemplifies tasks that could achieve human agreement in factuality evaluation, despite the fact that the paired [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Means and 95% CIs of F1s achieved with reasoning models and non-reasoning models and with four prompts. Dashed [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: An example of a typical LLM hallucination observed [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Means and 95% CIs of precisions achieved with [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Means and 95% CIs of recalls achieved with [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 21 canonical work pages

  1. [1]

    Eunice Akani, Benoit Favre, Frederic Bechet, and Romain Gemignani. 2024. In- creasing faithfulness in human-human dialog summarization with Spoken Lan- guage Understanding tasks. doi:10.48550/arXiv.2409.10070 arXiv:2409.10070

    arXiv doi:10.48550/arxiv.2409.10070 2024

  2. [2]

    Anthropic. 2024. Claude 3.5 Sonnet Model Card Addendum. https://www- cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_ Claude_3_Addendum.pdf

    work page 2024

  3. [3]

    Anthropic. 2025. Claude 3.7 Sonnet System Card. https://assets.anthropic.com/ m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf

    work page 2025

  4. [4]

    Anthropic. 2025. System Card: Claude Opus 4 & Claude Sonnet 4. https://www- cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf

    work page 2025

  5. [5]

    Lora Aroyo and Chris Welty. 2015. Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine 36, 1 (March 2015), 15–24. doi:10. 1609/aimag.v36i1.2564

    work page 2015

  6. [6]

    Forrest Sheng Bao, Miaoran Li, Renyi Qu, Ge Luo, Erana Wan, Yujia Tang, Weisi Fan, Manveer Singh Tamber, Suleman Kazi, Vivek Sourabh, Mike Qi, Ruixuan Tu, Chenyu Xu, Matthew Gonzales, Ofer Mendelevitch, and Amin Ahmad. 2024. FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs. doi:10.48550/arXiv.2410.13210 arXiv:2410.13210

    work page Pith review arXiv doi:10.48550/arxiv.2410.13210 2024

  7. [7]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexan- der Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2024. L...

    arXiv doi:10.48550/arxiv.2406 2024

  8. [8]

    Cheng-Han Chiang and Hung-yi Lee. 2023. A Closer Look into Automatic Evaluation Using Large Language Models. doi:10.48550/arXiv.2310.05657 arXiv:2310.05657 [cs]

    arXiv doi:10.48550/arxiv.2310.05657 2023

Show all 47 references
  1. [9]

    DeepSeek. 2025. DeepSeek-R1 Release. https://api-docs.deepseek.com/news/ news250120

    2025

  2. [10]

    Google. 2025. Gemini 2.5 Flash Preview Model Card. https://storage.googleapis. com/model-cards/documents/gemini-2.5-flash-preview.pdf

    2025

  3. [11]

    Google. 2025. Gemini 2.5 Pro Preview Model Card. https://storage.googleapis. com/model-cards/documents/gemini-2.5-pro-preview.pdf

    2025

  4. [12]

    Gordon, Michelle S

    Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In CHI Conference on Human Factors in Computing Systems . ACM, New Or...

    2022

  5. [13]

    Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Nenkov Georgiev, Jiahui Geng, Iryna Gurevych, and Preslav Nakov. 2024. OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processin...

    2024 doi

  6. [14]

    Jeremy Greer, Manos Koukoumidis, Konstantinos Aisopos, and Michael Schuler

  7. [15]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo

  8. [16]

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen

  9. [17]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. https://arxiv.org/abs/2109.07958v2

    2021 arXiv

  10. [18]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguist...

    2023 doi

  11. [19]

    Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. 2024. HalluDial: A Large-Scale Benchmark for Automatic Dialogue- Level Hallucination Evaluation. doi:10.48550/arXiv.2406.07070 arXiv:2406.07070 version: 1

    2024 doi arXiv

  12. [20]

    Meta. 2025. Llama 4. https://github.com/meta-llama/llama-models/blob/main/ models/llama4/MODEL_CARD.md

    2025

  13. [21]

    Dasha Metropolitansky and Jonathan Larson. 2025. Towards Effective Extraction and Evaluation of Factual Claims. doi:10.48550/arXiv.2502.10855 arXiv:2502.10855 [cs]

    2025 doi arXiv

  14. [22]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing . Association ...

    2018 doi

  15. [23]

    OpenAI. 2024. GPT-4.1 nano. https://platform.openai.com/docs/models/gpt-4.1- nano

    2024

  16. [24]

    OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence. https://openai. com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  17. [25]

    OpenAI. 2024. GPT-4o System Card. https://cdn.openai.com/gpt-4o-system- card.pdf

    2024

  18. [26]

    OpenAI. 2024. OpenAI o1 System Card. https://cdn.openai.com/o1-system-card- 20241205.pdf FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts KDD’25, August 4, 2025, Toronto, ON, Canada

    2024

  19. [27]

    OpenAI. 2025. GPT-4.1. https://platform.openai.com/docs/models/gpt-4.1

    2025

  20. [28]

    OpenAI. 2025. GPT-4.1 mini. https://platform.openai.com/docs/models/gpt-4.1- mini

    2025

  21. [29]

    OpenAI. 2025. OpenAI o3 and o4-mini System Card. https://cdn.openai.com/ pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf

    2025

  22. [30]

    Oumi Community. 2025. Oumi: an Open, End-to-end Platform for Building Large Foundation Models. https://github.com/oumi-ai/oumi

    2025

  23. [31]

    Panos Achlioptas, Jeremy Greer, Konstantinos Aisopos, Michael Schuler, Ous- sama Elachqar, and Emmanouil Koukoumidis. 2025. HallOumi-8B-classifier. https://huggingface.co/oumi-ai/HallOumi-8B-classifier

    2025

  24. [32]

    Mike Schaekermann, Graeme Beaton, Elaheh Sanoubari, Andrew Lim, Kate Lar- son, and Edith Law. 2020. Ambiguity-aware AI Assistants for Medical Data Analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Comput- ing Systems. ACM, Honolulu HI USA, 1–14. doi:10.11...

    2020

  25. [33]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Han- naneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset Cartogra- phy: Mapping and Diagnosing Datasets with Training Dynamics. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural L...

    2020 doi

  26. [34]

    Zoltán Gendler Szabó. 2020. Compositionality. https://plato.stanford.edu/ archives/fall2024/entries/compositionality/

    2020

  27. [35]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024. MiniCheck: Efficient Fact- Checking of LLMs on Grounding Documents. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . Association for Computa- tional Linguistics, Miami, Florida, USA, ...

    2024 doi

  28. [36]

    Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. 2021. CON- FIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Con- trastive Fine-tuning. doi:10.18653/v1/2022.naacl-main.415

    2021 doi

  29. [37]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 5008–...

    2020 doi

  30. [38]

    Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, and Haizhou Li. 2022. Analyz- ing and Evaluating Faithfulness in Dialogue Summarization. doi:10.48550/arXiv. 2210.11777 arXiv:2210.11777

    2022 doi arXiv

  31. [39]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese

  32. [40]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. doi:10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs]

    2023 doi arXiv

  33. [41]

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025. A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well? doi:10.48550/arXiv.2503...

    2025 doi arXiv

  34. [42]

    Kivlichan, Vinodkumar Prabhakaran, Davis Yadav, and Amulya Yadav

    Wenbo Zhang, Hangzhi Guo, Ian D. Kivlichan, Vinodkumar Prabhakaran, Davis Yadav, and Amulya Yadav. 2023. A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity. doi:10.48550/arXiv.2311.04345 arXiv:2311.04345 [cs]

    2023 doi arXiv

  35. [43]

    doi:10.48550/arXiv.2504.12516 arXiv:2504.12516

    BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. doi:10.48550/arXiv.2504.12516 arXiv:2504.12516

    doi arXiv

  36. [47]

    What did the customer call about?

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. doi:10.48550/arXiv.2306.05685 arXi...

    2023 doi arXiv

  37. [2023]

    doi:10.48550/arXiv.2305.11747 arXiv:2305.11747

    HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. doi:10.48550/arXiv.2305.11747 arXiv:2305.11747

    doi arXiv

  38. [2024]

    doi:10.48550/arXiv.2310.08491 arXiv:2310.08491 [cs]

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. doi:10.48550/arXiv.2310.08491 arXiv:2310.08491 [cs]

    doi arXiv

  39. [2025]

    https: //oumi.ai/blog/posts/introducing-halloumi

    Introducing HallOumi: A State-of-the-Art Claim-Verification Model. https: //oumi.ai/blog/posts/introducing-halloumi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.