Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

On-device distillation recovers most of a large model's summary quality only when the teacher is a reasoning model, and different teachers hand the student different skills.

2026-07-10 10:19 UTC pith:QTBJJBCS

load-bearing objection Useful on-device distillation measurement with a real teacher control and honest per-field routing; the “reasoning nature, not scale” headline is a bit cleaner than the control design. the 3 major comments →

arxiv 2607.08268 v1 pith:QTBJJBCS submitted 2026-07-09 cs.AI cs.CL

Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

Vinay Kumar Chaganti This is my paper
classification cs.AI cs.CL
keywords knowledge distillationon-device inferencesmall language modelsstructured output generationreasoning-model distillationLLM-as-a-judgetext summarizationmodel compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

High-volume structured enrichment of news articles—one JSON object with a short summary and five labels—usually pays a large model's latency on every item. This paper shows that distilling that job into a 0.6B on-device model can cut time from roughly 39 seconds to about 0.8 seconds per article while recovering 58% of the quality gap on summaries, but only when the teacher is a reasoning model. A same-size non-reasoning teacher produces a student no better than the untuned base, so the writing gain comes from the teacher's reasoning nature rather than its size. A larger managed pipeline with synthetic data instead transfers label diversity. Because no single engine wins every field, the practical deliverable is a per-field routing map: run the distilled student for most work, fall back on few-shot for tone and on a larger model for thin short sources where the reasoning student fabricates.

Core claim

Distilling an 8B reasoning teacher into a 0.6B student recovers 58% of the base-to-teacher gap on summary quality and beats both non-distillation controls, while a same-size non-reasoning teacher yields a student no better than the untuned base; thus the summary gain depends on the teacher's reasoning nature, not its scale, and capabilities split by teacher—reasoning transfers writing quality, a managed larger pipeline transfers label diversity, and the instruction lineage stays more grounded on thin sources.

What carries the argument

Same-size non-reasoning-teacher control: an 8B instruction model re-labels the identical training set under the same prompt and recipe, isolating reasoning nature from scale so that any student gap can be attributed to the teacher's kind rather than its size.

Load-bearing premise

The three-judge language-model panel, validated only by a gross mismatched-article test and without human gold labels, is taken as a faithful enough proxy for summary quality and label truth, including the thin-source faithfulness ordering that rests on only 22 articles.

What would settle it

A human-gold anchor on roughly 50 test items that reverses either the checklist win of the reasoning student over the non-reasoning student or the claim that the non-reasoning student is no better than base would collapse the central attribution to reasoning nature.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies black-box distillation of a structured news-enrichment task (short summary + five closed-set labels) from an 8B reasoning teacher (deepseek-r1:8b) into a 0.6B on-device student (Qwen3-0.6B, QLoRA, three seeds). Against a same-size non-reasoning 8B teacher, a larger managed pipeline with synthetic expansion, and two non-distillation baselines (few-shot and constrained decoding), a blinded three-judge panel scores all arms reference-free against the full article. The student runs at ~0.8 s vs ~39 s for the teacher, recovers 58% of the base-to-teacher summary checklist gap, and beats constrained decoding by +16.8 points (paired bootstrap p<0.001). The authors attribute the summary gain to the teacher’s reasoning nature rather than scale, report a capability split (reasoning → writing quality; managed pipeline → label diversity; instruction lineage → thin-source grounding as a direction on n=22), and deliver a per-field routing map rather than a single-model recommendation.

Significance. If the measured results hold, the work is a useful, practice-facing contribution to on-device structured enrichment: it shows that distillation can beat the cheap levers practitioners try first on free-text summary quality, that gains are field-dependent, and that a same-size non-reasoning control is informative even when imperfect. Strengths that raise the paper above a routine distillation report include three training seeds with seed-range reporting, paired bootstrap over 93 items, two non-distillation baselines, family-independent judges, a negative-control check on mismatched articles, full-source vs truncated grading as a methodological finding, offline-reproducible metrics from a released scorecard, and an honest reduction from an exhaustive routing map to a minimal deployable configuration. The capability-split framing and the full-source faithfulness lesson are transferable beyond this corpus.

major comments (3)
  1. Section VI-D and the abstract/title claim that “the summary gain follows from the teacher’s reasoning nature rather than its scale.” The control is valuable but does not isolate reasoning nature: Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Llama-8B differ in family lineage, pretraining/instruction data, writing style, and label distributions, and the non-reasoning teacher yielded 396 vs 401 usable training JSONs (Appendix B, Section X). The student gap could be driven by teacher writing quality or label specificity rather than chain-of-thought training per se. The paper already notes these confounds in Limitations; the load-bearing fix is to restate the claim throughout (title, abstract, VI-D, conclusion) as “reasoning-lineage / higher-ceiling teacher” rather than “reasoning nature rather than scale,” and to avoid causal language that the design does not support.
  2. Section V-C and VI-B: the product-critical faithfulness and thin-source claims rest on an LLM panel validated only by a gross mismatched-article negative control (0% faithful, n=30), with no human gold and no injected-fabrication test. Aggregate faithfulness is non-significant (reasoning vs non-reasoning students 73.1 vs 77.8, p=0.19); the short-source ordering uses only 22/93 articles and is grading-sensitive (Section VII). The manuscript already labels this a “direction,” which is appropriate, but the abstract still leads with “74 versus 55 faithful.” Either add a small human-anchored or entailment-verified subset (as Future Work already proposes) or demote the thin-source numbers further so they cannot be read as a measured transfer coefficient.
  3. Section VI-C / Table VI: classification accuracy is scored against panel-consensus proxy-gold, not human labels, and several fields are heavily imbalanced (depth ~71% standard; urgency majority baseline ~74%). Urgency “wins” largely by predicting the majority class; depth sits below the majority baseline with inter-judge agreement 0.38. Macro does not beat few-shot. These caveats are present in the text, but the capability-split claim that the managed pipeline “transfers label diversity” needs to be stated as relative to a weak, judge-prior reference rather than as recovered classification skill. A majority-class and chance baseline should be foregrounded in every classification table, and any claim of label transfer should be restricted to fields that clear those lines under per-judge robustness.
minor comments (5)
  1. Figure 3 and gap-closure reporting: flag axes where teacher ≈ base more visibly in the figure itself; the text already notes instability of the ratio, but the plot can still be misread as a uniform recovery scale.
  2. Table III latency column mixes ~39,200 ms with sub-second student times; a log-scale or separate efficiency table would make the 5.4 h → ~7 min claim easier to verify at a glance.
  3. Section VII notes that judges disagree on mechanically checkable predicates (sentence count, opening phrase). Consider computing those two checks by rule in the released scorecard and reserving the panel for content checks only.
  4. Appendix C: individual per-judge votes are not reconstructable from the released scorecard. For a methods contribution that stresses per-judge robustness, releasing anonymized per-judge majorities (or a scripted re-grade path) would strengthen reproducibility claims.
  5. Typo/consistency: “Plat.” in Table VI is defined only in the caption as managed 120B; spell out once in the table header. Also “gpt-oss-120b” naming is opaque to readers outside that commercial stack—add a one-line description.

Circularity Check

0 steps flagged

Empirical distillation study with measured arms and explicit controls; no derivation reduces to its inputs by construction.

full rationale

This paper is a controlled measurement study of on-device distillation, not a first-principles derivation. The load-bearing claims are comparative empirical results: a QLoRA student of deepseek-r1:8b recovers 58% of the base-to-teacher summary checklist gap and beats constrained decoding by +16.8 points, while an identically trained student of a same-size non-reasoning 8B teacher does not beat the untuned base. Gap-closure is defined as (tuned−base)/(teacher−base) over independently measured arm scores; it is a reporting scale, not a fitted parameter renamed as a prediction. Classification accuracy is scored against a three-judge panel consensus that the authors explicitly label proxy-gold/silver, with majority-class baselines, per-judge robustness, and a mismatched-article negative control—limitations of evaluation validity, not circular reduction of a claimed derivation. Routing configurations are scored post hoc on the same graded outputs and are labeled engineering validation, not confirmatory prediction. Citations are to external distillation, judging, and model papers; there is no self-citation chain, uniqueness theorem, or ansatz smuggled from the author’s prior work. Confounds in the reasoning-vs-scale isolation (distinct 8B families, label distributions, 396 vs 401 usable labels) are identification/correctness issues the paper itself flags in Limitations, not circularity. No step reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

Empirical ML paper. Load-bearing assumptions are evaluation and data choices, not free physical constants. The central claims rest on treating panel majority as quality truth, on the RSS-News corpus length and label skews, and on the QLoRA recipe and seed set. No new physical entities are postulated.

free parameters (4)
  • LoRA rank = 32
    Fixed at 32 for all distillation runs; not swept. Affects student capacity and what can be transferred.
  • Training seeds = 42, 123, 7
    Three seeds (42/123/7); seed variance is first-order on subjective labels, so the three-seed mean is itself a design choice that defines reported tuned metrics.
  • Short-article length threshold = 1200 chars
    1200 characters defines the thin-source subgroup where faithfulness regression is claimed; threshold choice partitions the 22-item subgroup.
  • Summary checklist composition = 8 checks post-pilot
    Eight binary checks set after a 12-article pilot (two checks dropped for lack of headroom). Checklist definition directly determines the primary +16.8-point win.
axioms (5)
  • domain assumption A three-judge LLM panel majority against the full article is a valid reference-free measure of summary quality and classification correctness.
    Section V-B/C; validated only by mismatched-article negative control, not human gold or subtle fabrication injection.
  • domain assumption Teacher final answers (thinking disabled) are the right supervision signal for deployment-realistic distillation of a structured enrichment task.
    Section IV and II; rationale/trace supervision left to future work, so the reasoning-nature claim is about the teacher’s answer distribution, not CoT transfer.
  • domain assumption The 93-item held-out set from known feeds is representative of the intended production distribution (new items from fixed feeds).
    Section IV; generalization to unseen sources is explicitly out of scope.
  • ad hoc to paper Closed label vocabularies observed in the teacher’s own outputs define the classification task.
    Section III; vocabularies are imbalanced (e.g., depth 71% standard), which bounds what accuracy can mean.
  • standard math Bootstrap over 93 articles with three-seed means adequately characterizes uncertainty for primary comparisons.
    Section V-D; paper notes seed variance is not fully propagated and is first-order on subjective fields.
invented entities (3)
  • RSS-News corpus (494 labeled articles, 24 feeds) no independent evidence
    purpose: Provides the train/test distribution on which all gap-closure and faithfulness claims are measured.
    Custom corpus; properties (bimodal length, thin excerpts, label skew) drive central findings. No independent public benchmark comparison.
  • Eight-item binary summary checklist (faithful, thesis, takeaway, length, opening, teacher-lens, tech-lens, tone) no independent evidence
    purpose: Primary summarization metric replacing a saturating 0–5 rubric.
    Defined after pilot; not a standard public metric. Pass-rates are the headline numbers.
  • Per-field routing map / assembled routing configurations (Routers A, B, B′, C) no independent evidence
    purpose: Deployment deliverable that assigns engines per field and scores hybrid systems.
    Constructed post hoc from per-field results; engineering validation, not a held-out confirmatory finding (Section VIII-A).

pith-pipeline@v1.1.0-grok45 · 22814 in / 3877 out tokens · 38883 ms · 2026-07-10T10:19:46.820236+00:00 · methodology

0 comments
read the original abstract

High-volume structured extraction pays a large model's latency on every item, so distilling the task into a small on-device model is attractive: comparable output at a fraction of the time and cost. We measure what that distillation actually delivers, per sub-task. Each news article is mapped to one JSON object with a short summary and five categorical labels. We distill an 8B reasoning teacher (deepseek-r1:8b) into a 0.6B student (Qwen3-0.6B; QLoRA, three seeds), and add two teacher controls: a same-size non-reasoning teacher and a larger managed pipeline. A blinded, reference-free, three-judge panel scores every arm against the full article, alongside two non-distillation baselines, few-shot prompting and constrained decoding. The student runs at about 0.8 s per article against the teacher's 39 s, and recovers 58% of the base-to-teacher gap on summary quality, beating its primary baseline (constrained decoding) by +16.8 points and few-shot prompting by a secondary +4.9. A same-size non-reasoning teacher trains a student no better than the untuned base, so the summary gain follows from the teacher's reasoning nature rather than its scale. Capabilities then split by teacher: the reasoning teacher transfers writing quality and the managed pipeline transfers label diversity, while a same-size instruction teacher's students stay more grounded on the 22 short, thin-source articles in the 93-item test set (74 versus 55 faithful), where the reasoning-lineage student fabricates. That grounding difference is a consistent ordering rather than a significant aggregate effect, and the subgroup is small, so we report it as a direction. Because no single engine wins every field, the deliverable is a per-field routing map for on-device enrichment.

Figures

Figures reproduced from arXiv: 2607.08268 by Vinay Kumar Chaganti.

Figure 1
Figure 1. Figure 1: The structured-enrichment task. Each article maps to one JSON object, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The experimental pipeline. 494 labeled articles are split 401/93; three teachers, a reasoning 8B, a same-size non-reasoning 8B, and a larger managed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-axis gap-closure toward the teacher (0 = untuned base, 100 = [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Three teachers and their 0.6B students under one rubric (full 12-arm [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Left: batch wall-clock versus volume, the 5.4 h to [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · 37 internal anchors

  1. [1]

    Distilling the Knowledge in a Neural Network

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv:1503.02531, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  2. [2]

    Sequence-level knowledge distillation,

    Y . Kim and A. M. Rush, “Sequence-level knowledge distillation,” in Proc. EMNLP, 2016

    work page 2016

  3. [3]

    A Survey on Knowledge Distillation of Large Language Models

    X. Xuet al., “A survey on knowledge distillation of large language models,” arXiv:2402.13116, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    A Survey on Model Compression for Large Language Models

    X. Zhuet al., “A survey on model compression for large language models,” arXiv:2308.07633, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Qwen3 Technical Report

    A. Yanget al., “Qwen3 technical report,” arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

    Z. Liuet al., “MobileLLM: Optimizing sub-billion parameter language models for on-device use cases,” arXiv:2402.14905, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Revealing the power of post-training for small language models via knowledge distillation,

    M. Ranget al., “Revealing the power of post-training for small language models via knowledge distillation,” arXiv:2509.26497, 2025

    work page arXiv 2025

  8. [8]

    UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition

    W. Zhouet al., “UniversalNER: Targeted distillation from large language models for open named entity recognition,” arXiv:2308.03279, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events

    Y . Guet al., “Distilling large language models for biomedical knowledge extraction: A case study on adverse drug events,” arXiv:2307.06439, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    DeepSeek-AI, “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning,” arXiv:2501.12948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models

    C. Zhanget al., “100 days after DeepSeek-R1: A survey on repli- cation studies and more directions for reasoning language models,” arXiv:2505.00551, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks

    Y . Liet al., “NaturalThoughts: Selecting and distilling reasoning traces for general reasoning tasks,” arXiv:2507.01921, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning

    X. Chenet al., “Unveiling the key factors for distilling chain-of-thought reasoning,” inFindings of the Assoc. Comput. Linguistics: ACL 2025, 2025, arXiv:2502.18001

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    On the Generalization vs Fidelity Paradox in Knowledge Distillation

    S. K. Rameshet al., “On the generalization vs fidelity paradox in knowledge distillation,” inFindings of the Assoc. Comput. Linguistics: ACL 2025, 2025, arXiv:2505.15442

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    Distillation Scaling Laws

    D. Busbridgeet al., “Distillation scaling laws,” arXiv:2502.08606, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    Improved Knowledge Distillation via Teacher Assistant

    S.-I. Mirzadehet al., “Improved knowledge distillation via teacher assistant,” inProc. AAAI, 2020, arXiv:1902.03393

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  17. [17]

    Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

    X. Chenet al., “Do not think that much for 2+3=? On the overthinking of o1-like LLMs,” arXiv:2412.21187, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    The curse of CoT: On the limitations of chain-of-thought in in-context learning,

    T. Zhenget al., “The curse of CoT: On the limitations of chain-of-thought in in-context learning,” arXiv:2504.05081, 2025

    work page arXiv 2025

  19. [19]

    OptimalThinkingBench: Evaluating over and underthinking in LLMs,

    P. Aggarwalet al., “OptimalThinkingBench: Evaluating over and underthinking in LLMs,” arXiv:2508.13141, 2025

    work page arXiv 2025

  20. [20]

    Are Reasoning Models More Prone to Hallucination?

    Z. Yaoet al., “Are reasoning models more prone to hallucination?,” arXiv:2505.23646, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes,

    C.-Y . Hsiehet al., “Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes,” inFindings of ACL, 2023

    work page 2023

  22. [22]

    Self-Instruct: Aligning Language Models with Self-Generated Instructions

    Y . Wanget al., “Self-Instruct: Aligning language models with self- generated instructions,” inProc. ACL, 2023, arXiv:2212.10560

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data

    A. Shirgaonkaret al., “Knowledge distillation using frontier open-source LLMs: Generalizability and the role of synthetic data,” arXiv:2410.18588, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization

    G. Chrysostomouet al., “Investigating hallucinations in pruned large language models for abstractive summarization,” arXiv:2311.09335, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [25]

    Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

    S. Ramprasadet al., “Evaluating the factuality of zero-shot summarizers across varied domains,” arXiv:2402.03509, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Benchmarking Large Language Models for News Summarization

    T. Zhanget al., “Benchmarking large language models for news summarization,” arXiv:2301.13848, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    A systematic evaluation of on-device large language models: Quantization, performance, and resources,

    Q. Songet al., “A systematic evaluation of on-device large language models: Quantization, performance, and resources,” arXiv:2505.15030, 2025

    work page arXiv 2025

  28. [28]

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    J. Linet al., “AWQ: Activation-aware weight quantization for LLM compression and acceleration,” arXiv:2306.00978, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    L. Zhenget al., “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Large Language Models are not Fair Evaluators

    P. Wanget al., “Large language models are not fair evaluators,” arXiv:2305.17926, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  31. [31]

    Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

    J. Yeet al., “Justice or prejudice? Quantifying biases in LLM-as-a-judge,” arXiv:2410.02736, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

    Y . Liuet al., “G-Eval: NLG evaluation using GPT-4 with better human alignment,” arXiv:2303.16634, 2023. PREPRINT, JULY 2026 12

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists,

    Y . Leeet al., “CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists,” arXiv:2403.18771, 2024

    work page arXiv 2024

  34. [34]

    TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation

    J. Cooket al., “TICKing all the boxes: Generated checklists improve LLM evaluation and generation,” arXiv:2410.03608, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    InFoBench: Evaluating Instruction Following Ability in Large Language Models

    Y . Qinet al., “InFoBench: Evaluating instruction following ability in large language models,” inFindings of ACL, 2024, arXiv:2401.03601

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

    P. Vergaet al., “Replacing judges with juries: Evaluating LLM generations with a panel of diverse models,” arXiv:2404.18796, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  37. [37]

    Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution

    Z. Tianet al., “Overconfidence in LLM-as-a-judge: Diagnosis and confidence-driven solution,” arXiv:2508.06225, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    LLM Evaluators Recognize and Favor Their Own Generations

    A. Panickssery, S. R. Bowman, and S. Feng, “LLM evaluators recognize and favor their own generations,” arXiv:2404.13076, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    Self-Preference Bias in LLM-as-a-Judge

    K. Wataokaet al., “Self-preference bias in LLM-as-a-judge,” arXiv:2410.21819, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

    S. Minet al., “FActScore: Fine-grained atomic evaluation of factual precision in long form text generation,” arXiv:2305.14251, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  41. [41]

    MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents

    L. Tang, P. Laban, and G. Durrett, “MiniCheck: Efficient fact-checking of LLMs on grounding documents,” arXiv:2404.10774, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks

    D. Jianget al., “TIGERScore: Towards building explainable metric for all text generation tasks,” arXiv:2310.00752, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  43. [43]

    SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

    P. Manakul, A. Liusie, and M. J. F. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” arXiv:2303.08896, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models

    S. Genget al., “JSONSchemaBench: A rigorous benchmark of structured outputs for language models,” arXiv:2501.10868, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025