REVIEW 11 cited by
MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recognizing if LLM output can be grounded in evidence is central to many tasks in NLP: retrieval-augmented generation, summarization, document-grounded dialogue, and more. Current approaches to this kind of fact-checking are based on verifying each piece of a model generation against potential evidence using an LLM. However, this process can be very computationally expensive, requiring many calls to a model to check a single response. In this work, we show how to build small fact-checking models that have GPT-4-level performance but for 400x lower cost. We do this by constructing synthetic training data with GPT-4, which involves creating realistic yet challenging instances of factual errors via a structured generation procedure. Training on this data teaches models to check each fact in the claim and recognize synthesis of information across sentences. For evaluation, we unify datasets from recent work on fact-checking and grounding LLM generations into a new benchmark, LLM-AggreFact. Our best system MiniCheck-FT5 (770M parameters) outperforms all systems of comparable size and reaches GPT-4 accuracy. We release LLM-AggreFact, code for data synthesis, and models.
Forward citations
Cited by 11 Pith papers
-
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts
HAMLET introduces a three-level key-fact hierarchy with automated query-focused summarization to evaluate LLM recall and faithfulness in book-length contexts, achieving over 90% agreement with experts.
-
Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment
Distilling an 8B reasoning teacher into a 0.6B student recovers most summary quality at ~50× speed, but teacher type—not scale alone—determines which capabilities transfer.
-
DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
An audit framework and empirical study showing that generative search engines and deep research agents frequently produce one-sided answers and weakly supported citations, with citation accuracy between 40 and 80%.
-
TalkLess: Blending Extractive and Abstractive Speech Summarization for Editing Speech to Preserve Content and Style
TalkLess blends extractive and abstractive speech summarization through LLM candidate generation and a weighted scoring function, then converts transcript edits to audio with VoiceCraft, evaluating favorably against a...
-
Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
Automated factuality metrics for AI outputs are inconsistent with each other, misestimate system hallucination rates, and penalize paraphrase and long-document reasoning, so they should not be trusted without human va...
-
Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization
RHIO improves long-form QA faithfulness by training models with negative samples created by masking retrieval heads, then contrasting faithful and unfaithful decoding.
-
Paladin-mini: A Compact and Efficient Grounding Model Excelling in Real-World Scenarios
A compact fine-tuned grounding model and a purpose-built benchmark aim to expose and close gaps in numerical, temporal, and logical fact-checking.
-
Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
MSumBench is a new English/Chinese benchmark that grades summaries across six domains on faithfulness, completeness, and conciseness, using domain-specific key-facts and multi-agent-debate-assisted human annotations, ...
-
Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection
Using hallucinated benchmark answers as rejected DPO pairs, ordered by an external fact-checker's grounding score, improves hallucination detection in 1B-3B Llama models.
-
Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptation
A deterministic, proposition-level fact-checker that compares clinical summaries against electronic health records via (entity, attribute, value, time) claims and hard-coded logical checks reports 0.8904 precision and...
-
Multi-agent Systems for Misinformation Lifecycle : Detection, Correction And Source Identification
A conceptual multi-agent architecture for classifying, detecting, correcting, and sourcing misinformation is proposed but not implemented or evaluated.
Discussion (0). Continue with ORCID to comment.