REVIEW 16 cited by
The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
read the original abstract
We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at https://www.kaggle.com/facts-leaderboard.
Forward citations
Cited by 16 Pith papers
-
Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator
Hallucination Self-Play co-evolves a generator and detector from one base LLM via RLAIF and RLVR, lifting a 7B model to match advanced LLMs on RAGTruth faithfulness detection.
-
WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
WorldReasoner supplies 345 resolved forecasting tasks built from 14,141 articles to score LM agents on outcome quality, evidence quality, and reasoning quality against time-bounded evidence and hindsight graphs.
-
Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification
NEI competence does not transfer reliably across evidence constructions in fact verification; mixed training narrows but does not close the gap, and aggregate scores can mask specific weaknesses.
-
Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights
TRIVIA+ is a new long-context RAG hallucination benchmark with four noisy label variants that shows current detectors have substantial room for improvement and are hindered by label noise.
-
TRACE: Tourism Recommendation with Accountable Citation Evidence
TRACE is a new benchmark dataset and evaluation suite for conversational tourism recommenders that requires systems to suggest POIs, cite verifiable review spans, and recover from rejections, revealing a Three-Compete...
-
ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence
Introduces ConflictScore (CS-C and CS-R) to quantify how language model responses acknowledge conflicting evidence in grounding documents, plus ConflictBench for systematic evaluation.
-
Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures: A Multi-Stage Pipeline with Deterministic Reliability Scoring
PROBE pipeline with deterministic PCAP normalization, verdict-aware evidence ensembles, and composite reliability scoring raises weighted evidence F1 to 0.957 on 87 Wi-Fi captures while avoiding LLM self-confidence an...
-
OpenAaaS: An Open Agent-as-a-Service Framework for Distributed Materials-Informatics Research
OpenAaaS is a hierarchical agent-as-a-service system that enables secure multi-agent collaboration for materials informatics by moving code to data rather than data to code.
-
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Nearly half of 60 widely used LLM benchmarks show saturation, which increases with age; private test access does not prevent saturation, while expert curation does.
-
A Neurosymbolic Approach to Natural Language Formalization and Verification
A neurosymbolic guardrail reports 99.2% soundness on a 522-item policy QA benchmark, mainly by rejecting 84% of correct answers; unvetted real-world policies score 96.8%.
-
AI and Authenticity in Islamic Research: A Critical Evaluation of Generative AI Reliability, Hallucination, and Source Fidelity in Quranic, Hadith, and Fiqh Knowledge
Leading generative AIs are useful for introductory Islamic learning but unreliable as authorities on Fiqh, citations, and madhhab-sensitive rulings without human verification.
-
Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search
Empirical study of RLAIF for portable query generation finds reward shaping controls performance more than optimizer choice and a rule-based reward floor yields +0.147 quality gain.
-
APEX: Automated Prompt Engineering eXpert with Dynamic Data Selection
APEX dynamically tiers data into Easy/Hard/Mixed based on optimization lineage and prioritizes Mixed examples, reporting 11.2% and 6.8% average gains over baseline prompts on two models under a 5,000-call budget.
-
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
A step-level reinforcement-learning reward combining a process reward model with truncated resampling reduces chain-of-thought faithfulness hallucinations in small reasoning models.
-
Kimi K2: Open Agentic Intelligence
Kimi K2 is a 1-trillion-parameter MoE model that leads open-source non-thinking models on agentic benchmarks including 65.8 on SWE-Bench Verified and 66.1 on Tau2-Bench.
-
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Gemini 2.5 Pro and Flash models are presented as achieving frontier performance in reasoning, coding, and long-context multimodal tasks while spanning a cost-capability Pareto curve.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.