{"id":"438921c2-0612-48e1-875c-330bf0b64cb6","arxiv_id":"2501.03200","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FACTS Grounding is a benchmark and leaderboard that scores LLMs on producing long-form answers fully grounded in up to 32k-token documents, using a validated panel of judge models.","lead":"This paper introduces FACTS Grounding, a leaderboard that tests whether language models can write long answers that stick strictly to an attached document. The benchmark uses a panel of judge models to score each response and disqualifies responses that do not answer the user's request.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judge validation on N=402 shows FPR 32-56%, and no human validation is reported on final leaderboard responses, so reported grounding scores are likely inflated and rankings may be biased.","rationale":"The reader's weakest_assumption correctly identifies the judge models' alignment with human judgment as the load-bearing condition for the leaderboard's validity. The paper's own Table 2 shows the selected judge templates have FPRs of 32-56%, meaning the ensemble is lenient toward unfaithful responses. The validation set is small and heavily imbalanced, so these FPR estimates are uncertain, and no human validation is reported on the final leaderboard splits. This directly threatens the central claim that the scores measure grounding ability: if the judges systematically mark inaccuracies as accurate, the absolute scores in Tables 5 and 6 are overestimates, and the rankings could shift once the bias varies across models. The self-preference bias, while disclosed, is not corrected and could account for the 0.7-point gap between the top two ranked models. The eligibility filter is an additional weakness: its high FNR and consensus rule make disqualification rare, so the 'while fulfilling the user request' component is weakly enforced. However, the primary measurement of grounding is the more fundamental issue. A human-annotation spot-check on leaderboard responses would empirically settle whether the judge ensemble's FPR remains high in the deployment setting. If the FPR is low, the CONDITIONAL verdict could be upgraded; if high, the reported scores are not a faithful measure, and the conditional verdict is fully justified. The paper is transparent and the test is feasible because the leaderboard is public and the annotation protocol is described.","tokens_in":14544,"tokens_out":9095,"duration_ms":82567,"concrete_test":"Sample ~100 responses from the leaderboard across models and both splits (e.g., 50 open, 50 blind), have three expert annotators label each response as grounded/not grounded and eligible/ineligible using the paper's definitions, and compare with the judge ensemble. Compute the false positive rate on responses humans label as not grounded and the disqualification recall on human-ineligible responses. If the FPR exceeds ~20% or recall is below ~50%, Tables 5 and 6 overstate grounding accuracy and the rankings are unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The leaderboard's central claim—that scores in Tables 5 and 6 measure grounding as humans would judge it—rests on the three judge models' binary verdicts. Table 2 reports the selected templates' false positive rates: Gemini 1.5 Pro JSON 56.45%, Claude 3.5 Sonnet implicit span-level 45.16%, GPT-4o JSON 32.26%. Because the final factuality score is the average of these judges, a large fraction of unfaithful responses are marked accurate, inflating absolute scores. The validation set is small (N=402) and imbalanced (87:13), so the negative class has ~52 examples and FPR estimates are noisy; no human validation is reported on the actual public/blind leaderboard responses. The self-preference bias (+3.23% on average) is disclosed but not removed, which can shift close rankings: the top two models differ by 0.7 points, within the CI. Additionally, the eligibility filter (Table 4) has high FNR (62.67-80.67%) and uses consensus, so most ineligible responses are not disqualified, weakening the 'while fulfilling user request' component. Together, these gaps mean the reported scores are not demonstrably calibrated to human judgments of grounding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FACTS Grounding, an online leaderboard and benchmark for evaluating whether LLMs generate long-form responses that are grounded in a provided context document of up to 32k tokens while satisfying a user request. The dataset consists of 860 public and 859 private human-written examples spanning several domains and task types. Responses are evaluated in two stages: an eligibility filter disqualifies responses that do not sufficiently fulfill the user request, and three judge models (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) then produce binary grounding verdicts, with the final factuality score being the average of the three judges' accuracy rates. The judge prompts were selected on a private N=402 human-labeled set using Macro-F1, and the eligibility prompt was selected on a separate N=450 set. The paper reports unadjusted and final factuality scores for nine models across Open and Blind splits, and presents a fused ranking derived from Condorcet aggregation.","tokens_in":14750,"tokens_out":3337,"duration_ms":30708,"significance":"If the measurement pipeline is trustworthy, FACTS Grounding would be a valuable community resource: it is one of the first benchmark-plus-leaderboard efforts for long-form grounding at 32k-token context, it uses fresh human-written requests rather than repurposed academic tasks, it includes public and private splits to limit overfitting, and it explicitly validates judge prompts against human labels. The paper also reports self-preference bias rather than ignoring it, and the decision to average three judge models is a reasonable step toward robustness. However, the central measurement claim rests on judge models whose validation shows high false positive rates and for which no human validation is reported on the actual leaderboard responses. Because the reported absolute scores and close ranking gaps in Tables 5 and 6 depend on these judges, the paper's central contribution is not yet fully established.","major_comments":[{"comment":"The chosen judge templates have false positive rates of 32.26% (GPT-4o JSON), 45.16% (Claude 3.5 Sonnet implicit span-level), and 56.45% (Gemini 1.5 Pro JSON). Since the final factuality score is the average of these three judges, a large fraction of unfaithful responses are labelled accurate, which likely inflates the absolute scores in Tables 5 and 6. The validation set has only 13% negative examples, so these FPR estimates rest on roughly 52 cases and are quite noisy, and no human validation is reported on the public or blind leaderboard responses. This missing calibration check is load-bearing for the claim that FACTS Grounding scores reflect human judgments of grounding.","section":"Section 3.1 and Table 2"},{"comment":"The eligibility filter selects prompt templates with false negative rates of 62.67% (Claude, user request only), 74.00% (Gemini, user request only), and 80.67% (GPT-4o, user request only), and then applies a consensus rule that requires all three judges to label a response ineligible. The combined effect is that most responses that fail to fulfill the user request are not disqualified, so the 'while fulfilling the user request' component of the benchmark is not effectively enforced. The 1%–5% score reduction reported in Section 4 is consistent with this concern, but the benchmark would be more convincing if it also reported the disqualification rate and the judges' agreement on the final split.","section":"Section 3.2 and Table 4"},{"comment":"The paper reports a mean self-preference bias of +3.23% but does not correct for it, and the judge set includes the same model families that are ranked (Gemini, GPT-4o, and Claude). Since the top two models in Table 6 differ by only 0.7 points, the reported ranking may be sensitive to this bias. The authors should either provide a bias-corrected leaderboard, or analyze how the fused ranking changes when each judge is dropped or when self-preference is removed, so readers can assess whether the ordering in Tables 5 and 6 is robust.","section":"Section 4, 'On aggregating multiple judge models'"}],"minor_comments":[{"comment":"There are several spacing and capitalization errors in the text, for example 'TheFACTS Grounding leaderboard' in the abstract and 'theF ACTS Groundingleaderboard' in Section 1; these should be corrected.","section":"Abstract and Section 1"},{"comment":"The sentence 'As all frontier language models models were trained on large corpora of web data' contains a duplicated word 'models'; please remove the repetition.","section":"Section 2.2, data contamination point 3"},{"comment":"The selection criterion is reported as Macro-F1, but the chosen templates have very different FPR/FNR trade-offs; a short explanation of why Macro-F1, rather than a false-positive-sensitive criterion, is appropriate for this benchmark would help readers interpret the chosen judges.","section":"Table 2 and Section 3.1"},{"comment":"The final factuality score is described as the average of the three judge scores, but the table presents the three individual judge columns and an 'Average Fused' column; the caption should clarify that the 'Average Fused' column is the unweighted mean of the three judge-specific percentages.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper proposes a potentially useful benchmark, but the evaluation methodology is the central contribution and the current validation is insufficient to support the leaderboard's claims. The high false positive rates, small validation set, and absence of human validation on the final leaderboard responses are substantive, not cosmetic, issues. I concur with the stress-test assessment that these gaps affect the central measurement claim. The resource may still be valuable after the authors provide additional validation, bias analyses, and a clearer statement of the limitations of the judge models."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. It builds a genuinely useful benchmark resource: 1,719 human-written long-form grounding tasks over documents up to 32k tokens, with public and private splits, and a two-stage judge pipeline. And its reported numbers should not be taken at face value, because the judges are validated on a small set and are demonstrably lenient on the exact dimension they are supposed to measure.\n\nWhat's actually new: the combination of long context, complex human prompts, public/private splits, and an eligibility filter is not something prior benchmarks offer. The authors validate judge prompt templates against human labels on a private set, report per-template precision/recall, disclose the self-preference bias, and discuss data contamination. That level of transparency is welcome, and the Kaggle leaderboard with open and blind splits is a real community resource.\n\nThe soft spots are real but not fatal. The judge validation set is N=402 with only about 52 negative examples, so the false positive rates in Table 2 — 32% to 56% for the chosen templates — are both high and noisy. A judge that marks half of genuinely unfaithful responses as accurate will inflate absolute scores. The authors also report a +3.23% self-preference bias but do not correct for it, and the top two models are within 0.7 points, inside the confidence intervals. The eligibility filter is weak in the opposite direction: its false negative rate is 62–80%, and disqualification requires all three judges to agree, so most ineligible responses survive. That means the \"fulfills the user request\" part of the benchmark is mostly aspirational.\n\nWhere I land: the benchmark is worth having, and the ranking signal may still be roughly right despite the lenient judges, but the absolute scores and close rankings are not trustworthy as measurements of grounding ability. The authors should add human validation on the actual leaderboard responses, report per-model self-preference corrected scores, and either tighten the eligibility consensus rule or be explicit that the leaderboard measures grounding only, not instruction-following.\n\nI'd send this to review. It's a solid, honest benchmark paper with a real weakness in the evaluation loop that the authors have largely disclosed. A good referee can push them to fix the calibration.","headline":"Useful and transparent benchmark, but the judge calibration is too weak to trust absolute scores or close rankings.","tokens_in":15399,"tokens_out":2183,"would_cite":true,"duration_ms":20621,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FACTS Grounding is an online benchmark and leaderboard that measures whether LLM long-form responses are fully grounded in a supplied context document while still answering the user request, using a validated ensemble of judge models.","keywords":["factuality","grounding","long-form generation","LLM-as-a-judge","evaluation benchmark","leaderboard","instruction following","hallucination"],"falsifier":"Take a few hundred responses sampled from the public leaderboard, have human raters label each response as fully grounded or not under the paper's own rubric, and compare those labels with the aggregate verdicts of the three judge models; if the macro-F1 of the judges against human labels on this sample falls materially below the 68.9–71.5 range found on the validation set, the leaderboard's scores and rankings are not measuring grounding as claimed.","tokens_in":14327,"feed_emoji":"🏆","tokens_out":10077,"duration_ms":86349,"temperature":0.7,"pith_summary":"The paper introduces FACTS Grounding, an online benchmark and leaderboard for a specific failure mode of large language models: generating confident long-form answers that drift away from the source document in the prompt. Each of the 1,719 human-written test items pairs a non-trivial user request with a document of up to 32k tokens, and a model's response is scored by whether every information-bearing claim can be verified against that document. Scoring runs in two phases: a response that fails to address the request is first disqualified and counted as inaccurate, and the remaining responses are judged by three different LLM judges whose prompt templates were selected on a human-annotated held-out set. The paper's claim is that this pipeline gives a usable, continuously maintained measurement of grounding ability, and it reports a fused ranking of current models on that measure.","feed_headline":"FACTS Grounding ranks nine LLMs on source-faithful long answers","feed_subtitle":"A two-phase judge pipeline checks every claim against the document and disqualifies evasive replies before scoring.","key_machinery":"The load-bearing mechanism is the two-phase judge pipeline. In the eligibility phase, three prompted LLM judges (Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet) each compare the response against the user request, and a response is declared ineligible only by consensus, so only the worst instruction-following failures are filtered out. In the factuality phase, the same three judge models each classify the full response as 'accurate' or 'inaccurate' against the context, using prompt templates chosen for the highest Macro-F1 (balanced class-averaged F1) against human labels on a private 402-response test set; the unadjusted and final factuality scores are averages of the judges' accuracy percentages. The benchmark itself supplies the task: 860 public and 859 private human-written prompt-document pairs with contexts up to 32k tokens, curated to avoid creative writing, expert knowledge, and heavy reasoning.","core_discovery":"The paper's central claim is a measurement recipe: long-form grounding can be scored at scale by combining an instruction-following filter with three independent LLM judges whose prompts were tuned to match human judgments. Each judge labels a response 'accurate' only if every information-bearing claim is supported by the supplied document or needs no grounding, and the unadjusted factuality score is the average of the three judges' accuracy rates; a response is disqualified and counted as inaccurate only when all three judges agree that it fails the user request. The judge prompts were selected by Macro-F1 (the average of the F1 scores for accurate and inaccurate classes) on a held-out set of 402 human-annotated responses, and the final score is the average of judge percentages after disqualification. On this measure, the paper reports Gemini 2.0 Flash Experimental first and Gemini 1.5 Flash second, with scores in the low-to-mid 80s on both the public and blind splits.","pith_inferences":["Because the selected judge prompts show false positive rates of 32–56% on the paper's own validation set, a natural next step is to re-measure human agreement on live leaderboard responses; if the same leniency persists, the leaderboard scores would overstate true grounding accuracy.","The reported self-preference bias of +3.23% is not removed before ranking; recomputing scores with judges that are not themselves among the evaluated models would show how much of the final order depends on judge family.","The web-sourced context documents may already be memorized by many models; constructing a parallel split with synthetic or fresh documents would separate true grounding from retrieval from parametric memory.","The same public/private split and multi-judge scoring scheme could be applied to other long-context abilities, such as following complex instructions or ignoring misleading context, since the infrastructure is task-agnostic."],"forward_implications":["Models that answer strictly from the supplied document, even when doing so conflicts with knowledge stored during pretraining, will score higher on FACTS Grounding.","Short, evasive responses that would otherwise dodge factuality checks are penalized by the eligibility filter, which treats them as inaccurate and lowers the score.","Because the benchmark spans domains such as medical, legal, financial, and retail text and request types from fact-finding to summarization, a high score indicates general grounding skill rather than task-specific summarization ability.","The public and blind splits allow external model submissions to be compared while keeping a private set that resists overfitting to the leaderboard.","The reported fused ranking, produced by Condorcet aggregation of six judge-split rankings, is the paper's answer to which current models are best at long-form grounded generation."],"supporting_citations":[{"why":"Supplies Gemini 1.5 Pro and Gemini 1.5 Flash, two of the judge models and evaluated response models whose outputs drive the factuality scores.","marker":"Gemini Team: R. Anil et al., 2023"},{"why":"Supplies GPT-4o, one of the three judge models and a response model on the leaderboard.","marker":"Achiam et al., 2023"},{"why":"Supplies Claude 3.5 Sonnet and Claude 3.5 Haiku, one judge model and two response models.","marker":"Anthropic, 2024"},{"why":"Documents self-preference bias in LLM-as-a-judge, the motivation for aggregating multiple judges.","marker":"Wataoka et al., 2024"},{"why":"Provides the definition of faithfulness and attribution that the 'fully grounded' criterion builds on.","marker":"Rashkin et al., 2023"},{"why":"Identifies the short-context or reasoning limits of specialized factuality classifiers that the document-level judge design targets.","marker":"Jacovi et al., 2024"}],"fun_headline_variants":["Benchmark ranks LLM grounding on long-document responses","Automated judges score LLM fidelity to full documents","FACTS Grounding: three-judge benchmark verifies every claim","Leaderboard evaluates if LLM responses are grounded in provided text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings are trustworthy only if the three judge models, taken together, judge grounding the way human raters would on the leaderboard responses themselves; the paper validates the judges on only one held-out set of 402 responses.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark ranks LLM grounding on long-document responses","Automated judges score LLM fidelity to full documents","FACTS Grounding: three-judge benchmark verifies every claim","Leaderboard evaluates if LLM responses are grounded in provided text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000984,"raw_usage":{"total_tokens":4175,"prompt_tokens":948,"completion_tokens":3227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":3157}},"tokens_in":564,"tokens_out":3227,"duration_ms":20484,"temperature":1.0,"reasoning_tokens":3157,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:44.281709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a few hundred responses sampled from the public leaderboard, have human raters label each response as fully grounded or not under the paper's own rubric, and compare those labels with the aggregate verdicts of the three judge models; if the macro-F1 of the judges against human labels on this sample falls materially below the 68.9–71.5 range found on the validation set, the leaderboard's scores and rankings are not measuring grounding as claimed.","supporting_citations":[{"cited_title":"The Claude 3 model family: Opus , Sonnet , Haiku , 2024","cited_arxiv_id":null,"evidence_quote":"Supplies Claude 3.5 Sonnet and Claude 3.5 Haiku, one judge model and two response models."}],"review_version":1}