{"id":"0b5dcf2c-6f51-4b9a-9aef-5ef20052c869","arxiv_id":"2607.18240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A tool-using LLM fact-checker that abstains on 6 of 95 claims reaches 97.8% accuracy on answered claims, but a simpler search baseline remains more accurate and better calibrated.","lead":"A fact-checking AI that can answer \"not sure\" instead of true or false stays very accurate on the claims it does answer. On a 95-claim test, a search-based baseline still answers everything more accurately, so the selective system's real-world advantage is not yet proven.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deferral concentration at L4 (5/6) is not significant after base-rate adjustment (L4=60/95; per-level rate 8.3% vs L2's 10%), and L4 tags are self-assigned, so the safety-oriented abstention claim is unsupported.","rationale":"The reader's weakest assumption was that model-generated source-level tags are a reliable proxy for evidence quality, making the safety interpretation partly tautological. My stress-test confirms this and sharpens it with a base-rate statistical argument: even if the L4 labels were ground truth, the 5/6 concentration is not significant given that L4 covers 63% of the benchmark. Moreover, per-level abstention rates at L2 and L4 are nearly identical (10% vs 8.3%), so the data do not show that abstention preferentially targets lower-reliability evidence. This reinforces the reader's conditionality: the paper should either add base-rate-corrected analyses, independent source annotations, or explicit error analysis of deferred cases. The core numbers (97.8% selective accuracy at 93.7% coverage) remain internally consistent, so the verdict should stay CONDITIONAL rather than move to REJECT; the issue is an over-interpretation that can be fixed with additional analysis, not a fatal error in the experimental setup. Therefore I recommend no change to the reader's verdict.","tokens_in":7608,"tokens_out":4275,"duration_ms":37194,"concrete_test":"Perform a two-sided Fisher exact test on the 2×2 table formed from Fig. 4: L4 abstain=5, L4 answer=55; non-L4 abstain=1, non-L4 answer=34. If p>0.05 (estimated p≈0.29), the claim that deferrals concentrate in L4 is not statistically supported. Also compare the L2 abstention rate (1/10) with the L4 rate (5/60) using a test of proportions; if the difference is not significant, the 'lower-reliability' interpretation fails. Optionally, have independent human annotators rate the evidence quality of the 6 deferred claims blind to the model's L-tags, and test whether their ratings are lower than for the 89 answered claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central interpretive claim (Abstract, §6.1) is that ECE's abstention is a safety-oriented mechanism for handling weak evidence, supported by 5/6 deferrals at source level L4. This is not established. First, L4 is the modal source level (60/95 claims); under random abstention the expected number of L4 deferrals is 6×60/95 ≈ 3.8, and the probability of observing ≥5 is ≈0.29 (exact binomial/Fisher). Thus the observed concentration is fully consistent with chance. Second, the per-level abstention rate at L4 (5/60 = 8.3%) is not higher than at L2 (1/10 = 10%), where L2 corresponds to academic papers — a putatively higher-reliability source. Third, the L4 labels are assigned by the system itself at inference time (§4.4, Fig. 4), so the same agent that decides to abstain also assigns the source-level tag; the apparent correlation may simply reflect the model's own confidence rather than independent evidence quality. Without a base-rate-corrected analysis or independent source-quality annotation, the conclusion in §6.1 overreaches the data. This matters because the 'safety-oriented' interpretation is the paper's distinctive qualitative contribution; the quantitative results alone show only that ECE defers 6 of 95 claims while performing worse than the Search baseline on standard accuracy and all aggregate calibration metrics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Evidence Chain Evaluation (ECE), a tool-using LLM verification agent that can output an 'uncertain' verdict in addition to confirmed/refuted. The agent is evaluated on a newly introduced 95-claim benchmark, ECE-Bench, comparing against Vanilla, CoT, and Search baselines. The central quantitative result is that ECE achieves 91.6% standard accuracy, 97.8% selective accuracy, and 93.7% coverage by deferring 6 of 95 claims; the Search baseline is better on standard accuracy and all reported calibration metrics (ECE, Brier, AURC). The paper's distinctive qualitative claim is that abstention is safety-oriented, with 5 of 6 deferrals falling in source-level L4, which the authors interpret as evidence that abstention targets epistemically weak evidence.","tokens_in":7962,"tokens_out":5957,"duration_ms":47410,"significance":"If the results held, the paper would provide a modest demonstration that a selective fact-checking agent can maintain high accuracy on the claims it chooses to answer while abstaining on a small subset. The metric definitions in Section 3 are clearly stated, and the paper is commendably honest in reporting that ECE does not beat the strongest retrieval baseline on aggregate calibration metrics. The release of code is a positive feature. However, the significance is limited by the small benchmark (95 claims, 6 deferrals) and, more importantly, by the fact that the paper's distinctive interpretive claim about 'safety-oriented' abstention is not statistically supported. The quantitative selective-accuracy result is internally consistent, but its practical value is undercut by ECE's worse AURC relative to Search, meaning ECE does not improve the risk–coverage frontier. The central contribution therefore reduces to the existence of an explicit abstention mechanism rather than a demonstrated reliability advantage.","major_comments":[{"comment":"The claim that deferrals are 'concentrated in lower-reliability evidence settings' is not supported by the data. With 60 of 95 claims at L4, under random abstention the probability that 5 or more of 6 deferrals fall at L4 is approximately 0.28 (hypergeometric: C(60,5)C(35,1)+C(60,6) over C(95,6)), so the observed 5/6 is fully consistent with chance. The per-level abstention rate at L4 (5/60 = 8.3%) is not higher than at L2 (1/10 = 10%), and Fig. 4b shows L4 accuracy (90.0%) is identical to L2 accuracy (90.0%). The safety-oriented interpretation in §6.1 therefore overreaches the data; a base-rate-corrected analysis and, ideally, independent source-quality annotation are required.","section":"§5.4, Fig. 4, Abstract, §6.1"},{"comment":"The paper's framing that ECE 'delivers a clear selective-prediction trade-off' is difficult to reconcile with its own aggregate metrics. Search has a better AURC (0.0082 vs 0.0182), and the risk–coverage curves in Fig. 3 show ECE at higher risk at essentially all coverage levels. Thus ECE does not improve the risk–coverage frontier relative to the strong baseline; it simply has an explicit abstention action. The selective-accuracy comparison at 93.7% vs 100% coverage is not apples-to-apples. Please provide coverage-matched comparisons (e.g., by thresholding Search confidence to the same coverage) or reframe the conclusion to acknowledge that the selective-prediction trade-off is not superior to the baseline.","section":"§5.3, Table 1, Fig. 3"},{"comment":"The source-level tags are assigned by the same VerificationAgent that decides whether to abstain, so the conclusion that abstention targets 'epistemically weak evidence' rests on the model's self-assessment of evidence quality. The paper acknowledges this in the Limitations section but still uses the self-assigned tags to support the abstract's main interpretive claim. This is a circularity concern: the L4 tag may be another expression of the model's low confidence rather than an independent measure of evidence reliability. The paper should either obtain independent source-quality annotations or present evidence that L4 claims are objectively harder (e.g., lower accuracy for all systems) before drawing safety conclusions.","section":"§4.4, Fig. 4, §6.1"},{"comment":"The evaluation is based on only 95 claims and the key abstention analysis uses just 6 deferrals. The 95% confidence interval for the selective accuracy of 97.8% on 89 answered claims is roughly [92.2%, 99.6%], and the deferral-level analysis is severely underpowered. ECE-Bench is introduced without a detailed description of claim construction, gold-labeling procedure, or annotation quality. At minimum, report exact confidence intervals and provide the benchmark's constitution (or a datasheet); more importantly, the paper's stronger qualitative conclusions should be softened given the small n.","section":"§5.1, §5.4"}],"minor_comments":[{"comment":"The acronym ECE is used both for the system (Evidence Chain Evaluation) and the metric (Expected Calibration Error), e.g., §3.2 vs §4. This is confusing; please use distinct labels or spell out the metric at first use.","section":"Throughout"},{"comment":"The 'Uncertain Rate by Source Level' panel appears to show bars for L2 and L4 only, with L1 and L3 at zero not visibly labeled. A small table with counts and rates per level would be clearer.","section":"Figure 4(c)"},{"comment":"The mechanism by which the model produces the scalar confidence score is not described. Is it a verbalized confidence, a logit-based probability, or a heuristic? This matters for interpreting ECE, Brier, and AURC.","section":"§4.3"},{"comment":"Typo: 'anuncertain verdict' should be 'an uncertain verdict.' Also, the GitHub URL contains a space after 'github.com'; please fix.","section":"Abstract"},{"comment":"Several references contain 'and 1 others' (e.g., Huang et al., 2023; Nakano et al., 2021; Schick et al., 2023). Please complete the author lists according to the journal style.","section":"References"},{"comment":"For the Brier score, the treatment of 'uncertain' as P(true)=0.5 is reasonable, but the paper should state whether the scalar confidence is also used for uncertain verdicts in the ECE and AURC computations, as §3.2 implies.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and the metric definitions are clear, but the distinctive qualitative contribution—the safety-oriented interpretation of abstention—is not supported by the reported statistics. The small benchmark size and self-assigned source-level labels compound the problem. The authors should either significantly reframe the contribution to a more modest claim about the feasibility of abstention or provide the additional analysis requested (base-rate-corrected, independent labels, coverage-matched comparisons)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's measurements look internally consistent, the authors are admirably clear that their strongest baseline beats them on standard accuracy and calibration, and the code is out. But the paper's distinctive qualitative claim—that abstention is a 'safety-oriented mechanism' targeting weak evidence—rests on a concentration of deferrals at source level L4 that is statistically indistinguishable from chance once you account for the base rate.\n\nWhat's new: a tool-using LLM agent that can return 'uncertain' plus a confidence score, evaluated on a new 95-claim benchmark. That is a direct application of existing selective classification and retrieval-augmented fact-checking, not a new capability, but the empirical data are new and the selective accuracy numbers are clearly reported.\n\nWorth credit: metric definitions in §3 are correct; the paper explicitly notes that uncertain verdicts count as incorrect in standard accuracy; and the Limitations section concedes the main weaknesses. The authors do not oversell the aggregate metrics.\n\nThe soft spot is the safety interpretation. In §4.4 and Figure 4, 5 of 6 deferred claims fall in L4. But L4 is the modal source level (60/95 claims). By chance, with 6 abstentions, the expected number of L4 deferrals is 3.8, and P(≥5) ≈ 0.29. So the 'concentration' is entirely compatible with random abstention. Worse, the per-level abstention rate at L4 (5/60 = 8.3%) is not even the highest—L2, academic papers, abstains 1/10 = 10%. And the L4 tags are assigned by the same agent that decides to abstain, so the correlation may just reflect the model's own confidence rather than an independent measure of evidence quality. The paper flags the tags as system-generated, but still uses them to support the central 'safety-oriented' conclusion. That conclusion is not established.\n\nThe paper also lacks a deferred-case error analysis: we never learn whether the 6 abstained claims are actually hard or error-prone for the baselines. Without that, 'safety-oriented' is a label, not a result. The small benchmark (95 claims) and missing benchmark release further limit precision.\n\nI think this deserves serious refereeing because the quantitative core is honest and the flaw is fixable: add a base-rate-adjusted analysis, use independent source annotation, and compare baseline errors on the deferred set. As is, the right verdict is conditional on those revisions. I wouldn't cite it as evidence for safety-oriented abstention, but for a reading group on selective prediction and statistical fallacies it's a useful case study.","headline":"Honest small study of selective fact-checking, but the safety-oriented abstention claim does not survive a base-rate check.","tokens_in":8442,"tokens_out":2911,"would_cite":false,"duration_ms":23900,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tool-using fact-checking agent can answer 'uncertain' when evidence is weak, and the paper shows it reaches 97.8% accuracy on the claims it does answer while deferring 6 of 95 cases.","keywords":["selective prediction","fact-checking","abstention","uncertain verdicts","LLM agents","tool use","evidence quality","calibration"],"falsifier":"Ask independent annotators to rate the reliability of the evidence available for each of the 95 claims without seeing ECE's outputs or source tags. If the six deferred claims are not rated lower in evidence quality than the answered claims, or if fewer than five of six fall into the weakest web/news tier, then the claim that abstention is safety-oriented toward weak evidence is refuted. A cheaper check: re-run ECE with a different model backend and observe whether the L4 abstention concentration reappears.","tokens_in":7461,"feed_emoji":"🔎","tokens_out":5685,"duration_ms":46632,"temperature":0.7,"pith_summary":"The paper's central claim is that fact-checking systems should be allowed to abstain, and that a tool-using agent built around this idea can stay accurate on what it answers while flagging what it cannot support. On a 95-claim benchmark, the Evidence Chain Evaluation (ECE) agent gathers evidence through web search, scholarly search, and executable checks, then returns a verdict of confirmed, refuted, or uncertain. ECE answers 89 claims with 97.8% accuracy and defers 6, and five of the six deferrals occur when the evidence is at the low-reliability web/forum level (L4). The paper reads this as a safety-oriented mechanism: instead of forcing a confident true/false answer on weak evidence, the system conservatively withholds judgment. The paper is explicit that ECE does not beat the strongest search baseline on overall accuracy or aggregate calibration; its contribution is the coverage–risk trade-off and a transparent abstention channel.","feed_headline":"Fact-checking agent abstains on weak evidence, hits 97.8% accuracy","feed_subtitle":"Withholding 6 of 95 claims lets the system nearly match the strongest baseline on the answers it gives.","key_machinery":"The load-bearing object is Evidence Chain Evaluation (ECE), a tool-routed verification agent built on an LLM with function calling. It plans and executes up to eight rounds of evidence gathering across web search, scholarly search, and a code-execution engine, then emits a structured verdict consisting of a ternary label (confirmed, refuted, or uncertain), a confidence score, a source-level tag (L1 code, L2 papers, L3 documentation, L4 web/news/forum, L5 internal knowledge), and a short reasoning string. The uncertain verdict is the selective-abstention mechanism: rather than a separately computed uncertainty score, the model itself decides whether evidence is decisive, which turns abstentio","core_discovery":"On the paper's own terms, the discovery is that a selective fact-checking protocol — one that lets the model emit uncertain instead of forcing a binary verdict — achieves high accuracy on exactly the subset it chooses to answer. Concretely, ECE reaches 97.8% selective accuracy at 93.7% coverage on ECE-Bench by deferring 6 of 95 claims; of those six, five are tagged L4 (web pages, news, or forums), the weakest non-internal evidence tier. The deferred set is split evenly between true and false gold labels (3 and 3), so abstention is not a directional bias. The paper's intended conclusion is that uncertain functions as a safety valve against overconfident verdicts when the evidence chain is wea","pith_inferences":["The 95-claim benchmark makes the six-deferral pattern statistically fragile; a natural extension is to test on a larger fact-checking corpus and see whether the abstention rate and the L4 concentration persist.","Because the L4 tags are produced by the same agent that decides to abstain, the 5-of-6 concentration may partly reflect the model labeling its own uncertain cases; an independent evidence-quality annotation would separate genuine evidence weakness from model self-signaling.","The paper's framing suggests a deployment pattern: pair ECE's abstention with a separate confidence-ranked retriever for the answered subset, potentially improving aggregate calibration while retaining selective safety.","One testable extension is to vary the model backend: if deferrals move away from L4 with a different LLM, that would indicate the concentration is model-specific rather than a property of the evidence chain."],"forward_implications":["In a real fact-checking pipeline, ECE's 6 deferred claims would be routed to human reviewers, shrinking the audit surface from 95 to 6 while nearly matching the strongest baseline's accuracy on the answered subset.","Because deferrals are balanced between true and false gold labels, users can interpret uncertain as 'evidence insufficient' rather than 'likely false'.","The risk–coverage curve gives operators a dial: accepting lower coverage should yield even higher accuracy on answered claims, or the curve quantifies the cost of adding coverage.","The abstention mechanism requires no retraining or separate uncertainty module; it is a change in output protocol, so it can be layered onto existing tool-using LLM agents."],"fun_headline_variants":["Fact-checker abstains on weak evidence, hits 97.8% accuracy","Selective fact-checking: 97.8% accuracy by deferring 6 weak claims","Abstention boosts fact-check accuracy to 97.8% on answered claims","Fact-check agent skips weak evidence, keeps 97.8% accuracy","Deferring weak evidence gives fact-checker 97.8% selective accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The safety-oriented reading depends on the system-generated L4 tag being a reliable measure of evidence quality; if L4 is mainly a label the agent assigns to cases it already doubts, then the 5-of-6 abstention concentration does not independently show that abstention targets genuinely weak evidence.","fun_headline_variants_meta":{"raw":{"variants":["Fact-checker abstains on weak evidence, hits 97.8% accuracy","Selective fact-checking: 97.8% accuracy by deferring 6 weak claims","Abstention boosts fact-check accuracy to 97.8% on answered claims","Fact-check agent skips weak evidence, keeps 97.8% accuracy","Deferring weak evidence gives fact-checker 97.8% selective accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1156,"prompt_tokens":792,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":256}},"tokens_in":536,"tokens_out":364,"duration_ms":3365,"temperature":1.0,"reasoning_tokens":256,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T16:14:59.664541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask independent annotators to rate the reliability of the evidence available for each of the 95 claims without seeing ECE's outputs or source tags. If the six deferred claims are not rated lower in evidence quality than the answered claims, or if fewer than five of six fall into the weakest web/news tier, then the claim that abstention is safety-oriented toward weak evidence is refuted. A cheaper check: re-run ECE with a different model backend and observe whether the L4 abstention concentration reappears.","supporting_citations":[],"review_version":1}