{"id":"092939dc-c580-4d16-8117-83b5d313bfee","arxiv_id":"2507.06195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Longer context windows and right-to-left number tokenization do not improve numerical fact verification; evidence quality is the main bottleneck.","lead":"This paper tests whether giving a fact-checking AI more evidence text or a special number-reading trick makes it better at checking claims with numbers. Neither helped; the authors found that the quality of the retrieved evidence matters most, and their system still placed fourth in a shared competitor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'evidence quality is the bottleneck' conclusion rests on the Benchmark→Our-Data comparison where retrieval, claim decomposition, reranker, and training epochs all change at once; evidence quality is never measured, so the attribution is unsupported.","rationale":"The reader's weakest_assumption is exactly the load-bearing gap I identified: the paper attributes the performance drop to evidence quality without measuring evidence quality or isolating retrieval, decomposition, reranker, and training choices. My stress-test confirms this is the least secure condition for the central claim. I also noted that the R2L comparison could be undermined by applying a reversed tokenization to a pretrained model without re-pretraining, but the evidence-quality attribution is more central because the abstract and conclusion use it to justify the overall negative finding about context and tokenization. The inconsistent validation/test F1 values in the text versus Tables 1 and 3 compound the problem by making the quantitative basis of the claim difficult to verify. Because the reader already returned a CONDITIONAL verdict at moderate confidence, my concern reinforces that verdict rather than moving it; the paper should be accepted only if the authors supply direct evidence-quality metrics and an ablation that controls the confounded variables, or soften the bottleneck claim to a hypothesis.","tokens_in":8074,"tokens_out":4803,"duration_ms":52395,"concrete_test":"Add an ablation that isolates evidence quality: take the Our-Data pipeline but feed it (a) the organizer's official evidence and (b) the recreated evidence, and separately ablate claim decomposition on/off, while keeping the classifier, reranker, and number of training epochs fixed; report macro-F1 over at least 3 seeds plus retrieval quality metrics (Recall@3, MRR, and overlap with official evidence) for each condition. If official evidence restores the Benchmark-level macro-F1, the bottleneck claim is supported; if not, the drop is due to pipeline differences, not evidence quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 1, the Benchmark row (MathRoBERTa, organizer pipeline, validation macro-F1 0.56) is contrasted with Our-Data (same classifier, recreated pipeline, validation 0.52), and the drop is interpreted as 'poorer evidences.' But at least four variables change simultaneously: claim decomposition via a GPT-4o-mini prompt with custom temperature/penalties; BM25 retrieving 50 documents per decomposed sub-claim; reranking with MiniLM-L-12-v2; and training epochs (Table 2: 5 epochs for Benchmark vs 7 for Our-Data, with different fine-tuning schedules). The paper reports no direct evidence-quality measurement, such as recall of the official QuanTemp gold evidence, MRR of the reranker, or overlap between retrieved and organizer-provided evidence. Consequently, the central conclusion that longer context and R2L are ineffective 'in absence of high-quality evidence' is an interpretation of a confounded comparison rather than a demonstrated bottleneck. The conclusion is made even harder to verify by inconsistent reported F1 values: the text says validation 0.63 and test 0.58, while Table 1 lists submission validation macro-F1 0.57 and Table 3 lists test macro-F1 0.52. A single checkpoint's single-run result without significance testing cannot carry the load of the paper's headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This CLEF 2025 working-notes paper describes a system for Task 3 of CheckThat! on numerical and temporal fact verification over the QuanTemp corpus. The authors reimplement the QuanTemp evidence pipeline (GPT-4o-mini claim decomposition, BM25 retrieval, MiniLM cross-encoder reranking) and fine-tune ModernBERT as the NLI/veracity classifier, with MathRoBERTa and few-shot GPT-4o-mini baselines. They report ablations that vary input context length (256 vs. 1,024 tokens, with 1 vs. 3 evidence snippets), left-to-right vs. right-to-left numerical tokenization, and the use of PEFT or focal loss. The headline conclusion is that neither longer context nor R2L tokenization improves veracity prediction, and the authors attribute this to poor evidence quality from their recreated retrieval pipeline. Their best system reaches a validation macro-F1 of 0.57 and a reported test macro-F1 of 0.52/0.58 in different parts of the text, ranking 4th out of 10 submissions.","tokens_in":8502,"tokens_out":3477,"duration_ms":39522,"significance":"If the central conclusion were fully supported, the paper would provide a useful negative result for numerical fact verification: that model-side interventions such as longer context windows and altered numerical tokenization cannot compensate for weak retrieved evidence. The paper has strengths worth acknowledging: it is evaluated on a public shared-task benchmark, it reports a systematic set of ablations, it reproduces a substantial retrieval pipeline, and it makes code available. Its results also complement prior work by suggesting that the R2L-tokenization benefit observed in arithmetic reasoning does not transfer to NLI on numerical claims. However, the paper's main attribution of performance differences to 'evidence quality' is not supported by any direct measurement, and the reported numbers are internally inconsistent. As a result, the significance of the headline claim is currently limited by methodological and reporting issues rather than by the absence of a clear evaluation setup.","major_comments":[{"comment":"The central 'evidence quality is the bottleneck' conclusion is based on the Benchmark (0.56) vs. Our-Data (0.52) comparison, but at least four pipeline components change simultaneously: the claim-decomposition prompt and sampling parameters, the BM25 retrieval depth over decomposed sub-claims, the reranker model, and the training schedule (5 vs. 7 epochs in Table 2). The paper never directly measures evidence quality, for example by computing recall of the official QuanTemp gold evidence, reranker MRR, or overlap between the retrieved and organizer-provided evidence sets. Without such a measure, the paper cannot distinguish poorer evidence from other pipeline differences, so the abstract and conclusion statements about evidence quality as 'the dominant bottleneck' are unsupported as stated.","section":"Section 5, Table 1"},{"comment":"The manuscript contains irreconcilable numeric discrepancies in the main result: the text states a validation macro-F1 drop from 0.63 to a test value of 0.58, while Table 1 lists the Submission validation macro-F1 as 0.57 and Table 3 lists the test macro-F1 as 0.52. The text also describes the validation result as 0.57 in the preceding paragraph. The authors must identify which checkpoint and which table row correspond to each reported number, and correct the inconsistent values, because the discrepancy prevents a reader from verifying the claimed validation-to-test generalization drop.","section":"Section 5, Tables 1 and 3"},{"comment":"RQ1 asks whether longer context improves veracity prediction, but the proposed comparison conflates two factors: Short-Context uses 1 evidence snippet and a 256-token window, whereas Long-Context uses 3 evidence snippets and a 1,024-token window. Any performance difference (or lack of it) could be due to the number of evidences rather than the context-window size. The experiment should either hold the evidence count fixed while varying the token budget, or the paper should explicitly acknowledge that RQ1 as tested is about the joint effect of evidence quantity and context length.","section":"Section 5, RQ1"},{"comment":"The paper claims that focal-loss results are 'not statistically significant' from the cross-entropy Submission results, but no statistical test, confidence interval, or variance estimate is reported anywhere, and each configuration appears to be a single run. With differences on the order of 0.01-0.05 in macro-F1, and no seed variation, the statement is not verifiable. The absence of repeated runs or significance testing also weakens the RQ2/RQ3 conclusions, which are based on the same single-run comparison.","section":"Section 5, Focal Loss"}],"minor_comments":[{"comment":"The sentence 'epochs until fine-tuning is finished (early-stopping after 2 epochs)' is ambiguous with respect to the epoch counts of 5-7 listed in the table; please clarify whether the early-stopping patience is 2 epochs and report the actual stopping epoch, or explain why the epoch counts include the patience window.","section":"Table 2"},{"comment":"The paper cites GPT-4o-mini with reference [12], which is the GPT-4 technical report; please provide the correct citation or documentation for GPT-4o-mini.","section":"References"},{"comment":"The claim of 'a noticeable separation of False claims from the other two categories' is based only on a qualitative inspection of a UMAP projection; consider adding a quantitative measure such as silhouette scores or pairwise centroid distances, or soften the wording to reflect the exploratory nature of the plot.","section":"Figure 1"},{"comment":"References [6] and [13] list the author 'A. Anand' twice; please correct the author lists.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task notebook paper, and the bar for publication in that venue is typically lower than for a full research paper. My concerns are not about the negative result itself, which is reasonable to report, but about the unsupported causal attribution and the unreproducible numeric reporting. If the authors can supply evidence-quality metrics, fix the reported numbers, and add at least basic repeated-run or significance information, the paper would be a solid workshop contribution. I do not see a reason to reject outright, because the issues are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing in this paper is the negative result: right-to-left tokenization does not transfer from arithmetic benchmarks to NLI-style numerical fact verification, and longer context windows don't help either within their setup. That's a result worth having on the record, and it's the first empirical test of R2L on this task. The paper is honest about its limitations, makes code available, and engages directly with the QuanTemp baseline. The ablation structure is clear, and they correctly note that the long-context null result agrees with QuanTemp's original findings.\n\nThe soft spots are real, but they don't wipe out the core negative findings. The headline conclusion that 'evidence quality is the dominant bottleneck' is an interpretation, not a measurement. The Benchmark→Our-Data comparison changes claim decomposition, retrieval, reranker, and training epochs all at once; no retrieval recall, MRR, or evidence overlap metric is reported to support the claim that their pipeline produces poorer evidence. The paper also contains internal inconsistencies: the text reports validation macro-F1 of 0.63 and test 0.58, while Table 1 lists the submission run at 0.57 and Table 3 lists 0.52. Those numbers need to be reconciled. And all comparisons rest on single runs with no error bars or significance tests; the flat statement that focal loss results are 'not statistically significant' with no procedure is a red flag.\n\nNone of that makes the RQ1/RQ2 conclusions wrong. The ablations showing no benefit from longer context or R2L are internally consistent, assuming the table typos are fixed. The evidence-quality attribution is the weak link, and the authors should either measure retrieval quality directly or soften the claim. As a shared-task notebook, this is fine; as a formal paper, it would need revision.\n\nThis paper is for the shared-task and numerical fact-checking community. A careful reader gets value from the negative results. I'd send it to peer review rather than desk reject, with the expectation that the authors fix the numbers, add proper uncertainty quantification, and either justify or retract the evidence-quality bottleneck claim.","headline":"Useful negative result on R2L tokenization and long context for numerical fact verification, but the headline claim about evidence quality being the bottleneck is asserted rather than demonstrated.","tokens_in":8883,"tokens_out":3346,"would_cite":false,"duration_ms":33422,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Numerical fact-checking is limited by the quality of retrieved evidence, not by how much context the model sees or how numbers are tokenized.","keywords":["numerical fact verification","QuanTemp dataset","ModernBERT","right-to-left tokenization","context window","evidence retrieval","veracity prediction","natural language inference"],"falsifier":"Use oracle evidence (the snippets the benchmark labels as supporting each claim) and the paper's BM25-plus-reranker evidence for the same claims, keeping classifier, context length, and tokenizer fixed. If longer context or R2L tokenization improves macro-F1 only when oracle evidence is used, the paper's claim is refuted; if it never improves even with oracle evidence, the claim is strengthened.","tokens_in":7883,"feed_emoji":"🔢","tokens_out":8993,"duration_ms":90049,"temperature":0.7,"pith_summary":"Automated fact-checking of numerical claims fails for a different reason than many model-centered approaches assume. This paper shows, through a controlled ablation on numerical claims and an evidence corpus of over 400,000 snippets, that giving the veracity model more retrieved evidence (a longer context window) or changing the order in which numbers are tokenized (right-to-left instead of left-to-right) does not improve veracity prediction. The authors attribute the limiting factor to the quality of the retrieved evidence: when they rebuilt the evidence-retrieval pipeline themselves, performance fell relative to the original benchmark, and no model-side intervention recovered the gap. If the claim holds, effort in numerical fact-checking should shift from bigger inputs and special number tokenizers to stronger evidence retrieval and reranking.","feed_headline":"Extra evidence and digit reversal don't help number fact-checking","feed_subtitle":"Evidence quality, not context size or token order, sets the ceiling for veracity prediction.","key_machinery":"The load-bearing machinery is an ablation that varies exactly two model-side factors while holding the classifier architecture fixed. Evidence is retrieved by decomposing each claim into sub-questions with GPT-4o-mini, running BM25 sparse retrieval, and reranking with a cross-encoder; the reconstructed pipeline is then fed to a ModernBERT natural-language-inference classifier. The experiment contrasts one evidence snippet with a 256-token context against three snippets with a 1,024-token context, and swaps the tokenizer to right-to-left digit order. The drop from the organizer's benchmark macro-F1 (0.75 train, 0.56 validation) to the recreated pipeline (0.56 train, 0.52 validation) carries the paper's inference about evidence quality.","core_discovery":"The paper's central claim is that in the absence of high-quality evidence, neither a longer context window (three evidence snippets at 1,024 tokens versus one snippet at 256 tokens) nor right-to-left tokenization of numbers improves veracity prediction for numerical and temporal claims. This directly contradicts earlier findings on arithmetic reasoning, where right-to-left number tokenization helped. The authors further argue that the performance drop between the original benchmark pipeline and their recreated retrieval pipeline is a sign that evidence quality is the dominant bottleneck, since identical classifiers perform worse when fed evidence from their own BM25-plus-reranker retrieval. Their best system, a ModernBERT classifier with one evidence snippet per decomposed sub-question and a 1,024-token context, reaches a macro-averaged F1 of 0.57 on validation and 0.52 on test.","pith_inferences":["If the paper is right, the fastest way to confirm the bottleneck is to hold classifier and retrieval fixed and swap in oracle evidence; the paper does not run this control.","Task-family transfer is the deeper lesson: a tokenization change that helps arithmetic can fail on numerical NLI, so results should not be carried across benchmarks without re-testing.","Part of the ceiling may be label-side rather than evidence-side: the paper's own embedding plot shows True and Conflicting claims overlapping heavily, suggesting coarse label mapping could cap any retrieval improvement.","A practical test of the paper's implication would gate evidence by reranker confidence, adding a snippet only when it clears a threshold, rather than always feeding three."],"forward_implications":["Veracity systems for number-heavy claims should concentrate effort on retrieval and reranking quality rather than on enlarging the input window.","R2L tokenization, which helps arithmetic reasoning, does not transfer to natural-language inference over numerical claims; tokenizer choices need task-specific validation.","Supplying additional weak evidence snippets neither helps nor hurts: with poor retrieval, three snippets behave like one.","A single evidence snippet per decomposed sub-question with a 1,024-token context can match the benchmark's validation macro-F1 of 0.57.","LoRA-style parameter-efficient fine-tuning and focal loss do not recover the gap left by weak evidence."],"supporting_citations":[{"why":"Provides the QuanTemp dataset, the claim-decomposition approach, and the benchmark results that the paper recreates and compares against.","marker":"[6]"},{"why":"Reports the right-to-left tokenization benefit for arithmetic reasoning that the paper directly tests and contradicts for natural language inference.","marker":"[7]"},{"why":"Introduces ModernBERT, the long-context encoder whose context-window support enables the comparison of short versus long contexts.","marker":"[8]"},{"why":"Supplies the two-step sparse retrieval plus reranking method that determines evidence quality in the paper's pipeline.","marker":"[16]"},{"why":"Provides the MS MARCO data used to train the cross-encoder reranker the paper relies on.","marker":"[18]"},{"why":"Introduces focal loss, which the paper tests as a remedy for class imbalance and finds unhelpful.","marker":"[9]"}],"fun_headline_variants":["Longer context, reversed digits: no boost for number fact-checking","Evidence quality trumps context and token order in number claims","R2L tokenization fails for numerical NLI, despite arithmetic wins","For number fact-checking, extra evidence won't fix poor retrieval","Top-4 in CheckThat 2025: Quality beats quantity in evidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never directly measures evidence quality; its key conclusion rests on the assumption that its recreated retrieval pipeline differs from the original only in evidence quality, not in claim decomposition, reranker choice, or training details.","fun_headline_variants_meta":{"raw":{"variants":["Longer context, reversed digits: no boost for number fact-checking","Evidence quality trumps context and token order in number claims","R2L tokenization fails for numerical NLI, despite arithmetic wins","For number fact-checking, extra evidence won't fix poor retrieval","Top-4 in CheckThat 2025: Quality beats quantity in evidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1672,"prompt_tokens":902,"completion_tokens":770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":677}},"tokens_in":518,"tokens_out":770,"duration_ms":7472,"temperature":1.0,"reasoning_tokens":677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:07:45.112234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use oracle evidence (the snippets the benchmark labels as supporting each claim) and the paper's BM25-plus-reranker evidence for the same claims, keeping classifier, context length, and tokenizer fixed. If longer context or R2L tokenization improves macro-F1 only when oracle evidence is used, the paper's claim is refuted; if it never improves even with oracle evidence, the claim is strengthened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports the right-to-left tokenization benefit for arithmetic reasoning that the paper directly tests and contradicts for natural language inference."}],"review_version":1}