{"id":"49864bbb-da17-4952-a2af-f2b2f103da62","arxiv_id":"2506.13328","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A coarse-to-fine LLM framework for document-level numerical semantic matching achieves about 90% F1 on financial disclosure documents, outperforming the prior AutoCheck system by roughly ten points.","lead":"This paper presents CoFiTCheck, a two-stage system that uses an embedding-based filter and a specialized language model to check whether numbers appearing in different tables of the same document mean the same thing. It reports around 90% F1 on three types of Chinese financial disclosure documents, roughly ten points above the previous method, while processing a document in under a minute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison to AutoCheck may not be apples-to-apples; re-run AutoCheck on the same new splits and check recall upper bounds before trusting the ~10 point margin.","rationale":"I agree with the reader's weakest-assumption analysis. The paper itself states the AutoCheck results are 'as presented in the original paper [12]' while claiming 'the same training and test splits', which is internally ambiguous at best, and the three datasets are new collections that did not exist in AutoCheck's KDD 2020 work. Since the headline claim is a comparative one, this is a load-bearing provenance problem rather than a style issue. The efficiency numbers (Table IV) and the internal ablations (Figures 5-6) are plausibly sound and are not the primary risk. A clean re-run of AutoCheck on the same split is the single test that would settle the matter. I keep the verdict at CONDITIONAL because the concern is about comparability and reproducibility, not about the method being incoherent; the paper should be accepted only if the baseline comparison is repaired or explicitly caveated.","tokens_in":18961,"tokens_out":1507,"duration_ms":12404,"concrete_test":"Obtain AutoCheck's released implementation or, failing that, re-implement its cell embedding network and heuristic filtering per [12], and run it on the authors' 10% test splits of the auditor, IPO, and annual report datasets. If the re-run F1 on auditor's reports and IPO prospectuses is within ~2 points of the Table II AutoCheck values (75.4 / 81.3) and the annual report F1 is reported, the comparison stands; if the re-run differs materially or annual reports cannot be run, the headline margin should be re-quantified with matched baselines.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CoFiTCheck surpasses prior methods by ~10 F1 points (e.g., 11.6 on auditor's reports, 9.0 on IPO prospectuses), but Table II reports AutoCheck's numbers 'as presented in the original paper [12]', not recomputed on the authors' new Chinese IPO, auditor, and annual report test splits. The reader's concern is well-founded: AutoCheck's KDD 2020 evaluation used different documents and different annotation pipelines, so unless those exact splits match, the comparison conflates method strength with dataset differences. The paper also omits AutoCheck's F1 on annual reports entirely, even though annual reports are one of the three central benchmarks and where the headline 90.8% is achieved. Separately, Section IV-A.4 says 'we use the same training and test splits' — but this is ambiguous: the authors' own splits were newly constructed (8:1:1 at document level on their three collected corpora), so AutoCheck could not have been trained on 'the same' splits in its original paper. If the AutoCheck row is taken from the original paper, the 'same splits' clause cannot be literally true. The observed positive-negative ratios are also extreme (1:73,362), so even a small number of spurious true positives or a slightly different filtering recall can move F1 by several points; without running AutoCheck on these exact splits, the margin is not established. To settle this, the most direct check is to re-run AutoCheck's released pipeline on the authors' test splits (or, if code/data are unavailable, to quote the exact provenance of each AutoCheck number in Table II).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoFiTCheck, a two-stage coarse-to-fine framework for document-level tabular numerical cross-checking. Stage 1 uses an LLM-based embedding filter (CIPE with a decoupled InfoNCE objective) to prune candidate numerical mention pairs; Stage 2 uses a specialized discriminative LLM (ClsLLM) to classify the remaining pairs. A cross-table numerical alignment pretraining (CNAP) method is also proposed to improve ClsLLM without manual annotations. Evaluation on new Chinese IPO prospectuses, auditor's reports, and annual reports reports F1 scores around 87-91% with a 7B ClsLLM and an average processing time of 40.8 seconds per document, purportedly surpassing the previous AutoCheck system by about 10 F1 points.","tokens_in":19250,"tokens_out":5758,"duration_ms":57961,"significance":"If the results hold, this would be a meaningful advance: CoFiTCheck appears to be the first LLM-based coarse-to-fine system that makes document-level numerical consistency checking both accurate and practical. The CIPE parallel-encoding idea, the decoupled InfoNCE objective for isolated mentions, and the CNAP weak-supervision pretraining are all well-motivated and potentially reusable beyond this specific task. The paper also contributes three annotated Chinese financial-disclosure benchmarks and a detailed efficiency analysis. However, the headline superiority over AutoCheck is currently undermined by an unclear baseline comparison, and the embedding threshold is selected on the test set, so the reported F1 numbers are not clean held-out results. The technical core is defensible, but the empirical claims need substantial verification before the paper can be accepted.","major_comments":[{"comment":"Table II reports AutoCheck's P/R/F1 \"as presented in the original paper [12]\" and also states \"We use the same training and test splits,\" but the three corpora in this paper are newly constructed in Section IV-A.1; AutoCheck's original KDD 2020 evaluation could not have used these splits. The AutoCheck row for annual reports is empty, so the claimed 11.6- and 9.0-point F1 improvements are at best comparisons against a different dataset and annotation pipeline, not controlled comparisons on the new benchmarks. Please re-run AutoCheck's released pipeline on the exact same training/test splits (or clearly identify which numbers were recomputed and how), and provide the missing annual-report numbers. Without this, the abstract's claim of \"surpassing previous methods by around 10 points\" is unsupported.","section":"IV-A.4, Table II"},{"comment":"The embedding similarity threshold t=0.5 is fixed in Section IV-A.3 based on the analysis in Section IV-E, and Section IV-E explicitly states: \"Using 126 test set documents as benchmark, we vary the embedding similarity threshold from 0.1 to 0.9.\" Thus t is selected on the test set, so the end-to-end F1 scores in Table II are not clean held-out numbers and are likely optimistic. The threshold (and any hyperparameters read off these curves) should be selected on the validation split or via nested cross-validation, with the test split reserved for final evaluation.","section":"IV-A.3, IV-E"},{"comment":"CNAP pretrains on 11,635 annual reports \"crawled from a stock exchange website,\" while the annual-report benchmark is one of the three evaluation corpora collected by the authors. The paper does not state that the test (or validation) annual reports were excluded from the pretraining corpus. If any overlap exists, the CNAP gains shown in Fig. 6 and Table II would be confounded by data leakage. Please state explicitly how the pretraining documents were deduplicated against the training, validation, and test sets, or re-run CNAP with all benchmark documents removed from pretraining.","section":"IV-A.1, IV-B, Fig. 6"}],"minor_comments":[{"comment":"There are typos such as \"heuristc\" and \"capabilites\" in the baselines subsection; please proofread.","section":"IV-A.4"},{"comment":"The 7B CoFiTCheck row presents values without spacing (e.g., \"87.086.987.092.8\"), making the table hard to read; use proper column alignment or additional vertical lines.","section":"Table II"},{"comment":"The 1k-sample evaluation of SOTA LLMs is a single random draw with no reported seed or confidence intervals; please report the sampling procedure and variability (e.g., multiple draws or bootstrap intervals).","section":"IV-C"},{"comment":"The runtime comparison with AutoCheck (166.6 seconds) appears to come from the original paper and may have been measured on different hardware; please add a caveat or, ideally, measure AutoCheck on the same four-GPU setup.","section":"IV-D"},{"comment":"The rows for \"Heuristic-based Filtering\" and \"Removing Stage 1\" are estimates based on average processing times, not measured runs; the table should clearly mark these rows as estimates rather than only mentioning it in a footnote.","section":"IV-D"},{"comment":"The relevance score R(t_i,t_j) counts equal numerical mentions; if a table contains repeated identical values that are not semantically equivalent cross-table pairs, this could overestimate relevance. Please clarify how repeated values are handled in the equality count.","section":"III-D, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the AutoCheck baseline: the manuscript appears to compare against numbers copied from AutoCheck's original paper rather than recomputed on the new splits, and the annual-report row is missing. If the authors cannot provide a true apples-to-apples comparison, the \"around 10 points\" claim should be removed or substantially weakened. The test-set threshold selection also needs to be corrected. The core methodology and the efficiency analysis are publishable in principle, but the current evaluation does not support the headline claims as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper before trusting the abstract: the CoFiTCheck system is interesting and the efficiency is real, but the claimed ~10 point F1 margin over AutoCheck is not established. The key issue is Table II: AutoCheck's numbers are reported 'as presented in the original paper' while CoFiTCheck is evaluated on the authors' new Chinese IPO, auditor, and annual report corpora. Those are not the same splits, and the 'same training and test splits' sentence in IV-A.4 cannot be literally true because the authors constructed their own 8:1:1 splits. Without re-running AutoCheck's released pipeline on the same test documents, the comparison conflates method with data and annotation differences.\n\nThere is real novelty here. The contextualized instructional parallel encoding (CIPE) with attention masking is a legitimate engineering contribution for encoding many mentions in one pass, and the decoupled InfoNCE objective sensibly addresses the isolated-mention problem that standard contrastive losses mishandle. The CNAP pretraining idea – reordering tables so equal-valued mentions appear in the same context window – is clever, and the ablation against reading-order pretraining, while small, points in its favor. The runtime numbers are credible: 12.4s for stage 1 and 40.8s total with a 7B ClsLLM, with a 25x slowdown when parallel encoding is removed.\n\nThe soft spots, in order of severity. First, the embedding similarity threshold t=0.5 is selected using the 126 test documents in Section IV-E, so the reported F1s are not clean held-out numbers. That needs a validation split or at least a sensitivity analysis. Second, the AutoCheck row is missing entirely for annual reports, the very corpus where CoFiTCheck claims its best F1 (90.8%). Third, ROPT/CNAP deltas are 0.1–0.7 F1 points; without error bars or multiple seeds, those could be noise. Fourth, the CNAP pretraining corpus is 11,635 annual reports from a stock exchange, and the annual report test set is presumably drawn from the same source; the paper does not say the test documents were excluded from pretraining. That is a possible leakage that should be clarified. Finally, no code or data link is given, so the results are not independently reproducible.\n\nAll of this is fixable. The method is worth engaging with, and the paper deserves a serious referee, but the referee should require a re-run of AutoCheck on the same splits, a principled threshold selection, and explicit deduplication between pretraining and test documents. I'd be cautious about citing the F1 numbers as-is.\n\nRegards.","headline":"Potentially useful coarse-to-fine system for numerical consistency checking, but the headline margin over AutoCheck rests on a baseline from a different corpus and a test-set-tuned threshold.","tokens_in":19827,"tokens_out":2746,"would_cite":false,"duration_ms":25189,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-stage AI checks cross-table numbers with 90% F1 in 41 seconds","keywords":["numerical cross-checking","tabular fact-checking","document-level verification","numerical semantic matching","large language models","contrastive learning","financial disclosure documents","coarse-to-fine retrieval"],"falsifier":"Take the three new benchmarks (auditor's reports, IPO prospectuses, annual reports), retrain AutoCheck on the same training split, and run it on the same test split with the same set-level precision, recall, and F1 metrics. If its F1 comes out materially higher than the numbers quoted in Table II, the claimed roughly 10-point improvement over prior work is not established.","tokens_in":1600,"feed_emoji":"📊","tokens_out":1858,"duration_ms":70830,"temperature":0.7,"pith_summary":"The paper proposes CoFiTCheck, a coarse-to-fine system for checking that the same numerical fact is stated consistently across the tables of a disclosure document. It claims the system reaches roughly 90% F1 on auditor's reports, IPO prospectuses, and annual reports, beating the previous end-to-end method by about 10 points, and that it processes one document in 40.8 seconds on four RTX 4090 GPUs. The significance is that document-level numerical verification, previously dominated by either heuristic filtering that caps recall or full pairwise LLM inference that is too slow, becomes accurate and fast enough for routine financial-disclosure review.","feed_headline":"Two-stage AI checks cross-table numbers with 90% F1 in 41 seconds","feed_subtitle":"The pipeline filters millions of candidate pairs, then a specialized 7B model checks what remains.","key_machinery":"The load-bearing object is the coarse-to-fine cascade. It has three named mechanisms: Contextualized Instructional Parallel Encoding (CIPE), which jointly encodes all numerical mentions of a table in one LLM forward pass with attention masking; a decoupled InfoNCE objective, which separately trains non-isolated mentions to cluster with their semantic equivalents and isolated mentions to stay apart; and Cross-Table Numerical Alignment Pretraining (CNAP), which reorders tables in documents so that tables sharing equal numbers appear adjacent and then continues next-token pretraining. Together they make candidate pruning high-recall and cheap, and make the classifier specific to numerical semantics in financial documents.","core_discovery":"The central claim is that treating numerical cross-checking as two sequential stages, embedding-based candidate filtering followed by discriminative classification with a specialized LLM, solves both the combinatorial explosion of candidate pairs and the need to understand multi-faceted numerical semantics. The embedding stage encodes every numerical mention in a table in a single forward pass and prunes pairs by embedding similarity; the classification stage then reads the full table context of each surviving pair and decides semantic equivalence, with values masked so the model cannot cheat by matching raw numbers. The paper reports F1 of 87.0% on auditor's reports, 90.3% on IPO prospectuses, and 90.8% on annual reports with a 7B classifier, and shows that its cross-table numerical alignment pretraining adds gains without requiring manual annotation.","pith_inferences":["Because CIPE and CNAP are not tied to Chinese financial language, the same two-stage recipe should transfer to tables in scientific papers, regulatory filings, and technical reports; that transfer is an extrapolation, since the paper only evaluates Chinese financial disclosures.","The embedding similarity threshold (chosen as 0.5 here) makes the system tunable: a deployment that fears missing an inconsistency can lower it at the cost of more classifier calls, and one that must bound latency can raise it, with the paper's recall-versus-candidate-pairs curves giving the trade-off.","CNAP's table-relevance graph, built from equality of numeric mentions, is a reusable weak-supervision signal: any downstream task that benefits from placing tables that share values near each other in the language-model context could adopt the same maximum-path ordering."],"forward_implications":["At 7B scale the system reaches F1 of 87.0% on auditor's reports, 90.3% on IPO prospectuses, and 90.8% on annual reports, so the method works across three document genres.","A full document takes 40.8 seconds on four RTX 4090 GPUs, versus 166.6 seconds reported for the prior system and many hours of manual review, making routine deployment plausible.","Removing the embedding stage would inflate per-document processing to roughly 1.5 days with the 0.5B classifier or 12.9 days with the 7B classifier, so the recall-preserving filter is what makes the classifier affordable.","CNAP raises F1 without labeled data and lets the 3B classifier match the 7B classifier without pretraining on two of the three document types.","A 0.5B task-finetuned classifier beats the strongest general-purpose and reasoning-specialized LLMs tested in zero-shot classification, by up to roughly 14 F1 points."],"supporting_citations":[{"why":"Supplies the only prior end-to-end system for the task and the baseline whose F1 and runtime are compared in Tables II and IV.","marker":"[12]"},{"why":"Provides the Qwen2.5 backbone used for both the embedding model and the classifier.","marker":"[21]"},{"why":"Supplies the hierarchical navigable small-world graph algorithm used to make candidate-pair search near-linear.","marker":"[41]"},{"why":"Provides the approximate nearest-neighbor search library used to retrieve candidate pairs in the filtering stage.","marker":"[42]"},{"why":"Supplies the in-context pretraining idea of ordering related content into a sequence, which CNAP adapts to cross-table numerical alignment.","marker":"[47]"},{"why":"Defines extractive parallel encoding, the ablation alternative that CIPE is compared against.","marker":"[29]"},{"why":"Defines the standard InfoNCE objective used as the ablation baseline for the decoupled InfoNCE loss.","marker":"[56]"},{"why":"Supports the instruction-tuned format rationale behind the contextualized instructional parallel encoding prompt.","marker":"[40]"}],"fun_headline_variants":["Coarse-to-fine LLM method verifies numbers across tables fast","Filter-then-classify AI catches numeric errors in disclosures","Two-stage LLM pruning checks millions of table pairs in seconds","Pretrained 7B model cross-checks financial tables with 90% F1","Efficient LLM pipeline detects inconsistent numbers in documents"],"cache_read_input_tokens":21888,"weakest_assumption_plain":"The reported gains over AutoCheck assume the baseline numbers quoted from AutoCheck's original paper were computed on the same training and test splits and the same three document benchmarks; if those numbers came from a different evaluation, the claimed margin is not apples-to-apples.","fun_headline_variants_meta":{"raw":{"variants":["Coarse-to-fine LLM method verifies numbers across tables fast","Filter-then-classify AI catches numeric errors in disclosures","Two-stage LLM pruning checks millions of table pairs in seconds","Pretrained 7B model cross-checks financial tables with 90% F1","Efficient LLM pipeline detects inconsistent numbers in documents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1332,"prompt_tokens":966,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":276}},"tokens_in":582,"tokens_out":366,"duration_ms":3642,"temperature":1.0,"reasoning_tokens":276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:05:09.148965+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the three new benchmarks (auditor's reports, IPO prospectuses, annual reports), retrain AutoCheck on the same training split, and run it on the same test split with the same set-level precision, recall, and F1 metrics. If its F1 comes out materially higher than the numbers quoted in Table II, the claimed roughly 10-point improvement over prior work is not established.","supporting_citations":[{"cited_title":"Cracking tabular pre- sentation diversity for automatic cross-checking over numerical facts,","cited_arxiv_id":null,"evidence_quote":"Supplies the only prior end-to-end system for the task and the baseline whose F1 and runtime are compared in Tables II and IV."},{"cited_title":"In-context pretraining: Language modeling beyond document boundaries,","cited_arxiv_id":null,"evidence_quote":"Supplies the in-context pretraining idea of ordering related content into a sequence, which CNAP adapts to cross-table numerical alignment."},{"cited_title":"Generative representational instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Defines extractive parallel encoding, the ablation alternative that CIPE is compared against."}],"review_version":1}