{"id":"10d8d138-ffb0-4b70-9904-c5047b1387e8","arxiv_id":"2507.02009","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Applying conformal prediction to TSR+OCR outputs flags incorrect table cells; the paper reports 53% labor savings and a 30-point accuracy gain, but the flagging threshold is fitted to the test data.","lead":"An automated table-reading pipeline can flag likely errors using conformal prediction, so humans only need to verify the risky cells instead of every cell. The authors report checking 47% of cells raises accuracy from 53% to 83%, but the flag threshold is tuned on the same test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 47%/30% headline is an in-sample threshold selection result: Sec. 4.1 picks the flagging threshold by maximizing F1 on the evaluation data, so the claimed verification savings are not shown to generalize and lack a random-verification control.","rationale":"The reader's weakest assumption and my concern coincide: the threshold used to define 'flagged' is selected on the same data that produces the headline numbers. This is load-bearing because the entire contribution is quantitative: 53% labor savings, 47% verification, 30% quality gain. Section 4.1 explicitly says the optimal threshold was identified by maximizing F1 on the dataset, and Sections 4.2–4.3 then use the same evaluation setup to compute the metrics. The paper's framework and code release are real supports, and the comparison of score functions is useful, but neither addresses whether τ=0.03 would remain optimal on an unseen test set. A second control—random verification at the same 47% rate—is absent, so the efficiency claim lacks a direct comparison to non-UQ effort allocation. I do not see an internal contradiction in the framework itself; the issue is the validity of the empirical estimate. A re-evaluation with held-out threshold selection and a random baseline would settle it. The reader's CONDITIONAL verdict remains appropriate; no change in verdict is needed.","tokens_in":11168,"tokens_out":5763,"duration_ms":69933,"concrete_test":"Use a table-level split into calibration (50%), validation (25%), and test (25%). Compute the CP quantile q on calibration only; select the APS flagging threshold τ on validation only by maximizing F1; then report precision/recall, labor savings, and accuracy-after-correction on test. Also emulate human correction on a randomly sampled 47% of cells as a control. If out-of-sample F1 or the 30% accuracy improvement drops materially relative to Table 3 and Fig. 4, the headline claim must be restated as conditional on threshold tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that verifying only 47% of extracted cells (53% labor savings) improves accuracy by 30%. For that to be true, the uncertainty scores must flag the cells that are actually wrong. The APS flagging rule uses a threshold τ=0.03, and Sec. 4.1 states that this threshold was selected by scanning 0.01–1 and choosing the value that maximizes F1 'for our dataset'; Sec. 4.3 then reports Table 3 and the 47%/30% figures on that same data. Choosing τ on the evaluation set makes the headline a fitted optimum rather than an estimate of a deployable rule. With only 59 tables and cell-level labels drawn from the same tables, this selection effect is not negligible. The CP quantile q from Eq. 10 is not the final decision threshold; τ is an extra free parameter tuned on the test set. In addition, no random-verification baseline is reported, so 'only 47%' is not compared with correcting a random 47% of cells, which would remove roughly 0.47 × error-rate of the errors. Both omissions bear directly on the central quantitative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TSR-OCR-UQ, a pipeline that integrates table structure recognition (TATR) and OCR (PaddleOCR) with uncertainty scores derived from conformal-prediction-style calibration, in order to flag likely-incorrect extracted cells for human verification. The method is evaluated on the GenTSR and ICDAR-2013 datasets, with several scoring functions (LAC, APS, HSS) compared. The central claim is that by manually verifying only 47% of extracted cells, the data accuracy improves by about 30% (from 53% to 83%), and the verification effort is reduced by 53%. The paper also reports domain-wise precision/recall/F1 for error flagging and includes a discussion of failure modes. Code and data are publicly available.","tokens_in":11448,"tokens_out":6353,"duration_ms":71144,"significance":"If the quantitative claims were established, this would be a practically useful contribution to scientific table extraction, since it would provide a way to prioritize human verification effort using uncertainty information. The paper is also valuable for comparing several uncertainty scoring functions and for releasing an open implementation of the pipeline. However, the headline numbers are currently supported only by an evaluation procedure that selects key thresholds on the test set and lacks a random-verification control, and the connection to conformal prediction's formal coverage guarantee is not established as written. These issues are central, not cosmetic, so the paper needs a substantive revision before the main claims can be accepted.","major_comments":[{"comment":"The APS flagging threshold τ=0.03 is selected by maximizing the F1-score on 'our dataset' (Section 4.1), and the same data is then used to report Table 3 and the headline 47%/30% figures in Section 4.3. Since the labor savings in Eq. (15) is a direct function of the number of flagged cells, selecting τ on the evaluation set makes the reported labor savings and accuracy gains in-sample optima. The paper must split the data into calibration, validation, and test (or use nested cross-validation), select τ on a validation set, and then report the metrics on a held-out test set. The sensitivity of the 47% and 30% numbers to τ should also be reported.","section":"Section 4.1, Table 3"},{"comment":"No random-verification baseline is reported. Correcting a randomly chosen 47% of the cells would be expected to remove roughly 47% of the existing errors, assuming errors are not concentrated. The paper should compare UQ-informed correction against this random baseline (repeated over multiple random seeds) to demonstrate that the flagged cells are enriched for errors. Without this comparison, the 30% accuracy improvement cannot be attributed to the uncertainty scoring rather than to the mere act of verifying a fraction of the cells.","section":"Section 4.3.2, Section 4.3.3"},{"comment":"The scores s_LAC and s_APS are defined using the confidence of the predicted cell Y, not the true label. In conformal prediction, the conformity score must be a function of the true label (e.g., 1 - f(X)_Y for LAC) for the quantile in Eq. (3) to yield the coverage guarantee stated in Eq. (2). As written, the scores are label-independent functions of the model's confidence for its own prediction, so the procedure is a heuristic calibration of confidence scores rather than conformal prediction, and the coverage guarantee does not follow. The authors should either reformulate the scores to depend on the true label during calibration or remove the claim of conformal prediction and describe the method as a calibrated uncertainty-scoring heuristic.","section":"Section 3.3.1, Eq. (5), Eq. (6), Eq. (10)"},{"comment":"The calibration set is described as 'a randomly selected 50% subset of the extracted cells from each domain,' and it is not stated whether the test tables listed in Section 4.2 are disjoint from these calibration cells. If calibration and test cells come from the same tables, within-table correlation can inflate the apparent flagging performance and can break the exchangeability assumption needed for the conformal guarantee. The paper should use table-level disjoint splits for calibration and testing, and should also verify and report the empirical coverage of the prediction sets at the chosen α level, which is currently missing.","section":"Section 3.4, Eq. (2), Eq. (3)"}],"minor_comments":[{"comment":"The '30% improvement' is in absolute percentage points (from 53% to 83%); please state this explicitly to avoid confusion with a relative improvement.","section":"Abstract and Section 4.3.3"},{"comment":"The HSS weight parameters w_row, w_col, and w_text are tuned by a grid search; please state whether this tuning is performed on the calibration set or on the test set, since tuning on the test set would inflate the HSS results.","section":"Section 3.3.1, Eq. (8)"},{"comment":"The treatment of cells with no OCR match (OCR confidence set to 0) should be justified, because blank cells may be correctly empty and setting confidence to 0 may cause systematic false flagging of legitimate blank cells.","section":"Section 3.2.3"},{"comment":"Table 2 reports an APS mean F1-score of 0.761 while Table 3 reports an ALL-domain F1 of 0.590; please clarify whether these are computed over different units (e.g., tables vs cells, or with vs without threshold selection) so that readers can reconcile the two numbers.","section":"Table 2 vs Table 3"},{"comment":"The Levenshtein accuracy can become negative when the Levenshtein distance exceeds the maximum string length; clipping the value to [0,1] would make the metric more interpretable.","section":"Equation (1)"},{"comment":"There is a typo in reference [39], which reads 'ICDAR-2103' and should be 'ICDAR-2013'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central quantitative claims (47% verification effort, 30% accuracy improvement, 53% labor savings) are not yet supported because the flagging threshold is selected on the evaluation set and no random-verification control is reported. The conceptual issue with the conformity scores being label-independent is also important: if the authors cannot reformulate the scores to use true labels, they should reframe the contribution as a calibrated confidence-based flagging heuristic rather than conformal prediction. These are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my quick take on arXiv:2507.02009. The paper applies conformal prediction to integrated TSR+OCR table extraction, and that combination is genuinely new. The grid-cell integration between TSR and OCR is a reasonable engineering step, and comparing LAC, APS, and their HSS score on this task is a sensible ablation. They also ship code and data, which is more than many papers in this area.\n\nThat said, the central quantitative claims do not hold up as reported. The 47% verification / 30% quality improvement numbers come from a threshold (0.03 for APS) that was selected by maximizing F1 on the very same dataset used to compute those metrics. Section 4.1 is explicit about that. So the headline is a fitted optimum, not an estimate of deployable performance. Without a separate validation split or cross-validation, we have no idea what the real savings would be. In addition, there is no random-verification baseline: correcting a random 47% of cells would remove roughly 0.47 × error rate of errors, and the paper never compares its flagging against that. The conformal prediction coverage guarantee is also not verified; they use the quantile, but they never report actual coverage on the test cells.\n\nThe new HSS score performs worse than the existing APS (F1 0.514 vs 0.761), so the novelty of that piece is limited. The error flagging results in Table 3 are plausible, but again they are threshold-selected in-sample, so the F1/precision/recall numbers are optimistic.\n\nOn the positive side, the framework is model-agnostic and clearly described, the ablation across scoring functions is useful, and the small dataset (59 tables) is at least diverse across domains. The finding that APS beats OCR-only confidence is interesting, though it would be stronger with confidence intervals.\n\nWho gets value from this? Researchers working on human-in-the-loop document extraction, and anyone thinking about how to evaluate UQ methods for structured outputs. But the authors need to fix the evaluation protocol: pick the threshold on a validation fold, report random baseline, and check actual coverage. With those changes the paper could be solid.\n\nMy recommendation: send it to peer review, but expect heavy revision. It's a real contribution with a fixable flaw, not a desk reject.","headline":"Interesting idea, but the headline verification savings are in-sample because the flagging threshold was tuned on the test set.","tokens_in":11920,"tokens_out":2183,"would_cite":false,"duration_ms":22295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An uncertainty-aware pipeline for scientific tables flags likely OCR and structure errors, so verifying just 47% of extracted cells raises data accuracy by about 30%.","keywords":["table structure recognition","optical character recognition","uncertainty quantification","conformal prediction","scientific table extraction","adaptive prediction sets","human-in-the-loop","data quality"],"falsifier":"Run the pipeline with the flagging threshold selected on a separate validation split or via nested cross-validation, then count the fraction of cells requiring human review and the post-correction accuracy; if the 47%, 53%, and 30% figures degrade materially, the headline savings are an artifact of threshold selection on the evaluation set.","tokens_in":10991,"feed_emoji":"📊","tokens_out":4321,"duration_ms":46323,"temperature":0.7,"pith_summary":"The paper tries to show that uncertainty quantification can make human-in-the-loop extraction from complex scientific tables practical. It builds a TSR-OCR pipeline and wraps it in conformal prediction, giving each extracted cell an uncertainty score; cells above a threshold are flagged for review. On GenTSR and ICDAR-2013 tables, it reports that reviewing only 47% of cells improves data accuracy by about 30% (from roughly 53% to 83%), and cuts verification work by 53%. A reader should care because scientific table extraction currently demands full manual verification to be usable; this is a concrete, model-agnostic way to target verification where errors actually are.","feed_headline":"Verifying 47% of table cells lifts extraction accuracy 30%","feed_subtitle":"Conformal prediction scores flag likely OCR and layout errors, cutting manual review effort by 53%.","key_machinery":"The central object is the conformal score assigned to each constructed grid cell. TSR row and column boxes are intersected to form grid cells; OCR text boxes are matched to cells by intersection-over-area exceeding 50%; each cell inherits a TSR confidence (average of row and column confidences) and an OCR confidence (average of matched text-box confidences). Conformal prediction on a calibration half of the cells yields a threshold q-hat, and each cell receives uncertainty U(X)=max(0, s(X)-q-hat). The paper compares three score functions: LAC (minimum of TSR and OCR confidence), APS (cumulative confidence), and a hybrid spatial score, with APS giving the best F1 and becoming the primary choice.","core_discovery":"The central claim is that conformal prediction applied to combined TSR and OCR confidences can reliably flag incorrectly extracted cells in scientific tables. On the paper's own terms, with the APS score the pipeline flags most errors (overall recall about 65%) while keeping most flagged cells genuinely wrong (overall precision about 70%), and if a human corrects only the flagged cells, overall data accuracy improves by about 30%, with Computer Science tables improving from 53% to 97%. The framework is model-agnostic: any TSR or OCR model that outputs confidence scores can be plugged in, and the uncertainty score is produced by comparing a conformal score against a calibration-set threshold.","pith_inferences":["Editorial inference: because the 0.03 flagging threshold was selected on the evaluation data itself by maximizing F1, the headline 47% and 30% figures likely overstate out-of-sample performance; a validation-set or nested threshold selection would give a fairer estimate of the savings.","Editorial inference: the same per-cell uncertainty scores could be used to order the human review queue rather than binary flagging, so a reviewer with a fixed time budget could stop after the highest-uncertainty cells and still capture most errors.","Editorial inference: the grid-cell representation with uncertainty attached could feed downstream database loading or scientific knowledge bases by marking low-confidence cells as requiring review instead of silently inserting potentially wrong values."],"forward_implications":["If the UQ signal is trusted, extraction pipelines can stop verifying every cell and instead route only flagged cells to a human reviewer, preserving most of the accuracy gain at half the labor cost.","Because the framework only requires confidence outputs from the underlying models, replacing the OCR or TSR engine with a better one should directly reduce both true errors and false flags without re-engineering the UQ layer.","APS is the recommended score function among those tested; OCR-only confidence is close but slightly worse, and TSR-only confidence is clearly worse, so both structure and text uncertainty matter for error detection.","UQ-informed correction works unevenly across domains: tables with clean structure and short text (Computer Science) benefit most, while Biology and Materials Science retain more errors due to symbols and long content, pointing to a need for domain-adaptive scoring."],"supporting_citations":[{"why":"Supplies the GenTSR in-house dataset of 386 complex scientific tables with ground-truth cell boxes, used for the main evaluation.","marker":"[18]"},{"why":"Provides the ICDAR 2013 standard TSR benchmark tables used as a second evaluation set.","marker":"[19]"},{"why":"Provides PaddleOCR, the OCR engine whose per-text confidence scores feed the uncertainty computation.","marker":"[33]"},{"why":"Provides TATR, the table structure recognition model whose row, column, and cell boxes and confidences form the grid-cell representation.","marker":"[23]"},{"why":"Supplies the conformal prediction machinery of quantile thresholds and coverage guarantees that the uncertainty scores are built on.","marker":"[13]"},{"why":"Defines the LAC conformal score, one of the baseline score functions compared in the paper.","marker":"[41]"},{"why":"Defines the APS conformal score, which the paper selects as the best-performing score function.","marker":"[42]"}],"fun_headline_variants":["Conformal prediction targets bad cells, cutting manual checks","Uncertainty-aware extraction: verify 47%, gain 30% accuracy","Flag likely errors in tables, halve verification workload","UQ for tables: human verifies only half, accuracy up 30%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported verification savings and quality gains assume the flagging threshold can be chosen on the same data used to measure success; if the threshold must be fixed on separate validation data, the 47% and 30% figures would likely change.","fun_headline_variants_meta":{"raw":{"variants":["Conformal prediction targets bad cells, cutting manual checks","Uncertainty-aware extraction: verify 47%, gain 30% accuracy","Flag likely errors in tables, halve verification workload","UQ for tables: human verifies only half, accuracy up 30%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1642,"prompt_tokens":896,"completion_tokens":746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":672}},"tokens_in":512,"tokens_out":746,"duration_ms":8559,"temperature":1.0,"reasoning_tokens":672,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:54:20.144406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline with the flagging threshold selected on a separate validation split or via nested cross-validation, then count the fraction of cells requiring human review and the post-correction accuracy; if the 47%, 53%, and 30% figures degrade materially, the headline savings are an artifact of threshold selection on the evaluation set.","supporting_citations":[{"cited_title":"A Study on Reproducibility and Replicability of Table Structure Recognition Methods","cited_arxiv_id":null,"evidence_quote":"Supplies the GenTSR in-house dataset of 386 complex scientific tables with ground-truth cell boxes, used for the main evaluation."},{"cited_title":"ICDAR 2013 Table Competition","cited_arxiv_id":null,"evidence_quote":"Provides the ICDAR 2013 standard TSR benchmark tables used as a second evaluation set."},{"cited_title":"Table Transformer, 06 2021","cited_arxiv_id":null,"evidence_quote":"Provides TATR, the table structure recognition model whose row, column, and cell boxes and confidences form the grid-cell representation."},{"cited_title":"Least ambiguous set-valued classifiers with bounded error levels","cited_arxiv_id":null,"evidence_quote":"Defines the LAC conformal score, one of the baseline score functions compared in the paper."},{"cited_title":"Classification with valid and adaptive coverage","cited_arxiv_id":null,"evidence_quote":"Defines the APS conformal score, which the paper selects as the best-performing score function."}],"review_version":1}