{"id":"71107127-5583-460d-8190-cd9de3a4e300","arxiv_id":"2603.18652","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-as-a-judge semantic scoring of PDF table extraction correlates with humans at r=0.93, beating TEDS (0.68) and GriTS (0.70), on a synthetic arXiv-table benchmark of 21 parsers.","lead":"This paper proposes using large language models as judges to score how well PDF parsers extract tables, backed by synthetic PDFs with known LaTeX ground truth. It matters because table extraction underpins scientific data mining, and current string-matching metrics miss when two tables mean the same thing.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The load-bearing concern is transfer of the LLM-judge superiority claim (r=0.93 vs ~0.7) from synthetic LaTeX PDFs to real-world PDFs; the abstract asserts representativeness without a held-out real-PDF human study.","rationale":"The Reader already isolated the precise soft spot: synthetic-to-real transfer is asserted rather than measured, and that premise underwrites both the metric-superiority claim and the parser-ranking guidance. Full-text inspection is still required for protocol details, judge model identity, error bars, and whether any real-PDF subset was used, so confidence remains LOW and CONDITIONAL is the correct holding verdict. No stronger internal inconsistency is visible from the abstract alone; the concern is domain validity, not circularity or arithmetic error. The concrete test above is the minimal experiment that would settle whether the concern lands.","tokens_in":2118,"tokens_out":574,"duration_ms":6706,"concrete_test":"Collect a held-out set of ~50 real (non-arXiv-synthetic) scientific PDFs containing tables, extract with the same 21 parsers, obtain ~200 human quality judgments on the resulting pairs under the identical protocol, recompute Pearson r for LLM-judge vs TEDS/GriTS, and test whether the LLM advantage remains statistically significant (e.g., bootstrap CI for r_LLM - r_TEDS excludes 0). If the gap collapses or reverses, the transfer claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM-as-a-judge yields substantially higher human correlation (Pearson r=0.93) than TEDS (0.68) or GriTS (0.70) on >1,500 judgments, thereby providing a superior metric for ranking the 21 parsers on 451 tables. That claim is only as strong as the domain on which the judgments were collected. The framework uses synthetically regenerated PDFs from arXiv tables with precise LaTeX ground truth. The abstract asserts these ensure \"realistic complexity and diversity,\" yet supplies no quantitative comparison of layout quirks, multi-page spans, font embedding artifacts, nested structures, or OCR-like noise against production PDFs. If the human validation pairs were drawn exclusively (or predominantly) from the synthetic distribution, the elevated correlation may simply reflect that LLM judges and humans both find the residual errors easy to score when ground-truth structure is clean and generation artifacts are limited. In that case the metric ranking and the practical parser-selection guidance do not automatically transfer, which is exactly the use case the paper advertises. This is the single most load-bearing unvalidated premise; everything else (matching pipeline, 21-parser ranking) inherits its risk.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript proposes a benchmarking framework for PDF table extraction that uses synthetically regenerated PDFs built from arXiv tables with precise LaTeX ground truth. Its central methodological claim is that an LLM-as-a-judge semantic metric, embedded in a matching pipeline that tolerates parser output inconsistencies, correlates substantially more strongly with human quality judgments (Pearson r=0.93) than standard structure-based metrics TEDS (r=0.68) and GriTS (r=0.70), based on a human validation study of over 1,500 judgments. The framework is then used to rank 21 contemporary PDF parsers on 100 synthetic documents containing 451 tables, with the stated goals of practical parser-selection guidance and a reproducible evaluation methodology. Code and data are released.","tokens_in":2360,"tokens_out":972,"duration_ms":14023,"significance":"If the reported human-correlation advantage and the parser rankings transfer beyond the synthetic setting, the work would address a genuine gap: rule-based table metrics often penalize semantically equivalent extractions and mis-rank systems for scientific data mining. The combination of a large human validation study, multi-parser comparison, and public code/data is a clear strength and would give the community a usable, scalable alternative to TEDS/GriTS for semantic table evaluation. The result is therefore of practical interest to document AI and scientific knowledge-base construction, contingent on domain validity of the synthetic benchmark.","major_comments":[{"comment":"Abstract: the claim that arXiv-sourced synthetic PDFs ensure 'realistic complexity and diversity' is load-bearing for both the metric superiority result and the parser-selection guidance, yet no held-out human study on real production PDFs is reported. Without quantitative comparison (layout quirks, multi-page spans, font/embedding artifacts, nested structures, OCR-like noise) or a real-PDF human correlation check, it remains unshown that r=0.93 vs ~0.7 and the 21-parser ranking transfer to the use case the paper advertises. A real-PDF validation subset or an explicit transfer experiment is needed.","section":null},{"comment":"Abstract (human validation, n>1500): the Pearson correlations are the central evidence for preferring LLM-as-a-judge over TEDS/GriTS, but the abstract does not specify judge model/family, prompt, blinding, inter-annotator agreement, pair sampling, score scale, or whether humans and the LLM saw identical matched pairs. These design choices determine whether r=0.93 is interpretable and whether residual circularity exists if the judge family overlaps LLM-based parsers among the 21 systems. Full study protocol and IAA must be reported and, if overlap exists, ablated.","section":null},{"comment":"Abstract (r=0.93 vs 0.68/0.70): the superiority claim needs uncertainty quantification—confidence intervals on the correlations, a test of difference between dependent correlations, and stratification by error type (structure vs content vs alignment). Without this, it is unclear whether the gap is robust across table difficulty and whether TEDS/GriTS fail systematically on the same cases the LLM judge handles.","section":null}],"minor_comments":[{"comment":"Abstract: clarify how the matching pipeline defines table-level pairs when parsers split, merge, or drop tables; this affects both human labels and metric scores.","section":null},{"comment":"Abstract: state the number of human raters, rating scale, and aggregation method for the >1,500 judgments so readers can assess reliability before the full text.","section":null},{"comment":"Abstract: name which of the 21 parsers are rule-based, ML-based, or LLM-based so the ranking can be interpreted relative to the LLM judge.","section":null},{"comment":"Provide, in the full manuscript, error bars or significance tests on parser rankings across the 451 tables and document-level aggregation details.","section":null}],"recommendation":"major_revision","confidential_remarks":"This is effectively an abstract-only review; confidence is limited until the full study design, IAA, and any real-PDF transfer evidence appear. The synthetic-to-real transfer issue is the single point that could move the paper from major_revision toward reject if unaddressable within scope; if the full paper already contains a real-PDF human study, that major comment can be dropped. Scope fit for cs.CV is reasonable but borderline with IR/document-AI venues—editor may wish to confirm audience."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know: this is a methods/benchmark paper that claims LLM-as-a-judge for PDF table extraction correlates with humans at r=0.93 versus ~0.7 for TEDS/GriTS, on >1500 judgments, plus a ranking of 21 parsers on 451 arXiv-sourced synthetic tables. That is the load-bearing result.\n\nWhat is new and done well is the packaging. They build synthetic PDFs from real arXiv tables with LaTeX ground truth, add a matching pipeline for messy parser outputs, run a sizable human correlation study, and ship code/data links. LLM-as-judge is not novel as a pattern, and TEDS/GriTS are the standard baselines they position against, but the combination—arXiv tables, matching, large human study, 21-parser ranking—is a concrete empirical contribution for document AI. If the study design is clean, people will use the ranking and the metric. Credit for shipping the artifacts and for grounding the metric claim against external human judgments rather than against the LLM itself.\n\nSoft spot, in proportion: we only have the abstract. The transfer premise is the real one. They assert the synthetic PDFs have “realistic complexity and diversity,” but there is no held-out real-PDF human study mentioned. If the 1500 judgments are all on the synthetic distribution, the high correlation may not travel to production PDFs with font quirks, multi-page spans, nested mess, and OCR-like noise—the exact setting where people need parser guidance. That is load-bearing for the practical claims. Secondary unknowns we cannot check: judge model identity, prompt, blinding, inter-annotator agreement, sampling of pairs, error bars, and whether any of the 21 parsers share an LLM family with the judge. Circularity risk looks low from the abstract structure, but residual family overlap is possible.\n\nWho it is for: people who extract tables from scientific PDFs or who build/evaluate document parsers. A serious referee should see it; the numbers and the public ranking are important enough inside the subfield even if the synthetic-to-real gap forces revision. I would not desk-reject. I would want the full protocol and a real-PDF check before treating the ranking as production advice.\n\nRecommendation: send to peer review. Ask specifically for human-study details and for evidence that the metric superiority and parser order hold outside the synthetic set.","headline":"Useful subfield benchmark and metric study if the human correlation holds; main risk is synthetic-only validation for real-PDF claims.","tokens_in":3028,"tokens_out":579,"would_cite":false,"duration_ms":6018,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM judges of extracted PDF tables match human quality scores far better than TEDS or GriTS, ranking 21 parsers on synthetic arXiv tables.","keywords":["PDF table extraction","LLM-as-a-judge","semantic evaluation","TEDS","GriTS","synthetic benchmarks","arXiv tables","parser ranking"],"falsifier":"A held-out human study on real (non-synthetic) scientific PDFs in which the same LLM judge and the same parsers are scored against human quality labels; if the LLM–human correlation falls near TEDS/GriTS levels or the parser ranking reorders, the central claim fails.","tokens_in":2952,"feed_emoji":"📊","tokens_out":769,"duration_ms":6592,"temperature":0.7,"pith_summary":"Reliably pulling tables out of PDFs matters for scientific data mining, but current scores treat tables as rigid trees or grids and miss when two tables mean the same thing. This paper builds a benchmark of synthetic PDFs whose tables come from arXiv and whose exact LaTeX source serves as ground truth, then scores extractions with an LLM acting as a semantic judge inside a matching pipeline that tolerates messy parser output. A human study of more than 1,500 quality judgments shows the LLM scores correlate with people at Pearson r=0.93, well above the 0.68–0.70 range of TEDS and GriTS. Running 21 contemporary parsers on 100 documents with 451 tables then yields a practical ranking and a reusable evaluation method.","feed_headline":"LLM table judges beat TEDS and GriTS at matching human scores","feed_subtitle":"On 1,500+ human judgments, semantic scoring ranks 21 PDF parsers more reliably than structural metrics.","key_machinery":"LLM-as-a-judge semantic scoring, integrated into a matching pipeline that accommodates structural and formatting inconsistencies between parser outputs and LaTeX ground truth; the judge supplies a continuous quality score that tracks human perception of table equivalence.","core_discovery":"An LLM-as-a-judge semantic evaluation of PDF table extractions, placed inside a matching pipeline that handles inconsistent parser outputs, reaches Pearson r=0.93 with human quality judgments—substantially higher than Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70)—and thereby supplies a more reliable metric for ranking parsers on a synthetic arXiv-derived benchmark.","pith_inferences":["The same LLM-judge pattern could be reused for other semi-structured PDF elements (figures with captions, multi-column equations, nested lists) where structural metrics also diverge from human notions of correctness.","If the synthetic-to-real transfer holds only for clean academic LaTeX, commercial or scanned PDFs may still need a separate, harder calibration set.","High human correlation of the LLM judge suggests automated leaderboards for table extraction could be updated continuously without repeated large-scale human studies."],"forward_implications":["Parser selection for large-scale scientific table mining can be guided by the reported ranking of 21 systems rather than by TEDS or GriTS alone.","Future table-extraction papers can adopt the LLM-judge pipeline and the synthetic arXiv-derived corpus as a reproducible evaluation standard.","Semantic equivalence, not just structural edit distance, becomes the default notion of correctness for table extraction metrics.","Matching pipelines that tolerate layout and serialization differences become necessary scaffolding around any semantic judge."],"fun_headline_variants":["LLM judges hit r=0.93 with humans on PDF table quality","Semantic LLM scoring beats TEDS and GriTS for table extraction","LLM-as-judge ranks PDF parsers more like humans than TEDS","Human judgments align tighter with LLM table metrics than GriTS","Beyond structure: LLM evaluates PDF tables at r=0.93 human match"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That synthetically regenerated PDFs built from arXiv tables with perfect LaTeX ground truth are representative enough of real-world PDF table difficulty that the metric correlations and parser rankings transfer to production documents.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges hit r=0.93 with humans on PDF table quality","Semantic LLM scoring beats TEDS and GriTS for table extraction","LLM-as-judge ranks PDF parsers more like humans than TEDS","Human judgments align tighter with LLM table metrics than GriTS","Beyond structure: LLM evaluates PDF tables at r=0.93 human match"]},"model":"grok-4.5","effort":"low","cost_usd":0.008502,"raw_usage":{"total_tokens":2019,"prompt_tokens":807,"num_sources_used":0,"completion_tokens":94,"cost_in_usd_ticks":85020000,"prompt_tokens_details":{"text_tokens":807,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1118,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":807,"tokens_out":94,"duration_ms":9198,"temperature":1.0,"reasoning_tokens":1118,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T22:28:48.729829+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A held-out human study on real (non-synthetic) scientific PDFs in which the same LLM judge and the same parsers are scored against human quality labels; if the LLM–human correlation falls near TEDS/GriTS levels or the parser ranking reorders, the central claim fails.","supporting_citations":[],"review_version":1}