{"id":"1a449089-c70d-4215-b3e0-87bd41a0f3ab","arxiv_id":"2506.14583","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A synthetic two-column document generator for table detection is introduced, and TableNet is re-evaluated on it and on Marmot with pixel-wise XOR errors reported.","lead":"The authors built an automated LaTeX pipeline that renders two-column document pages containing tables with aligned pixel masks, then trained the TableNet detector on synthetic and real data and compared pixel-wise error at 256x256 and 1024x1024 inputs. The stated 'augmentation' of the Marmot benchmark is not directly tested, since synthetic and real data are never combined in training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'augmentation' claim is untested: no experiment combines synthetic and real training data, so the paper's headline claim does not follow from the reported results.","rationale":"I agree with the reader's weakest-assumption identification: the word 'augments' in the Abstract and Introduction implies a benefit from combining synthetic and real data, but the implementation (Section V.A) never combines them. This is the single most load-bearing gap because the paper's stated contribution and novelty hinge on the augmentation framing; the resolution study and the synthetic data generator are useful and reproducible (code/dataset released, per Sections V.B and VII), but they do not by themselves establish the headline claim. The reader also flagged two further issues that I weigh: (1) test-set leakage from reporting the minimum error across epochs in Table I, which inflates all headline numbers and weakens the resolution comparison; and (2) the use of XOR error alone, without IoU/F1 or a comparison against existing synthetic corpora such as PubTables-1M, which leaves the realism/transfer claim under-supported. These are all correctness risks internal to the paper's protocol, not merely disagreements with consensus. The proper fix is conditional: add a combined-training experiment and validation-based epoch selection, or revise the claim to 'a synthetic corpus enabling a resolution study of TableNet' rather than 'augments the Marmot benchmark.' Given the pipeline's public availability and the clear protocol description, the conditional verdict is appropriate; no evidence here suggests the synthetic data cannot work, only that it has not been shown to work as augmentation.","tokens_in":7096,"tokens_out":1682,"duration_ms":14387,"concrete_test":"Run a combined-training experiment: train TableNet on (a) Marmot only, (b) synthetic only, (c) Marmot + synthetic together, and (d) synthetic pre-training then Marmot fine-tuning; evaluate all on a fixed Marmot test split (and, if feasible, on PubTables-1M) using a single validation split for early stopping, then report IoU/F1 in addition to XOR. If (c) or (d) does not beat (a) by a meaningful margin on real documents, the augmentation claim should be revised; if (c)/(d) clearly improves over (a), the claim is supported. Also rerun the reported numbers with test-set-based epoch selection replaced by validation-based early stopping.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The Abstract and Introduction frame the contribution as data augmentation ('The generated corpus augments the real-world Marmot benchmark'), but Section V.A trains and evaluates models separately on synthetic and Marmot data, with no combined-training experiment (e.g., Marmot + synthetic, or fine-tuning on Marmot after synthetic pre-training). The resolution study is internally coherent, but the augmentation claim requires showing that adding synthetic data improves real-world table detection (or reduces annotation effort while matching performance). As reported, the Marmot results (9.18% best at 256x256) are from a model trained only on Marmot, so they neither support nor refute an augmentation benefit. A second load-bearing issue compounds this: Table I reports the lowest error across training epochs (e.g., Marmot 256x256 at 500 epochs, synthetic at 1540 epochs) selected from the test set, despite the text in Section V.A saying EarlyStopping uses validation loss and Section VI saying 'the lowest XOR error rate... was evaluated at several epoch intervals.' Selecting the best epoch on the test set is test-set leakage, so the reported error rates (4.04%, 4.33%, 9.18%, 13.83%) are optimistic lower bounds. The conclusion 'higher resolution does not universally improve performance' rests partly on this selection. The paper's central claim is therefore best read as a dataset/resolution study, not a demonstrated augmentation result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces an automated LaTeX-based pipeline that renders two-column scientific-style pages containing diverse table layouts with pixel-level ground-truth masks. The authors train the TableNet segmentation model separately on the synthetic corpus and on the Marmot benchmark at two input resolutions (256x256 and 1024x1024), evaluate with a pixel-wise XOR error rate, and report that the synthetic-trained model reaches 4.04% error on the synthetic test set at 256x256, while the Marmot-trained model reaches 9.18% at 256x256. They conclude that the synthetic corpus augments the real-world Marmot benchmark and that higher input resolution does not universally improve table detection.","tokens_in":7460,"tokens_out":3120,"duration_ms":34283,"significance":"If the central claims were fully demonstrated, the paper would make a useful practical contribution: a fully automated, publicly released synthetic document generation pipeline that reduces manual annotation effort and provides a controlled environment for studying architecture and resolution effects in table detection. The authors provide code and data, which is a genuine strength for reproducibility, and the motivation of moving from isolated table crops to full-page layouts is well placed. The significance is currently limited, however, because the augmentation claim is not tested by the experiments as designed, and the reported error rates are selected on the test set, which makes the quantitative claims optimistic lower bounds rather than reliable performance estimates.","major_comments":[{"comment":"The abstract and introduction state that the generated corpus 'augments' the real-world Marmot benchmark, but Section V.A describes only separately trained models: 'For each corpus—both the synthetic images introduced in this work and the original Marmot pages—we randomly split the data, using 90% of the images for training and reserving the remaining 10% for testing.' No experiment combines synthetic and real training data, such as training on Marmot plus synthetic images or pre-training on synthetic data and fine-tuning on Marmot. Consequently, the reported Marmot results (e.g., 9.18% at 256x256) come from a model trained only on Marmot and neither support nor refute an augmentation benefit. The paper should either add a combined-training experiment or reframe the contribution as a dataset and resolution study rather than a demonstrated augmentation result.","section":"Abstract, Section I, Section V.A"},{"comment":"The reported error rates are selected as the minimum over a grid of training epochs using the test set. Section VI states, for example, 'The lowest XOR error rate on the Marmot dataset was 9.18% at 500 epochs' and Table I shows the lowest error for each resolution/dataset combination in bold. Selecting the best epoch from test-set performance is a form of test-set leakage, because the test set is used for model selection. The text mentions EarlyStopping based on validation loss in Section V.A, but the reported numbers are not the validation-selected models. This makes all headline numbers (4.04%, 4.33%, 9.18%, 13.83%) optimistic and invalidates the comparative claims about resolutions and epoch counts. The authors should use a held-out validation set for epoch selection and report the corresponding test errors, or report the final model after training for a fixed number of epochs.","section":"Table I, Section VI"},{"comment":"The resolution comparison rests on single point estimates with no uncertainty quantification and with different batch sizes and epoch ranges for the two resolutions (e.g., batch size 256 for 256x256 versus 64 for 1024x1024). A difference such as 4.04% versus 4.33% on the synthetic test set, or 9.18% versus 13.83% on Marmot, is not established as meaningful without multiple runs and a common model-selection rule. The conclusion that 'higher resolution does not universally improve performance' should be supported by confidence intervals or repeated runs, and by a protocol that treats both resolutions symmetrically.","section":"Section VI, Table I"}],"minor_comments":[{"comment":"The paper does not report any corpus statistics, such as the total number of generated document pages, the number of pages per table style, the number of tables per page, or the sizes of the training and test splits for both the synthetic corpus and Marmot. These numbers are necessary for reproducibility and for interpreting the reported error rates.","section":"Section IV.B"},{"comment":"The rationale for using XOR error over IoU, F1, or mAP is plausible, but the paper reports no standard region-level metric. Adding at least one conventional metric (e.g., IoU or F1) would allow readers to compare the method with prior table-detection work and would show that the XOR-based conclusions are not metric-specific.","section":"Section IV.D.1"},{"comment":"The sentence 'To ensure the uniform input size of 256x256 pixels for images and 1024x1024 pixels for masks' is confusing, because the resolution study consistently compares 256x256 and 1024x1024 for both images and masks. Please clarify the intended image and mask dimensions for each configuration.","section":"Section V.A"},{"comment":"The epoch-interval selection procedure is described only as 'progressive validation'; please specify the exact epoch grid, the EarlyStopping patience and threshold, and whether any validation split was used before the reported test evaluation, so that the experiments can be reproduced exactly.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the real contribution here is a reproducible LaTeX-based generator for full-page synthetic documents with table masks, and a narrow but honest resolution study of TableNet. The 'augmentation' claim in the title and abstract does not follow from the experiments: no run ever mixes synthetic and real training data. That is a framing gap you can fix, not a fatal flaw.\n\nWhat's new: existing synthetic corpora like SynthTabNet, PubTables-1M, and DocBank mostly release isolated table crops. This generator renders two-column pages with surrounding text, captions, figures, five table styles, and aligned masks. The code and data are public. The resolution study is also new for TableNet, and the finding that 1024×1024 doesn't beat 256×256 on Marmot is a useful counterpoint to the assumption that higher resolution always helps.\n\nThe soft spots are real. First, the abstract says the corpus 'augments' Marmot, but Section V.A splits each corpus into train/test separately; there's no combined-training or fine-tuning experiment. The 9.18% Marmot number is from a model trained on Marmot alone, so it says nothing about whether synthetic pages help real-world detection. The conclusion carefully frames the work as synthetic-to-synthetic evaluation, but the title and abstract don't. Second, Table I reports the lowest XOR error across several evaluated epochs (9.18% at 500, 12.14% at 240, 11.69% at 1540 for Marmot 256). Choosing the best epoch on the test set makes the headline numbers optimistic lower bounds. The text says EarlyStopping monitors validation loss, but the reported results aren't tied to that stopping rule. Third, the synthetic test set is generated by the same pipeline as the training set, so the 4.04% is a domain-matched measurement; the paper acknowledges this but still leads with that number.\n\nMinor: the metric choice (XOR) is defensible for boundary sensitivity, but adding IoU/F1 would help comparison with the broader literature. The evaluation stops at TableNet, so the resolution conclusion is architecture-specific. Citation pattern is fine; the relevant synthetic corpora are cited, and the only self-citation is contextual.\n\nWho this is for: people building synthetic document corpora and anyone re-baselining TableNet. The dataset itself is a usable resource. With a revised claim (or a real augmentation experiment) and validation-based early stopping, it would be a solid workshop/short-conference paper.\n\nRecommendation: send it to peer review. The work is reproducible, the dataset fills a gap, and a good referee will ask for the mixed-training experiment or a corrected claim—that's a normal revision path, not a rejection.","headline":"A useful synthetic page generator and a defensible resolution study, but the 'augmentation' claim in the title and abstract isn't tested.","tokens_in":7845,"tokens_out":4062,"would_cite":true,"duration_ms":38091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A LaTeX pipeline that renders synthetic document pages can train TableNet to a 4.04% pixel-wise XOR error on synthetic test pages and a 9.18% error on the real Marmot benchmark.","keywords":["table detection","synthetic data generation","LaTeX document rendering","semantic segmentation","TableNet","Marmot benchmark","XOR error rate","input resolution"],"falsifier":"Run the same TableNet training on the synthetic corpus and score predictions on Marmot with region-level metrics such as IoU or mAP; if the model shows low XOR error but high false-positive or false-negative rates at the bounding-box level, the central transfer claim fails under a different metric, whereas strong region-level scores would support it.","tokens_in":6907,"feed_emoji":"📄","tokens_out":6024,"duration_ms":55828,"temperature":0.7,"pith_summary":"This paper introduces an automated LaTeX-based pipeline that renders synthetic two-column document pages containing diverse tables, with ground-truth masks generated automatically at render time. The authors train TableNet on this corpus and report pixel-wise XOR errors of 4.04% on their synthetic test set and 9.18% on the real Marmot benchmark, both at 256x256 resolution. They present the corpus as a way to cut manual annotation effort and to enable a controlled comparison of input resolutions. The paper's findings are that higher resolution does not universally help and that synthetic-to-synthetic evaluation can expose architecture-dependent trends without domain noise.","feed_headline":"Synthetic pages train table detector to 9.18% error on real data","feed_subtitle":"Automated LaTeX pipeline cuts manual annotation and shows 256x256 beats 1024x1024 on Marmot.","key_machinery":"The load-bearing mechanism is the generation pipeline: a Python script drives LaTeX, taking four parameters (number of rows, number of columns, datatypes, and table style) and producing a rasterized page image plus an aligned binary ground-truth mask. Tables are placed either alone or surrounded by two-column text, forcing a detector to reason about the page layout rather than classify isolated table crops. The trained model is TableNet, an encoder-decoder semantic-segmentation network with a shared VGG-19 encoder and separate table and column decoder branches, and the evaluation metric is the pixel-wise XOR error rate, which counts every misclassified pixel.","core_discovery":"The paper's central claim is that a fully automated LaTeX-based pipeline can generate page-level synthetic documents with diverse table styles and exact ground-truth masks, and that this corpus is sufficient to train TableNet to competitive table-detection performance. The headline numbers are a 4.04% pixel-wise XOR error on the synthetic test set at 256x256 input resolution, 4.33% at 1024x1024, and a best 9.18% on the real Marmot benchmark, also at 256x256. From these experiments the authors conclude that higher resolution does not universally improve performance and that synthetic-to-synthetic evaluation can reveal architectural behaviour while holding domain noise fixed.","pith_inferences":["A direct test of the paper's 'augmentation' framing would mix synthetic and Marmot images in one training set; the reported experiments train and evaluate separately, so the added value of combining the corpora remains untested.","If pixel-wise XOR is as sensitive to thin cell borders as the paper argues, then ablating generation parameters (for example, removing borders or color) could reveal which visual cues TableNet actually relies on.","Comparing models trained on this page-level corpus against models trained on isolated-crop corpora such as PubTables-1M under the same evaluation protocol could quantify how much page context helps detection."],"forward_implications":["Extending training on synthetic data continues to reduce XOR error, reaching 4.04% at 256x256 after 1540 epochs, while the same model on Marmot overfits after 500 epochs.","At 1024x1024, the model reaches 4.33% on synthetic pages but only 13.83% on Marmot, so resolution is not a reliable lever for real-world transfer.","Because ground-truth masks are generated rather than hand-labeled, the pipeline can cheaply produce large training sets for layout-analysis models.","The synthetic-to-synthetic protocol offers a sandbox for isolating the effect of data distribution from architecture when tuning detectors."],"supporting_citations":[{"why":"Supplies the VGG-19 encoder that TableNet uses for shared feature extraction.","marker":"[4]"},{"why":"Defines the TableNet architecture that all training and evaluation runs in the paper use.","marker":"[6]"},{"why":"One of the synthetic-corpus baselines the paper contrasts with its page-level images.","marker":"[25]"},{"why":"A large-scale synthetic table corpus the paper distinguishes from its page-level approach.","marker":"[26]"},{"why":"Another synthetic document-layout dataset used to position the paper's contribution.","marker":"[27]"},{"why":"The real-world Marmot benchmark that provides the external test set for synthetic-to-real evaluation.","marker":"[31]"}],"fun_headline_variants":["Synthetic pages train TableNet to 9.18% real-data error","Automated LaTeX pipeline builds table corpus, boosts accuracy","TableNet sees gains from synthetic docs, 256px beats 1024px","Synthetic data slashes table detection error to 9.18%","Automated synthetic generation improves table detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LaTeX-rendered synthetic pages are realistic and diverse enough that performance measured on synthetic test pages and pixel-wise XOR error transfers to real-world document images and meaningful detection quality.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic pages train TableNet to 9.18% real-data error","Automated LaTeX pipeline builds table corpus, boosts accuracy","TableNet sees gains from synthetic docs, 256px beats 1024px","Synthetic data slashes table detection error to 9.18%","Automated synthetic generation improves table detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1166,"prompt_tokens":813,"completion_tokens":353,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":264}},"tokens_in":429,"tokens_out":353,"duration_ms":4210,"temperature":1.0,"reasoning_tokens":264,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:50:13.111269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same TableNet training on the synthetic corpus and score predictions on Marmot with region-level metrics such as IoU or mAP; if the model shows low XOR error but high false-positive or false-negative rates at the bounding-box level, the central transfer claim fails under a different metric, whereas strong region-level scores would support it.","supporting_citations":[{"cited_title":"Tablenet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images,","cited_arxiv_id":null,"evidence_quote":"Defines the TableNet architecture that all training and evaluation runs in the paper use."},{"cited_title":"Supervised multi-specialist topic model with applications on large-scale electronic health record data","cited_arxiv_id":"2105.01238","evidence_quote":"One of the synthetic-corpus baselines the paper contrasts with its page-level images."},{"cited_title":"Marmot dataset","cited_arxiv_id":null,"evidence_quote":"The real-world Marmot benchmark that provides the external test set for synthetic-to-real evaluation."}],"review_version":2}