{"id":"37dbe0fc-e1fa-4bc3-8388-f47512248d06","arxiv_id":"2412.04262","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 100k-table synthetic financial dataset with ground-truth word and cell bounding boxes, plus a LayoutLM fine-tune that reaches 89% on a 100-question real-world extraction test.","lead":"The authors introduce SynFinTabs, a set of 100,000 machine-generated financial tables with pixel-level labels for words, cells, and rows. They show that a small layout model fine-tuned on this synthetic data can answer questions about real financial statements, though a tuned GPT-4V prompt performs better.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world accuracy is scored with first-occurrence and zero-fallback spans; this can mislabel answers and does not cleanly support the transfer claim.","rationale":"The reader's weakest assumption was that synthetic CSS templates and random content transfer to real financial tables, with no baseline comparison. That is a valid concern, but the more immediately load-bearing problem is the evaluation metric used on the real-world test set. The paper itself identifies the first-occurrence ambiguity in §4.1 and then uses the same rule in §4.2, while also allowing degenerate (0,0) ground-truth spans when OCR fails. Because the central claim is supported primarily by the 89% real-world accuracy, a flaw in how that number is computed directly undermines the conclusion. I partially agree with the reader: the missing baseline is important, but the metric issue is more specific and testable, and it affects both the synthetic and real-world experiments. I still regard the dataset contribution as potentially valuable, and the paper includes useful self-criticism in the Limitations section, so the appropriate outcome remains a conditional acceptance pending correction of the evaluation protocol rather than outright rejection. The proposed test would settle whether the concern actually changes the reported accuracy.","tokens_in":11738,"tokens_out":3842,"duration_ms":40950,"concrete_test":"Re-score the 100 real-world QA pairs with human-verified target spans: for each pair, manually identify the intended answer cell in the OCR token stream, ignoring the automatic first-occurrence rule, and record cases where the answer string occurs multiple times or is absent from the OCR output. Exclude or separately score the zero-span cases, and compare FinTabQA's accuracy under this corrected scoring with the reported 89%. If the corrected accuracy differs by more than a few points, or if a large share of correct predictions are degenerate zero/zero spans, the central effectiveness claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's support for the central claim rests on FinTabQA's 89% accuracy on the 100 real-world QA pairs (Table 2). However, the evaluation protocol in §4.1/§4.2 has a validity gap: ground-truth start/end positions are obtained by searching the OCR token stream for the answer text and taking the first occurrence, and when OCR fails to recognize the answer, both ground-truth positions are set to 0. In financial tables, the same numeric string (e.g., a subtotal or a repeated year value) frequently appears in multiple cells, so the first occurrence need not be the intended answer cell. The paper itself acknowledges this exact problem in §4.1 as a reason to prefer SynFinTabs's known spans, but then applies the same first-occurrence rule to the real-world test. Conversely, when OCR misses the answer entirely and the target span is set to (0,0), a model that also predicts (0,0) is counted as correct even though it extracted no answer. This can inflate or deflate the headline 89% and, more importantly, means the experiment does not cleanly demonstrate that training on SynFinTabs transfers to real tables. No comparison against a model trained on an existing real table dataset (e.g., FinTabNet or PubTables-1M) is provided, so the observed absolute accuracy is hard to interpret.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SynFinTabs, a dataset of 100,000 synthetic financial table images with word-, cell-, and row-level bounding boxes, cell types, and generated question-answer pairs. The authors fine-tune LayoutLM to create FinTabQA, an extractive question-answering model over table images, and evaluate it on a synthetic test split and on a manually created real-world set of 100 questions over 50 Companies House tables. They report 95.87% accuracy on the synthetic test split and 89% on the real-world set, compare against GPT-4V, and analyze the impact of OCR errors on end-to-end performance.","tokens_in":11988,"tokens_out":4305,"duration_ms":39366,"significance":"If validated, SynFinTabs would be a valuable resource: it is large-scale, has accurate positional annotations (including full cell boxes, rows, and empty cells), and the generation code is public. The idea of using synthetic financial tables to avoid OCR noise in training is worth pursuing. However, the empirical evidence for the central claim that the dataset transfers to real-world tables is weakened by the evaluation protocol (first-occurrence span matching, zero-fallback) and by the lack of a comparison against training on an existing real table dataset. The paper's strengths are the dataset generation process and the release of code/data; the main weakness is the insufficient validation of the transfer claim.","major_comments":[{"comment":"The real-world evaluation protocol is not a clean measure of extractive QA. When OCR fails to recognize the answer, both ground-truth span positions are set to zero, so a model that also predicts (0,0) is counted as correct even though it extracted no answer; conversely, when the answer text appears in multiple cells, first-occurrence span matching can mark a correct extraction as wrong or an incorrect cell extraction as correct. The paper itself identifies this first-occurrence problem in §4.1 for other OCR-based datasets, so applying the same rule to the real-world test undermines the validity of the reported 89% accuracy as evidence of transfer. The authors should either manually annotate the answer spans for the real-world questions or evaluate with a measure that compares the predicted cell text against the intended cell.","section":"§4.2 / Table 2"},{"comment":"The central claim that SynFinTabs is effective for training information extraction models is not compared against a baseline trained on an existing real-world table dataset (e.g., FinTabNet or PubTables-1M) under the same finetuning and evaluation setup. Without such a comparison, the absolute 89% accuracy on a small, manually evaluated set (100 questions, 50 tables) is difficult to interpret, since the model's performance could reflect properties of LayoutLM or the QA formulation rather than the quality of SynFinTabs. A matched comparison (same model, same evaluation, different training data) is needed to substantiate the transfer claim.","section":"§4.2 / Table 2"},{"comment":"The EasyOCR parameters were selected to maximize accuracy on the SynFinTabs test split (Table 1) and then applied to the real-world tables. Tuning preprocessing parameters on the test split makes the reported 95.87% an optimistic estimate of model performance on synthetic data, and it is unclear whether these parameters generalize to the real-world domain. For rigor, the parameter search should be done on a validation split or the default-parameter accuracy (75.27%) should be reported as the primary test result.","section":"§4.3"}],"minor_comments":[{"comment":"The description of the 'end position limited' heuristic is informal; please give the exact procedure, for example by masking end logits for positions before the predicted start.","section":"§4.1"},{"comment":"The Companies House style was based on visual inspection of a random sample of extracted tables; because the transfer claim depends on the similarity of the synthetic layout distribution to real tables, a quantitative comparison (e.g., distributions of cell widths, text lengths, number of rows/columns) would be helpful.","section":"§3.1"},{"comment":"Figure 5's example question contains a nonsensical row header ('Idle ver learning satisfied') due to random word generation; this is acknowledged in Limitations, but an example with a more realistic header would clarify the intended task.","section":"Appendix A"},{"comment":"The paper uses 'layout LLM' for LayoutLM, which is a layout-aware language model but not a large language model; please use consistent terminology.","section":"Throughout"},{"comment":"Given the small number of questions, it would be useful to report confidence intervals or per-table performance alongside the aggregate accuracy.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The dataset and generation code are genuinely useful contributions, and the paper's scope (a dataset resource) is appropriate. However, the empirical validation needs strengthening before the central claim is convincing. The parameter-search-on-test-set issue in §4.3 is a form of test-set adaptation that the editor should be aware of; it should be addressed in revision. The comparison to GPT-4V is informative but does not replace a same-architecture baseline trained on existing real table datasets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the SynFinTabs paper. The dataset is a real contribution: 100k synthetic financial tables with exact word, cell, and row boxes, plus QA pairs, generated from HTML with known ground truth. That fills a genuine gap—existing synthetic table datasets lack word-level annotations, and FinTabNet has only partial cell boxes. The release of code and data (if it all works) makes it a practical resource. The error analysis showing OCR failure modes is also a nice touch.\n\nThe soft spot is the evaluation of the central claim. The real-world test is 100 manually-written QA pairs over 50 Companies House tables. Ground-truth spans are computed by taking the first occurrence of the answer text in the OCR token stream, and when OCR misses the answer the span is set to (0,0). The paper itself acknowledges the first-occurrence problem in §4.1, then uses the same rule for the real-world test. In financial tables with repeated numbers, that can mislabel both the model and the ground truth; the (0,0) convention means a model that outputs nothing can be counted correct. So the 89% accuracy for FinTabQA is not a clean measure of transfer. There is also no baseline: they don't train the same model on FinTabNet, PubTables-1M, or another real dataset to compare. Without that, we don't know how much SynFinTabs helps relative to the obvious alternatives. The EasyOCR parameter search is described only as \"a parameter search\" with no detail on the grid or selection criterion, which makes the 20-point jump hard to trust.\n\nNone of this kills the dataset. If I were refereeing, I would ask for a baseline comparison, a more careful real-world evaluation protocol (e.g., human-verified answer cells, or matching by cell location rather than first occurrence), and a description of the OCR parameter search. The paper is honest about its limitations and the artifact itself is likely useful for training and for generating more diverse document data.\n\nSo: worth a serious referee, but the revision needs to strengthen the evidence. The dataset deserves to be out there; the current effectiveness claim does not.","headline":"A genuinely useful synthetic dataset with word-level annotations, but the transfer-to-real-world claim rests on a 100-question evaluation with a first-occurrence span rule that can mislabel answers.","tokens_in":12488,"tokens_out":2561,"would_cite":true,"duration_ms":25559,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces SynFinTabs, a 100,000-table synthetic financial dataset with exact word-, cell-, and row-level bounding boxes, and argues that it can train a layout language model to answer questions about real financial table images.","keywords":["synthetic data","financial tables","table extraction","document understanding","layout language models","visual question answering","OCR","bounding box annotations"],"falsifier":"Train the same LayoutLM configuration on real labelled tables only, evaluate it on the authors' 100 real-world table questions, and compare with FinTabQA; if the real-data model matches or exceeds FinTabQA's 89%, the claim that synthetic tables are effective for training would be weakened.","tokens_in":11551,"feed_emoji":"📊","tokens_out":7075,"duration_ms":66382,"temperature":0.7,"pith_summary":"SynFinTabs is a proposed solution to the scarcity of accurately labelled financial table images: 100,000 synthetic tables, each rendered from a structured specification so that the true bounding boxes of every word, cell, and row are known without OCR. The paper argues that this synthetic ground truth is sufficient to train a layout language model, FinTabQA, to answer extractive questions about table contents, and that the model transfers to real-world financial statements. On a hand-built test set of 100 questions about real tables from UK company filings, FinTabQA scores 89% exact match with table-cropped images, compared with 76% for GPT-4V given the questions alone and 94% for GPT-4V with an added extraction instruction. The authors also report near-perfect accuracy (99.98%) when ground-truth words are supplied instead of OCR output, which they use to attribute most end-to-end errors to OCR rather than to the model. The broader claim is that the generation method, not just the dataset, can be transferred to other document domains.","feed_headline":"Synthetic financial tables train a table-QA model to 89%","feed_subtitle":"A 100,000-table synthetic dataset with exact word positions beats GPT-4V on real financial filings.","key_machinery":"The mechanism that carries the argument is the table-generation and annotation pipeline. Each table begins as a structured object (sections, rows, cells, words), is serialised to HTML with a unique ID on every row, cell, and word, and is rendered in a headless browser at A4 page size; the browser's layout engine provides the pixel coordinates for every element, giving exact word-, cell-, and row-level bounding boxes as well as cell types ('section title', 'currency unit', 'row header', 'column header', 'data'). A question-answer pair is generated for every non-empty cell from its row and column headers, with the answer span located by start and end positions in the flattened word list. Fine-tuning LayoutLM on these pairs with exact span supervision, and evaluating with strict start/end exact match, is the demonstration task.","core_discovery":"The central discovery is that a large set of procedurally generated financial tables, labelled at word, cell, and row granularity at creation time, can stand in for real labelled financial tables in training an extractive table-QA model. The authors build SynFinTabs from six visual themes modelled on real filings and reports, randomise content and style, and render each table in a headless browser to record exact bounding boxes. Fine-tuning LayoutLM on this data yields FinTabQA, which reaches 89% exact match on 100 real-world questions from financial statements, outperforming GPT-4V (76%) in the question-only condition and approaching GPT-4V (94%) when the comparison model receives a task-specific instruction. The paper frames the result as evidence that the synthetic dataset is effective for information extraction from financial tables, and that the main bottleneck in the end-to-end pipeline is OCR quality, not the synthetic training data.","pith_inferences":["A direct quantitative test of the transfer assumption would be to train the same LayoutLM recipe on a real labelled table dataset and compare on the same real-world question set; the paper does not include this comparison.","The QA templates all share one grammatical form, so the dataset is likely to support models that answer template-style questions about tables; adapting the released code to generate more varied questions would test whether the learned skill generalises to free-form questions.","Because the synthetic content is random, models trained on it can learn layout, syntax, and span-selection behaviour but not domain semantics, suggesting the method transfers best to tasks where visual table structure carries the answer and less well to tasks requiring financial reasoning.","The public generation code could be adapted to produce full synthetic document pages with tables placed in known positions, enabling table detection experiments without manual annotation."],"forward_implications":["A model trained on SynFinTabs can answer real-world table questions at 89% exact match, showing synthetic financial tables are a viable substitute for manually labelled ones.","The same pipeline can be reused to generate larger or more diverse datasets, since the generation code is released alongside the data.","Because the ground-truth bounding boxes do not depend on OCR, the dataset can serve as clean supervision for table structure recognition, cell classification, and OCR training on tabular text.","The paper's error analysis indicates that most remaining end-to-end errors come from OCR rather than from the synthetic training data, pointing to OCR improvement as the next bottleneck."],"supporting_citations":[{"why":"Supplies the LayoutLM architecture that FinTabQA fine-tunes and the virtual-coordinate convention for scaling bounding boxes.","marker":"(Xu et al., 2020)"},{"why":"Provides Table Transformer, used to extract real tables whose style informed the main CSS template, and PubTables-1M as a real-data reference.","marker":"(Smock et al., 2022)"},{"why":"FinTabNet is the dataset whose incomplete cell annotations motivate SynFinTabs' full cell, row, and empty-cell boxes.","marker":"(Zheng et al., 2021)"},{"why":"GPT-4V is the comparison model on the real-world question set and the reference point for the reported accuracy differences.","marker":"(OpenAI, 2023)"},{"why":"ICDAR 2013 is the small early benchmark whose limitations motivate the call for large labelled table data.","marker":"(Göbel et al., 2013)"},{"why":"SQuAD supplies the extractive question-answering setup and exact-match convention adapted for span evaluation.","marker":"(Rajpurkar et al., 2016)"}],"fun_headline_variants":["Synthetic financial tables train a QA model to 89% exact match","100k synthetic tables beat GPT-4V on real financial QA","FinTabQA: synthetic tables outperform GPT-4V on real filings","SynFinTabs: synthetic data for table extraction, no OCR needed","Table extraction gets a synthetic boost: 89% on real financial questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the CSS templates used to generate SynFinTabs resemble real financial tables closely enough that a model trained on synthetic tables transfers to real-world tables; the paper supports this by visual inspection of a sample, not by a quantitative transfer comparison.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic financial tables train a QA model to 89% exact match","100k synthetic tables beat GPT-4V on real financial QA","FinTabQA: synthetic tables outperform GPT-4V on real filings","SynFinTabs: synthetic data for table extraction, no OCR needed","Table extraction gets a synthetic boost: 89% on real financial questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1771,"prompt_tokens":945,"completion_tokens":826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":731}},"tokens_in":561,"tokens_out":826,"duration_ms":7698,"temperature":1.0,"reasoning_tokens":731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:33:41.126584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LayoutLM configuration on real labelled tables only, evaluate it on the authors' 100 real-world table questions, and compare with FinTabQA; if the real-data model matches or exceeds FinTabQA's 89%, the claim that synthetic tables are effective for training would be weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4V is the comparison model on the real-world question set and the reference point for the reported accuracy differences."}],"review_version":1}