{"id":"3f7d3ece-c7d8-4411-b5ce-e60e13ea4b61","arxiv_id":"2412.20662","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"NGTR, a training-free framework that retrieves a similar neighbor, scores multiple image-preprocessing toolchains on that neighbor's labels, and reflects before each tool step, lifts VLLM table recognition accuracy on low-quality datasets.","lead":"This paper evaluates how well vision-language models read tables from images and proposes a framework, NGTR, that uses a similar sample's past tool choices to clean up low-quality table images before recognition. It reports large accuracy gains on hard, blurry, real-world table datasets, but the improvements rest on single-run experiments without error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The neighbor-transfer premise is the load-bearing weak point: NGTR selects a toolchain by scoring it on a retrieved neighbor's ground-truth HTML, and the reported gains stand or fall on whether that selection transfers to the test image.","rationale":"Reader's conditional verdict is reasonable; the empirical claim could be true. My stress-test converges on the same load-bearing assumption: transfer of toolchain selection from the retrieved neighbor to the test image. This is not a stylistic objection. The method's own ablation attributes a large share of the PubTabNet improvement to the experience-learning module, which is exactly the module that performs neighbor-ground-truth-guided selection. If ORB/Hamming retrieval is noisy or the neighbor's optimal toolchain is not predictive, that ablation result would not reproduce on a fresh evaluation, and the central claim would collapse to 'some image preprocessing plus a quality filter helps,' which is a much weaker and less interesting claim. The reflection module cannot rescue the premise because its acceptance rule is about preserving information in the image, not about producing a correct table parse; it can approve a visually clean but structurally destructive operation. I considered the absence of error bars and the hyperparameter tuning on the evaluation set as alternative concerns; they are real reporting weaknesses, but they weaken confidence rather than specifying a mechanism that could falsify the central claim. The proposed check is retrospective and uses only the released code and existing labels, so it can be run without new data. If the check shows strong transfer, the conditional verdict should stand or be upgraded; if it fails, the framework's contribution is not established. Because the concern is already embedded in the reader's weakest_assumption and the verdict is already CONDITIONAL, I recommend no verdict change.","tokens_in":17743,"tokens_out":11179,"duration_ms":117245,"concrete_test":"Use the released GitHub code on a random sample of 200 PubTabNet validation images. For each image, run the full Section 4.3-4.4 pipeline to obtain the ORB/Hamming neighbor, the candidate plan set, and the neighbor-selected plan. Then compute, using the test image's own ground truth, (i) the TEDS of the neighbor-selected plan on the test image, (ii) the TEDS of a randomly selected plan on the test image, and (iii) the TEDS of the oracle best plan on the test image. Report the Spearman correlation between the plan's neighbor-side TEDS and its test-side TEDS across the 200 images. If the neighbor-selected plan is not significantly better than random, or the correlation is near zero, the transfer premise fails and the reported gains cannot be attributed to the claimed mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3-4.5 implements the core of NGTR as follows: retrieve one neighbor via ORB/Hamming (Eq. 1), generate candidate toolchains from tool descriptions (Eq. 2), execute each plan on the neighbor and score the VLLM's HTML against the neighbor's ground truth with TEDS (Section 4.4), then transfer the winner to the test image while a reflection module (Eq. 4) accepts or rejects each tool step. The reflection prompt (Figure 19) asks whether the processed image 'contains more information' and tells the model to choose the second image if no information is lost; it does not check whether the operation improves downstream table recognition. This matters because a tool such as binarization can make an image look cleaner while erasing faint text or thin borders, and the reflection module can approve it. The PubTabNet gain of +10.57 TEDS (Table 4) and the EXP ablation (Table 5: 85.03 -> 80.57 without neighbor scoring) both depend on the selected plan being the right one for the test image. The paper itself acknowledges in the Limitation section that an inappropriate neighbor can lead to suboptimal performance, but it never measures how often this occurs or whether the neighbor's best plan correlates with the test image's best plan. Without that measurement, the central claim is a bet on unverified transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a benchmark for evaluating VLLM-based table recognition under a training-free paradigm and proposes NGTR, a framework that preprocesses table images by retrieving a similar training neighbor, generating candidate toolchains via the VLLM, scoring those toolchains by executing them on the neighbor and comparing the VLLM's output against the neighbor's ground-truth HTML with TEDS, and finally applying the winning toolchain to the test image with a reflection module that accepts or rejects each tool step. The authors report large gains on PubTabNet and WTW (e.g., GPT-4o TEDS from 74.46 to 85.03 on PubTabNet and TEDS-Struct from 40.01 to 52.03 on WTW) and smaller gains on SciTSR. The paper also contributes a hierarchical task benchmark and an analysis of VLLM robustness to image degradation.","tokens_in":18046,"tokens_out":4488,"duration_ms":46775,"significance":"If the reported results hold, the paper makes a useful contribution to table recognition with vision LLMs by demonstrating that lightweight image preprocessing, guided by retrieved neighbors, can substantially improve zero-shot VLLM performance on low-quality inputs. The manuscript ships code, proposes a training-free framework with modular tools, and provides a systematic benchmark across multiple VLLMs and three public datasets. The bottleneck analysis (Section 3.4) is a helpful diagnostic. However, the central mechanism relies on an unverified transfer assumption, and the experimental reporting lacks statistical rigor; the small SciTSR gains may not be significant.","major_comments":[{"comment":"The core of NGTR is the transfer of a toolchain selected by scoring it on a retrieved neighbor's ground-truth HTML to the test image. The manuscript never measures whether the neighbor's best toolchain actually matches the test image's best toolchain. The limitation section acknowledges that an inappropriate neighbor can degrade performance, but no analysis quantifies how often this occurs. Please provide an oracle comparison on a labeled subset: for each test image, compute the toolchain that maximizes TEDS on the test image itself (using the test ground truth), and compare its performance against the NGTR-selected toolchain. Report the agreement rate between the two selections and the TEDS gap. Without this, the reported gains could be driven by plans that are optimal for the neighbor but not for the test image; the w/o EXP ablation in Table 5 does not isolate this, as it compares no scoring against neighbor scoring rather than neighbor scoring against test-image scoring.","section":"Sections 4.3–4.5, Eqs. (1)–(4)"},{"comment":"The reflection module accepts or rejects each tool step based on whether the post-processing image 'contains more information' or whether 'no information is lost.' This criterion is not directly tied to downstream table recognition accuracy; for example, binarization can make an image look cleaner while erasing faint text or thin borders, and the current prompt would likely accept it. Please analyze whether the reflection decisions correlate with downstream TEDS improvements, for instance by labeling accepted/rejected steps on a held-out subset with known ground truth, or by modifying the reflection prompt to incorporate a recognition-aware check (e.g., asking the VLLM whether any table content would be lost). Reporting the acceptance rate and the proportion of accepted steps that improve TEDS would make the module's contribution measurable.","section":"Section 4.5, Eq. (4), Figure 19"},{"comment":"The hyperparameters L (maximum toolchain length) and N (number of plans) are chosen based on the sensitivity analysis in Figure 6, but the manuscript does not state which data split was used for this tuning. Since the SciTSR and WTW evaluations use their complete test sets, any tuning of L or N on these test sets would inflate the reported results. Please specify the tuning split explicitly. If hyperparameter selection used test-set information, either report results with hyperparameters fixed a priori (e.g., L=3, N=3 without tuning) or use a validation split from the training data for tuning. The same transparency is needed for the PubTabNet evaluation subset: it is a random selection of 1,500 validation images, but no seed or exact list is provided, which affects reproducibility.","section":"Section 5.5 and Implementation Details"},{"comment":"All reported numbers are single-run point estimates without error bars, confidence intervals, or significance tests. The SciTSR gains are small (GPT-4o TEDS +0.92, Gemini TEDS +1.88, and +1.36/+1.58 for TEDS-Struct) and may be within run-to-run variability, especially for closed-source APIs where temperature 0 does not guarantee deterministic output. Please report the variance across multiple runs (e.g., 3–5 runs) or, at minimum, the number of test samples and a paired significance test for the smaller gains. The large PubTabNet and WTW improvements are likely robust, but the 'significantly enhances' claim in the abstract and Section 5.2 should be qualified for datasets with small margins.","section":"Tables 4 and 5"}],"minor_comments":[{"comment":"The row/column sensitivity analysis uses a subsample where the difference between rows and columns does not exceed three, but the exact filtering procedure and the resulting sample sizes are not described; please clarify how 'difference' is computed and report the number of samples in each bar.","section":"Section 3.3, Figure 3"},{"comment":"The tool usage rates are described as 'the proportion of samples that invoke a particular tool among all samples where tools are invoked,' but the denominator is not stated numerically; please specify whether the denominator is all test samples or only samples where at least one tool was accepted, and how 'invoked' is counted when a tool is rejected by the reflection module.","section":"Section 5.4, Table 6"},{"comment":"The parameter settings mention temperature 0.8 for generating tool invocation plans and temperature 0 for other experiments; please clarify whether the reflection module also uses temperature 0, and whether the VLLM-based tool-plan generation in Eq. (2) uses any special decoding parameters beyond temperature.","section":"Appendix D"},{"comment":"The notation N(Itest) is used to denote the set of retrieved neighbor images, but it is not formally defined as a function returning a set; please define it explicitly, and clarify whether the planning prompt includes the test image itself or only the neighbor images, since Figure 5 suggests the test image is also shown to the VLLM during planning.","section":"Equation (2) and Figure 18"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the code release is a plus. The main concern is the unvalidated neighbor-transfer assumption at the heart of NGTR; the suggested oracle experiment is feasible and would substantially strengthen the paper. I would also ask the editor to ensure the hyperparameter tuning procedure is clarified, as the current text risks test-set contamination. The benchmark portion is solid and could be published on its own, but the framework claims need the additional analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a useful applied paper, not a breakthrough. It adds a hierarchical benchmark for VLLM table recognition and a training-free framework, NGTR, that retrieves a similar neighbor image, generates several toolchains, scores each by running the VLLM on the neighbor and comparing to the neighbor's ground truth with TEDS, then applies the best toolchain to the test image with a reflection module that accepts or rejects each tool step. The headline numbers are a +10.57 TEDS gain for GPT-4o on PubTabNet and +12.02 on WTW, with smaller gains on SciTSR. Ablations show both the experience-learning and reflection modules matter.\n\nWhat's new: the combination of neighbor retrieval, tool-plan scoring, and reflection is original for this task, and the hierarchical benchmark is a real contribution. The paper is honest about the low-quality image bottleneck, and the code is released. That gives it practical value for anyone building training-free VLLM document pipelines.\n\nThe soft spots are real but proportionate. All numbers are single-run point estimates; the SciTSR gains of about 1–2 TEDS points could easily be noise. Second, hyperparameters L and N are tuned on the evaluation sets (Figure 6), and PubTabNet uses a random 1,500-image validation subset with no separate validation split for these choices, so some optimism in the reported numbers is likely. Third, the neighbor-transfer premise—the toolchain that scores best on the retrieved neighbor is the right one for the test image—is acknowledged in the Limitation section but never directly measured. The reflection module checks relative image quality, not downstream recognition; a tool like binarization can look cleaner while erasing faint text. The paper does not quantify how often the selected plan is suboptimal for the test image, so the mechanism is plausible rather than proven.\n\nThe central empirical claim, though, is not contradicted by the paper's own evidence. The gains on low-quality datasets are large, and the ablations support the architecture. The missing transfer analysis is a weakness, but not fatal; the paper names it as a limitation.\n\nWho should read this: anyone working on training-free VLLM pipelines for document processing, especially table recognition. It deserves a serious referee—the empirical claim is substantial and would be useful if confirmed—but the paper needs stronger statistical reporting, a validation split for hyperparameters, and ideally a direct check of neighbor-transfer fidelity before publication.\n\nRecommendation: send to peer review, with major revisions requested.","headline":"Solid applied framework for VLLM table recognition with large gains on low-quality images, but thin statistics and an unverified neighbor-transfer mechanism keep it from being more than a moderate contribution.","tokens_in":18574,"tokens_out":4312,"would_cite":true,"duration_ms":40321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a training-free, neighbor-guided toolchain reasoner improves vision large language models' table recognition by selecting image-preprocessing plans through retrieval and reflection.","keywords":["table recognition","vision large language models","neighbor-guided toolchain reasoner","retrieval-augmented generation","TEDS","image preprocessing","reflection module","hierarchical benchmark"],"falsifier":"Take a test image whose nearest neighbor in the retrieval set is visually similar but structurally different (for instance, similar lighting and borders but a very different number of columns or merged-cell layout), apply the neighbor-selected toolchain, and compare the VLLM's TEDS against both no toolchain and a toolchain selected by a deliberately different neighbor; the transfer assumption would fail if the neighbor-selected plan scores worse than no preprocessing.","tokens_in":17549,"feed_emoji":"🧰","tokens_out":8498,"duration_ms":68101,"temperature":0.7,"pith_summary":"The paper sets out to show that vision large language models can recognize tables more accurately when the input image is preprocessed by a short toolchain chosen with hindsight from a similar example. It first builds a hierarchical benchmark and finds that poor image quality, not missing borders, is the main bottleneck for VLLMs. The proposed NGTR framework retrieves a visually similar neighbor from the training set, has the VLLM propose several preprocessing toolchains, scores each toolchain by running it on the neighbor and comparing the VLLM's HTML output against the neighbor's ground truth with TEDS, then applies the best-scoring plan to the test image while a reflection module accepts or rejects each step by comparing before-and-after image quality. On GPT-4o this raises TEDS on PubTabNet from 74.46 to 85.03 and TEDS-Struct on WTW from 40.01 to 52.03, with smaller gains on SciTSR. If this holds, it suggests that a large share of VLLM error in table recognition is recoverable without any fine-tuning.","feed_headline":"Neighbor-guided toolchains lift GPT-4o table TEDS to 85","feed_subtitle":"A training-free framework tries preprocessing plans on a similar table, then applies the winner to the target image.","key_machinery":"The carrying mechanism is the neighbor-guided toolchain with TEDS-scored experience transfer and reflection-based step acceptance. A toolchain is an ordered list of lightweight image operations (upscale, border enhancement, binarization, noise reduction, detection-and-cropping) drawn from a fixed toolkit. For a test image, the framework retrieves its nearest neighbor from the training set by ORB keypoint matching with Hamming distance, has the VLLM propose several toolchains, executes each toolchain on the neighbor, scores the VLLM's HTML output against the neighbor's ground-truth HTML using the tree-edit-distance similarity TEDS, and transfers the highest-scoring plan to the test image. The reflection module then runs each tool in order and asks the VLLM whether the processed image retains at least as much information as the previous one, accepting or discarding the operation. This combination turns the VLLM itself into both the planner and the quality judge, with the neighbor's label supplying the only external supervision.","core_discovery":"The central discovery is that a training-free preprocessing \"toolchain reasoner\" can close much of the gap between naive VLLM table recognition and specialized OCR systems, provided the toolchain is chosen by experience transfer from a similar neighbor rather than by prompting alone. The framework's key step is to treat the neighbor image as a rehearsal: generate several candidate toolchains, execute each one on the neighbor, score the resulting HTML against the neighbor's ground truth using TEDS, and transfer the winner to the test image. A reflection module then supervises execution by having the VLLM compare the image before and after each tool operation, rejecting operations that lose information. The paper reports that this pipeline lifts GPT-4o from 74.46 to 85.03 TEDS on PubTabNet and from 40.01 to 52.03 TEDS-Struct on WTW, while ablations show both the experience-learning and reflection modules contribute. The paper also claims that VLLMs rely little on table borders and that column tasks are easier than row tasks.","pith_inferences":["Because the framework is training-free and tool-agnostic, the same neighbor-guided scoring loop could be applied to other output formats besides HTML (e.g., Markdown or JSON) and to other document-understanding tasks where a ground-truth label and a structural similarity metric exist.","The neighbor-transfer premise suggests a cheap diagnostic: if retrieval quality is measured (e.g., by TEDS between neighbor and test ground truths), the framework's ceiling on a new dataset can be predicted before running the VLLM.","The reflection module's reliance on image-quality judgment rather than recognition accuracy implies that a tool which preserves visual information but confuses the VLLM (e.g., upscaling that changes font rendering) would pass the reflection gate yet still hurt downstream TEDS; testing this could reveal whether reflection should be conditioned on the recognition target.","The benchmark's finding that VLLMs ignore borders is testable in reverse: inserting strong but false border cues into a borderless table image could measurably change recognition output, indicating whether structural priors are learned from text rather than vision."],"forward_implications":["GPT-4o's PubTabNet TEDS rises from 74.46 to 85.03 with NGTR, a gain larger than switching between any of the six evaluated VLLMs.","On the wild-image WTW dataset, NGTR lifts GPT-4o TEDS-Struct from 40.01 to 52.03, but the framework does not close the gap to specialized structure-recognition models like LORE.","The reflection module contributes independently: removing it drops PubTabNet TEDS from 85.03 to 82.08, and removing the experience-learning module drops it to 80.57.","Tool invocation adapts to dataset characteristics: image upscaling is preferred on low-resolution PubTabNet, detection-and-cropping on wild WTW images, and border enhancement on clean SciTSR tables.","VLLMs show asymmetric structure sensitivity: column-related recognition is more accurate than row-related recognition, and border visibility has little effect on performance."],"supporting_citations":[{"why":"Supplies the PubTabNet dataset and the TEDS and TEDS-Struct metrics used for all main results.","marker":"[Zhong et al., 2020]"},{"why":"Supplies the SciTSR dataset used to measure performance on high-quality scientific tables.","marker":"[Chi et al., 2019]"},{"why":"Supplies the WTW wild-scenario dataset used to test robustness on extreme images.","marker":"[Long et al., 2021]"},{"why":"LGPMA, the bottom-up cell-detection baseline for comparison on SciTSR and PubTabNet.","marker":"[Qiao et al., 2021]"},{"why":"LORE, the structure-recognition baseline for comparison on WTW and PubTabNet.","marker":"[Xing et al., 2023]"}],"fun_headline_variants":["Neighbor rehearsal picks best table preprocessing toolchain","Training-free toolchain reasoner boosts VLLM table recognition","Rehearse on a neighbor, then apply: table OCR trick","Reflective toolchain selection lifts GPT-4o table TEDS by 10","Neighbor-guided toolchains crack low-quality table images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the preprocessing toolchain that scores best on a retrieved neighbor image will also be the best toolchain for the test image, even though the reflection module only checks relative image quality and never checks recognition accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Neighbor rehearsal picks best table preprocessing toolchain","Training-free toolchain reasoner boosts VLLM table recognition","Rehearse on a neighbor, then apply: table OCR trick","Reflective toolchain selection lifts GPT-4o table TEDS by 10","Neighbor-guided toolchains crack low-quality table images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1300,"prompt_tokens":936,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":552,"tokens_out":364,"duration_ms":3696,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:13:52.492747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a test image whose nearest neighbor in the retrieval set is visually similar but structurally different (for instance, similar lighting and borders but a very different number of columns or merged-cell layout), apply the neighbor-selected toolchain, and compare the VLLM's TEDS against both no toolchain and a toolchain selected by a deliberately different neighbor; the transfer assumption would fail if the neighbor-selected plan scores worse than no preprocessing.","supporting_citations":[],"review_version":1}