{"id":"9a170146-442a-4f26-81e1-93278d230dab","arxiv_id":"2605.27298","paper_version":1,"verdict":"UNVERDICTED","confidence":"UNKNOWN","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Self-ensembling of VLM outputs improves chart-to-table extraction accuracy by up to 23% on a new complex benchmark via repeated sampling and median aggregation.","lead":"This paper proposes a self-ensembling approach for vision-language models that samples multiple table outputs from a chart image and aggregates them using per-cell medians after alignment. It also introduces a new benchmark with more complex charts to test the method.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Repeated VLM sampling may lack sufficient output diversity for median aggregation to deliver the claimed accuracy gains.","rationale":"The reader's weakest assumption matches the identified load-bearing point exactly. Because the full text is referenced but the core claim's validity hinges on an unverified diversity condition, the UNVERDICTED status is appropriate and no adjustment is warranted.","tokens_in":1752,"tokens_out":276,"duration_ms":18374,"concrete_test":"On 20 WB-ChartExtract charts, draw 10 independent samples each at the temperature used in the paper; compute mean pairwise cell agreement (exact match on value and position after alignment). If mean agreement >85%, recompute the ensembled accuracy on those charts and check whether the 23% relative gain still holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline improvement (up to 23% relative on WB-ChartExtract) rests on the assumption that independent samples produce uncorrelated errors that per-cell medians can correct after alignment. If the VLM exhibits low stochasticity or systematic biases on complex charts (many datapoints, stylistic variation), samples will largely agree on the same mistakes and aggregation will add negligible value. The abstract mentions convergence detection and uncertainty estimation but supplies no quantitative evidence on sample diversity, pairwise disagreement rates, or error correlation.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that repeatedly sampling multiple tabular outputs from a fixed VLM on a chart image, aligning the tables, and taking per-cell medians over numerical values produces more accurate extractions than single-pass VLM outputs. It introduces convergence detection to halt sampling and uncertainty estimation from sample dispersion. A new benchmark WB-ChartExtract is presented with charts averaging 7x more datapoints than ChartQA; the method yields up to 23% relative accuracy improvement on this benchmark.","tokens_in":1822,"tokens_out":563,"duration_ms":27309,"significance":"If the gains are robust, the approach supplies a practical, training-free route to improve VLM chart extraction by exploiting output stochasticity, directly addressing reuse of quantitative data locked in images. The new WB-ChartExtract benchmark is a clear contribution because existing suites are too simple to measure further progress. The paper receives credit for grounding the method in statistical aggregation rather than additional fine-tuning.","major_comments":[{"comment":"§3.2 (Aggregation procedure): the claim that per-cell medians reduce errors rests on the untested premise that independent samples produce sufficiently uncorrelated mistakes; no pairwise disagreement rates, error-correlation statistics, or diversity metrics across samples are reported, leaving the 23% relative gain on WB-ChartExtract without direct support.","section":"§3.2"},{"comment":"Table 2 / WB-ChartExtract results: the headline relative improvement is presented without an ablation that isolates the contribution of median aggregation versus other factors (e.g., prompt variation or post-processing), so it is impossible to confirm that the reported lift is load-bearingly due to ensembling rather than implementation details.","section":"Table 2"},{"comment":"§4.1 (Benchmark construction): the statement that WB-ChartExtract charts contain “7 times more datapoints” is used to justify the new benchmark, yet no quantitative comparison of stylistic variation, axis complexity, or extraction difficulty against ChartQA is supplied, weakening the argument that prior benchmarks leave insufficient headroom.","section":"§4.1"}],"minor_comments":[{"comment":"The convergence-detection threshold and uncertainty formula are described only at a high level; adding the exact stopping criterion and dispersion metric (e.g., inter-quartile range) would improve reproducibility.","section":"§3.3"},{"comment":"Figure 3 caption does not state the number of samples used for the visualized uncertainty bands, making it hard to interpret the plotted dispersion.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. The recognition of both the practical utility of self-ensembling and the value of the WB-ChartExtract benchmark is appreciated. We respond point-by-point to the major comments below.","responses":[{"response":"We agree that explicit statistics on error correlation and sample diversity would strengthen the justification for median aggregation. While the reported accuracy gains are empirical, we will add pairwise disagreement rates, error-correlation statistics, and diversity metrics to §3.2 in the revision.","revision_made":"yes","referee_comment":"[§3.2] §3.2 (Aggregation procedure): the claim that per-cell medians reduce errors rests on the untested premise that independent samples produce sufficiently uncorrelated mistakes; no pairwise disagreement rates, error-correlation statistics, or diversity metrics across samples are reported, leaving the 23% relative gain on WB-ChartExtract without direct support."},{"response":"Table 2 compares the full pipeline against single-pass baselines. To isolate the median aggregation step, we will add an ablation study (new table or subsection) that holds prompting and alignment fixed while varying the aggregation operator.","revision_made":"yes","referee_comment":"[Table 2] Table 2 / WB-ChartExtract results: the headline relative improvement is presented without an ablation that isolates the contribution of median aggregation versus other factors (e.g., prompt variation or post-processing), so it is impossible to confirm that the reported lift is load-bearingly due to ensembling rather than implementation details."},{"response":"The 7× datapoint increase is the primary justification and is directly tied to measured difficulty in our experiments. We will augment §4.1 with quantitative comparisons of stylistic variation, axis complexity, and per-datapoint error rates versus ChartQA to further support the benchmark’s contribution.","revision_made":"yes","referee_comment":"[§4.1] §4.1 (Benchmark construction): the statement that WB-ChartExtract charts contain “7 times more datapoints” is used to justify the new benchmark, yet no quantitative comparison of stylistic variation, axis complexity, or extraction difficulty against ChartQA is supplied, weakening the argument that prior benchmarks leave insufficient headroom."}],"tokens_in":1466,"tokens_out":495,"duration_ms":33432,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a self-ensembling procedure for chart-to-table extraction: draw multiple independent outputs from the same VLM, align the resulting tables, and replace each cell with its median value. It adds convergence detection to stop early and a dispersion-based uncertainty score. They also release WB-ChartExtract, a new benchmark whose charts average seven times more data points than ChartQA and include more stylistic variety.\n\nThat benchmark is useful on its own; existing sets are too easy to show real headroom. The ensembling idea itself is not conceptually deep, but applying it at the cell level after alignment is a reasonable engineering choice for this task.\n\nThe soft spot is the lack of evidence that the samples actually vary in useful ways. The abstract reports up to 23% relative improvement on the new benchmark, yet supplies no numbers on pairwise disagreement rates, error correlation across samples, or how often the median actually corrects a mistake rather than just averaging the same error. If the VLM tends to make the same systematic mistakes on complex charts, the aggregation adds little. The stress-test concern lands here.\n\nThis is the kind of incremental systems paper that belongs in an application track. A reader working on chart parsing or data extraction from figures would get value from the benchmark and the uncertainty estimates. It deserves a serious referee because the method is reproducible in principle and the new data set is a clear step up, even if the gains need tighter validation on sample diversity.","headline":"The self-ensembling approach of repeated VLM sampling plus per-cell median after alignment is a straightforward practical tweak, but the claimed gains rest on an untested assumption that the samples disagree enough on errors.","tokens_in":2305,"tokens_out":385,"would_cite":false,"duration_ms":17978,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Repeated sampling from a vision-language model followed by per-cell median aggregation improves chart-to-table extraction accuracy over single passes.","keywords":["chart data extraction","vision-language models","self-ensembling","table aggregation","median aggregation","benchmark","uncertainty estimation"],"falsifier":"Run the single-pass and ensembled pipelines on WB-ChartExtract and compare extraction accuracy; if the relative improvement is statistically insignificant or if sample outputs show near-zero variation, the benefit of ensembling is not shown.","tokens_in":2633,"feed_emoji":"📊","tokens_out":650,"duration_ms":22969,"temperature":0.7,"pith_summary":"The paper establishes that a vision-language model extracts tabular data from chart images more accurately when its outputs are collected across multiple independent samples and combined through table alignment and per-cell medians rather than accepted from one forward pass. The method adds convergence detection to limit samples once the aggregate stabilizes and dispersion-based uncertainty scores for each cell. This matters because many charts contain quantitative information that remains inaccessible for analysis or reuse unless recovered reliably. Gains are larger on a new benchmark of complex, stylistically varied charts with seven times more datapoints on average than prior tests. The approach therefore converts image-locked data into usable tables with measurable reliability signals.","feed_headline":"Ensembling VLM samples lifts chart extraction accuracy up to 23%","feed_subtitle":"Cell-level medians from repeated draws recover more accurate tables from complex charts with seven times the datapoints of earlier benchmark","key_machinery":"Self-ensembling via repeated independent VLM samples aggregated by table alignment and per-cell median over numerical values.","core_discovery":"By drawing multiple independent tabular outputs from the same VLM on a fixed chart image, aligning the candidate tables, and computing per-cell medians over numerical values, the procedure yields a consensus table whose accuracy exceeds that of any single sample. Convergence detection stops further sampling once the aggregate table stabilizes, while dispersion across samples supplies an uncertainty estimate per cell.","pith_inferences":["The sampling-and-median procedure could be tested on other VLM structured-output tasks such as diagram-to-graph or form parsing where single-pass errors are common.","If the base VLM is already fine-tuned on charts, measuring whether ensembling still adds value would show whether the technique complements specialized training.","Low variance across samples would limit gains, suggesting experiments that deliberately increase output diversity through prompt variation or temperature settings."],"forward_implications":["Extraction accuracy improves over single-pass VLM outputs on both ChartQA and the new WB-ChartExtract benchmark.","Convergence detection reduces the number of samples required once the aggregate stabilizes.","Per-cell uncertainty estimates derived from sample dispersion allow users to assess reliability of the extracted table.","Tabular data previously locked inside chart images becomes available for downstream analysis and reuse."],"fun_headline_variants":["VLM self-ensembling lifts chart extraction accuracy up to 23%","Repeated VLM samples with cell medians raise chart table accuracy","Self-ensemble aggregates VLM outputs for better chart data recovery","Multiple VLM draws and per-cell medians improve extraction accuracy"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Repeated independent samples from the VLM differ enough that their median corrects errors instead of reinforcing the same mistakes.","fun_headline_variants_meta":{"raw":{"variants":["VLM self-ensembling lifts chart extraction accuracy up to 23%","Repeated VLM samples with cell medians raise chart table accuracy","Self-ensemble aggregates VLM outputs for better chart data recovery","Multiple VLM draws and per-cell medians improve extraction accuracy"]},"model":"grok-4.3","cost_usd":0.003726,"raw_usage":{"total_tokens":1945,"prompt_tokens":694,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":37262000,"prompt_tokens_details":{"text_tokens":694,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1180,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":694,"tokens_out":71,"duration_ms":14547,"temperature":1.0,"reasoning_tokens":1180,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T17:47:10.264177+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run the single-pass and ensembled pipelines on WB-ChartExtract and compare extraction accuracy; if the relative improvement is statistically insignificant or if sample outputs show near-zero variation, the benefit of ensembling is not shown.","supporting_citations":[],"review_version":1}