{"id":"f0c08d1f-c18d-4fcd-addd-4aa78b7638a0","arxiv_id":"2505.19360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ChartLens uses segmentation and set-of-marks prompting to attribute chart-based answers to specific visual elements, and the authors release a new benchmark for evaluating such attribution.","lead":"This paper introduces ChartLens, a method that highlights the specific bars, lines, or pie slices in a chart that support a given answer, enabling verification of chart-based responses. It also introduces ChartVA-Eval, a benchmark with fine-grained attribution annotations for synthetic and real-world charts from finance, policy, and economics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is undercut by a benchmark-circularity risk: GPT-4o generated the PlotQA/ChartQA ground-truth attributions while also serving as ChartLens's backbone, and no precision check was performed; the reported numbers also do not reproduce the 26-66% headline.","rationale":"The paper proposes a plausible pipeline: heuristic/SAM segmentation plus set-of-marks prompting is a reasonable way to obtain fine-grained chart attributions, and the qualitative examples look consistent with the method's design. However, the central claim—that ChartLens improves fine-grained attributions by 26-66%—depends entirely on the validity of ChartVA-Eval. The benchmark's ground truth for PlotQA and ChartQA was initially generated by GPT-4o, the same model family that ChartLens uses for attribution, and the human check did not enforce the precision criterion that defines fine-grained attribution. This creates a concrete risk that ChartLens is measuring agreement with the annotation model rather than genuine visual-evidence quality. The reader's weakest assumption identifies exactly this issue, and I agree it is the most load-bearing concern. The additional arithmetic discrepancy in the reported improvements strengthens the case for caution: the abstract's range is not recoverable from Tables 2-4 under the definitions given. The reader's CONDITIONAL verdict is appropriate; I do not see grounds to reject the method outright, but the performance claims need correction and the benchmark needs an independent precision audit before they can be accepted.","tokens_in":14900,"tokens_out":6422,"duration_ms":58908,"concrete_test":"Independently re-annotate a random sample of 100 ChartVA-PlotQA and 100 ChartVA-ChartQA instances with a second annotator instructed to mark all chart elements that are necessary and sufficient for the answer and to explicitly remove any irrelevant or extraneous elements (a precision check). Then recompute Precision, Recall, and F1 for ChartLens and for Kosmos-2 and LISA on this subset. If ChartLens's F1 advantage over the non-GPT-4o baselines shrinks by more than 10 relative points, or the ranking changes, the GPT-4o-generated ground truth is biasing the benchmark in ChartLens's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that ChartVA-Eval's ground-truth attribution sets are correct and unbiased. Section 4.2 states that GPT-4o generated initial annotations from the data tables, and the human verification covered only Relevance and Completeness, not Precision—the third criterion defined in the problem statement. ChartLens uses ChatGPT-4o as its attribution MLLM (Section 6.3). If GPT-4o's own preferences determine which chart elements it counts as supporting an answer, ChartLens is rewarded for agreeing with the annotation model, while non-GPT-4o baselines (Kosmos-2, LISA) are disadvantaged for reasons unrelated to attribution quality. This affects 943 of the 1,244 benchmark queries (PlotQA + ChartQA), so it is not a corner case. The paper's tables also show a separate red flag: relative F1 gains in Tables 2 and 4 are 204-950% and 315%, and line detection gains in Table 3 are 223-2195%; no reported metric yields the claimed '26-66%'. Table 3 additionally appears to contain a copy error (LISA's PlotQA and ChartQA rows are identical). These issues do not disprove the method, but they make the headline quantitative claim unreliable as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Post-Hoc Fine-grained Visual Attribution for Charts, a task that grounds a chart-associated textual response to the specific visual elements (bars, lines, pie sectors) that support it. To address the task, the authors propose ChartLens, a pipeline that segments chart elements using heuristic methods, SAM, and LineFormer, then applies set-of-marks prompting to ChatGPT-4o to generate attributions. They also introduce ChartVA-Eval, a benchmark of 1,244 queries built from MATSA-AITQA, PlotQA, and ChartQA, with human-verified attribution annotations. Experiments compare ChartLens against zero-shot ChatGPT-4o, Kosmos-2, and LISA, reporting substantially higher F1 scores on bar and pie charts and better precision-recall tradeoffs on line charts. The abstract and conclusion claim a 26–66% improvement over baselines.","tokens_in":15187,"tokens_out":4800,"duration_ms":45240,"significance":"If the results are valid, the paper makes a useful contribution: it formalizes a practically important task (chart attribution for hallucination mitigation), releases a new benchmark with real-world and synthetic charts, and demonstrates a modular method that combines segmentation with set-of-marks prompting. The qualitative examples in Figure 3 show clear localization improvements over the baselines, and the paper explicitly limits the method's scope to visual elements, which is honest. The main significance is conditional, however, on resolving the mismatch between the headline improvement figure and the reported tables, and on addressing the risk that the benchmark's ground truth is biased by the same model family used in the proposed method. The benchmark itself, if properly validated, would be a valuable resource for evaluating chart grounding.","major_comments":[{"comment":"The headline claim of a 26–66% improvement is not supported by the reported tables. Relative F1 improvements of ChartLens over the best baseline are approximately 204% on ChartVA-AITQA bars (69.28 vs 22.77), 950% on ChartVA-PlotQA bars (34.65 vs 3.30), 728% on ChartVA-ChartQA bars (64.14 vs 7.75), and 315% on ChartVA-ChartQA pies (48.56 vs 11.70). Absolute F1 point gains range from 31 to 56 points. No subset of Tables 2–4 yields the claimed 26–66% range. Please specify how this number was computed or correct it, since it is the central quantitative claim of the paper.","section":"Abstract, §1, §7"},{"comment":"The annotation protocol in §4.2 states that GPT-4o generated initial attribution annotations for PlotQA and ChartQA, and human annotators reviewed only Relevance and Completeness, not Precision, even though Precision is the third criterion in the problem statement (§3). ChartLens's attribution MLLM is ChatGPT-4o (§6.3). This creates a same-model bias risk: ground truth may reflect GPT-4o's notion of which chart elements support an answer, rewarding ChartLens for agreeing with the annotation model while disadvantaging non-GPT-4o baselines (Kosmos-2, LISA) for reasons unrelated to attribution quality. This affects 943 of the 1,244 benchmark queries (§4.1, Table 1). Please add a precision-oriented human verification step, report inter-annotator agreement on precision, or provide evidence that this bias does not affect the ranking.","section":"§4.2, §6.3, §3"},{"comment":"The statement in §7 that ChartLens reduces Chart% area by approximately 3–50 times is contradicted by Table 3. On PlotQA, ChartLens has a higher Chart Ar (9.98%) than zero-shot ChatGPT-4o (8.63%), and on AITQA the reduction is only about 1.55× (1.25% vs 1.94%). Only the ChartQA row (5.34% vs 1.15%) approaches the claimed reduction. Please correct this statement with the actual ratios or clarify which baselines and datasets support the 3–50× claim.","section":"§7, Table 3"}],"minor_comments":[{"comment":"The LISA rows for ChartVA-PlotQA and ChartVA-ChartQA are identical (Detection 50.21 and Chart Ar 40.92 for both), which appears to be a copy-paste error. Please verify the entries.","section":"Table 3"},{"comment":"The sentence 'This process ensured high-quality and precise attribution annotations' overstates what was checked, since the protocol assessed only Relevance and Completeness; the word 'precise' should be removed unless Precision is added to the verification criteria.","section":"§4.2"},{"comment":"The abstract contains a superscript '1' after the '26–66%' claim but no corresponding footnote or link in the main text; if this refers to code/data, please provide the repository URL.","section":"Abstract footnote"},{"comment":"Several typos and formatting issues should be corrected: 'methods that leverages' in §1 should be 'method that leverages'; 'LLms' in §5.2 should be 'LLMs'; 'O(104)' in §4.1 should be '10^4'; and the dataset name 'ChartV A-Eval' is inconsistently spaced.","section":"Throughout"},{"comment":"For line charts, Detection Rate and Average Area are reported separately without a combined metric, making it difficult to compare line-chart performance with the P/R/F1 results for bars and pies; consider reporting a single precision-recall composite or an F1-style score.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's main weakness is the inconsistency between the 26–66% headline and the actual table numbers, which the authors must resolve. The benchmark-circularity concern is serious because the new benchmark is partially annotated by the same model family used in the method, and the annotation checks omit Precision. These issues are fixable within the manuscript's scope (e.g., by correcting the headline, adding precision verification, and re-running or re-analyzing the affected comparisons), so I do not recommend rejection. I would also ask the editor to ensure the Table 3 copy error is corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ChartLens defines a real task and ships a useful benchmark, but the reported headline gain doesn't match the tables, and the benchmark construction has a same-model bias risk that needs to be addressed before the numbers are taken at face value.\n\nWhat's new: the post-hoc fine-grained visual attribution task for charts is a good idea, and ChartVA-Eval is a decent-sized benchmark (1200+ queries) drawn from synthetic and real-world sources. The ChartLens pipeline—heuristic preprocessing, SAM for bars/pie, LineFormer for lines, set-of-marks prompting—is a sensible composition, and the qualitative examples show it actually does something: it localizes the relevant bars or sectors rather than dumping a whole-region bounding box. The decoupling of attribution from response generation is a useful property for verification workflows.\n\nWhat's soft: The abstract says 26-66% improvement, but the tables show relative F1 gains of roughly 204-950% on bar charts and 315% on pie charts. Those are not the same numbers, and the paper doesn't explain the discrepancy. Table 3 has LISA's PlotQA and ChartQA rows identical, which looks like a copy error. The benchmark annotations for PlotQA and ChartQA (943 of 1244 queries) were generated by GPT-4o from data tables, then humans checked only relevance and completeness, not precision. Since ChartLens uses ChatGPT-4o as its backbone, there's a legitimate concern that the ground truth is partly tuned to GPT-4o's preferences, making the comparison against Kosmos-2 and LISA less clean. The paper also provides no code/data, no ablations, no error bars, so the robustness of the approach is unquantified.\n\nNone of these are fatal. The core idea stands, and the direction of the results is credible. But the specific claims need to be rechecked and the benchmark construction documented more transparently. I'd send this to peer review—the task and benchmark are worth refereeing—but I'd ask the authors to fix the reported numbers, correct the table, release the benchmark, and add at least a small ablation and error bars. If the numbers hold up after that, this could be a useful resource for chart understanding and hallucination work.\n\nRead it if you care about chart grounding; otherwise skimmable.","headline":"A genuinely new task and a useful benchmark, but the headline numbers don't match the tables and the benchmark has a same-model bias risk.","tokens_in":15721,"tokens_out":2843,"would_cite":false,"duration_ms":24395,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces ChartLens, a post-hoc visual attribution method that grounds chart-related answers in the specific bars, lines, and pie slices that support them, and claims a 26-66% improvement over existing baselines on a new…","keywords":["chart attribution","fine-grained visual attribution","set-of-marks prompting","multimodal large language models","hallucination mitigation","chart segmentation","visual grounding","benchmark"],"falsifier":"Take a random sample of ChartVA-Eval and have new human annotators mark the ground truth from scratch, enforcing precision by deleting any element that does not directly support the answer; recompute ChartLens's F1 against these revised sets. If the 26-66% margin over baselines shrinks or disappears, the headline improvement is partly an artifact of annotation bias.","tokens_in":14726,"feed_emoji":"📊","tokens_out":8669,"duration_ms":67521,"temperature":0.7,"pith_summary":"The paper tries to establish that chart answers from multimodal language models can be verified by attributing each answer to the exact chart elements supporting it, and that a segmentation-plus-prompting pipeline can do this reliably. It introduces ChartLens, which detects and labels bars, pie sectors, and line segments, then asks a multimodal LLM to point to the labels that justify the answer. To measure this, it builds ChartVA-Eval, a benchmark of over 1,200 chart-question-answer triples with fine-grained attribution annotations drawn from synthetic and real-world sources. The paper reports that ChartLens improves fine-grained attribution by 26-66% over zero-shot GPT-4o, Kosmos-2, and LISA. If true, this offers a plug-and-play verification layer for chart-VQA systems that does not require retraining the underlying model.","feed_headline":"ChartLens points to the exact bars and slices behind an answer","feed_subtitle":"A post-hoc attribution method grounds chart answers to specific visual elements.","key_machinery":"The mechanism that carries the argument is set-of-marks prompting coupled with segmentation-based mark generation. Segmentation turns a chart into discrete candidate objects: bars and pie sectors emerge from heuristic preprocessing (thresholding, contour extraction, radial unrolling) refined by the Segment Anything Model, while lines are extracted by LineFormer and divided into segments. Each candidate receives an alphanumeric mark overlaid on the image. The multimodal LLM is then given the marked chart plus a prompt instructing it to validate the answer and identify the specific marks that substantiate it. This makes attribution a selection problem over labeled visual anchors rather than a free-form localization problem.","core_discovery":"The central claim is that post-hoc fine-grained visual attribution for charts can be solved by a two-stage pipeline: segment the chart into candidate elements, then use set-of-marks prompting, where candidate regions are overlaid with alphanumeric labels, so that a multimodal LLM selects the elements that validate the answer. ChartLens instantiates this pipeline: bars are found by Otsu thresholding and contour analysis refined with SAM; pie sectors by radial unrolling and edge detection refined with SAM; and lines by LineFormer, split into equally spaced segments. The MLLM is prompted with a marked chart, a chain-of-thought instruction to first validate the answer and then name the marked elements that support it. Against zero-shot GPT-4o bounding boxes, Kosmos-2, and LISA, the paper reports F1 of 69.28, 34.65, and 64.14 on bar-chart splits of AITQA, PlotQA, and ChartQA respectively, and 48.56 on pie charts, with line-chart detection rates of 52-78% at drastically reduced chart-area coverage.","pith_inferences":["The method's dependence on GPT-4o both as annotator and as attribution engine means the reported gains could partly reflect self-consistency; a re-annotation study with humans enforcing precision would measure that bias.","A natural extension is to use attribution sets as corrective feedback: when the selected marks do not support the answer, feed that signal back to the MLLM to revise its response, extending post-hoc attribution to hallucination repair.","The benchmark currently ignores textual chart components such as titles, axis labels, and legends; extending attribution to these textual marks could improve real-world applicability and is explicitly left for future work.","The segmentation bottleneck suggests that attribution quality will track advances in chart element segmentation; replacing SAM and LineFormer with stronger segmenters is a direct way to push the reported numbers higher."],"forward_implications":["ChartLens can serve as a plug-and-play verification layer for any chart question-answering MLLM: responses are checkable by looking at which marked elements support the answer.","The ChartVA-Eval benchmark provides a standardized way to measure attribution quality, so future methods can be compared on grounding rather than only answer accuracy.","If an answer has no supporting marks, the pipeline flags it as likely inconsistent with the chart, enabling hallucination detection.","The line-chart metrics (detection rate and area covered) make precision explicit for continuous chart elements, where bounding-box overlap is not well-defined.","Because segmentation is modular, improving the segmentation stage should improve attribution without changing the prompting stage."],"supporting_citations":[{"why":"Supplies the set-of-marks prompting technique that lets the MLLM select labeled chart elements.","marker":"(Yang et al., 2023)"},{"why":"Provides the Segment Anything Model used to refine bar and pie masks from heuristic contours.","marker":"(Kirillov et al., 2023)"},{"why":"Provides LineFormer, the transformer-based line extractor used to obtain line segments for attribution.","marker":"(Lal et al., 2023)"},{"why":"GPT-4o is the base MLLM for ChartLens and the zero-shot bounding-box baseline it must beat.","marker":"(OpenAI, 2024)"},{"why":"Kosmos-2 serves as a grounded-MLLM baseline with markdown-link bounding boxes.","marker":"(Peng et al., 2023)"},{"why":"LISA serves as a reasoning-segmentation baseline that produces masks from textual queries.","marker":"(Li et al., 2023b)"},{"why":"Provides the MATSA-AITQA tables and QA pairs that become the synthetic chart split of ChartVA-Eval.","marker":"(Mathur et al., 2024)"},{"why":"PlotQA supplies synthetic scientific charts, crowd-sourced questions, and data sources for the PlotQA split.","marker":"(Methani et al., 2020)"},{"why":"ChartQA supplies real-world charts and human-authored QA pairs for the ChartQA split, including pie charts.","marker":"(Masry et al., 2022)"}],"fun_headline_variants":["ChartLens pinpoints the bars and slices behind any answer","Fine-grained chart attribution: ChartLens names the deciding elements","ChartLens marks the exact chart parts that support a response","ChartLens: see which chart elements justify an answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth attributions in ChartVA-Eval are reliable: they were drafted by GPT-4o from data tables and human-checked only for relevance and completeness, not for precision, so the benchmark may contain extra elements and may favor whatever GPT-4o treats as supporting evidence.","fun_headline_variants_meta":{"raw":{"variants":["ChartLens pinpoints the bars and slices behind any answer","Fine-grained chart attribution: ChartLens names the deciding elements","ChartLens marks the exact chart parts that support a response","ChartLens: see which chart elements justify an answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000132,"raw_usage":{"total_tokens":1103,"prompt_tokens":888,"completion_tokens":215,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":145}},"tokens_in":504,"tokens_out":215,"duration_ms":3258,"temperature":1.0,"reasoning_tokens":145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:14:50.507165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ChartVA-Eval and have new human annotators mark the ground truth from scratch, enforcing precision by deleting any element that does not directly support the answer; recompute ChartLens's F1 against these revised sets. If the 26-66% margin over baselines shrinks or disappears, the headline improvement is partly an artifact of annotation bias.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LineFormer, the transformer-based line extractor used to obtain line segments for attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o is the base MLLM for ChartLens and the zero-shot bounding-box baseline it must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MATSA-AITQA tables and QA pairs that become the synthetic chart split of ChartVA-Eval."}],"review_version":1}