{"id":"c1c2bb4f-c080-4481-98da-f27f1f84c0c7","arxiv_id":"2501.04303","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multimodal scene graph with contrastive learning, injected as a decoder soft prompt, gives small ChartQA gains, but ablations show the visual graph alone often matches or beats it.","lead":"This paper adds a graph-based contrastive learning layer to chart question answering, aligning visual and textual scene graphs and feeding them into a transformer decoder as soft prompts. It reports modest gains on ChartQA and OpenCQA, but its own ablations show the visual graph alone often performs just as well.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported GCL gains are not causally identifiable because Table 2 lacks a matched no-graph fine-tuning baseline.","rationale":"The reader's weakest assumption focuses on noisy Mask R-CNN/OCR alignment corrupting contrastive pairs. That is plausible, but the more load-bearing problem is earlier in the inference chain: the experiments do not isolate the proposed module from ordinary fine-tuning. A method that injects a graph prompt and then fine-tunes the whole backbone on the evaluation datasets can improve for reasons unrelated to graph structure or contrastive alignment; the published baselines in Table 2 are from different training runs/papers. The paper's own Table 5 shows that the visual graph alone often matches or beats the full multimodal variant, which suggests the textual graph and cross-modal alignment are not obviously the active ingredient. These observations do not prove the method is ineffective; they show the evidence as presented does not license the causal claim. A matched baseline is cheap and decisive, so CONDITIONAL remains the right verdict: accept only if that control is provided and the summary inconsistencies in Section 3.2/Table 5 are corrected. My concern partially overlaps with the reader's because both point to missing evidence for the claim's mechanism, but the missing baseline is the more fundamental gap.","tokens_in":10425,"tokens_out":6055,"duration_ms":58842,"concrete_test":"Run the authors' exact training recipe (same data, optimizer, epochs, batch size, learning rate, and seed) on UniChart and VL-T5 with the graph prompt and GCL removed, training only the standard QA loss on ChartQA and OpenCQA. Compare those no-graph numbers directly to UniChart(O) and VL-T5(O) in Table 2. If the no-graph baseline closes most of the 9.04/7.59 improvements, the paper must abandon the causal attribution; if it does not, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: graph contrastive learning 'significantly boosts' backbone performance. But Table 2 supports this only by comparing UniChart(O) and VL-T5(O) to an original-paper result and a Hugging Face checkpoint (Section 3.2, Table 2). Those baselines are not matched to the authors' fine-tuning protocol — the same train/validation splits, optimizer, epochs, batch size, learning rate, and random seeds. The headline improvements (up to 9.04 on ChartQA and 7.59 on OpenCQA) could therefore come from fine-tuning the backbone on the target datasets or from other training choices, not from the graph layer or GCL. The row UniChart[2] (graph without GCL, 61.4 avg / 11.91 BLEU) is the only available GCL ablation, but the paper does not state that it was rerun under identical conditions; if it is taken from reference [2], the comparison is also uncontrolled. Table 5 further weakens the causal story: visual-graph-only prompts outperform the full multimodal prompt on OpenCQA for both backbones and on ChartQA-human for VL-T5, so the multimodal alignment component is not consistently responsible for the gains. Without a same-pipeline no-graph baseline, the central claim is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a multimodal scene-graph framework for chart question answering. The method extracts visual objects with Mask R-CNN and text with Google OCR, constructs visual and textual graphs, aligns their node representations with a graph contrastive InfoNCE objective, and injects the learned graph representations into the decoder of a vision-language backbone as soft prompts. The authors evaluate the approach on ChartQA and OpenCQA for fine-tuned backbones (UniChart, VL-T5) and on ChartQA and ChartX for zero-shot MLLMs with chain-of-thought prompts. They report gains over published baselines, including up to a 9.04-point improvement on ChartQA and 7.59 on OpenCQA for UniChart relative to a Hugging Face checkpoint, and they present ablations on intra-modality contrastive learning and on visual versus textual graph prompts.","tokens_in":10656,"tokens_out":4515,"duration_ms":45441,"significance":"If the central claim were supported, the contribution would be useful: injecting graph-structured scene information as decoder soft prompts is a plausible way to recover object-level information lost by patch-based encoders, and the GCL objective offers a principled alignment mechanism. The paper also provides a compact CoT-prompting study for MLLMs on chart tasks, which is of practical interest. The framework is described in enough detail to reimplement in broad strokes, and the ablation structure shows awareness of the key design choices. However, the experimental design does not currently isolate the effect of the proposed graph layer and GCL from the effect of fine-tuning itself, and the ablations partly contradict the multimodal-alignment premise.","major_comments":[{"comment":"The central claim that graph contrastive learning 'significantly boosts the performance of backbone models' is not causally identifiable from the reported comparisons. UniChart(O) is compared against UniChart's original-paper result, a Hugging Face checkpoint, and the previous MSG-Chart system of reference [2], none of which was trained under the authors' fine-tuning protocol. The differences of 9.04 points on ChartQA and 7.59 points on OpenCQA are computed against the checkpoint, not against a matched no-graph baseline. The row labeled UniChart[2] is the only graph-without-GCL comparison, but the paper does not state whether it was rerun under identical conditions or is quoted from reference [2]. A matched baseline consisting of the same backbone fine-tuned on the same splits with the same optimizer, epochs, batch size, learning rate, and seeds, but without any graph prompt or GCL loss, is required to attribute the gains to the proposed method.","section":"Section 3.2, Table 2"},{"comment":"The full multimodal VG+TG configuration is not consistently the best, which undercuts the premise that cross-modal alignment is what drives the improvement. On OpenCQA, visual-graph-only prompts outperform the full multimodal prompt for both backbones (16.68 vs 16.35 for UniChart; 19.03 vs 18.42 for VL-T5), and on VL-T5 ChartQA the visual-only variant is better on both the human set (38.16 vs 34.96) and the average (65.04 vs 63.60). The authors themselves concede that predicted labels from Mask R-CNN are 'highly noisy, particularly on OpenCQA.' Since the positive pairs in Eq. (2) are defined by aligning detected objects with OCR-derived textual nodes, noisy detections can corrupt the contrastive objective. The manuscript should provide evidence that the node alignments are reliable, for example by measuring alignment accuracy against ground-truth chart annotations or by ablating with ground-truth object locations.","section":"Section 3.3, Table 5"},{"comment":"All reported scores are single numbers with no variance or significance testing. Several key comparisons are small (e.g., 67.44 vs 66.64 average on ChartQA; 16.35 vs 16.38 BLEU on OpenCQA in Table 5), and without repeated runs or error bars these differences are not distinguishable from noise. The paper should report means and standard deviations over multiple seeds and, where appropriate, a significance test for the headline gains in Table 2.","section":"Section 3.2 and Section 3.3"},{"comment":"The decoder integration is described only for the textual graph: the paper says the embeddings of 36 special tokens are replaced with 'object node representations from the textual graph.' Yet Table 5 reports prompts built from the visual graph only, and the abstract claims a joint multimodal graph is integrated. The manuscript should specify how the visual-graph representation is injected into the decoder and whether the 36-token prompt is used for both modalities, as this is essential to reproducing the reported configurations.","section":"Section 2.4 and Table 5"}],"minor_comments":[{"comment":"The abstract says evaluations include ChartX, but Section 3.1 clarifies that GCL is evaluated on ChartQA and OpenCQA while ChartX is used only for CoT prompting of MLLMs; this distinction should be stated more explicitly in the abstract or introduction.","section":"Abstract and Section 3.1"},{"comment":"The phrase '9.04% increase' refers to an absolute percentage-point difference in relaxed accuracy (67.44 vs 58.40), not a relative increase; the paper should use 'percentage points' or give relative percentages for clarity.","section":"Section 3.2"},{"comment":"Hyperparameters for the graph and contrastive components are only partially specified: K is stated as three in the text but not defined as a hyperparameter, and the temperature tau in Eq. (1), GCN hidden dimensions, number of training epochs, learning rate, and batch size are not reported. Reporting these values is necessary for reproducibility.","section":"Section 2.2 and Section 2.3"},{"comment":"The manuscript contains ACM template placeholder text, including the CCS Concepts line 'Do Not Use This Code' and the keywords 'Do, Not, Us, This, Code...'; these must be replaced with the correct metadata.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The novelty relative to the authors' own MSG-Chart paper (reference [2]) needs to be made precise, and the comparison to that prior work must be controlled rather than quoted. The manuscript also has the appearance of a conference-style paper with template placeholders; if it is being considered for a journal, the experimental section needs substantially more rigor, including matched baselines, variance reporting, and a complete specification of the training protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the graph-contrastive soft prompt is a legitimate, incremental extension of your MSG-Chart line, but the headline claim that GCL 'significantly boosts' the backbones is not supported by the experiments as reported. Treat the paper as a method proposal with an ablation that cuts against it, not as a demonstration.\n\nWhat's actually new: unlike MSG-Chart, the unified graph representation is injected into the decoder as 36 soft-prompt tokens rather than into the encoder, and an inter-modality InfoNCE loss aligns visual and textual graph nodes. The visual graph uses KNN edges instead of a fully connected graph. That is a reasonable design shift, and the CoT prompting study, though mixed, is a separate minor contribution. The paper is clearly written and the authors deserve credit for running a graph-setting ablation (Table 5) and for admitting that Mask R-CNN labels are noisy, especially on OpenCQA.\n\nThe soft spots are not minor. First, there is no matched no-graph baseline. Table 2 compares UniChart(O) and VL-T5(O) to the original paper's published number or a Hugging Face checkpoint, not to the same backbone fine-tuned on the same splits with the same optimizer, epochs, batch size, and seeds. The 9.04-point ChartQA gain could come from fine-tuning protocol alone. The rows taken from MSG-Chart, if they are taken from [2], are also uncontrolled. Second, the paper's own Table 5 shows that using only the visual graph often beats the full visual-plus-textual graph: on OpenCQA for both backbones and on ChartQA-human for VL-T5. The sentence in Section 3.2 claiming both backbones 'attain their best scores' with both graphs contradicts that table. Third, there are no error bars or significance tests, and several of the decisive differences are around one point. Fourth, the alignment premise is fragile: positive pairs in InfoNCE assume Mask R-CNN boxes and OCR text refer to the same objects, and the authors concede the labels are highly noisy.\n\nThe central idea is not dead. Decoder-side graph prompts could work with cleaner detections, and intra-modality CL hurting is an honest, useful ablation result. But as it stands, the load-bearing causal claim is unverified. The paper needs a same-pipeline no-graph baseline, matched reruns of the MSG-Chart comparisons, error bars, code, and a rewrite that does not oversell.\n\nI would send this to review rather than desk reject—there is a real method here and the limitations are fixable—but I would ask for major revision and a much more careful evaluation before anything is accepted.","headline":"Plausible method, unproven headline: the GCL gains aren't causally identifiable without a matched no-graph baseline, and the authors' own ablation says visual-only is often best.","tokens_in":11200,"tokens_out":2684,"would_cite":false,"duration_ms":26280,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that injecting a contrastively trained multimodal scene-graph layer as a decoder soft prompt materially improves chart question answering, reporting a 9.04-point ChartQA gain and a 7.59-point OpenCQA gain over the…","keywords":["chart question answering","multimodal scene graph","graph contrastive learning","soft prompt","chain-of-thought prompting","hallucination mitigation","chart understanding"],"falsifier":"Scramble the pairing between visual and textual graph nodes before contrastive training and rerun the same pipeline: if chart QA performance stays at the same level as the correctly aligned model, then cross-modal alignment is not the cause of the gain. Alternatively, build the graphs from ground-truth chart tables instead of detector and OCR output; if the improvement disappears, the bottleneck is perception quality rather than graph reasoning.","tokens_in":10193,"feed_emoji":"📊","tokens_out":7043,"duration_ms":65669,"temperature":0.7,"pith_summary":"Chart question answering asks a model to read a chart and answer questions that often require locating values and comparing components, something patch-based vision encoders do poorly because they blur object-level detail. The paper proposes to recover that detail explicitly: it detects chart objects, builds a visual scene graph of spatial neighbours and a textual scene graph of labels and OCR text, aligns the two graphs with contrastive learning, and inserts the aligned graph representations into a transformer decoder as a soft prompt. The central claim is that this graph layer is what lifts performance, and the paper reports gains up to 9.04 points on ChartQA and 7.59 BLEU points on OpenCQA over the released UniChart checkpoint, alongside smaller gains over the original UniChart results. The paper also tries chain-of-thought prompting for multimodal LLMs, observes that it helps on the more diverse ChartX benchmark but not consistently on ChartQA, and takes that as evidence that explicit graph structure still has a role to play.","feed_headline":"Gain 9 points on Chart QA with graph contrastive prompts","feed_subtitle":"Aligning visual and textual chart elements as soft prompts beats strong baselines on ChartQA and OpenCQA.","key_machinery":"The load-bearing object is the multimodal scene-graph soft prompt: a pair of graphs, one visual and one textual, whose node representations are aligned by graph contrastive learning and then injected into a text decoder as trainable prompt tokens. The visual graph encodes spatial proximity between detected chart objects using k-nearest-neighbour edges and distance-based edge weights; the textual graph encodes labels and OCR text. Two graph convolutional encoders compute node representations, a contrastive InfoNCE-style objective maximizes similarity between nodes representing the same object across modalities, and the aligned textual node representations replace special graph tokens prepended to the question, turning the graph into a soft prompt without modifying the backbone encoder. This mechanism is what the paper credits for the reported performance gains, and it is also what the ablations probe when they remove one graph or add intra-modality contrastive loss.","core_discovery":"The central claim, stated on the paper's own terms, is that multimodal graph contrastive learning improves chart question answering when the learned graph is delivered to the decoder as a soft prompt. In the proposed architecture, a visual graph connects each detected object to its three nearest neighbours with edge weights that decay with bounding-box distance, while a textual graph connects chart labels and OCR text; two graph convolutional encoders produce node representations, and a contrastive objective pulls together node pairs that refer to the same chart component across the two modalities. The textual node representations are then written into special graph tokens prepended to the question, giving the decoder structured information about chart components and their relationships. The paper reports that this addition improves both UniChart and VL-T5 on ChartQA and OpenCQA in most settings, that it outperforms the released UniChart checkpoint by 9.04 points on ChartQA and 7.59 points on OpenCQA, and that dropping either graph or adding intra-modality contrastive loss tends to reduce performance, which the paper reads as evidence that the cross-modal alignment is the active ingredient.","pith_inferences":["One testable extension is to build the graphs from the chart's underlying data table rather than from detected objects and OCR text; if the gain grows, perception noise is the current bottleneck.","The design does not isolate the contrastive objective from the graph prompt itself, so a comparison against a graph prompt trained with plain reconstruction or matching loss would reveal whether the contrastive alignment is necessary or merely sufficient.","The same patch-loss problem exists for tables, infographics, and other structured documents, so the decoder soft-prompt mechanism may transfer beyond charts if the node alignment can be supplied from structured sources.","The ChartX results with the chart-to-table prompt suggest that grouping results by chart type could identify which chart families still need perceptual structure rather than reasoning prompts."],"forward_implications":["If the graph soft prompt is the cause of the gains, existing chart QA backbones can be upgraded by adding a contrastively trained scene-graph layer without retraining from scratch.","The ablations imply that inter-modality alignment, not extra intra-modality structure, is what helps, so future graph-based chart models should concentrate on cross-modal node alignment.","Since adding intra-modality contrastive loss hurts both backbones, the reported results argue against simply stacking more contrastive objectives in generation tasks.","The CoT experiments indicate that prompting alone is unreliable for chart reasoning on diverse chart types, pointing to explicit graph structure as a complementary route to reduce hallucination."],"supporting_citations":[{"why":"Supplies the UniChart backbone whose released checkpoint is the main baseline for the reported 9.04-point ChartQA and 7.59-point OpenCQA gains.","marker":"[14]"},{"why":"Provides the ChartQA dataset and evaluation protocol, and the fine-tuned object detector used to initialize visual graph nodes.","marker":"[15]"},{"why":"Defines the multimodal scene graph construction and prior graph integration approach that this paper modifies with KNN edges and decoder soft-prompt injection.","marker":"[2]"},{"why":"Supplies the multimodal contrastive learning idea that the graph contrastive loss adapts to align visual and textual graph nodes.","marker":"[19]"},{"why":"Provides the InfoNCE-based graph contrastive learning formulation and augmentation strategy used for the inter-modality alignment loss.","marker":"[28]"},{"why":"Contributes the OpenCQA open-ended benchmark and BLEU-based metric used to measure the 7.59-point improvement.","marker":"[9]"},{"why":"Gives the chain-of-thought prompting method the paper adapts for MLLM zero-shot evaluation.","marker":"[25]"},{"why":"Provides the ChartX benchmark and evaluation metric that the CoT experiments use with an exact-match modification.","marker":"[26]"}],"fun_headline_variants":["Graph contrastive learning lifts ChartQA by 9 points","Align chart graphs across modalities for +9 ChartQA","Multimodal graph soft prompts gain 9 points on ChartQA","ChartQA up 9 points via graph contrastive alignment","Graph contrastive prompts add 9 points to ChartQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that detected chart objects and OCR-extracted text can be aligned into trustworthy visual-textual node pairs; if the detector or OCR produces wrong labels or positions, the contrastive loss connects mismatched components and the graph prompt carries corrupted structure into the decoder.","fun_headline_variants_meta":{"raw":{"variants":["Graph contrastive learning lifts ChartQA by 9 points","Align chart graphs across modalities for +9 ChartQA","Multimodal graph soft prompts gain 9 points on ChartQA","ChartQA up 9 points via graph contrastive alignment","Graph contrastive prompts add 9 points to ChartQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":2987,"prompt_tokens":884,"completion_tokens":2103,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":2021}},"tokens_in":500,"tokens_out":2103,"duration_ms":14895,"temperature":1.0,"reasoning_tokens":2021,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:37:00.335558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scramble the pairing between visual and textual graph nodes before contrastive training and rerun the same pipeline: if chart QA performance stays at the same level as the correctly aligned model, then cross-modal alignment is not the cause of the gain. Alternatively, build the graphs from ground-truth chart tables instead of detector and OCR output; if the improvement disappears, the bottleneck is perception quality rather than graph reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UniChart backbone whose released checkpoint is the main baseline for the reported 9.04-point ChartQA and 7.59-point OpenCQA gains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the multimodal scene graph construction and prior graph integration approach that this paper modifies with KNN edges and decoder soft-prompt injection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multimodal contrastive learning idea that the graph contrastive loss adapts to align visual and textual graph nodes."}],"review_version":1}