{"id":"33c380ac-0c24-400f-bc35-a747befbedfb","arxiv_id":"2506.07643","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A GPT-4V/GPT-4o pipeline for completing and refining scene graph annotations yields a dense synthetic dataset that, after instruction tuning, gives a 3B model strong relationship understanding and grounding results.","lead":"The authors build ROBIN, a 3B-parameter vision-language model trained on a synthetic scene graph dataset created by completing human annotations with GPT-4V and refining them with GPT-4o. It reports strong relationship reasoning and state-of-the-art referring expression scores, but several of those benchmarks were also present in the training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's headline comparisons on VSR and GQA are invalid because both benchmarks appear in ROBIN's Stage 1/2 training mixtures; the clean VSR score (69.7, Table 6) falls below BLIP-3's 72.6, so the claimed cross-model superiority is not yet established.","rationale":"Reading in good faith, the paper's central claim is that dense synthetic scene graphs enable a 3B model trained on fewer than 3M instances to beat larger-scale models on relationship understanding. For that claim to hold, the reported benchmarks must measure genuine transfer rather than memorization of training distributions. The training-data tables show otherwise for two of the headline benchmarks: VSR and GQA appear in the Stage 1 and Stage 2 Visual Instruction mixtures, while Table 2 reports them as zero-shot or development-set accuracy. The paper's own VSR-excluded ablation quantifies the contamination: the clean Stage 1 VSR score is 69.7, below the compared baselines, so the cross-model superiority claim on VSR is not merely overstated but reversed under fair evaluation. A parallel concern applies to GQA, though the paper does not provide a GQA-excluded ablation. The reader's stated weakest assumption was the model-based filter (Appendix E.2, Balanced accuracy 0.6165), which is a legitimate data-quality issue, and I agree it deserves scrutiny; however, the benchmark contamination is more load-bearing because it directly invalidates the evidence for the strongest claim, independent of filter quality. I do not see this as a rejection-worthy flaw: the pipeline is novel, the dataset is released, the SG-EDIT ablation with VSR removed gives partial evidence that scene graph data helps region classification and some relation benchmarks, and the uncontaminated columns still show gains. But the headline must be corrected, the contaminated rows re-run with training data removed, and the comparisons restated accordingly. That is a conditional acceptance, matching the reader's verdict, so I recommend UNCHANGED.","tokens_in":30956,"tokens_out":3580,"duration_ms":42576,"concrete_test":"Retrain or reuse the Table 6 last-row variant (no VSR in the training mixture) and evaluate it on the VSR zero-shot test, then compare against BLIP-3-single-image (72.6) and Phi-3-Vision (72.5); if the clean VSR score is ~69.7, the Table 2 VSR column is an artifact. Separately, retrain a variant that removes GQA and GQA-CoT from the Stage 1/2 VQA mixture and re-evaluate GQA Dev-EN; if the score drops or the gap to baselines reverses, remove the contaminated columns from the headline and restrict the claim to benchmarks never used in training or filtering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires fair, zero-shot evaluation on relationship benchmarks. That condition fails for at least two reported benchmarks. Table 7 and Table 8 list VSR and GQA (and GQA-CoT) inside the Visual Instruction VQA data used in Stage 1 and Stage 2 training, yet Table 2 reports VSR \"ZS-test\" and GQA Dev-EN as headline results. The paper itself acknowledges the overlap by training an ablation that excludes VSR from instruction data (Table 6, last row), but the model reported in Table 2 uses the contaminated mixture. Under the clean evaluation, ROBIN-3B (Stage 1) scores 69.7 on VSR rather than 73.7, which is below BLIP-3-single-image (72.6) and Phi-3-Vision (72.5). The GQA result (61.6) is likewise in-distribution because GQA and GQA-CoT appear in the training data, so it cannot be read as evidence of transfer from dense scene graphs. The remaining uncontaminated columns (MMBench, SeedBench, CRPE, SugarCrepe, What's Up) still favor ROBIN, and the VSR-excluded ablation in Table 6 provides some independent support for the scene graph data, which is why the work retains value. But the headline assertion that ROBIN-3B outperforms models trained on over 300M instances on relationship understanding is not supported until the contaminated columns are removed or replaced with genuinely held-out evaluations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SVG (Synthetic Visual Genome), a pipeline for creating dense scene graph annotations at scale: Stage 1 uses GPT-4V to complete missing relationships for selected objects in seed images with human annotations, followed by rule-based and model-based filtering; Stage 2 describes SG-EDIT, a self-distillation framework in which ROBIN generates scene graphs that GPT-4o edits, and these edited graphs are used for further training. The authors train ROBIN-3B, a Qwen2.5-3B-based multimodal model, on about 1.73M Stage 1 instances and 1.23M Stage 2 instances, then report results on relationship understanding benchmarks, referring expression comprehension, region recognition, and panoptic scene graph generation. The central claim is that training on refined dense scene graphs yields strong relationship understanding with far fewer training instances than larger-scale baselines.","tokens_in":31345,"tokens_out":5548,"duration_ms":67727,"significance":"If the central claim holds, the paper makes a valuable contribution: dense synthetic scene graphs could be an efficient training signal for multimodal relationship reasoning, and the proposed filtering and self-distillation pipeline is a concrete recipe for generating such data at scale. The paper also includes useful ablations (e.g., the role of scene graph data in Stage 1, shown in Table 6, and the effect of GPT-4 editing in Figure 4) and the authors state that data, code, and checkpoints will be released. However, the headline evaluation is materially weakened by training/benchmark overlap for VSR, GQA, and PSG, and by filter selection on the same SugarCREPE/CREPE benchmarks that are then reported as results. The remaining clean evidence (e.g., What's Up?, MMBench, and the VSR-excluded ablation) still suggests some benefit, but the paper's strongest claims of cross-model superiority on relationship understanding are not yet established.","major_comments":[{"comment":"Table 2 reports VSR ZS-test and GQA Dev-EN as headline results, but Tables 7 and 8 show that VSR, GQA, and GQA-CoT are included in the Stage 1 and Stage 2 visual instruction VQA training mixtures. The reported scores (76.4 on VSR, 61.6 on GQA) therefore cannot support the claim that ROBIN-3B outperforms models trained on over 300M instances on relationship understanding. The paper itself acknowledges the overlap in Sec. 4.5, where excluding VSR from training lowers Stage 1 VSR to 69.7 (Table 6, last row), which is below BLIP-3's 72.6. Please re-report Table 2 using only genuinely held-out benchmarks, or clearly label the in-distribution columns and remove them from the headline comparison.","section":"Table 2; Tables 7 and 8"},{"comment":"The model-based filtering method is selected by evaluating on SugarCREPE and CREPE (Table 17), yet CRPE and SugarCrepe are then reported as benchmark results in Table 2. This is a selection-on-test-set loop: the reported gains on these benchmarks are not independent evidence for the filtering method. The human evaluation in Table 18 shows only a modest advantage for the chosen LLaVA+Qwen filter (balanced accuracy 0.6165 vs. 0.5484 for majority voting), which is a weak validation for a filter that determines the quality of the entire SVG dataset. Please validate the filter on a held-out set that is not subsequently reported as a benchmark, and either remove or explicitly relabel the CRPE/SugarCrepe columns as in-domain diagnostics.","section":"Appendix E.2; Table 2"},{"comment":"Table 5 reports panoptic scene graph generation on PSG, but PSG appears as scene graph detection data in both Stage 1 (Table 7) and Stage 2 (Table 8), and Stage 2 uses PSG as a source for SVG-SG generation. The comparison with closed-set models that are fine-tuned on PSG is therefore not a fair open-ended comparison: ROBIN has also been trained on PSG scene graph annotations. Please either evaluate on a held-out scene graph benchmark, or explicitly state the training overlap and avoid claiming superiority over PSG-fine-tuned models in an open-ended setting.","section":"Table 5; Tables 7 and 8"},{"comment":"Table 4 reports ADE20K open-vocabulary segmentation results, but Sec. 2 states that SVG-SG includes 25K ADE20K images used in Stage 2 training. The ADE20K scores are therefore obtained on images whose scene graph annotations were part of the training data, so the comparison against Osprey-7B on ADE20K is not an out-of-distribution evaluation. Please either re-evaluate on a held-out split or clearly identify the ADE20K result as in-domain.","section":"Table 4; Sec. 2"}],"minor_comments":[{"comment":"The text in Sec. 3.2 says Stage 2 training uses 1.23M samples, while the caption of Table 8 says 1.19M instances; please reconcile these numbers.","section":"Sec. 3.2; Table 8"},{"comment":"The text refers to the 'CPRE dataset' but the benchmark is CRPE; this typo appears in the first paragraph of Sec. 4.1.","section":"Sec. 4.1"},{"comment":"The header of the CRPE column cites reference [81] (ASM-V2), but the benchmark described in Sec. 4.1 and Appendix E.2 is CREPE from reference [56]; please correct the citation.","section":"Table 2"},{"comment":"The phrase 'identify at lest K = 5 subjects' should read 'at least K = 5 subjects'.","section":"Appendix E.1"},{"comment":"The model name is written as 'LLaVa-v1.6-vicuna-13b', while elsewhere it appears as 'LLaVA'; please use a consistent notation.","section":"Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong pipeline and useful ablations, but the evaluation contains multiple train/test overlaps that directly affect the headline claims. The issues are fixable by re-running or re-labeling evaluations, so I do not recommend rejection. I would ask the authors to remove or clearly mark in-distribution benchmark columns, re-validate the filter on a held-out set, and temper the abstract and conclusion claims accordingly. The remaining clean evidence, especially the VSR-excluded ablation in Table 6 and the gains on What's Up?, suggests the core idea still has merit after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline on this one is not the headline they wrote. The data pipeline is a real contribution, but the central comparison in Table 2 is compromised because VSR and GQA appear in the Stage 1 and Stage 2 training mixtures (Tables 7 and 8) and are then reported as held-out benchmarks. The paper's own ablation that removes VSR from training (Table 6, last row) gives a clean VSR score of 69.7, below BLIP-3's 72.6 and Phi-3-Vision's 72.5. So the claim that ROBIN-3B outperforms models trained on 300M instances is not supported as stated. GQA (61.6) is equally in-distribution.\n\nThat said, there's a lot here worth taking seriously. The core idea—completing existing human scene graph annotations with GPT-4V instead of generating from scratch, then filtering with rules and a VQA model, then self-distilling through a student model with GPT-4o edits—is new and sensible. The SVG dataset (146K images, 5.6M relations) is a useful resource, and the pipeline details (prompts, filtering rules) are transparent. The ablations are also better than average: they explicitly remove VSR from training and still show gains from adding scene graph data (+1.5 on VSR, +0.7 on CRPE in Table 6). Filtering itself is shown to help substantially (Table 15). So there is independent evidence that dense scene graphs, not just the benchmark contamination, drive part of the improvement.\n\nThe soft spots beyond the overlap: the model-based filter is validated on only 600 human examples, with balanced accuracy 0.6165 versus 0.5484 for majority voting (Table 18). That's modest, though precision is high and the downstream gains suggest it does more good than harm. The same filter-selection evaluation is done on SugarCREPE and CREPE, which later appear in Table 2; that's another form of test-set tuning. Also, the paper reports no variance or multiple seeds, so it's hard to know how robust the small deltas are.\n\nNet: the work deserves a serious referee, but the authors need to re-run on genuinely held-out benchmarks, remove the contaminated columns from the headline, and report variance. If they do that, the dataset and pipeline should be useful to anyone working on relationship understanding or scene graph generation.\n\nRecommendation: send to peer review, but with major revisions and a request for clean evaluation.","headline":"The data pipeline is a real contribution, but Table 2's headline comparison is contaminated by training/benchmark overlap.","tokens_in":31904,"tokens_out":4114,"would_cite":true,"duration_ms":43023,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ROBIN-3B, trained on fewer than 3 million synthetic scene-graph instances, outperforms same-size models trained on over 300 million instances and surpasses models up to 13B on relationship understanding and referring expression…","keywords":["scene graph generation","visual relationship understanding","synthetic data","instruction tuning","multimodal language model","self-distillation","referring expression comprehension","data filtering"],"falsifier":"Have human annotators label a larger random sample (say 5,000) of relations that the two-model VQA filter keeps versus discards, and compare per-type precision; the dense-scene-graph claim weakens if kept relations are not clearly more accurate than discarded ones, or if gains on relationship benchmarks vanish when the model is trained on an equal-size random subset of unfiltered relations.","tokens_in":30763,"feed_emoji":"🖼️","tokens_out":8650,"duration_ms":99729,"temperature":0.7,"pith_summary":"The paper claims that dense, machine-generated scene graphs—not ever-larger datasets—are the missing training signal for visual relationship reasoning. It builds SVG, a synthetic dataset of 146K images with 5.6M relationships over 2.6M objects, by having a teacher multimodal model fill in missing relations in existing human-annotated scene graphs and then filtering the results. Training a 3B mask-aware model, ROBIN, on this data yields better relationship understanding than same-size models trained on 300M+ instances, and the best referring-expression score (88.9 average) among models up to 13B. A second stage, SG-EDIT, has ROBIN generate scene graphs and a frontier model refine them, and each stage adds consistent benchmark gains. If the claim holds, synthetic relational annotation becomes a cheap, scalable substitute for massive human-curated relational data.","feed_headline":"Dense synthetic scene graphs lift a 3B model past 13B rivals","feed_subtitle":"A 3B model trained on fewer than 3M examples tops relationship-reasoning and grounding benchmarks.","key_machinery":"The load-bearing object is the two-stage synthetic scene-graph data engine, with filtering at each stage. Stage 1 uses human-annotated seed regions (COCO/LVIS detections, VG/GQA scene graphs, region captions, depth maps) and prompts a teacher model to complete missing relationships in five categories; spatial relations survive seven geometric rules, and non-spatial relations survive only if both a LLaVA-based and a Qwen-VL-based binary VQA check answer 'Yes'. Stage 2 is a self-distillation loop in which ROBIN generates candidate dense scene graphs from segmentation masks, GPT-4o edits them by removing unlikely relations and adding relevant ones, and the edited graphs become the next training set. The model's architecture represents each region by both pixel-level masks and normalized text coordinates, which lets one forward pass handle up to 99 regions per image for dense relation generation.","core_discovery":"The paper's central discovery is that a 3B multimodal model can reach or exceed the relationship reasoning of much larger models when it is instruction-tuned on dense, synthetic scene graphs rather than on hundreds of millions of web-scale image-text pairs. Starting from 33K seed images whose regions are grounded by human labels and segmentation masks, the pipeline asks GPT-4V to list at least five relationship categories per subject, producing SVG-RELATIONS with roughly five relations per region; then ROBIN is trained and used to generate scene graphs for ADE20K, PSG, and VG images, which GPT-4o edits before a second training pass. The final ROBIN-3B reports 61.6 GQA accuracy, 77.6 MMBench, 90.1 SugarCrepe replace-relation, 86.2 What's Up controlled, 88.8 average RefCOCO/+/g, and 21.0 R@20 on PSG scene graph generation, surpassing same-size baselines such as Phi-3-Vision and BLIP-3 and larger models such as ASM-V2-13B. The paper's own ablations attribute the gains to adding scene-graph data to the instruction mixture and to GPT-4o-edited self-distillation.","pith_inferences":["I would not yet generalize the data-efficiency claim to images far outside the COCO-style seed distribution; the paper itself lists in-the-wild evaluation as future work, and all seed images come from datasets with human annotations.","The VQA filter's balanced accuracy on the paper's own 600-example human audit (0.6165 vs. 0.5484 for majority voting) is thin support for the claim that filtering is what makes the data high quality; a larger human audit of kept versus discarded relations would separate true filtering gains from an easier training distribution.","An implication the authors leave implicit is that the same five-relation taxonomy could be carried into video or 3D scene graphs; if the efficiency result is real, adding scene-graph objectives to other mid-size MLMs is a natural next test.","The referring-expression gains may be inflated by RefCOCO-family grounding data in the training mixture; an evaluation on referring expressions from a held-out source would show how much the scene-graph data alone contributes."],"forward_implications":["If ROBIN-3B's results are taken at face value, dense synthetic scene graphs can replace hundreds of millions of web-scale training instances for relationship reasoning.","Scene-graph instruction tuning transfers to grounding and region recognition: the same model improves RefCOCO/+/g, ADE20K, LVIS, and PACO scores, so relational data helps beyond relation-specific benchmarks.","The SG-EDIT loop shows a model's own generations, when edited by a stronger teacher, are a reusable data source; training on edited graphs beats training on raw generations on every relationship benchmark.","A mask-aware 3B model can produce open-ended panoptic scene graphs competitive with closed-set models fine-tuned on the target dataset, which suggests general-purpose MLMs can absorb scene graph generation as a standard task."],"supporting_citations":[{"why":"Visual Genome supplies the human-annotated scene graphs and region captions used as seed data for relationship completion and as training data for ROBIN.","marker":"[40]"},{"why":"Panoptic Scene Graph dataset provides the scene graph generation benchmark and one of the additional datasets distilled in Stage 2.","marker":"[87]"},{"why":"Osprey contributes the mask-aware architecture and progressive training recipe that ROBIN extends to dense scene graphs.","marker":"[96]"},{"why":"ASM-V2 is the main open-ended scene graph/relation baseline whose 13B model ROBIN-3B surpasses.","marker":"[81]"},{"why":"LLaVA establishes visual instruction tuning and serves as one of the two VQA filter models in data filtering.","marker":"[52]"},{"why":"Qwen-VL serves as the second VQA filter model whose 'Yes' answers are required to keep a generated relation.","marker":"[3]"},{"why":"Segment Anything produces the segmentation masks used to select semantically significant regions in seed images.","marker":"[39]"},{"why":"Semantic-SAM supplies multi-granularity masks for object and part proposals in region selection.","marker":"[44]"},{"why":"GQA contributes seed scene graphs and is one of the relationship understanding benchmarks used for evaluation.","marker":"[28]"}],"fun_headline_variants":["3B model beats 13B on relationships using synthetic scene graphs","Synthetic scene graphs lift 3B model past 13B rivals","3B outperforms 13B via refined scene graphs","Dense synthetic graphs give 3B model edge over 13B","Scene graphs: quality data lets 3B beat 13B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the two-model 'Yes/Yes' VQA filter actually removes bad relationships, but on the paper's own 600-example human evaluation that filter's balanced accuracy of 0.6165 barely beats the 0.5484 of simply always answering 'Yes', so if this filter mostly keeps easy relations and throws out hard but valid ones, the reported gains could come from an easier training distribution rather than dense scene graphs.","fun_headline_variants_meta":{"raw":{"variants":["3B model beats 13B on relationships using synthetic scene graphs","Synthetic scene graphs lift 3B model past 13B rivals","3B outperforms 13B via refined scene graphs","Dense synthetic graphs give 3B model edge over 13B","Scene graphs: quality data lets 3B beat 13B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1611,"prompt_tokens":1077,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":443}},"tokens_in":693,"tokens_out":534,"duration_ms":6024,"temperature":1.0,"reasoning_tokens":443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:29:16.803158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators label a larger random sample (say 5,000) of relations that the two-model VQA filter keeps versus discards, and compare per-type precision; the dense-scene-graph claim weakens if kept relations are not clearly more accurate than discarded ones, or if gains on relationship benchmarks vanish when the model is trained on an equal-size random subset of unfiltered relations.","supporting_citations":[{"cited_title":"Panoptic scene graph generation","cited_arxiv_id":null,"evidence_quote":"Panoptic Scene Graph dataset provides the scene graph generation benchmark and one of the additional datasets distilled in Stage 2."},{"cited_title":"Os- prey: Pixel understanding with visual instruction tun- ing, 2024","cited_arxiv_id":null,"evidence_quote":"Osprey contributes the mask-aware architecture and progressive training recipe that ROBIN extends to dense scene graphs."},{"cited_title":"The all-seeing project v2: Towards gen- eral relation comprehension of the open world, 2024","cited_arxiv_id":null,"evidence_quote":"ASM-V2 is the main open-ended scene graph/relation baseline whose 13B model ROBIN-3B surpasses."}],"review_version":1}