{"id":"1760a1cb-4484-48e6-9a95-726777b43678","arxiv_id":"2412.08125","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A training and inference framework that decomposes visual queries into nested phrases and grounds them progressively outperforms baseline LVLMs on compositional grounding and reasoning benchmarks.","lead":"This paper introduces PromViL, a framework that improves vision-language models on compositional visual questions by grounding simple phrases first and then using those grounded phrases as clues to find more complex targets. It also contributes a new training dataset of nested image-text pairs derived from Visual Genome, called CompoVL.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CompoVL-hard's 8.7-point gain may rest on a dependency-parser head-noun heuristic: if the parser picks the wrong referent or the expression denotes a set, the label is wrong, and PromViL may be rewarded for learning parser patterns rather than compositional grounding.","rationale":"The reader's weakest_assumption identifies the same point I would attack, so I agree. This is load-bearing because the headline 8.7-point gain is measured on CompoVL-hard, a benchmark constructed by the same dependency-parser heuristic used to build training labels. A model trained on those labels could learn a parser-correlated shortcut (for example, 'pick the nsubj's box'), inflating both the training signal and the evaluation. This does not by itself invalidate the paper: the human evaluation suggests most boxes are correct, and zero-shot gains on RefCOCO*, GQA, and Visual7W use externally defined targets and support real generalization. Nor is the claim circular in the sense of fitting those external benchmarks. But since the method's signature result is on CompoVL-hard, the parser-label assumption must be tested before the 9-point claim can be accepted. I did not find a fatal flaw, and a CONDITIONAL verdict with this test as a condition is appropriate; hence verdict_should_be is UNCHANGED relative to the reader's CONDITIONAL. A secondary caveat is that the inference-time ablations do not isolate progressive decoding, but that is less decisive than the benchmark-label concern.","tokens_in":12521,"tokens_out":6063,"duration_ms":64621,"concrete_test":"Stratify CompoVL-hard by complexity level and sample about 500 expressions. For each, show annotators the two VG predicates used to generate the expression and ask them to select the intended visual referent; compare with the dependency-parser head that assigned the training/evaluation bounding box. Then recompute PromViL and MiniGPTv2 accuracy on the subset where the parser head matches the human referent, and separately on the mismatch subset. If the parser-mismatch rate is high (more than 10%) and increases with level, or if PromViL's advantage shrinks materially (more than 3 points) on the matched subset, the CompoVL-hard result is substantially label-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is in 'Multi-granular Compositional V-L Dataset': 'The visual bounding boxes are then chosen as bounding boxes of the identified main entities in VG.' Every nested training pair and every CompoVL-hard evaluation target inherits this choice. If the dependency parser selects the wrong head noun, or if the generated text refers to a set of objects rather than a single entity, the expression is paired with an incorrect or ambiguous region. The same parser is reused at inference to prune level-one expressions, so parsing errors propagate into the prompting chain. The paper's human evaluation (92.48% bbox accuracy, 87.61% uniqueness on 2% of CompoVL-hard) mitigates but does not settle this: annotators judged the box against the generated caption, not against the VG predicate's intended subject, and the 12.4% ambiguous cases may be concentrated in higher-complexity items. If parser errors increase with complexity, the widening accuracy gap in Fig. 5 could be a label artifact rather than evidence of progressive reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PromViL, a method that augments grounded LVLMs with multi-granular, nested vision-language supervision and a progressive decoding procedure. It constructs CompoVL from Visual Genome predicates by chaining related triples through an LLM and assigning each generated expression to the VG bounding box of the dependency-parser-selected head noun. Training uses LoRA fine-tuning of Kosmos-2 (or MiniGPTv2) with a next-token loss over expression levels, conditioning each level on the previous level's expression and box tokens. At inference, a constituency/dependency parser recovers nested subsequences and the model is prompted level by level. The authors report gains over base and fine-tuned baselines on CompoVL-hard (about 9 points over Kosmos-2), zero-shot RefCOCO/RefCOCO+/RefCOCOg grounding, and zero-shot GQA/GQA-OOD/Visual7W, plus ablations showing that all nesting levels matter.","tokens_in":12648,"tokens_out":9860,"duration_ms":98868,"significance":"The central idea—hierarchical visual-language alignment plus progressive decoding—is timely and, if the concerns below are addressed, could yield a lightweight recipe (4.9% LoRA parameters, 60K fine-tuning samples, single V100). The paper includes useful controls: F-Kosmos-2 uses the same CompoVL data without nested subsequences; Kosmos-8K/16K control for the amount of VQA data; and the zero-shot RefCOCO*, GQA, and Visual7W benchmarks are not generated by the CompoVL pipeline. The Table 5 ablation clearly shows that intermediate nesting levels contribute beyond merely having the highest-level expression. These strengths make the paper potentially publishable, but the load-bearing data-labeling heuristic and several missing controls require additional work before the claims are fully established.","major_comments":[{"comment":"The sentence \"The visual bounding boxes are then chosen as bounding boxes of the identified main entities in VG\" is load-bearing: every CompoVL training target and every CompoVL-hard evaluation label inherits the dependency parser's head-noun choice. The manuscript does not validate that this choice agrees with the intended subject of the VG predicate chain, and the 2% human evaluation asks annotators whether the box is correct for the generated caption, not whether the parser selected the VG predicate's subject. It also reports only aggregate naturalness/uniqueness/bbox rates, with no breakdown by complexity level. Because the same parser is reused at inference to prune level-one expressions, parser errors can transfer into the prompting chain. Since CompoVL-hard is produced by the same pipeline that creates the training labels, the 8.7-point CompoVL-hard gain is not by itself evidence of compositional grounding; the external zero-shot results carry that claim. I ask for a direct parser validation (head-noun agreement with VG subject annotations, bbox agreement per complexity level, and ambiguity rates on higher-complexity items) and for an analysis of whether the widening accuracy gap in Fig. 5 survives after conditioning on label quality.","section":"Methods — Multi-granular Compositional V-L Dataset / Annotation's reliability"},{"comment":"The supervised RefCOCO/RefCOCO+/RefCOCOg results are potentially inflated by image overlap between Visual Genome and MS COCO. RefCOCO* test images come from COCO, and CompoVL is derived from VG, which shares many images with COCO; the fine-tuned PromViL‡ has therefore plausibly seen the test images during CompoVL training. Even if the Pink and MiniGPTv2 baselines were also trained on VG region-caption data, the comparison does not separate generalization from memorization. Please quantify the overlap between the CompoVL training images and the RefCOCO* test splits, and report the supervised comparison on the non-overlapping subset (or otherwise control for overlap).","section":"Experiments — Referring expression tasks (Table 2)"},{"comment":"The experimental design does not isolate the inference-time progressive mechanism from the benefit of being trained on nested data. Table 5 varies the training data (all levels, intermediate removed, highest only, simplest only), but there is no condition that trains on all levels and then decodes flat, or trains without nesting and then decodes progressively. Moreover, Eq. (1) teacher-forces the previous level's ground-truth box at training time, while Algorithm 1 feeds generated boxes at inference; no analysis reports the accuracy of the intermediate generated boxes or how errors propagate. Please add the missing cross-condition (e.g., full training with single-prompt decoding, or a parser-based decomposition ablated) and report intermediate-box accuracy. It should also be stated explicitly whether F-Kosmos-2 in Table 1 uses flat or progressive decoding, since this determines what that comparison controls for.","section":"Methods — Training and Inference (Eq. 1, Algorithm 1); Model Analysis — Table 5"}],"minor_comments":[{"comment":"Several references to 'Sec.' and 'Appendix' are unresolved; for example, the text says 'See Sec. ' after presenting CompoVL-hard, and the prompts and evaluation interface are said to be in the Appendix but do not appear in this version. Please include the appendix or point to the repository.","section":"General"},{"comment":"There are typographical errors: 'dependancies' in the Inference paragraph, 'thile@@uark.edu' in the author list, 'spatial tokensyi' in the Training paragraph, and 'To quantify the the relations' in Model Analysis and Ablation Studies.","section":"General"},{"comment":"Algorithm 1 line 8 contains 'yc ← yi', which appears to be a typo for returning the final generated sequence; please correct.","section":"Methods — Algorithm 1"},{"comment":"The footnote '(*) are supervised on region-caption data from VG' is ambiguous: it is unclear which rows the asterisk applies to and whether PromViL‡ was additionally trained on the RefCOCO* training splits. Please state the exact training data for each supervised row.","section":"Experiments — Table 2"},{"comment":"The abstract's 'nearly 5 point and 10 point increases in accuracy and validity' should be calibrated to Table 3: the accuracy gap is 45.07 versus 40.55 (4.52 points) and the validity gap is 83.47 versus 74.82 (8.65 points).","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The zero-shot results on RefCOCO*, GQA, and Visual7W are the strongest evidence for the paper's central claim and should be foregrounded in the revision. The CompoVL-hard benchmark should be repositioned as a diagnostic rather than an independent evaluation unless the parser-label validation requested above is provided. Please also ensure that the dataset release includes the exact LLM prompts, parser versions, and the overlap analysis for COCO/VG."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper delivers a genuinely new mechanism—nested multi-granular V-L pairs from Visual Genome, plus a progressive decoding scheme that feeds lower-level grounding answers into higher-level prompts. The zero-shot results on RefCOCO*, GQA, and Visual7W are the strongest evidence; they're external benchmarks not built from the generation pipeline, so the central claim isn't circular in the obvious way. The paper is honest about its setup and the code/data release is a plus.\n\nWhat's new: the CompoVL dataset (60K instances, nested expressions up to level 3+) and the progressive prompting algorithm. The ablations in Table 5 show that removing intermediate levels hurts, which supports the multi-granular training idea. The gains over Kosmos-2 and MiniGPTv2 baselines are consistent across tasks.\n\nWhere it's soft: First, the stress-test about the dependency parser is real. Every bounding box for a complex expression is inherited from the main entity identified by the parser over LLM-generated text. If the parser picks the wrong head noun, the label is wrong. The human evaluation (92.48% bbox accuracy on 2% of CompoVL-hard) is reassuring but doesn't compare against the VG predicate's intended subject; it judges the box against the generated caption, so it can't detect systematic parser errors. This is a genuine weakness, though not necessarily fatal—the external zero-shot results suggest the training signal is still useful.\n\nSecond, the ablations change training data, not inference decoding. The reader is right: we don't learn whether progressive prompting at inference is what drives the gains, or just the multi-granular training data. A proper ablation would run the same model with and without the progressive chain at inference.\n\nThird, the supervised RefCOCO* comparison may be contaminated by image overlap between Visual Genome and MS COCO. The paper doesn't address this directly, so those numbers should be re-evaluated on non-overlapping splits. Minor: no error bars, and the claim of \"unbounded complexity\" is overstated; the data only goes to a few levels.\n\nOverall: a worthwhile paper with a load-bearing but not fatal assumption. The parser-labeling issue deserves scrutiny, but the external zero-shot evidence carries the core idea. I'd send it to peer review with a request for the overlap check, an inference-only ablation, and error bars. It's a solid contribution for anyone working on grounded LVLM reasoning; I'd cite the dataset and method if I were in that space.","headline":"PromViL is a solid, useful contribution to grounded LVLM reasoning: new nested dataset, progressive decoding, and credible zero-shot gains; the main soft spots are label noise from the parser heuristic and ablations that don't isolate inference-time prompting.","tokens_in":13250,"tokens_out":2235,"would_cite":true,"duration_ms":22928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a vision-language model on nested multi-granular expression pairs and decoding from simple to complex lets a 1.6B model beat larger grounded LVLMs on compositional reasoning, with gains up to 10 points.","keywords":["compositional visual reasoning","visual grounding","large vision-language models","multi-granular alignment","progressive reasoning","referring expression grounding"," Visual Genome","grounded question answering"],"falsifier":"Build a test set of compositional expressions whose referent is not a single head noun (coordinate noun phrases, plural sets, or expressions where the dependency parser's main entity is not the denoted region) and compare PromViL's grounding accuracy against human judgment. A sharp drop would show that the parser's head-noun box is the load-bearing crutch rather than progressive alignment.","tokens_in":12250,"feed_emoji":"🖼️","tokens_out":9512,"duration_ms":86025,"temperature":0.7,"pith_summary":"The paper claims that large vision-language models can be taught grounded compositional reasoning by training them on nested multi-granular vision-language pairs and then prompting them to decode from simple to complex expressions. It introduces PromViL, a framework consisting of the CompoVL dataset (nested expression-box pairs built from Visual Genome) and a progressive decoding scheme where each level's grounded output becomes the cue for the next. The reported results show a 1.6B model fine-tuned on 60K samples with only 4.9% trainable parameters outperforming larger grounded LVLMs: about 9 points on the new CompoVL-hard benchmark, up to 5.5 points on zero-shot referring expression grounding, and roughly 5 and 10 points on accuracy and validity for zero-shot compositional VQA. If these results hold, the path to compositional reasoning in multimodal models may lie in explicit hierarchical grounding rather than ever-larger scale.","feed_headline":"Nested multi-granular prompts boost grounded VQA by ~10 points","feed_subtitle":"A 1.6B model fine-tuned on 60K nested pairs beats larger zero-shot grounding baselines.","key_machinery":"The load-bearing mechanism is a chain of nested multi-granular alignments. CompoVL is built by chaining Visual Genome's ⟨subject, relation, object⟩ predicates that share entities, using an LLM to render the combined predicates as a complex expression and a dependency parser to assign the bounding box of the main head noun to the whole expression; the result is a list of nested expressions at multiple complexity levels. At inference, a constituency parser extracts nested noun-phrase subsequences of the input expression, a dependency parser prunes non-referential level-one items, and the model is prompted level by level with the previous grounded response as context, using next-token prediction trained across all levels. The core objects are the nested expression lists and the progressive prompt chain, which carry the argument: ablation shows that removing intermediate levels or using only one granularity degrades accuracy.","core_discovery":"The central discovery is that aligning language and vision at multiple nested granularities, and using simpler grounded expressions as stepping stones for more complex ones, lets a grounded LVLM reason compositionally where holistic or object-only alignments fail. The authors claim PromViL, fine-tuned on the CompoVL dataset with 4.9% tunable parameters and 60K samples, raises its base model Kosmos-2's accuracy on CompoVL-hard from 59.12 (fine-tuned on the same data without nesting) to 64.07, and on zero-shot RefCOCOg from 60.57 to 64.44, while also improving GQA accuracy and validity. The paper attributes these gains to the chain of progressive multi-granular alignments, supported by ablations that remove intermediate levels.","pith_inferences":["The same nested-data recipe could transfer to other grounded LVLMs without the parser's head-noun bias; a direct test would be to train PromViL on CompoVL with human-annotated expression-level boxes and compare the gain, isolating how much of the improvement is genuine compositional learning rather than learning the parser's heuristic.","Because the inference chain depends on constituency and dependency parses of the query, the method's accuracy on out-of-distribution syntax (coordinate subjects, non-canonical word order, or set-denoting phrases) would probably drop; evaluating on such a probe set would reveal the parse-sensitivity of progressive decoding.","The reported gains are measured on benchmarks that share Visual Genome's predicate vocabulary and box style, so a testable extension is to measure zero-shot grounding on datasets with different annotation conventions, such as one box per entity versus one box per entire phrase, to see whether the hierarchy transfers."],"forward_implications":["With the same 1.6B backbone, PromViL outperforms Kosmos-2 by about 4 points on RefCOCOg and about 5.5 points on RefCOCO in zero-shot grounding, and it beats larger zero-shot models like CoVLM 2.8B and Pink 7B.","The roughly 9-point gain on CompoVL-hard and the about 4-point gain on GQA over Kosmos-2 fine-tuned on the same data without nested structure indicate that the nested multi-granular structure, not extra VQA data, drives the improvement.","Ablations show every complexity level is needed: removing intermediate levels drops CompoVL-hard accuracy from 64.07 to 61.22, and keeping only the simplest or only the highest level yields 53.26 or 59.12 respectively, so neither fine-grained nor coarse-grained alignment alone is sufficient.","The performance gap over baselines widens with expression complexity and scene object count, meaning the method specifically targets harder compositional cases rather than providing a uniform boost.","CompoVL-hard serves as a diagnostic benchmark where current grounded LVLMs drop from roughly 84% to 62% accuracy, exposing the compositional grounding failure the paper addresses."],"supporting_citations":[{"why":"Supplies the Visual Genome region annotations and ⟨subject, relation, object⟩ predicates from which CompoVL's nested pairs are built.","marker":"(Krishna et al. 2017)"},{"why":"Is the Kosmos-2 base model that PromViL fine-tunes, providing the grounded ⟨p⟩ ⟨b⟩ token format and the zero-shot baseline.","marker":"(Peng et al. 2024)"},{"why":"Is the Mixtral8x7B LLM that generates complex expression text from pairs of Visual Genome predicates.","marker":"(Jiang et al. 2024)"},{"why":"Is the dependency-parsing algorithm used to assign the main-entity bounding box to each generated expression and to prune non-referential level-one expressions at inference.","marker":"(Nivre 2008)"},{"why":"Is the constituency parser used at inference to decompose an input expression into nested subsequences.","marker":"(Kitaev, Cao, and Klein 2019)"},{"why":"Is the LoRA method that keeps trainable parameters at 4.9% of the model.","marker":"(Hu et al. 2021)"},{"why":"Provides the LLaVA-Instruct150K instruction-following data included in CompoVL and is an example of a non-grounded LVLM baseline.","marker":"(Liu et al. 2024)"},{"why":"Is the GQA dataset on which zero-shot compositional VQA accuracy and validity are measured.","marker":"(Hudson and Manning 2019)"},{"why":"Is the RefCOCOg benchmark for zero-shot and supervised referring expression grounding.","marker":"(Mao et al. 2016)"}],"fun_headline_variants":["PromViL: Progressive multi-granular alignments for grounded reasoning","Nested alignments lift compositional visual reasoning in LVLMs","Progressive alignment chain lifts grounded VQA with few samples","Hierarchical vision-language alignment improves compositional reasoning","1.6B model with nested prompts beats zero-shot grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset and the benchmark both assume that a dependency parser's chosen main noun is what a complex expression refers to, so any expression about a set, or any parser mistake, is paired with the wrong or ambiguous region.","fun_headline_variants_meta":{"raw":{"variants":["PromViL: Progressive multi-granular alignments for grounded reasoning","Nested alignments lift compositional visual reasoning in LVLMs","Progressive alignment chain lifts grounded VQA with few samples","Hierarchical vision-language alignment improves compositional reasoning","1.6B model with nested prompts beats zero-shot grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2262,"prompt_tokens":876,"completion_tokens":1386,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":1303}},"tokens_in":492,"tokens_out":1386,"duration_ms":12958,"temperature":1.0,"reasoning_tokens":1303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:11:08.029479+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set of compositional expressions whose referent is not a single head noun (coordinate noun phrases, plural sets, or expressions where the dependency parser's main entity is not the denoted region) and compare PromViL's grounding accuracy against human judgment. A sharp drop would show that the parser's head-noun box is the load-bearing crutch rather than progressive alignment.","supporting_citations":[{"cited_title":"A.; et al","cited_arxiv_id":null,"evidence_quote":"Supplies the Visual Genome region annotations and ⟨subject, relation, object⟩ predicates from which CompoVL's nested pairs are built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the Kosmos-2 base model that PromViL fine-tunes, providing the grounded ⟨p⟩ ⟨b⟩ token format and the zero-shot baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the dependency-parsing algorithm used to assign the main-entity bounding box to each generated expression and to prune non-referential level-one expressions at inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the constituency parser used at inference to decompose an input expression into nested subsequences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-Instruct150K instruction-following data included in CompoVL and is an example of a non-grounded LVLM baseline."},{"cited_title":"A.; and Manning, C","cited_arxiv_id":null,"evidence_quote":"Is the GQA dataset on which zero-shot compositional VQA accuracy and validity are measured."},{"cited_title":"L.; and Murphy, K","cited_arxiv_id":null,"evidence_quote":"Is the RefCOCOg benchmark for zero-shot and supervised referring expression grounding."}],"review_version":1}