{"id":"2feca031-8e96-4f53-aecc-19fc108d43bf","arxiv_id":"2509.04833","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PropVG is an end-to-end visual grounding model that generates foreground object proposals with a DETR-style decoder and scores them with sentence- and word-level contrastive learning, achieving state-of-the-art on ten referring expression datasets.","lead":"This paper introduces a new AI model, PropVG, that finds and outlines objects in images from text descriptions. It combines a detection step that proposes candidate objects with a scoring step that picks the object the sentence mentions, and it reports top results on ten standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposal recall is the unexamined ceiling: PropVG trains its proposal branch only on filtered foreground objects and gates both refer scoring and target existence on those proposals, yet never reports recall or size-stratified results.","rationale":"The reader identifies the proposal-stage recall ceiling as the weakest assumption; I agree and regard it as the most load-bearing issue. Eq. 5 makes the dependence explicit: the existence score is the product of a proposal-driven term, a segmentation term, and a learned scalar, so a missed proposal zeroes the existence judgment regardless of segmentation quality. The Appendix C.1 filter excludes from supervision exactly the objects that are hardest for a fixed-query detector to propose, and no experiment measures whether the model generalizes to them. This does not falsify the reported aggregate results, but it makes the broad claim of SOTA effectiveness contingent on an unreported diagnostic. The UNINEXT overlap noted by the reader is a secondary novelty concern; it weakens the 'first' phrasing but does not bear on the empirical mechanism as directly as the recall ceiling. I would keep the CONDITIONAL verdict and ask for the recall/size-stratified analysis before final acceptance.","tokens_in":22791,"tokens_out":10137,"duration_ms":88775,"concrete_test":"Run the released PropVG model on RefCOCO val and gRefCOCO val; for each ground-truth referent, check whether any predicted proposal before thresholding has IoU > 0.5 with the referent box. Stratify recall by relative area (<0.05, 0.05-0.15, >0.15), absolute area (<100 px, >=100 px), and occlusion/crowd flags, then compare with per-bin final accuracy. If recall is uniformly high, the filter is not limiting and the concern is resolved; if recall collapses on small/occluded referents, the headline results need a size/visibility qualifier and the paper should report proposal recall as a diagnostic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PropVG's proposal stage is the bottleneck for the entire pipeline, not just a component. CRS scores only the N decoder queries, and the MTD existence score is a product (Eq. 5): S_exist = Max(S_ref) * TAS(M_seg) * eps_exist. If the referred object is not among the proposals, Max(S_ref) is near zero, so the model cannot recover the target even when the segmentation branch produces a correct mask. The proposal branch is trained solely on COCO foreground instances that pass the Appendix C.1 filter: no crowd, absolute area > 100 px, relative area in [0.05, 0.8]. Small, occluded, or atypical referents are therefore never part of proposal supervision. No proposal recall, no per-size accuracy, and no failure analysis are reported on any of the ten datasets. The paper's qualitative claim that text can modulate foreground generation (Fig. 10) is suggestive but does not quantify recall on the filtered-out size regime. Since the same filter choice is justified in Appendix E.4 only by aggregate F1/gIoU, the reported SOTA may be an artifact of training and evaluating on predominantly prominent foreground objects. The fixed query count N is also not reported in the main text or appendix, so proposal capacity is undocumented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PropVG, an end-to-end, detector-free, proposal-based visual grounding framework. The architecture uses a BEiT-3 multimodal encoder with parallel segmentation and detection branches; the detection branch produces N foreground proposals with a multi-scale deformable decoder, and a Contrastive-based Refer Scoring (CRS) module scores each proposal against the expression at sentence and word levels. A Multi-granularity Target Discrimination (MTD) module predicts target existence by combining the maximum refer score, a top-K average segmentation score, and a learned existence score. The model is trained with detection, referring, segmentation, and existence losses and is evaluated on REC/RES, GREC/GRES, Ref-ZOM, and R-RefCOCO/+/g, with the claim of state-of-the-art performance on ten datasets.","tokens_in":23114,"tokens_out":7603,"duration_ms":64046,"significance":"If the results hold, PropVG is a meaningful contribution: it shows that a proposal-based grounding framework can be trained end-to-end without an external detector, provides foreground-level supervision beyond the referred target, and reports a controlled improvement over the same-backbone OneRef method on RefCOCO/+/g. The experimental design is broad (ten benchmarks, module ablations, sensitivity analyses for K and loss weights), and the release of code and models is a practical strength. However, the headline SOTA claims currently rest on two under-supported points: several key comparisons vary the backbone as well as the method, and the proposal-recall ceiling is not examined at all.","major_comments":[{"comment":"The proposal stage is the ceiling for the whole pipeline, and that ceiling is not measured. The proposal branch is supervised only with COCO foreground boxes that pass the filter (no crowd, absolute area > 100 pixels, relative area in [0.05, 0.8]), and the CRS and MTD stages score only the N decoder queries. By Eq. (5), S_exist is the product of Max(S_ref), TAS(M_seg), and eps_exist, so if the referred object is not among the proposals, Max(S_ref) is near zero and the model cannot recover the target even when the segmentation branch is correct. The paper reports no proposal recall, no size-stratified accuracy, and no failure analysis on any dataset, and the query count N is not stated in the main text or appendix. Please report recall of the referred targets among the proposals, especially for the filtered-out size regime, and discuss how any shortfall affects the reported numbers.","section":"Appendix C.1 and Eq. (5)"},{"comment":"Most headline comparisons pit BEiT3-ViT-B against Swin-B or ResNet baselines (e.g., HDC, ReLA, LAVT, MAttNet), so part of the reported margin may be an encoder/pretraining gain rather than a method gain. The controlled comparison with OneRef, which also uses BEiT3-ViT-B, is the right kind of evidence and shows small positive gains on REC, but no same-backbone specialist baseline is provided for gRefCOCO, Ref-ZOM, or R-RefCOCO/+/g. Please add same-backbone variants of the strongest baselines (at least HDC) for these benchmarks, or explicitly separate encoder gains from method gains in the conclusions.","section":"Section 4.2, Tables 2-4 (and Table 1)"},{"comment":"The foreground-object filter is justified only by aggregate F1/gIoU curves, which do not show what happens to precisely the objects the filter removes. Since the filter determines the proposal supervision, and small or occluded referents are the ones that fall outside it, the choice R_low = 0.05, R_high = 0.8, min area = 100 needs a per-size analysis. Please include area-stratified recall and accuracy (small/medium/large, or deciles) to demonstrate that the reported results are not confined to prominent foreground objects.","section":"Appendix E.4, Fig. 8"}],"minor_comments":[{"comment":"The heading reads 'Constrastive-based Refer Scoring' and should be 'Contrastive-based Refer Scoring'.","section":"Section 3.2 heading"},{"comment":"The text refers to 'our DeRIS' when describing the proposed method; this should read 'PropVG'.","section":"Appendix F.1, Fig. 9"},{"comment":"The number of proposal queries N is never reported; please include it in the implementation details.","section":"Appendix D"},{"comment":"The segmentation binarization threshold Thr_m is mentioned but its value is never stated; please report it.","section":"Appendix C.2"},{"comment":"The claim of '4× faster' than MAttNet is only backed by a single wall-clock number in Table 1; please report the hardware and settings used for the timing comparison.","section":"Section 4.2.1"}],"recommendation":"major_revision","confidential_remarks":"The proposal-recall concern is the main correctness risk and should be addressed before publication. The paper's own visualizations (Fig. 10) suggest that text can modulate proposal generation, but this is not quantified, and the existence score in Eq. (5) makes the model unable to recover missed proposals. The controlled same-backbone comparison with OneRef is a strong piece of evidence and should be made more prominent; conversely, the cross-backbone tables should be labeled or supplemented to avoid overclaiming method-level superiority."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine empirical contribution — an end-to-end proposal-driven grounding model that beats direct regression and several MLLM baselines on multiple benchmarks, with ablations that mostly hold up. But the headline novelty claim is too strong, and the proposal-stage recall ceiling is never examined, which matters because the training filter explicitly excludes small and occluded objects.\n\nWhat's actually new: the two-stage design where a DETR-style decoder generates foreground proposals and a contrastive refer-scoring module (CRS) scores them, plus the MTD module for existence prediction. The framework-level idea is structurally similar to UNINEXT's object discovery and retrieval, which the paper cites only in a table and never discusses; so \"first proposal-based framework without detectors\" needs qualification. The controlled comparison against OneRef on the same BEiT-3 backbone is the right way to isolate gains, and PropVG wins consistently there. The ablations are thorough, and the release of code and models helps reproducibility.\n\nSoft spots, in order. First, the proposal bottleneck. The proposal branch is trained only on COCO instances passing the Appendix C.1 filter (no crowd, absolute area >100, relative area 0.05–0.8), and the final refer score is a product with Max(S_ref). If the referent is not among the N proposals, the model cannot recover. The paper reports no proposal recall, no size-stratified accuracy, and no failure analysis. The Appendix E.4 justification for R_low=0.05 on aggregate metrics doesn't tell us what happens on the filtered-out size regime. This is a real gap, and I'd want it addressed before trusting the general claim. Second, several headline comparisons (gRefCOCO, Ref-ZOM, R-RefCOCO) pit BEiT3-ViT-B against Swin-B or ResNet backbones, so part of the margin may come from the backbone rather than the method. Third, no error bars or multiple seeds — common, but still worth asking for. These are not fatal, and the core results are probably sound, but they make the paper's strongest advertising (\"SOTA on ten datasets\") softer than the evidence supports.\n\nWho this is for: people working on referring expression comprehension, generalized grounding, and detector-free proposal architectures. A serious referee should engage with it. The revision should add proposal-recall analysis, size-stratified results, backbone-matched comparisons where feasible, and a fair discussion of UNINEXT. I'd send it to review with that expectation.","headline":"Solid empirical paper on end-to-end proposal-based grounding, but the proposal-recall ceiling is unexamined and the 'first detector-free' claim overlaps with UNINEXT.","tokens_in":23651,"tokens_out":2743,"would_cite":true,"duration_ms":24336,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A proposal-based visual grounding model can be trained end-to-end without any pre-trained detector, and the paper reports that it outperforms direct-referring and detector-dependent baselines across ten benchmarks.","keywords":["visual grounding","referring expression comprehension","referring expression segmentation","generalized visual grounding","end-to-end proposal-based framework","contrastive learning","target existence discrimination","DETR-style detection"],"falsifier":"Run PropVG on a test set constructed so that every referred object has a relative area below 0.05 or an absolute area below 100 pixels. If its localization accuracy stays high, the foreground filter is not a hard ceiling; if accuracy collapses while the same text on larger objects succeeds, the proposal stage's recall is confirmed as the binding constraint.","tokens_in":22612,"feed_emoji":"🎯","tokens_out":6360,"duration_ms":55277,"temperature":0.7,"pith_summary":"PropVG argues that the old two-stage recipe for visual grounding — propose candidate regions, then match them to the text — can be revived if the proposal generator is trained end-to-end inside the model instead of supplied by a pre-trained detector. The paper's central claim is that learning to propose all prominent foreground objects first, then scoring each proposal's referentiality, matches or beats both modern one-stage direct-referring systems and much larger multimodal language models on classic and generalized grounding benchmarks. Two new mechanisms carry the argument: a contrastive refer-scoring module that compares each proposal to the full sentence and to individual words, and a multi-granularity target-discrimination module that fuses proposal and segmentation evidence to judge whether the expression refers to anything at all. This matters because generalized visual grounding — zero or multiple referents, ambiguous or absent targets — needs reliable object-existence judgment in addition to localization.","feed_headline":"Detector-free proposal grounding tops ten visual grounding benchmarks","feed_subtitle":"An end-to-end two-stage model with contrastive refer scoring outruns direct-referring and MLLM baselines on ten datasets.","key_machinery":"The central object is the learnable proposal query. A DETR-style multi-scale deformable decoder turns a fixed set of queries into foreground boxes, with Hungarian matching providing the query-to-object assignment; those same queries are then reused for referential scoring. The Contrastive-based Refer Scoring module computes a similarity between each proposal and the whole sentence and a max-pooled similarity to each word, then mixes the two with a learned weight. The Multi-granularity Target Discrimination module predicts whether any target exists by applying two score-prior cross-attention blocks — one conditioned on the referring scores, one on the segmentation mask — and combining their output with a top-K average of the segmentation map. The whole pipeline is trained jointly with detection, referring, segmentation, and existence losses.","core_discovery":"The central discovery claimed is that foreground proposal generation and referential understanding do not need to be separate systems. A single network can emit candidate boxes for all salient objects, then treat the visual grounding task as binary classification over those candidates, and this division of labour improves accuracy and speed over prior two-stage methods because no external detector bottlenecks the pipeline. The paper further claims that training the proposal stage on all foreground objects, not only the referred target, sharpens the model's later judgments and suppresses false positives. Supported this way, the model is said to outperform existing direct-referring specialists and several multimodal-LLM baselines on ten datasets, including generalized benchmarks where expressions can describe zero or multiple objects.","pith_inferences":["Inference: the dataset construction's filter — absolute area above 100 pixels and relative area between 0.05 and 0.8 — effectively defines the model's competence envelope; a benchmark of very small referents would likely expose the proposal stage's recall as the hard ceiling.","Inference: because the text conditions the proposal branch early in the encoder, the design could transfer to tasks where the text selects which category of objects counts as foreground, such as open-vocabulary or all-instances-of-X grounding.","Inference: the score-prior cross-attention mechanism for existence classification is generic and could be dropped into other detection-based models that need an explicit absent-object prediction."],"forward_implications":["Two-stage, proposal-based grounding can be competitive again when the proposal stage is learned end-to-end, so future work does not have to choose between direct referring and detector-dependent pipelines.","Because the proposal stage is supervised by all salient foreground objects, the model gains a perception channel for non-referred objects, which direct-referring and MLLM baselines do not explicitly train for.","The same architecture handles classic single-target and generalized zero-or-multiple-target settings, suggesting a unified detector-free design can serve both task families.","For deployment, the approach offers a small-parameter model that avoids heavyweight LLM inference, making grounding available in latency-sensitive and resource-constrained settings."],"supporting_citations":[{"why":"The classic two-stage baseline that motivates the paper's performance and speed comparisons.","marker":"[79]"},{"why":"Supplies the vision-language encoder features used by both the segmentation and proposal branches.","marker":"[66]"},{"why":"Supplies the Hungarian matching strategy and detection losses used for query-to-foreground assignment.","marker":"[2]"},{"why":"Provides the multi-scale deformable decoder used to generate the foreground proposals.","marker":"[84]"},{"why":"The direct-referring model using the same encoder that the paper claims to surpass on classic benchmarks.","marker":"[70]"},{"why":"Defines the GRES benchmark and the gIoU, cIoU, and N-acc evaluation used for generalized segmentation.","marker":"[37]"},{"why":"Defines the GREC benchmark and the F1-based evaluation used for generalized expression comprehension.","marker":"[17]"},{"why":"Introduces the R-RefCOCO datasets and the negative-sentence robustness evaluation.","marker":"[67]"},{"why":"Introduces the Ref-ZOM benchmark covering zero, one, and many referents.","marker":"[22]"}],"fun_headline_variants":["First end-to-end proposal-grounding net skips detectors","Contrastive scoring plus multi-granularity beats ten benchmarks","One-shot proposal and referential understanding, no extra detectors","No detector, no problem: proposal grounding outdoes MLLMs","Proposal-grounding net fuses object and semantic cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every referred object is prominent enough to be generated as a foreground proposal; any referent smaller than the filtering cutoff, heavily occluded, or outside the detector-like categories will be missed before the scoring stages can consider it.","fun_headline_variants_meta":{"raw":{"variants":["First end-to-end proposal-grounding net skips detectors","Contrastive scoring plus multi-granularity beats ten benchmarks","One-shot proposal and referential understanding, no extra detectors","No detector, no problem: proposal grounding outdoes MLLMs","Proposal-grounding net fuses object and semantic cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000994,"raw_usage":{"total_tokens":4200,"prompt_tokens":921,"completion_tokens":3279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":3194}},"tokens_in":537,"tokens_out":3279,"duration_ms":20232,"temperature":1.0,"reasoning_tokens":3194,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:25:51.281751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PropVG on a test set constructed so that every referred object has a relative area below 0.05 or an absolute area below 100 pixels. If its localization accuracy stays high, the foreground filter is not a hard ceiling; if accuracy collapses while the same text on larger objects succeeds, the proposal stage's recall is confirmed as the binding constraint.","supporting_citations":[{"cited_title":"Mattnet: Modular at- tention network for referring expression comprehension","cited_arxiv_id":null,"evidence_quote":"The classic two-stage baseline that motivates the paper's performance and speed comparisons."},{"cited_title":"Image as a foreign language: BEiT pretraining for vision and vision-language tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the vision-language encoder features used by both the segmentation and proposal branches."},{"cited_title":"Deformable detr: Deformable transformers for end-to-end object detection.arXiv, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the multi-scale deformable decoder used to generate the foreground proposals."},{"cited_title":"Oneref: Unified one-tower expression grounding and segmentation with mask referring modeling","cited_arxiv_id":null,"evidence_quote":"The direct-referring model using the same encoder that the paper claims to surpass on classic benchmarks."},{"cited_title":"GRES: gen- eralized referring expression segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the GRES benchmark and the gIoU, cIoU, and N-acc evaluation used for generalized segmentation."},{"cited_title":"GREC: Generalized referring expression comprehension","cited_arxiv_id":null,"evidence_quote":"Defines the GREC benchmark and the F1-based evaluation used for generalized expression comprehension."},{"cited_title":"Towards robust referring image seg- mentation.TIP, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces the R-RefCOCO datasets and the negative-sentence robustness evaluation."},{"cited_title":"Beyond one-to-one: Re- thinking the referring image segmentation","cited_arxiv_id":null,"evidence_quote":"Introduces the Ref-ZOM benchmark covering zero, one, and many referents."}],"review_version":2}