{"id":"9b0b0c9b-e30d-42f7-97dd-eae77c69bddc","arxiv_id":"2608.07886","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Grounding can be treated as recovering all word-span to image-mask correspondences in one pass, and a bridge-token model, ConCor-1, does this better than existing grounding pipelines on the tested benchmarks.","lead":"This paper redefines vision-language grounding as a two-way matching problem: a model must find both which words in a caption refer to visible things and which pixels in the image those words point to. The authors' ConCor-1 model uses special bridge tokens to propose and score these word-pixel pairs, and it beats existing grounding pipelines on dense captions and large category lists.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Caption-benchmark ground truth is generated by the same LLM rewrite pipeline used to construct training labels, so the headline caption-grounding gains may partly reflect imitation of the annotation pipeline; only the category-list results provide a partially independent check.","rationale":"The reader's weakest assumption identifies the shared LLM rewrite pipeline as the main source of correctness risk, and my reading agrees. The strongest empirical claim is the measured JointF1 improvement on COCONut-PanCap (59.9 to 88.8) and, to a lesser degree, on zero-shot LVIS (23.1 to 29.9). The LVIS result uses standard instance masks and a category list, so it is a partially independent check and prevents the entire paper from collapsing on the pipeline concern. However, the caption benchmarks are the setting that motivates the 'bidirectional concept correspondence' formulation, and there the training and evaluation labels are produced by the same pipeline. The paper does not release code or data, does not report error bars, and does not evaluate on an independently annotated caption-grounding benchmark, so the magnitude of the claimed caption gains is not yet verifiable. Appendix F shows the architectural advantage is not purely a standardization artifact: with original annotations, ConCor-1 beats MM-GDINO-FT by 3.8 and 11.6 points on image-caption and image-category JointF1, respectively. This is real evidence in the paper's favor, but it does not quantify the Qwen3.5-FT comparison on the headline caption benchmark, nor does it test raw unrewritten captions with independent human labels. A secondary concern is that Qwen3.5-FT, the strongest Table 1 baseline, is forced to output masks as 16-point polygons, which may suppress its JointF1 due to mask fidelity rather than correspondence ability; this reinforces the need for the proposed independent re-annotation test. For these reasons I keep the reader's CONDITIONAL verdict unchanged and recommend that the specific pipeline-bias check above be added to the acceptance conditions.","tokens_in":25485,"tokens_out":6592,"duration_ms":85415,"concrete_test":"Independently re-annotate a random subset of 500 COCONut-PanCap and 500 Flickr30k validation images using only the original raw captions and original ground-truth masks, with two human annotators and no use of the Resolve/Rewrite pipeline; measure inter-annotator agreement, then evaluate ConCor-1, Qwen3.5-FT, and GDINO+SAM against this human ground truth. If ConCor-1's JointF1 margin over Qwen3.5-FT on COCONut-PanCap remains close to the reported 28.9 points, the pipeline-bias concern is refuted; if the margin drops toward the approximately 4-point original-annotation margin seen in Appendix F, the headline 48% improvement is substantially an artifact of shared annotation generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that ConCor-1 recovers complete text–image correspondences far better than existing systems — rests most heavily on the caption benchmarks in Table 1. In Section 3.2 and Appendix D.1, both the training captions (GoldG, COCONut-PanCap, GroundedRef) and the evaluation sets for Flickr30k and COCONut-PanCap are processed by the same two-stage LLM Resolve/Rewrite pipeline. This pipeline decides what is visually referential: it removes mentions without associated masks, rewrites captions so that only objects represented by annotation groups remain, and excludes examples that cannot be rewritten coherently. The evaluation ground truth is therefore not an independent sample of grounding behavior; it is a sample of the pipeline's annotation decisions. A model trained on the same rewrite outputs can learn the pipeline's filtering patterns and rewrite style, inflating measured gains without improving general grounding of raw, unprocessed captions. The paper's human filtering of evaluation labels ensures quality but does not break the pipeline's control over which spans and instances are considered groundable. Appendix F partially addresses this by showing ConCor-1 also outperforms MM-GDINO-FT under original source annotations, but that comparison does not cover the headline COCONut-PanCap vs. Qwen3.5-FT result and does not quantify how much of the reported 28.9-point JointF1 margin survives on independently annotated raw captions. Without code, data, or an independently curated caption-grounding benchmark, the measured gains on the caption setting cannot be cleanly attributed to the proposed bidirectional correspondence mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes reformulating vision-language grounding as 'bidirectional concept correspondence': given an image and a text input (caption, referring expression, or category list), the model must recover all pairs consisting of a binary mask over text spans and a binary instance-level image mask, without assuming the text spans are provided. The authors introduce ConCor-1, built on a pretrained Qwen3.5-0.8B VLM, which appends 385 learnable bridge tokens to the multimodal sequence; each bridge token predicts a text mask, an image mask, and a presence score, with a multi-scale spatial assignment used for training. A unified training set is constructed by converting several existing grounding, detection, and segmentation datasets, including a novel LLM-based Resolve/Rewrite pipeline for caption data. Experiments report large improvements over baselines on caption grounding (e.g., COCONut-PanCap JointF1 88.8 vs 59.9 for Qwen3.5-FT) and on category-list grounding (e.g., LVIS-minival JointF1 29.9 vs 23.1 for GLaMM), together with ablations and layer-wise attention analyses.","tokens_in":25739,"tokens_out":9978,"duration_ms":106232,"significance":"If the reported results are robust, the paper would make a useful contribution: it unifies several grounding tasks under a single output space and introduces an explicit, non-autoregressive correspondence prediction architecture that goes beyond phrase-given localization. The paper is notably transparent in its appendices: it provides detailed dataset statistics, a full baseline description, a component-level computational breakdown, and a candid limitations section. The category-list results on standard benchmarks (COCO, LVIS-minival, EntitySeg) are an independent validation path that does not rely on the LLM rewrite pipeline, and the multi-scale bridge-token assignment plus the analysis of gradual grounding emergence across VLM layers are concrete and reproducible-looking contributions. However, the headline caption-grounding numbers rest on an evaluation protocol whose ground truth is generated by the same LLM pipeline used to produce training labels, and the paper's own standardization analysis (Appendix F) shows the model-architecture advantage under original source annotations is far smaller than the headline margins.","major_comments":[{"comment":"The evaluation ground truth for the three caption benchmarks (Flickr30k, COCONut-PanCap, and GroundedRef) is produced by the same two-stage LLM Resolve/Rewrite pipeline that generates the training labels (Section 3.2; Appendix D.1). Because this pipeline decides which mentions are visually referential, merges coreferential mentions, removes mentions without associated masks, and excludes captions that it cannot rewrite coherently, a model trained on the same outputs can learn the pipeline's annotation style rather than general grounding ability. Appendix F (Table 11) mitigates this concern only partially: it shows a 3.8-point JointF1 gain over MM-GDINO-FT under original source annotations, but this is an average over all three caption benchmarks and does not report the COCONut-PanCap-specific comparison against Qwen3.5-FT, which is the source of the headline 28.9-point margin. The paper should report per-dataset results under original source annotations, and additionally evaluate on raw, unrewritten human captions (e.g., the original Flickr30k Entities annotations) so that the reader can see how much of the headline margin survives without the shared pipeline.","section":"§3.2, Appendix D.1"},{"comment":"The paper's own standardization analysis shows that switching from original source annotations to the standardized annotations improves image-caption JointF1 by 9.4 points for MM-GDINO-FT and 9.8 points for ConCor-1, whereas the model architecture contributes only about 4 points under either annotation setting. This suggests that a substantial part of the large margins in Table 1 may be attributable to the standardized annotations or to the evaluation protocol being more aligned with ConCor-1's token-level output format, rather than to the bidirectional-correspondence formulation itself. The paper should provide a per-dataset, per-baseline breakdown under both annotation settings, and clarify explicitly whether the Table 1 numbers are computed on the rewritten evaluation captions, so that the reader can assess the contribution of each factor.","section":"Appendix F, Table 11"},{"comment":"The loss weights (λtext, λbce, λdice, λpres) are selected by a sweep whose results are averaged over the evaluation benchmarks (Appendix B.4, Table 5), and the inference thresholds (presence 0.1, text 0.45, visual 0.45) and NMS IoU threshold (0.5) are reported as fixed choices without a held-out validation set. Since the main results are reported on the same validation sets, this constitutes tuning the reported numbers on the evaluation data. The paper should hold out a validation split for threshold and hyperparameter selection, or state which values were chosen before observing the final evaluation results, to allow the reported numbers to be interpreted as unbiased estimates.","section":"§B.4, §B.7, Table 5"},{"comment":"The model predicts binary masks over text tokens (Appendix B.3), but the evaluation metrics TextF1 and JointF1 are computed on character-level span IoU (Appendix C.2). The conversion from token-level predictions to character spans is not described. This is not a cosmetic issue: the resulting span boundaries determine whether a predicted span meets the 0.5 IoU threshold, especially for long, dense captions with many tokens per mention. The paper should specify how token logits are thresholded and mapped to character offsets (e.g., whether spans are expanded to token boundaries, and how non-contiguous predicted tokens are grouped into spans).","section":"§B.3, §C.2"},{"comment":"All reported numbers come from single runs without error bars or significance tests. For large margins on COCONut-PanCap this may not change the qualitative conclusion, but several important comparisons are small: the LVIS-minival JointF1 margin of 29.9 vs 23.1 in Table 2, the Appendix F margin of 3.8 points under original annotations, and the ablation differences in Table 3 (e.g., 51.3 vs 49.3 for bidirectional vs causal attention). For these, the absence of variance estimates makes it impossible to distinguish a real effect from run-to-run noise. Please report at least three seeds for the main configurations and for the ablations.","section":"§4, Tables 1–3"}],"minor_comments":[{"comment":"The abstract states 'improving correspondence F1 by 48%' and 'by 29%'; these are relative improvements (48.2% and 29.4% respectively). Please state explicitly that the percentages are relative gains to avoid confusion with absolute F1 improvements.","section":"Abstract, §4.1"},{"comment":"In Table 4, Attn-IoU uses the ground-truth mask area to determine the number of selected visual tokens. This is an oracle-informed procedure that makes the absolute values hard to interpret; while the comparison across models is fair under the same protocol, please state this limitation in the text.","section":"Table 4, §4.3"},{"comment":"The term 'zero-shot' for LVIS is not precisely defined; the training mixture includes COCO and other datasets whose categories overlap with LVIS. Please clarify what is zero-shot (e.g., no LVIS training images or no LVIS category names in the training text) in the main text.","section":"Table 2, §E"},{"comment":"The two-stage LLM rewrite pipeline is central to the data generation but does not specify which LLM is used (e.g., GPT-4o-mini, GPT-5.4) or the exact prompts. Without these details the pipeline is not reproducible; please provide the prompts or a pseudocode description in the appendix.","section":"Appendix D.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical core is currently hard to judge because the caption benchmarks share an annotation pipeline with the training data, and the only independent evidence (category-list results) shows a much smaller margin. The authors seem aware of the issue and include a candid limitations section, which is good. The main revisions needed are a per-dataset original-annotation evaluation, a discussion of the annotation-pipeline circularity, and error bars/hyperparameter discipline. If code and data are released, the detailed appendix will make this a strong candidate; without them, the empirical claims are difficult to verify."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper deserves a serious read. The core move is to redefine grounding as recovering all correspondences between visually referential text spans and instance-level image segments, without prespecified phrases. That is a genuinely new task-level framing, and it cleanly unifies phrase grounding, referring expression grounding, and open-vocabulary detection. The bridge-token architecture is a concrete, sensible way to instantiate it: each learnable token predicts a text mask, an image mask, and a presence score, with multi-scale spatial assignment during training. The appendix is unusually complete — full data mixture, loss-weight sweep, threshold choices, ablation details, and an honest limitations section. Credit where due: the category-list results (COCO, LVIS, EntitySeg) use standard ground-truth masks and category lists, and they show solid gains, e.g., LVIS JointF1 from 23.1 to 29.9. Those numbers are partially independent of the pipeline concern.\n\nThe soft spot is real: the caption evaluation ground truth is produced by the same two-stage LLM Resolve/Rewrite pipeline used to build training data. That pipeline decides what counts as visually referential, removes ungroundable mentions, and filters out unrewritable examples. A model trained on those outputs can learn the pipeline's annotation habits rather than general grounding. The appendix's own Appendix F partially answers this — with original source annotations, ConCor-1 still beats MM-GDINO-FT by 3.8 points on image-caption JointF1 — but that check does not cover the headline COCONut-PanCap margin over Qwen3.5-FT (88.8 vs 59.9), so the size of that gain is unexplained. Missing code and data, no error bars, and loss weights/thresholds tuned on the evaluation benchmarks add to the uncertainty. These are not fatal; the architecture clearly does something. But the caption-claim is not yet cleanly attributable to the correspondence mechanism.\n\nWho gets value: anyone working on VLM grounding, dense caption evaluation, or open-vocabulary detection, mainly for the formulation and the thorough appendix. It deserves a serious referee, with the request that the authors release code and data, add repeated runs or error bars, and evaluate on an independently curated caption-grounding benchmark. I'd cite the formulation in my own work and bring it to reading group.","headline":"A genuinely new task framing and a sensible bridge-token model, but the headline caption-grounding numbers rest on a self-consistent LLM-rewrite pipeline and need independent validation before I'd trust the 28-point margins.","tokens_in":26340,"tokens_out":4049,"would_cite":true,"duration_ms":40225,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grounding is a two-way problem: find which words point to things and where those things are.","keywords":["vision-language grounding","bidirectional concept correspondence","bridge tokens","text segmentation","instance segmentation","coreference resolution","open-vocabulary detection","grounded captions"],"falsifier":"Evaluate ConCor-1 and the baselines on a held-out set of human-annotated image-caption correspondences that were not produced by the rewrite pipeline, including captions the pipeline marked unrewritable; if the JointF1 gap shrinks to near zero on those cases, the bidirectional-correspondence claim is not a general grounding ability.","tokens_in":25232,"feed_emoji":"🔗","tokens_out":8234,"duration_ms":81012,"temperature":0.7,"pith_summary":"This paper claims that the standard way of evaluating vision-language grounding—hand the model a phrase and ask where it is in the image—skips the harder part of reference: deciding which parts of the text point to anything at all. It reformulates grounding as bidirectional concept correspondence: from an image and its paired text, recover every pair of a text mask and an instance mask that refer to the same entity, without assuming the text spans are given. The paper introduces ConCor-1, which appends learnable bridge tokens to a pretrained vision-language model and predicts a text mask, an image mask, and a validity score for each token. On dense long captions, ConCor-1 raises correspondence F1 from 59.9 to 88.8 over the strongest tuned baseline, and on zero-shot LVIS category lists from 23.1 to 29.9. If true, grounding becomes a single structured prediction problem rather than a collection of separate localization tasks.","feed_headline":"One model pairs every caption phrase with its instance mask","feed_subtitle":"Bidirectional grounding beats tuned baselines on dense captions and zero-shot categories without preselected phrases.","key_machinery":"Bridge tokens are learnable query embeddings appended after the image and text tokens in a pretrained vision-language backbone; with 385 tokens arranged in multi-scale spatial grids from $1\\times1$ to $10\\times10$, each token is a candidate correspondence hypothesis with an implicit spatial prior. During training, Hungarian matching assigns ground-truth correspondences to bridge tokens by mask overlap with grid cells. After self-attention over both modalities, each bridge token is decoded by three heads: a bilinear text segmentation head that scores every text token, a bilinear image segmentation head over reconstructed fine visual features, and an MLP presence head. The shared bridge token is the mechanism that forces text span, image segment, and alignment to be predicted jointly rather than by separate unidirectional retrievals.","core_discovery":"The paper's central claim is that grounding is not a one-way localization problem but a bidirectional correspondence problem over an image-text pair. The output is a set of pairs, each pairing a binary mask over text characters (which may cover several non-contiguous mentions of one entity) with an instance-level pixel mask; the model must also decide how many correspondences exist and reject mentions or objects that have no partner. ConCor-1 instantiates this with learnable bridge tokens as candidate correspondences: each token attends to the whole multimodal sequence, then three prediction heads produce a text mask, an image mask, and a presence score. Trained on a unified conversion of grounding, detection, and segmentation datasets, the model reports large gains on the long, densely referenced COCONut-PanCap captions and on LVIS category lists processed in a single forward pass.","pith_inferences":["The paper's own rewrite pipeline quietly defines the task, so a natural next test is to measure ConCor-1 on original, unrewritten annotations; that would separate grounding skill from pipeline mimicry.","Bridge tokens are a generic binding mechanism: the same slot-based correspondence interface could extend to video by requiring masks to persist over time, which the paper lists as future work but does not develop.","Because every correspondence carries a presence score, the model could also be used to flag ungrounded text in generated captions—low-presence mentions are candidates for hallucination—an application the paper mentions only as future work."],"forward_implications":["A single model of this kind can serve phrase grounding, referring expression grounding, and open-vocabulary detection and segmentation, because the paper treats each as a special case of the same text-mask/image-mask correspondence output.","Systems that need spatially attributable outputs—visual search, caption evaluation, assistive perception—can ask for the full set of correspondences instead of supplying a preselected phrase.","Large category vocabularies can be processed in one forward pass as the text input, avoiding the chunking that detector-based pipelines require.","The largest measured gains are on long, dense captions, which suggests that explicit bridge-token correspondence prediction matters most where many-to-many text-image alignment is hardest.","The bridge-token objective also sharpens the underlying vision-language model's text-to-image attention, so the trained model is more spatially attributable even before its prediction heads are read."],"supporting_citations":[{"why":"Supplies the pretrained vision-language backbone whose joint image-text representations carry ConCor-1.","marker":"[24]"},{"why":"Supplies COCONut-PanCap, the dense long-caption benchmark where the largest measured gains appear.","marker":"[50]"},{"why":"Provides Grounding DINO as a central baseline and contributes the GoldG training corpus with coreference groups.","marker":"[11]"},{"why":"SAM3 generates pseudo masks for box-level training data and converts baseline detections to masks.","marker":"[12]"},{"why":"Flickr30k Entities is a source of human-written caption-grounding evaluation data.","marker":"[9]"},{"why":"RefCOCOg provides referring expressions used to build the GroundedRef evaluation set.","marker":"[5]"},{"why":"GQA captions are a component of the GoldG grounding corpus used in training.","marker":"[51]"}],"fun_headline_variants":["No phrase presets: ConCor-1 pairs each caption span to a mask","Bidirectional grounding: one model, all image-text pairs","ConCor-1 recovers every caption-mask correspondence without presets","One pass: full caption text spans matched to instance masks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the LLM rewrite pipeline that regenerates captions and assigns ground-truth text masks encodes the right notion of what is visually referential; if that pipeline systematically filters hard cases or imposes its own annotation style, the reported gains may measure how well the model imitates the pipeline rather than how well it grounds language.","fun_headline_variants_meta":{"raw":{"variants":["No phrase presets: ConCor-1 pairs each caption span to a mask","Bidirectional grounding: one model, all image-text pairs","ConCor-1 recovers every caption-mask correspondence without presets","One pass: full caption text spans matched to instance masks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001154,"raw_usage":{"total_tokens":4806,"prompt_tokens":995,"completion_tokens":3811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":3736}},"tokens_in":611,"tokens_out":3811,"duration_ms":27653,"temperature":1.0,"reasoning_tokens":3736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:43:23.508057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate ConCor-1 and the baselines on a held-out set of human-annotated image-caption correspondences that were not produced by the rewrite pipeline, including captions the pipeline marked unrewritable; if the JointF1 gap shrinks to near zero on those cases, the bidirectional-correspondence claim is not a general grounding ability.","supporting_citations":[],"review_version":1}