{"id":"6cc411c1-d5e5-4efd-8a77-8e27a3f08b4a","arxiv_id":"1909.00421","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"VisPro and VisCoref: a dataset and model showing that adding visual object labels improves pronoun coreference resolution in dialogues, particularly when the antecedent is absent from the text.","lead":"This paper introduces a new dataset, VisPro, for resolving pronouns in image-based dialogues, and a model, VisCoref, that combines text and detected image objects. The model improves over text-only coreference systems, especially when the pronoun refers to something never mentioned in the dialogue.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold 'Not Discussed' antecedents are taken from captions rather than worker visual selections (Sec. 2.4 fn. 3); if caption phrases do not match the intended visual objects, VisCoref's visual branch learns caption-text patterns, not image grounding, so the central claim is conditional.","rationale":"The reader's weakest assumption exactly matches the concern I identify: the gold 'Not Discussed' labels are derived from captions rather than from workers' direct visual selection. This is the most load-bearing point because the dataset's stated purpose is to study visual grounding, and the strongest empirical evidence for that claim comes from the 'Not Discussed' subset, where the antecedents are not in the dialogue. If those labels are not actually grounded in the image, the model's visual branch is being trained and evaluated against caption text, not against the visual referents the speakers intended. The paper's own footnote 3 admits the antecedents come from the caption, and Section 5.1 confirms the mention pool includes caption-derived mentions, making the concern concrete rather than speculative. The reader's conditional verdict is appropriate: the paper makes a useful contribution, but the central claim cannot be fully accepted until the caption-derived labels are validated against direct visual judgments. No additional objection about significance testing or prior work seems more load-bearing than this, and the reader already flagged the same issue, so my stress test does not change the recommended verdict.","tokens_in":10165,"tokens_out":3951,"duration_ms":39466,"concrete_test":"Sample 100 'Not Discussed' test pronouns. For each, show the image and dialogue (without the caption) to fresh annotators and ask them to select the object in the image the pronoun refers to, or 'cannot tell'. Then check whether the gold caption-derived antecedent corresponds to the selected object (after resolving mention surface forms). If agreement is substantially below the reported IAA of 72.4 (Sec. 2.4), the gold labels are not visually grounded and the central empirical claim would need to be reassessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that visual information is crucial for resolving pronouns in dialogues, supported by VisCoref's gains on the 'Not Discussed' subset. That claim depends on the gold antecedents for those pronouns being the actual visual objects the speakers intended. The annotation procedure does not establish this: when workers select 'some concepts not present in text' (Sec. 2.2), they do not point to a region or object in the image; the system afterwards assigns the antecedent from the caption (Sec. 2.4, footnote 3). Thus the gold label for every 'Not Discussed' pronoun is a caption noun phrase, not an independently verified visual referent. The mention pool given to all models also contains caption-derived mentions (Sec. 5.1), so the 'Not Discussed' task can in principle be solved by matching dialogue context to caption text. VisCoref's visual scoring is based on detected object labels encoded as text (Sec. 4.2), so the model is trained to align mention embeddings to text labels, with the image only indirectly involved through the detector's output. If the caption phrase does not match the speaker's intended object, the training signal is wrong for the model's stated purpose, and the superiority over text-only baselines may reflect learning caption-dialogue regularities rather than grounding to the image. This is the load-bearing assumption: the validity of the caption-derived gold labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines the task of visual-aware pronoun coreference resolution (PCR) in dialogues, introduces a new dataset called VisPro built from VisDial, and proposes a model called VisCoref that combines contextual text representations with visual object information. The dataset contains 29,722 pronoun annotations across 5,000 dialogues, with anaphoricity and antecedent labels collected via MTurk. VisCoref scores candidate antecedents using a weighted sum of a context-based score and a visual score, where the visual score is computed from detected object labels encoded as text. Experiments compare VisCoref against deterministic, statistical, deep reinforcement learning, and end-to-end coreference baselines, and report F1 gains, especially on a 'Not Discussed' subset of pronouns whose antecedents do not appear in the dialogue text. The paper also includes a hyperparameter analysis of the visual weight lambda_vis, two case studies, and an assessment of dataset quality via inner-annotator agreement.","tokens_in":10464,"tokens_out":3574,"duration_ms":33631,"significance":"If the main empirical claim holds, this is a useful contribution: it is, to my knowledge, the first large-scale dataset specifically targeting pronoun coreference in visually grounded dialogues, and the proposed model is simple and reproducible. The authors provide dataset, code, and models, and they report a human performance baseline, which is commendable. The 'Not Discussed' split is a sensible way to isolate cases where visual grounding should matter. However, the central claim that visual information is crucial is more conditional than the text suggests, because the gold labels for the 'Not Discussed' subset are derived from image captions rather than from direct visual verification, and the mention pool given to all models also contains those caption-derived mentions. The experimental section also lacks variance estimates or significance tests. These issues do not invalidate the dataset contribution, but they do mean the central comparative claim needs additional support.","major_comments":[{"comment":"The gold antecedents for the 'Not Discussed' pronouns are taken from the image caption (Section 2.4, footnote 3), and the mention pool supplied to all models contains exactly those caption-derived mentions (Section 5.1). Since the visual scoring in Section 4.2 encodes detected object labels as text in the same vector space as dialogue spans, the reported gain of VisCoref on 'Not Discussed' (69.28 vs. 66.02 F1 in Table 1) could in principle be achieved by learning dialogue-to-caption textual associations rather than by grounding pronouns to the image. This is load-bearing for the paper's central claim in Section 6 that visual information is crucial. The authors should either collect gold labels in which workers select or verify the actual visual referent in the image, or provide an analysis (e.g., a human re-annotation sample, or a text-only model with access only to the caption-derived mention pool) showing that the visual alignment term is necessary beyond text matching.","section":"2.4 (footnote 3), 5.1, 6"},{"comment":"The statement that VisCoref 'outperforms all the baseline models significantly' is not supported by any statistical evidence. All results appear to be from single runs, with no standard deviations, no multiple seeds, and no significance tests. The overall F1 gap between VisCoref and the End-to-end model is 4.07 points, and the 'Not Discussed' gap is 3.26 points; without variance estimates, these differences could be within run-to-run noise. Please report mean and standard deviation over multiple random seeds and provide a significance test (e.g., a paired bootstrap or per-dialogue paired test).","section":"6, Table 1"},{"comment":"The visual evidence is mediated by a fixed object detector whose labels are encoded with the same text encoder used for dialogue spans. This design has two consequences: the model cannot refer to any object the detector does not label, and the 'visual' score is effectively a textual similarity between mention embeddings and detected-label embeddings. The paper does not report detector accuracy or coverage on VisPro images, nor does it include an oracle experiment with ground-truth object labels. Without such an analysis, the contribution of genuine visual grounding versus label-text matching is unknown. Please add detector coverage statistics and, ideally, an oracle-labels experiment to bound the effect of detection errors.","section":"4.2, Eq. (7)-(11)"}],"minor_comments":[{"comment":"There is a typo in 'somef noun phrases' that should read 'some noun phrases'.","section":"2.4"},{"comment":"The phrase 'whenλvis equals to 0.9 or 1' should have a space after 'when' and should read 'when λvis equals 0.9 or 1'.","section":"6.1"},{"comment":"The sentence 'Each mention selects the text span of the highest overall score among all previous text spans in the dialogue or the mention pool as its antecedent' is unclear about the training objective; please clarify whether this describes the inference-time clustering procedure or the training loss, and how it relates to the objective in Eq. (1).","section":"5.2"},{"comment":"The 'Human' row should specify how the human F1 was computed, in particular whether the same mention-pool and scoring protocol was used as for the models.","section":"Table 1"},{"comment":"The y-axis of the pronoun distribution plot is not labeled; please add an axis label such as 'count'.","section":"Figure 3"},{"comment":"The criterion 'noun phrases with a height of two in parse trees' could use a brief explanation or a reference, since it is not immediately obvious why this height specifically is chosen.","section":"2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid dataset-and-model contribution, but the main empirical claim is more conditional than the text suggests. The annotation procedure for 'Not Discussed' antecedents (caption-derived labels) is the key risk; I recommend requesting additional evidence that the visual branch is not simply learning caption-dialogue text regularities. The lack of significance testing is also a concern for a journal-level claim. Please also verify the novelty statement against prior visual coreference work in visual dialog; the related work section cites Kottur et al. (2018) but the text says visual-aware PCR is 'still unexplored,' which may overstate the novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: VisPro is a genuinely useful resource, and the paper deserves a serious referee. But the central claim—that visual information is crucial for pronoun resolution in dialogue—is only as strong as the 'Not Discussed' gold labels, and those labels come from captions, not from workers pointing at the image. That is the soft spot to press.\n\nWhat's new: the dataset. 29,722 pronoun annotations in 5,000 VisDial dialogues, with anaphoricity labels and mention annotations. That is real work and fills a real gap. The model (VisCoref) is a sensible extension of e2e-coref: it aligns detected object labels with candidate mentions and combines contextual and visual scores. The experiments are reasonable: several text-only baselines, a visual-feature-concatenation baseline, and a split into Discussed vs Not Discussed pronouns. The case studies are illustrative, and the data/code release is a plus.\n\nWhere it gets shaky. First, the novelty claim is overstated. The paper says visual-aware PCR is unexplored and cites Kottur et al. (2018) in the same breath—yet Kottur et al. introduced visual coreference resolution in visual dialog. No comparison, no discussion of how the two tasks differ. That is a citation-pattern problem the authors should fix.\n\nSecond, and more importantly, the 'Not Discussed' gold annotation is not visually grounded. Section 2.4 footnote 3 says that when workers select 'some concepts not present in text,' the antecedent is 'provided by the caption.' Workers do not point at the image. So the gold label for the exact subset that most supports the visual claim is a caption noun phrase, not a verified visual referent. Worse, the mention pool given to all models is built from the corresponding caption plus negatives from other captions (Section 5.1). That means the 'Not Discussed' test could partly be solved by matching dialogue context to caption text, with no image grounding at all. The model's visual scoring itself converts detected object labels into text strings (Section 4.2), so VisCoref aligns mention embeddings to label strings rather than to image regions. The case studies show selected examples where the visual reasoning looks plausible, but the overall gains (4.07 F1 on the full set, 3.26 on Not Discussed) could reflect caption-dialogue regularities rather than true grounding.\n\nThird, the results are single runs with no variance or significance tests. The word 'significantly' in Section 6 is unsupported. That is a minor issue in this subfield, but worth noting.\n\nThe annotation procedure otherwise looks careful: four workers per dialogue, checkpoints, 95% kept after filtering, and MUC IAA of 72.4. On balance this is a contribution worth publishing after revision. The dataset alone is useful. But the authors should validate the 'Not Discussed' labels—for instance by re-annotating a sample with direct image grounding—or at least analyze how many caption-derived antecedents correspond to detectable objects, and report variance. Until then the strong claim is conditional.\n\nMy recommendation: send it to review, and ask the authors to address the caption-label validity head-on.","headline":"Useful dataset, but the central claim about visual grounding is undercut by caption-derived gold labels for 'Not Discussed' pronouns; worth reviewing with revisions.","tokens_in":10986,"tokens_out":1935,"would_cite":true,"duration_ms":19027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual grounding resolves pronouns that text-only models miss.","keywords":["visual pronoun coreference resolution","VisPro dataset","visual grounding","dialogue understanding","mention-object alignment","coreference resolution","image-grounded dialogue"],"falsifier":"Re-annotate the \"Not Discussed\" subset by asking workers to click the actual object in the image that each pronoun refers to, then compare those clicks with the caption-derived antecedents. If a substantial fraction do not match, the visual supervision in VisPro is caption-bound rather than image-grounded, and the visual branch of VisCoref may be learning text associations, not vision.","tokens_in":9980,"feed_emoji":"🖼️","tokens_out":7815,"duration_ms":64533,"temperature":0.7,"pith_summary":"The paper claims that in dialogues about a shared image, speakers routinely use pronouns for objects they can all see, even when the object is never named in the conversation. To test this, the authors built VisPro, a dataset of 29,722 pronoun annotations from 5,000 image-grounded dialogues, and VisCoref, a model that scores candidate antecedents by combining contextual text evidence with evidence from detected image objects. On VisPro, VisCoref reaches an overall F1 of 77.66, beating the best text-only baseline's 73.59, and the improvement is largest on pronouns whose antecedents are absent from the dialogue. The paper takes this as evidence that visual information is not optional for pronoun resolution in situated dialogue.","feed_headline":"Visual grounding lifts dialogue pronoun F1 to 77.66","feed_subtitle":"Aligning image objects with text spans beats text-only coreference, especially when antecedents are never spoken.","key_machinery":"The load-bearing mechanism is a mention-object alignment module. Object labels detected in the image (plus a \"null\" label) are embedded with the same encoder used for text spans; a scoring network computes the probability that a mention aligns with each label, and the visual score for a pronoun-antecedent pair combines the probability that each mention refers to some image object and that both refer to the same object. This score is added to a contextual span-scoring score in $F(n,p) = (1-\\lambda_{\\mathrm{vis}})F_c(n,p) + \\lambda_{\\mathrm{vis}}F_v(n,p)$, with $\\lambda_{\\mathrm{vis}}=0.4$, forcing the model to use both text and vision rather than letting raw image features dominate.","core_discovery":"The central discovery is that aligning textual mentions with detected image objects lets a model resolve pronouns whose antecedents appear only in the shared visual scene. VisCoref does this by converting object labels from an image detector into the same embedding space as dialogue spans, computing for each pronoun and candidate mention a visual compatibility score, and adding it to a contextual score with a tuned weight. The result is consistent improvement over all text-only baselines, including the state-of-the-art End-to-end model: 77.66 versus 73.59 overall F1, and 69.28 versus 66.02 on the \"Not Discussed\" subset where the antecedent is never mentioned in the dialogue. Human performance on the same benchmark is 88.12 F1, so the paper concludes the task is meaningful but far from solved.","pith_inferences":["Editorial inference: Because the visual branch relies on a fixed object-detector vocabulary, any pronoun referring to an entity outside that vocabulary is pushed toward \"null\"; a model with open-vocabulary or region-level grounding should do better on the Not Discussed subset.","Editorial inference: The mention pool mixes caption noun phrases with random negatives, so the evaluation is a ranking task over 30 candidates rather than open generation; an end-to-end visual coreference system that proposes its own antecedents could be tested on the same data.","Editorial inference: The 13.45% share of anaphoric pronouns whose antecedents are absent from the dialogue suggests a text-only ceiling; if the same rate holds in other visually situated conversations, no amount of language modeling alone can close the gap.","Editorial inference: The caption is used only to supply candidate noun phrases, never as dialogue text; a model allowed to read the caption directly could inflate Not Discussed scores, so the dataset's design choice keeps the visual grounding test honest but also leaves caption-reading as an untested alternative."],"forward_implications":["Text-only coreference models, however strong, will stay near 66.02 F1 on pronouns whose antecedents are absent from the dialogue; any situated dialogue system needs a visual pathway.","Simply concatenating image features to text features can help with text-mentioned antecedents but can hurt when the antecedent is only in the image; the object-label alignment used in VisCoref is the more reliable way to fuse the two modalities.","The tuned weight $\\lambda_{\\mathrm{vis}}$ matters: giving the visual branch too little weight forfeits its benefit, and giving it too much (0.9 or 1.0) makes the model overfit to vision and drop in overall F1.","The VisPro benchmark provides a reusable testbed for future visual PCR models, with separate Discussed and Not Discussed splits and a mention pool of 30 candidate spans per dialogue."],"supporting_citations":[{"why":"Provides VisDial, the source of the 5,000 dialogues and images from which VisPro is built.","marker":"Das et al., 2017"},{"why":"The End-to-end coreference model that VisCoref extends and the strongest text-only baseline it beats.","marker":"Lee et al., 2018"},{"why":"Supplies GloVe word embeddings used with ELMo as input representations.","marker":"Pennington et al., 2014"},{"why":"Supplies ELMo contextual embeddings used in the span encoder.","marker":"Peters et al., 2018"},{"why":"ResNet-152 image features used by the End-to-end+Visual baseline that VisCoref outperforms.","marker":"He et al., 2016"},{"why":"Rule-based deterministic baseline that reaches only 33.73 F1, showing the need for learned models.","marker":"Raghunathan et al., 2010"},{"why":"Statistical baseline that reaches 36.76 F1 on VisPro.","marker":"Clark and Manning, 2015"},{"why":"Deep reinforcement learning baseline that reaches 37.62 F1 on VisPro.","marker":"Clark and Manning, 2016"}],"fun_headline_variants":["Visual grounding resolves pronouns never spoken in dialogue","Visual cues boost dialogue pronoun F1 to 77.66","Image-aware coreference outdoes text-only in dialogue","Seeing the antecedent: visual grounding lifts dialogue pronoun F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that when annotators mark a pronoun as referring to \"some concepts not present in text,\" the noun phrase taken from the image caption is the object the speaker actually intended, rather than merely a caption word that happens to be plausible.","fun_headline_variants_meta":{"raw":{"variants":["Visual grounding resolves pronouns never spoken in dialogue","Visual cues boost dialogue pronoun F1 to 77.66","Image-aware coreference outdoes text-only in dialogue","Seeing the antecedent: visual grounding lifts dialogue pronoun F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000399,"raw_usage":{"total_tokens":2044,"prompt_tokens":860,"completion_tokens":1184,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":1129}},"tokens_in":476,"tokens_out":1184,"duration_ms":11221,"temperature":1.0,"reasoning_tokens":1129,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:52:14.955866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the \"Not Discussed\" subset by asking workers to click the actual object in the image that each pronoun refers to, then compare those clicks with the caption-derived antecedents. If a substantial fraction do not match, the visual supervision in VisPro is caption-bound rather than image-grounded, and the visual branch of VisCoref may be learning text associations, not vision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides VisDial, the source of the 5,000 dialogues and images from which VisPro is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The End-to-end coreference model that VisCoref extends and the strongest text-only baseline it beats."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GloVe word embeddings used with ELMo as input representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rule-based deterministic baseline that reaches only 33.73 F1, showing the need for learned models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Statistical baseline that reaches 36.76 F1 on VisPro."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Deep reinforcement learning baseline that reaches 37.62 F1 on VisPro."}],"review_version":1}