Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Visual grounding resolves pronouns that text-only models miss.

desk verdict Useful dataset, but the central claim about visual grounding is undercut by caption-derived gold labels for 'Not Discussed' pronouns; worth reviewing with revisions. read the letter →

arxiv 1909.00421 v1 pith:77E3YIFY submitted 2019-09-01 cs.CL

classification cs.CL
keywords visualpronouncoreferenceresolutionVisProdatasetgroundingdialogueunderstandingmention-objectalignmentimage-grounded
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that in dialogues about a shared image, speakers routinely use pronouns for objects they can all see, even when the object is never named in the conversation. To test this, the authors built VisPro, a dataset of 29,722 pronoun annotations from 5,000 image-grounded dialogues, and VisCoref, a model that scores candidate antecedents by combining contextual text evidence with evidence from detected image objects. On VisPro, VisCoref reaches an overall F1 of 77.66, beating the best text-only baseline's 73.59, and the improvement is largest on pronouns whose antecedents are absent from the dialogue. The paper takes this as evidence that visual information is not optional for pronoun resolution in situated dialogue.

What carries the argument

The load-bearing mechanism is a mention-object alignment module. Object labels detected in the image (plus a "null" label) are embedded with the same encoder used for text spans; a scoring network computes the probability that a mention aligns with each label, and the visual score for a pronoun-antecedent pair combines the probability that each mention refers to some image object and that both refer to the same object. This score is added to a contextual span-scoring score in $F(n,p) = (1-\lambda_{\mathrm{vis}})F_c(n,p) + \lambda_{\mathrm{vis}}F_v(n,p)$, with $\lambda_{\mathrm{vis}}=0.4$, forcing the model to use both text and vision rather than letting raw image features dominate.

What would settle it

Re-annotate the "Not Discussed" subset by asking workers to click the actual object in the image that each pronoun refers to, then compare those clicks with the caption-derived antecedents. If a substantial fraction do not match, the visual supervision in VisPro is caption-bound rather than image-grounded, and the visual branch of VisCoref may be learning text associations, not vision.

Watch

Extended reading notes

Core claim

The central discovery is that aligning textual mentions with detected image objects lets a model resolve pronouns whose antecedents appear only in the shared visual scene. VisCoref does this by converting object labels from an image detector into the same embedding space as dialogue spans, computing for each pronoun and candidate mention a visual compatibility score, and adding it to a contextual score with a tuned weight. The result is consistent improvement over all text-only baselines, including the state-of-the-art End-to-end model: 77.66 versus 73.59 overall F1, and 69.28 versus 66.02 on the "Not Discussed" subset where the antecedent is never mentioned in the dialogue. Human performance on the same benchmark is 88.12 F1, so the paper concludes the task is meaningful but far from solved.

Load-bearing premise

The paper assumes that when annotators mark a pronoun as referring to "some concepts not present in text," the noun phrase taken from the image caption is the object the speaker actually intended, rather than merely a caption word that happens to be plausible.

Editorial extensions

If this is right

  • Text-only coreference models, however strong, will stay near 66.02 F1 on pronouns whose antecedents are absent from the dialogue; any situated dialogue system needs a visual pathway.
  • Simply concatenating image features to text features can help with text-mentioned antecedents but can hurt when the antecedent is only in the image; the object-label alignment used in VisCoref is the more reliable way to fuse the two modalities.
  • The tuned weight $\lambda_{\mathrm{vis}}$ matters: giving the visual branch too little weight forfeits its benefit, and giving it too much (0.9 or 1.0) makes the model overfit to vision and drop in overall F1.
  • The VisPro benchmark provides a reusable testbed for future visual PCR models, with separate Discussed and Not Discussed splits and a mention pool of 30 candidate spans per dialogue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the visual branch relies on a fixed object-detector vocabulary, any pronoun referring to an entity outside that vocabulary is pushed toward "null"; a model with open-vocabulary or region-level grounding should do better on the Not Discussed subset.
  • Editorial inference: The mention pool mixes caption noun phrases with random negatives, so the evaluation is a ranking task over 30 candidates rather than open generation; an end-to-end visual coreference system that proposes its own antecedents could be tested on the same data.
  • Editorial inference: The 13.45% share of anaphoric pronouns whose antecedents are absent from the dialogue suggests a text-only ceiling; if the same rate holds in other visually situated conversations, no amount of language modeling alone can close the gap.
  • Editorial inference: The caption is used only to supply candidate noun phrases, never as dialogue text; a model allowed to read the caption directly could inflate Not Discussed scores, so the dataset's design choice keeps the visual grounding test honest but also leaves caption-reading as an untested alternative.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper defines the task of visual-aware pronoun coreference resolution (PCR) in dialogues, introduces a new dataset called VisPro built from VisDial, and proposes a model called VisCoref that combines contextual text representations with visual object information. The dataset contains 29,722 pronoun annotations across 5,000 dialogues, with anaphoricity and antecedent labels collected via MTurk. VisCoref scores candidate antecedents using a weighted sum of a context-based score and a visual score, where the visual score is computed from detected object labels encoded as text. Experiments compare VisCoref against deterministic, statistical, deep reinforcement learning, and end-to-end coreference baselines, and report F1 gains, especially on a 'Not Discussed' subset of pronouns whose antecedents do not appear in the dialogue text. The paper also includes a hyperparameter analysis of the visual weight lambda_vis, two case studies, and an assessment of dataset quality via inner-annotator agreement.

Significance. If the main empirical claim holds, this is a useful contribution: it is, to my knowledge, the first large-scale dataset specifically targeting pronoun coreference in visually grounded dialogues, and the proposed model is simple and reproducible. The authors provide dataset, code, and models, and they report a human performance baseline, which is commendable. The 'Not Discussed' split is a sensible way to isolate cases where visual grounding should matter. However, the central claim that visual information is crucial is more conditional than the text suggests, because the gold labels for the 'Not Discussed' subset are derived from image captions rather than from direct visual verification, and the mention pool given to all models also contains those caption-derived mentions. The experimental section also lacks variance estimates or significance tests. These issues do not invalidate the dataset contribution, but they do mean the central comparative claim needs additional support.

major comments (3)
  1. [2.4 (footnote 3), 5.1, 6] The gold antecedents for the 'Not Discussed' pronouns are taken from the image caption (Section 2.4, footnote 3), and the mention pool supplied to all models contains exactly those caption-derived mentions (Section 5.1). Since the visual scoring in Section 4.2 encodes detected object labels as text in the same vector space as dialogue spans, the reported gain of VisCoref on 'Not Discussed' (69.28 vs. 66.02 F1 in Table 1) could in principle be achieved by learning dialogue-to-caption textual associations rather than by grounding pronouns to the image. This is load-bearing for the paper's central claim in Section 6 that visual information is crucial. The authors should either collect gold labels in which workers select or verify the actual visual referent in the image, or provide an analysis (e.g., a human re-annotation sample, or a text-only model with access only to the caption-derived mention pool) showing that the visual alignment term is necessary beyond text matching.
  2. [6, Table 1] The statement that VisCoref 'outperforms all the baseline models significantly' is not supported by any statistical evidence. All results appear to be from single runs, with no standard deviations, no multiple seeds, and no significance tests. The overall F1 gap between VisCoref and the End-to-end model is 4.07 points, and the 'Not Discussed' gap is 3.26 points; without variance estimates, these differences could be within run-to-run noise. Please report mean and standard deviation over multiple random seeds and provide a significance test (e.g., a paired bootstrap or per-dialogue paired test).
  3. [4.2, Eq. (7)-(11)] The visual evidence is mediated by a fixed object detector whose labels are encoded with the same text encoder used for dialogue spans. This design has two consequences: the model cannot refer to any object the detector does not label, and the 'visual' score is effectively a textual similarity between mention embeddings and detected-label embeddings. The paper does not report detector accuracy or coverage on VisPro images, nor does it include an oracle experiment with ground-truth object labels. Without such an analysis, the contribution of genuine visual grounding versus label-text matching is unknown. Please add detector coverage statistics and, ideally, an oracle-labels experiment to bound the effect of detection errors.
minor comments (6)
  1. [2.4] There is a typo in 'somef noun phrases' that should read 'some noun phrases'.
  2. [6.1] The phrase 'whenλvis equals to 0.9 or 1' should have a space after 'when' and should read 'when λvis equals 0.9 or 1'.
  3. [5.2] The sentence 'Each mention selects the text span of the highest overall score among all previous text spans in the dialogue or the mention pool as its antecedent' is unclear about the training objective; please clarify whether this describes the inference-time clustering procedure or the training loss, and how it relates to the objective in Eq. (1).
  4. [Table 1] The 'Human' row should specify how the human F1 was computed, in particular whether the same mention-pool and scoring protocol was used as for the models.
  5. [Figure 3] The y-axis of the pronoun distribution plot is not labeled; please add an axis label such as 'count'.
  6. [2.1] The criterion 'noun phrases with a height of two in parse trees' could use a brief explanation or a reference, since it is not immediately obvious why this height specifically is chosen.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's model and dataset are empirically evaluated against an independently annotated test set, and self-citations are not load-bearing.

full rationale

The paper's central claim is that visual information helps pronoun coreference resolution in dialogues, supported by a new dataset (VisPro) and a model (VisCoref) evaluated on held-out test dialogues. There is no derivation chain in which an output is identified with an input by construction: the model is trained on a training split and scored on a separate test split, and the baselines are standard released systems. The cited prior work by the authors (Zhang et al. 2019a,b) appears in the introduction and related work for background on coreference resolution; it is not invoked as the source of the model's architecture, dataset annotations, or experimental result, so it is not load-bearing. The only potentially questionable point is footnote 3, which states that when a worker selects 'some concepts not present in text,' the antecedent is provided by the caption, and Section 5.1 places caption-derived mentions into the candidate mention pool. This is a data-construction and annotation-validity concern: if the caption phrase does not match the speaker's intended visual object, then the gold labels for 'Not Discussed' pronouns may be noisy or textually solvable. That would weaken the interpretation of the visual gains, but it is not circular reasoning, because the model's predictions are not defined in terms of the gold labels and no fitted parameter is renamed as a prediction. The evaluation is independent of the model's training objective, and the paper does not reduce its central claim to its own assumptions. Therefore, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the reliability of the dataset construction (caption-derived gold labels, object detection coverage) and on the validity of the model's text-conditioned visual alignment. No new physical or mathematical entities are introduced.

free parameters (2)
  • lambda_vis = 0.4
    Weight balancing contextual and visual scoring; tuned on the validation set (Section 6.1).
  • mention_pool_size = 30
    Number of candidate mentions outside the dialogue provided to models (Section 5.1); a design choice that affects task difficulty.
assumptions (3)
  • domain assumption Object labels detected by SSD ResNet 50 FPN COCO capture the visual entities that pronouns refer to in these dialogues.
    The visual scoring branch depends entirely on detected object labels; if the referent object is not detected, visual grounding cannot work. Section 4.2.
  • domain assumption Caption noun phrases are valid gold antecedents for pronouns whose referents are not in the dialogue text.
    Footnote 3 in Section 2.4 says the antecedent labeled by the worker is provided by the caption; this grounds 'Not Discussed' labels in captions rather than direct image annotations.
  • domain assumption Majority voting and cluster merging on MTurk annotations yield correct coreference clusters.
    Section 2.4 describes the adjudication procedure; IAA MUC 72.4 suggests moderate, not perfect, agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues." pith.science (2026). https://pith.science/paper/77E3YIFY

@misc{pith2026190900421,
  author       = {Pith},
  title        = {Pith review of: What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77E3YIFY}},
  note         = {Machine review of arXiv:1909.00421}
}
read the original abstract

Grounding a pronoun to a visual object it refers to requires complex reasoning from various information sources, especially in conversational scenarios. For example, when people in a conversation talk about something all speakers can see, they often directly use pronouns (e.g., it) to refer to it without previous introduction. This fact brings a huge challenge for modern natural language understanding systems, particularly conventional context-based pronoun coreference models. To tackle this challenge, in this paper, we formally define the task of visual-aware pronoun coreference resolution (PCR) and introduce VisPro, a large-scale dialogue PCR dataset, to investigate whether and how the visual information can help resolve pronouns in dialogues. We then propose a novel visual-aware PCR model, VisCoref, for this task and conduct comprehensive experiments and case studies on our dataset. Results demonstrate the importance of the visual information in this PCR case and show the effectiveness of the proposed model.

Figures

Figures reproduced from arXiv: 1909.00421 by the authors.

Figure 1
Figure 1. An example of a visual-related dialogue. Two [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example syntax parsing result of the sentence [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Distribution of pronouns in VisPro. tle use to the task. For another, dialogues with too many pronouns often contain repeating pronouns referring to the same object, which makes the task too easy. The dialogues selected contain 5.94 pro￾nouns on average [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Overall structure of the VisCoref model. Text [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Effect of λvis. F1 scores of all categories are reported. gap between our model and human being. It in￾dicates that current models still cannot fully un￾derstand the dialogue even with the support of visual information and further proves the value and necessity of prop…
Figure 7
Figure 7. Figure 7: Randomly selected examples from VisPro. The image, dialogue record, prediction result, and heatmap [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution

    cs.HC 2025-09 conditional novelty 6.0 of 10

    Augmenting VR speech transcripts with gaze and pointing cues improved GPT-4 coreference resolution from 40.6% to 67.1% accuracy.

Reference graph

Works this paper leans on

31 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In Proceedings of ICCV, pages 2425--2433

  4. [4]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR

  5. [5]

    Yuille, and Shu Rong

    Hong Chen, Zhenhua Fan, Hao Lu, Alan L. Yuille, and Shu Rong. 2018. Preco: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of EMNLP, pages 172--181

  6. [6]

    Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL, pages 1405--1415

  7. [7]

    Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of EMNLP, pages 2256--2262

  8. [8]

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos \' e M. F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of CVPR, pages 1080--1089

Show all 31 references
  1. [9]

    Edens, Helen L

    Richard J. Edens, Helen L. Gaylard, Gareth J. F. Jones, and Adenike M. Lam - Adesina. 2003. An investigation of broad coverage automatic pronoun resolution for information retrieval. In Proceedings of SIGIR, pages 381--382

  2. [10]

    Kate Ehrlich. 1981. Search and inference strategies in pronoun resolution: an experimental study. In Proceedings of ACL, pages 89--93

  3. [11]

    Liane Guillou. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of EACL, pages 1--10

  4. [12]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770--778

  5. [13]

    Jerry R Hobbs. 1978. Resolving pronoun references. Lingua, 44(4):311--338

  6. [14]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

  7. [15]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR

  8. [16]

    Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of ACL, pages 423--430

  9. [17]

    Satwik Kottur, Jos \' e M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of ECCV, pages 160--178

  10. [18]

    Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL-HLT, pages 687--692

  11. [19]

    Yuille, and Kevin Murphy

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of CVPR, pages 11--20

  12. [20]

    Vincent Ng. 2005. Supervised ranking for pronoun resolution: Some recent improvements. In Proceedings of AAAI, pages 1081--1086

  13. [21]

    US NIST. 2003. The ace 2003 evaluation plan. US National Institute for Standards and Technology (NIST), pages 2003--08

  14. [22]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532--1543

  15. [23]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227--2237

  16. [24]

    Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Proceedings of EMNLP-CoNLL, pages 1--40

  17. [25]

    Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nate Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher D. Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of EMNLP, pages 492--501

  18. [26]

    Kabadjov, and Karel Jezek

    Josef Steinberger, Massimo Poesio, Mijail A. Kabadjov, and Karel Jezek. 2007. Two uses of anaphora resolution in summarization. Inf. Process. Manage., 43(6):1663--1680

  19. [27]

    Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL, pages 656--664

  20. [28]

    Michael Strube and Christoph M \" u ller. 2003. A machine learning approach to pronoun resolution in spoken dialogue. In Proceedings of ACL, pages 168--175

  21. [29]

    Courville, Ruslan Salakhutdinov, Richard S

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of ICML, pages 2048--2057

  22. [30]

    Hongming Zhang, Yan Song, and Yangqiu Song. 2019 a . Incorporating context and external knowledge for pronoun coreference resolution. In Proceedings of NAACL-HLT, pages 872--881

  23. [31]

    Hongming Zhang, Yan Song, Yangqiu Song, and Dong Yu. 2019 b . Knowledge-aware pronoun coreference resolution. In Proceedings of ACL, pages 867--876

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.