Searching for Ambiguous Objects in Videos using Relational Referring Expressions

Hazan Anayurt; Sezai Artun Ozyegin; Sinan Kalkan; Ulfet Cetin; Utku Aktas

REVIEW 4 major objections 5 minor 17 references

Searching for Ambiguous Objects in Videos using Relational Referring Expressions

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The proposed attention network reaches BLEU-4 0.2365 and mAP 0.65 on the new VIREF test set, beating the strongest baseline at 0.1498 and 0.46.

desk verdict A genuinely useful new dataset for relational referring expressions in ambiguous videos, but the paper's main quantitative claims are compromised by an unspecified data split. read the letter →

arxiv 1908.01189 v2 pith:TNJMZNI3 submitted 2019-08-03 cs.CV

Hazan Anayurt , Sezai Artun Ozyegin , Ulfet Cetin , Utku Aktas , Sinan Kalkan This is my paper · ORCID

classification cs.CV

keywords referringexpressionsrelationalvideoobjectsearchambiguousobjectsattentionmechanismencoder-decodercomprehensionandgenerationretrieval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Humans describe an object in a crowd by relating it to a unique nearby object. This paper argues that such relational referring expressions can be learned from video, and that one attention-based model can both generate them and use them to find the intended object. To test this, the authors build the VIREF dataset of surveillance and vehicle videos with many near-duplicate objects, collect referring expressions for object pairs, and train two baseline video extensions of an image-based generation-and-comprehension model. Their proposed VIREF model, an LSTM encoder-decoder with feature-level attention, reaches BLEU-4 0.2365 and mAP 0.65 on the test set, compared with 0.1498 and 0.46 for the stronger baseline. If the result holds, relational language becomes a usable query mechanism for searching objects in ambiguous videos without training a separate retrieval model.

What carries the argument

The load-bearing mechanism is the Feature Attention Network (FAN) inserted between the decoder and the encoder. At each decoding step, the decoder's hidden state produces attention weights over five feature streams: main-object crop, context-object crop, full frame, and binary masks of the two boxes, and the encoder LSTM is re-run on the rescaled features before the word-estimation network predicts the next word. Re-running the encoder with attention-weighted inputs lets relational information such as "walking near a white van" be distributed across object appearance, scene, and motion rather than summed once. The object-pair formulation itself, where every expression names a main object and a context object, is the other essential choice: it forces the model to learn relations rather than absolute attributes.

What would settle it

Run VIREF on the same test videos but replace the ground-truth bounding boxes with outputs of an automatic object detector, then measure comprehension mAP; if the score drops to near the VIREF-a baseline (0.46) or lower, the reported advantage depends on perfect annotations rather than on the relational attention mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single deep network can handle both directions of relational referring expressions in videos: given a main object and a context object, generate the phrase that identifies the main object with respect to the context one, and given the phrase, rank the candidate object pairs in a video so the intended pair surfaces. The model encodes each sampled frame as deep convolutional features of the main-object crop, the context-object crop, the whole frame, and binary masks of the two boxes, passes the sequence through an encoder LSTM, and decodes words with an LSTM whose hidden state drives a Feature Attention Network. At every decoding step the attention weights rescale the encoder's inputs and the encoder is run again, letting the model emphasize motion, appearance, or scene depending on the word it is producing. Using Bayes' rule with a uniform prior, the same trained generator ranks object pairs for comprehension, so no separate comprehension model is trained. On the new VIREF test set this model scores BLEU-4 0.2365 and mAP 0.65, compared with 0.1498 and 0.46 for the best baseline.

Load-bearing premise

The whole pipeline assumes that every object in the video has already been detected and given as a labeled box, and that the context object used in the expression is one of those labeled boxes; with missing or noisy detections the model's inputs are undefined.

Editorial extensions

If this is right

A single generator can serve as a comprehender: any improvement in generation quality should translate directly into better retrieval ranking, since comprehension simply ranks pairs by the probability the generator assigns to the input phrase.
Feature-level attention is the main driver of the reported gains: stripping it out (VIREF-a) drops mAP from 0.65 to 0.46, and stripping the LSTM encoder (VIREF-e) drops it to 0.55.
The VIREF dataset supplies a reusable benchmark for ambiguous video object search, with 125 surveillance and 37 vehicle videos, 3,170 object pairs, and 9,510 referring expressions, where non-relational descriptions are deliberately hard to use.
Because comprehension is evaluated as ranking over ground-truth pairs, the reported numbers are a lower bound in ambiguous cases: an expression can correctly identify more than one pair, while the evaluation credits only the original pair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not explore how the model behaves when object boxes come from an automatic detector instead of ground truth; a natural next test is whether the mAP gap over baselines survives noisy or missing boxes.
The paper leaves implicit that the same pair-based attention encoder-decoder could be adapted to other relational video-language tasks, such as referring-expression segmentation or spatio-temporal grounding, by changing only the output head.
A stress test beyond the paper's evaluation would present videos with more than 46 objects per frame or with the target object partially occluded; the relational strategy should degrade more gracefully than absolute-attribute baselines if learned relations are the actual driver.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a dataset-and-baselines paper, and the dataset is the real contribution. It gives us a new resource for relational referring expressions in videos where many objects look alike, covering both generation and comprehension. The model work is a competent combination of known pieces, not a new paradigm.

Credit where it's due: the VIREF dataset fills a genuine gap. The authors collect three REs per ordered pair, use both VIRAT and ILSVRC videos, and deliberately build in ambiguity. They are transparent about the assumption that tracks are already annotated. The comprehension-as-generation trick via Bayes with uniform priors is standard and not circular. The two baselines are reasonable ablations, and the attention model plausibly outperforms them.

Now the soft spots. First and most important, the train/validation/test split unit is never stated. Section 5.1 says only "60%, 10%, 30%." With multiple REs per pair and many pairs per video, an RE-level split would let the model see sibling expressions for the same pair during training. That would inflate BLEU-4 and mAP, and the reported gains (0.2365 vs 0.1498 BLEU-4; 0.65 vs 0.46 mAP) could reflect memorization rather than generalization. This is not a hypothetical; the paper gives no reason to believe the split is at the video or pair level. The promised GitHub link isn't verifiable from the preprint, so I can't check. Second, there are no variance estimates or significance tests—all comparisons rest on a single split. Third, they acknowledge the comprehension metric is conservative because an RE may match multiple pairs, but they still evaluate with a single correct pair; the reported numbers are lower bounds, which is fine, but the paper doesn't frame it that way in the abstract. Fourth, no comparison to existing video RE methods; the related work section cites three, but none are evaluated against. Minor issue: the text says six REs per object pair, but the totals (9,510 REs, 3,170 pairs) work out to three per pair, a small internal inconsistency.

Overall, the idea is sound and the dataset is useful. In its current form the empirical support is too thin to trust the quantitative claims. But it deserves a serious referee, not a desk reject. I'd recommend sending it to review with a request to clarify the split unit, add variance estimates or at least a second split, and reconcile the collection counts.

Who's this for: anyone working on referring expressions in video, especially benchmark builders. I wouldn't cite it until the split issue is resolved, but I'd put it on the reading list.

Take it to review.

Referee Report

4 major / 5 minor

Summary. The paper introduces VIREF, a dataset of relational referring expressions (REs) for object pairs in surveillance videos, collected on VIRAT and ILSVRC subsets. The authors propose an LSTM encoder-decoder with feature-level attention (VIREF) that generates REs for a given main/context object pair and, via Bayes' rule with a uniform prior, ranks object pairs for comprehension. Two baselines are introduced: VIREF-a (no attention) and VIREF-e (no LSTM encoder, using averaged VGG/C3D features). On a 60/10/30 split, VIREF reports BLEU-4 0.2365 vs 0.1498 (VIREF-a) and mAP 0.65 vs 0.46, and the paper claims significant improvement.

Significance. If the reported results withstand scrutiny, the paper contributes a new benchmark for an underexplored task—relational referring expressions in ambiguous videos—and a single model that performs both generation and comprehension. The attention mechanism and the reuse of the generator for comprehension via Bayes' rule are clean and reproducible design choices; the public dataset and code link further support follow-up work. The strengths are the task definition, dataset collection with explicit ambiguity, and the head-to-head comparison with two simple baselines. However, the significance is currently qualified by several unresolved technical issues in the dataset description and evaluation protocol.

major comments (4)

[Section 2 and Table 2] The text states 'We collected six REs each object pair (three for the straight and three for the reverse order)', but Table 1 reports 3,170 pairs and Table 2 reports 9,510 total REs, which is exactly three REs per pair, not six. This factor-of-two discrepancy means either the collection protocol, the pair count, or the RE count is misreported; since the dataset is a core contribution, the authors must correct this and ensure all statistics (including the '# RE per object' row) are consistent.
[Section 5.1] The description of the 60/10/30 split does not specify the unit of the split: individual referring expressions, ordered object pairs, or videos. Because the dataset contains multiple REs per pair and multiple pairs per video (Table 1), a split at the RE level could place sibling REs for the same pair, or pairs from the same video, in both training and test. That would allow the generation model to memorize video/pair-level patterns and inflate the BLEU-4 and METEOR scores in Table 3 and the comprehension results in Table 4. The authors should state the split unit, and ideally report results for a video-exclusive split.
[Section 5.3] The comprehension evaluation assumes that for each RE exactly one object pair is correct ('we assumed that only the object pair it was written for is the correct answer'), yet Section 2 explicitly states that REs were collected without requiring uniqueness: 'one RE could be correctly identifying more than one pair'. The single-answer AP/rank-k protocol therefore does not measure true comprehension for the intended task; it measures agreement with one collector's target. The paper should adopt a multi-answer evaluation or empirically measure the degree of ambiguity in the test set to justify the assumption.
[Tables 3 and 4 and Section 5.2] The abstract and conclusions state that VIREF 'significantly outperforms' the baselines, but the paper reports a single split with no variance estimates, confidence intervals, or statistical significance tests. With only one random 60/10/30 split, the observed gaps (BLEU-4 0.2365 vs 0.1498; mAP 0.65 vs 0.46) may not be stable across seeds or splits. The authors should either run multiple splits or provide bootstrap intervals to substantiate the word 'significantly'.

minor comments (5)

[Section 3.1, Eq. (1)] Please clarify the notation I_i(B_t_i): it appears to mean the image patch cropped to the bounding box, but as written it reads as the full frame; this makes the input construction ambiguous.
[Section 3.3] The sentence 'For both encoder and decoder, we use a six-layer LSTM' is surprising given the small dataset; please confirm the number of layers and whether dropout was applied to recurrent connections.
[Figure 2 caption] It is unclear whether the three listed REs under each pair are the three collected 'straight-order' responses or a selection; please label the figure accordingly.
[Section 5.1] Please report the random seed or state that no seed was fixed, to allow reproducibility of the split.
[Section 5.4] The comprehension time for VIREF (0.252 sec per query) is reported as an average over 100 samples; please state the hardware and whether this includes feature extraction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comprehension model is a Bayes-rule reuse of the generator, and all claims are evaluated on an external held-out split.

full rationale

The derivation chain is self-contained. The generation model directly maps object-pair features, conditioned on video frames, to word probabilities (Section 3.1) and is trained with cross-entropy loss. The comprehension model is the same generator applied through Bayes' rule under uniform priors, as stated in Eq. (6): argmax p(v,ot,oc|r) = argmax p(r|v,ot,oc). This is a valid scoring transformation, not a definitional equivalence with the target answer set: the RE-to-pair ranking is computed from probabilities learned on training pairs, and the test REs are not used for fitting. The baselines are architecturally simpler versions of the same model (Section 4), so the reported comparisons isolate the contributions of the attention module and the LSTM encoder rather than reducing to the model's own definitions. All external components (VGG16, C3D, GloVe, LSTM, BLEU/METEOR, mAP) are standard and independently defined. The paper does not invoke any self-citation as load-bearing evidence; the cited prior work is from other research groups. The decision to collect REs only for object pairs with meaningful relations is a task-definition choice, not a circular prediction. The principal weaknesses of the paper are the unspecified unit of the data split and the reliance on ground-truth boxes, but these are correctness or generalizability risks, not instances of circular reasoning. Therefore no circular step is present, and the score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical result about a dataset and a neural model. The main assumptions are practical domain assumptions about input annotations and feature quality rather than fitted scientific constants. The model's learned weights are not reported as free parameters; only key hyperparameters chosen by hand are listed.

free parameters (5)

learning_rate = 1e-4
Chosen for Adam optimizer in Section 5.1; performance depends on this value.
batch_size = 10
Section 5.1; affects optimization trajectory and final model.
num_lstm_layers = 6
Section 3.3; architecture choice for both encoder and decoder.
dropout_probability = 0.2
Section 3.3; regularization hyperparameter.
beam_size = 3
Section 3.1; decoding width used at test time for generation.

assumptions (5)

domain assumption Objects in videos are already detected or annotated, so the model inputs ground-truth bounding boxes.
Section 3 begins by assuming objects have been detected or annotated, following prior work. The method does not include detection and would fail without it.
domain assumption VGG16 fc1 and C3D fc6 feature representations are sufficient to capture object appearance and motion for generating and matching referring expressions.
The model input is fixed VGG16 and C3D features; if these features lack necessary visual details, the reported performance would drop.
standard math Bayes' rule with uniform priors over videos and object pairs is applicable for the comprehension task.
Equation (6) equates the posterior over pairs given the RE to the generative probability under a uniform prior; this is a standard probabilistic step.
domain assumption The manual filtering of object pairs to those with a meaningful relation yields a representative testbed for relational REs.
Section 2 describes visually observing each pair and keeping only those for which a meaningful RE could be written, which shapes the dataset distribution.
domain assumption REs longer than 25 words and words occurring once can be discarded without changing the task.
Section 5.1 excludes 94 of 9,510 REs (about 1%) and rare words, assuming they are not needed for training.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Searching for Ambiguous Objects in Videos using Relational Referring Expressions." pith.science (2026). https://pith.science/paper/TNJMZNI3

@misc{pith2026190801189,
  author       = {Pith},
  title        = {Pith review of: Searching for Ambiguous Objects in Videos using Relational Referring Expressions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNJMZNI3}},
  note         = {Machine review of arXiv:1908.01189}
}

read the original abstract

Humans frequently use referring (identifying) expressions to refer to objects. Especially in ambiguous settings, humans prefer expressions (called relational referring expressions) that describe an object with respect to a distinguishing, unique object. Unlike studies on video object search using referring expressions, in this paper, our focus is on (i) relational referring expressions in highly ambiguous settings, and (ii) methods that can both generate and comprehend a referring expression. For this goal, we first introduce a new dataset for video object search with referring expressions that includes numerous copies of the objects, making it difficult to use non-relational expressions. Moreover, we train two baseline deep networks on this dataset, which show promising results. Finally, we propose a deep attention network that significantly outperforms the baselines on our dataset. The dataset and the codes are available at https://github.com/hazananayurt/viref.

Figures

Figures reproduced from arXiv: 1908.01189 by the authors.

**Figure 2.** Two object pairs from our dataset and the three REs collected for them. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 4.** The VIREF-a model (VIREF without attention). Word Estimation Layer Decoder LSTM Feature Processing features Network hj outj [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: The VIREF model. Word Estimation Layer Decoder LSTM Encoder LSTM features hj outj [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]

**Figure 6.** Figure 6: Sample generated REs. Object with label 1 is the main object, and the one with [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Sample comprehension results. Object with label 1 is the main object, and the one [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

[1]

Object referring in videos with language and human gaze

Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object referring in videos with language and human gaze. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4129–4138, 2018

work page 2018
[2]

Cooking up referring expressions

Robert Dale. Cooking up referring expressions. In Proceedings of the 27th Annual Meeting on Association for Computational Linguistics , 1989

work page 1989
[3]

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compu- tation, 9(8):1735–1780, 1997

work page 1997
[4]

Referitgame: Referring to objects in photographs of natural scenes

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 787–798, 2014

work page 2014
[5]

Video object segmentation with referring expressions

Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with referring expressions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 7–12. Springer, 2018

work page 2018
[6]

Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments

Alon Lavie and Abhaya Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231. Association for Compu- tational Linguistics, 2007

work page 2007
[7]

Generation and comprehension of unambiguous object descriptions

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11–20, 2016

work page 2016
[8]

Modeling context between objects for referring expression understanding

Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. In Proceedings of the European Con- ference on Computer Vision (ECCV), pages 792–807. Springer, 2016

work page 2016

Show all 17 references

[9]

Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, J. K. Aggarwal, Hyungtae Lee, Larry Davis, Eran Swears, Xioyang Wang, Qiang Ji, Kishore Reddy, Mubarak Shah, Carl V ondrick, Hamed Pirsiavash, Deva Ramanan, Jenny Yuen,...

2011
[10]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , pages 311–318. Association for Computational Linguistics, 2002

2002
[11]

Glove: Global vectors for word representation

Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014. 12 ANA YURT ET AL.: SEARCHING FOR AMBIGUOUS OBJ...

2014
[12]

Berg, and Li Fei-Fei

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Inter- national Journal of Computer Vision (I...

2015
[13]

Very deep convolutional networks for large- scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale image recognition. In Proceedings of 3rd International Conference on Learning Representations (ICLR), 2015

2015
[14]

Learning spatiotemporal features with 3d convolutional networks

Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 4489–4497, 2015

2015
[15]

The use of spatial relations in referring expression generation

Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference, pages 59–67. Association for Computational Linguistics, 2008

2008
[16]

Referring to objects in videos using spatio-temporal identifying descriptions

Peratham Wiriyathammabhum, Abhinav Shrivastava, Vlad I Morariu, and Larry S Davis. Referring to objects in videos using spatio-temporal identifying descriptions. arXiv preprint arXiv:1904.03885, 2019

1904 arXiv
[17]

Mattnet: Modular attention network for referring expression comprehension

Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Object referring in videos with language and human gaze

Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object referring in videos with language and human gaze. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4129–4138, 2018

work page 2018

[2] [2]

Cooking up referring expressions

Robert Dale. Cooking up referring expressions. In Proceedings of the 27th Annual Meeting on Association for Computational Linguistics , 1989

work page 1989

[3] [3]

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compu- tation, 9(8):1735–1780, 1997

work page 1997

[4] [4]

Referitgame: Referring to objects in photographs of natural scenes

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 787–798, 2014

work page 2014

[5] [5]

Video object segmentation with referring expressions

Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with referring expressions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 7–12. Springer, 2018

work page 2018

[6] [6]

Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments

Alon Lavie and Abhaya Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231. Association for Compu- tational Linguistics, 2007

work page 2007

[7] [7]

Generation and comprehension of unambiguous object descriptions

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11–20, 2016

work page 2016

[8] [8]

Modeling context between objects for referring expression understanding

Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. In Proceedings of the European Con- ference on Computer Vision (ECCV), pages 792–807. Springer, 2016

work page 2016