Pith. sign in

REVIEW 4 major objections 5 minor 71 references

This paper introduces DiSG, a real-image benchmark for open-vocabulary gaze object prediction, along with a decoupled framework and a gradient-informed tuning method that can name gaze targets from categories never seen during training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:10 UTC pith:FENQ7GCJ

load-bearing objection Useful benchmark and solid baseline, but the 'open-vocabulary' claim is narrower than advertised: the label space is a fixed 86-class COCO+body-part inventory and the annotation pipeline filters to boxes enclosing the gaze point. the 4 major comments →

arxiv 2607.18827 v1 pith:FENQ7GCJ submitted 2026-07-21 cs.CV

Open-Vocabulary Gaze Object Prediction: Benchmark and Method

Binglu Wang , Sensen Niu , Ying Chen , Guangyu Guo This is my paper
classification cs.CV
keywords Gaze Object PredictionOpen-Vocabulary Object DetectionBenchmark DatasetGaze Target EstimationSaliency HeatmapSelective Parameter TuningLong-Tail DistributionBase/Novel Split
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to move gaze object prediction—finding and naming the object a person is looking at—from fixed category lists to open-vocabulary settings, where the target may be a category never seen in training. To make that possible, it introduces DiSG, a real-image benchmark of 13,041 images with 86 gaze-target categories, split into 60 base and 26 novel classes, and a protocol that prompts models with class names at test time. It then proposes a two-branch model: a text-driven detector generates candidate object boxes from an arbitrary vocabulary, and a gaze-guided selection branch scores those boxes by how strongly they align with a predicted saliency heatmap. To keep the detector's open-vocabulary knowledge while adapting it to small, subtle gaze targets, it adds Gradient-Informed Selection Tuning (GIST), which updates only the most gradient-sensitive 40% of transformer parameters per layer. On DiSG, the full model raises novel-category mSoC from 20.4 to 29.5 over its baseline and reaches 34.2 mSoC in the closed-vocabulary setting, surpassing prior GOP methods; the claim is that open-vocabulary GOP is feasible and this benchmark and baseline make it measurable.

Core claim

The paper's central claim is that gaze object prediction can be formulated and evaluated open-vocabulary: instead of training on a fixed label space, a model can localize candidate objects using text prompts and then pick the attended one using gaze. The paper supports this with DiSG, a benchmark built from COCO object boxes and GazeFollow gaze annotations plus six body-part classes, with a base/novel split where 26 categories are held out from training. On top of this, the proposed framework decouples discovery from selection: an open-vocabulary detector (initialized from Grounding DINO) proposes boxes for the queried vocabulary, and a gaze-guided branch predicts a class-agnostic saliency h

What carries the argument

The load-bearing mechanism is the decoupled two-stream design plus GIST. The discovery branch is a text-driven open-vocabulary detector (BERT text encoder plus Swin-T visual backbone, initialized from Grounding DINO) that produces candidate boxes for any vocabulary. The selection branch uses a CLIP-encoded head crop with eight directional prompts to predict a class-agnostic saliency heatmap. The two meet in a spatial-semantic disambiguation step: each candidate box receives the average saliency inside it, and the box with the maximum score wins. GIST is the piece that makes adaptation possible: during a short probing phase the model trains fully while accumulating AdamW bias-corrected second

Load-bearing premise

The benchmark assumes that every gaze target's box can be recovered from a pre-specified pool of COCO object boxes plus six body-part classes filtered by whether the box contains the gaze point; if the true target is outside that inventory or the gaze point falls outside its box, the annotation pipeline cannot represent it, so the open-vocabulary measurement is bounded by that inventory.

What would settle it

A decisive test would be to evaluate the same model on images whose gaze targets lie outside the 86-class inventory (for example, 'drone' or 'sushi') or to re-annotate a random subset of DiSG without the filter that keeps only candidate boxes enclosing the gaze point, then compare rankings; if novel-category mSoC collapses to chance or the ordering of methods flips, the open-vocabulary claim would not survive.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Open-vocabulary GOP becomes testable: future methods can report base/novel mSoC on DiSG under the same prompt-based protocol, so progress on unseen gaze targets is measurable rather than anecdotal.
  • The decoupling insight—separate 'what objects are present' from 'which one is attended'—means improvements in open-vocabulary grounding or gaze estimation can be composed without redesigning the whole pipeline.
  • GIST's sparse, gradient-sensitivity-based tuning suggests that only a minority of parameters (20–40%) in a pretrained detector need updating to shift it to a new domain without catastrophic forgetting; the paper's ablation shows larger update ratios hurt novel-category performance.
  • The benchmark's body-part categories let models distinguish attention on a head, hand, leg, or torso from attention on the whole person, addressing a semantic ambiguity closed-vocabulary GOP datasets leave unsolved.
  • Because the same framework outperforms prior GOP methods under the closed-vocabulary setting on DiSG-base, the open-vocabulary machinery does not come at the cost of conventional performance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's 'open' vocabulary is still bounded by the 80 COCO classes plus six body-part classes; a true open-world test would need gaze targets outside that inventory, so DiSG is best read as open-vocabulary within a fixed taxonomy.
  • Since the candidate pool is filtered by whether boxes enclose the gaze point, the benchmark may systematically exclude valid gaze targets whose box does not contain the point; re-annotating with a different candidate-generation rule could shift results.
  • The GIST principle—select parameters by bias-corrected second-moment—is generic and could transfer to other vision-language tasks facing the same adaptation-versus-forgetting tension, such as open-vocabulary detection in other specialized domains, though the paper only demonstrates it for gaze.
  • The gaze-guided selection module is class-agnostic; combining it with an even larger freely prompted vocabulary, or with descriptive attribute prompts rather than plain class names, might further improve novel-target selection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper addresses open-vocabulary gaze object prediction (OVGOP). It introduces DiSG, a benchmark of 13,041 real images with 86 categories (80 COCO classes plus 6 body-part classes) built from the intersection of COCO and GazeFollow, using a hybrid automatic/manual annotation pipeline and a base/novel split for zero-shot evaluation. It also proposes a framework that combines Grounding DINO-based text-driven proposal generation with a gaze-guided saliency selection module, and introduces GIST, a two-stage selective parameter-tuning strategy based on bias-corrected second moments of gradients. Experiments report improved open-vocabulary and closed-vocabulary performance on DiSG, with ablations on the contributions of the frozen text encoder and GIST, plus supplementary analyses including a quality audit and on GOO-Real.

Significance. If the results hold, DiSG would be a useful real-image complement to existing GOP benchmarks, and the proposed framework would provide a reproducible baseline for OVGOP. The paper ships code and benchmark, reports detailed ablations, and includes a supplementary appendix with algorithms, extra analyses, and a quality audit. However, several load-bearing issues must be resolved: the benchmark's 'open-vocabulary' scope is confined to a fixed COCO/body-part taxonomy; the GIST selection ratio appears to be tuned on the held-out novel split; and the benefit of gradient-informed selection over plain parameter freezing is not demonstrated. These issues affect the benchmark's central claim and the validity of some reported gains, so the manuscript needs major revision.

major comments (4)
  1. [Sec. 3 (Annotation summary); Supp. A.1] The DiSG label space is closed: candidate boxes are generated only from COCO's 80 classes plus 6 body-part classes, then filtered to boxes that enclose the gaze point, and annotators choose from this pool. Objects outside this inventory are never offered, so the 'open-vocabulary' protocol is a base/novel split over a fixed 86-term taxonomy, not an open-world benchmark. The paper's own counts (23,895→14,600→13,041 images) show 39% of overlapping images are discarded without labeling any gaze target, so coverage of out-of-inventory gaze targets is unmeasured. The abstract and Sec. 1 should qualify the claims of 'open-vocabulary' and 'unseen categories' to zero-shot transfer within the defined taxonomy, or add an analysis of the discarded images.
  2. [Sec. 5.2, Table 3] The GIST top-k ratio is selected by comparing novel-category metrics (mSoC_N, AP_N) across k=20–100 and choosing 40% as the best balance. Since novel categories are held out and intended only for zero-shot evaluation, tuning k on novel metrics leaks test information into model selection and biases the GIST gains reported in Table 1. Please either fix k using only base-category validation metrics or a separate validation split and report the resulting numbers, or explicitly state that the reported novel results are optimistic because k was tuned on them.
  3. [Sec. 4.2, Tables 1 and 3] The GIST ablation does not separate the effect of gradient-based selection from the effect of freezing 60% of parameters. A random-mask baseline with the same sparsity, or a simple 'freeze last-k layers' baseline, is needed to show that selecting high-variance parameters is what improves novel mSoC. Without this control, the 'gradient-informed' mechanism is not established; the observed improvements could simply reflect regularization from parameter freezing.
  4. [Sec. 5.3, Table 4; Supp. C.2] The closed-vocabulary comparisons on DiSG do not specify whether prior methods were retrained on DiSG-base or evaluated with official checkpoints, which is essential for a fair comparison. Additionally, on the existing GOO-Real benchmark (Supp. C.2) the method achieves 79.5 mSoC vs. TransGOP's 82.6, so the abstract's claim of 'outperforms existing methods in the conventional closed-vocabulary setting' is too broad. Please clarify the baseline training/evaluation protocol and qualify the claim to DiSG-base.
minor comments (5)
  1. [Eq. (1)] The gaze-alignment score is an average saliency over the box. Because object sizes vary widely (e.g., 'person' vs. 'hand'), the average may disadvantage small parts; consider also reporting max or a size-normalized aggregation.
  2. [Supp. A.2] The clarification that novel categories receive no box-level supervision during DiSG training is useful, but the pretrained Grounding DINO has likely seen COCO categories in image-text pretraining. The term 'unseen' should be defined relative to the DiSG training stage, not to the pretrained model, to avoid overclaiming.
  3. [Sec. 5.2, Table 2] The comparison with OV-DINO is not strictly controlled because OV-DINO uses additional pretraining data (CC1M) beyond the authors' setup. This should be noted in the table or text.
  4. [Table 1] The 'Baseline + Frozen TextEnc' row improves novel mSoC from 20.4 to 27.1, a larger gain than 'Baseline + GIST' (25.6). The paper should discuss why freezing the text encoder alone is so effective and how it interacts with GIST.
  5. [References] Reference [35] appears to duplicate [34]; please check citation numbering and remove duplicates.

Circularity Check

0 steps flagged

No significant circularity: the benchmark labels, GIST selection, and gaze-guided decoding are each produced by mechanisms distinct from the quantities they predict.

full rationale

The derivation chain is self-contained. DiSG is constructed from COCO boxes and GazeFollow gaze points, but adds body-part candidate generation and a human-in-the-loop verification stage (Supp. A.1); the final labels are not identical to the candidate-pool filter, as the paper explicitly reports that 5.73% of final gaze points fall outside the target box and that a 1,000-instance audit found 94.3% correct annotations. The method's selection rule (Eq. 1) aggregates a learned, class-agnostic saliency heatmap inside each detected proposal; it is trained with box-energy and MSE losses (Eq. 4) and is not an algebraic re-statement of the annotation rule. GIST selects parameters using bias-corrected second-moment gradient statistics (Eqs. 2-3) computed on base-category training only; novel categories receive no box supervision during DiSG task training, so the reported novel-category mSoC is not a fitted constant. The paper's self-citations (TransGOP [53], TransGOP-R [12], etc.) appear only as baselines, related work, or loss references and do not carry the central open-vocabulary claim. The one residual concern is benchmark validity rather than derivation circularity: DiSG's 86-class taxonomy is derived from COCO plus six body-part classes, and the Grounding DINO backbone's pretraining data overlaps in vocabulary; the authors explicitly avoid the supervised-COCO checkpoint to control this. That caveat does not make any equation or prediction equivalent to its input by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The paper introduces no novel physical or theoretical entities. Its new artifacts are a dataset (DiSG) and a tuning heuristic (GIST); the dataset is an empirical resource and GIST is accounted for in free_parameters and axioms, so invented_entities is empty.

free parameters (5)
  • GIST top-k selection ratio = 40%
    Chosen by scanning 20/40/60/100 in Table 3 and picking the value with the best across OVGOP, OVOD, and gaze metrics; the scan uses novel-category metrics, so it is tuned on the held-out split.
  • Loss weights λ_cls, λ_L1, λ_giou, λ_gb, λ_hm = not reported
    Equation (4) defines a multi-task loss with balancing weights, but no numerical values are given in main text or supplement; the training balance of detection vs. gaze losses is a free choice affecting all results.
  • GIST probing length = 5 epochs
    Justified by the probing-length sweep in Supplementary Fig. 2; 5 is chosen as a good compromise between novel GOP and AUC.
  • Head-crop enlargement factor = 1.3
    Input preprocessing in Sec. 5.1: head crops are enlarged by 1.3 before resizing to 224x224; chosen by hand and not ablated.
  • Body-part geometric proportions and part-detector confidence threshold = predefined proportions; unspecified threshold
    Supplementary A.1 uses “predefined body-part proportions” for skeleton boxes and retains “high-confidence” part-detector predictions, but the proportions and threshold are not specified.
axioms (6)
  • ad hoc to paper GIST sensitivity hypothesis: parameters with high bias-corrected second-moment estimates are task-sensitive, while stable parameters encode task-agnostic semantic knowledge.
    The core justification for GIST in Sec. 4.2 is a heuristic that high gradient variance indicates gaze-domain adaptation and low variance indicates preserved knowledge; no proof or controlled causal test is provided beyond Table 3.
  • ad hoc to paper Candidate-pool filter: only boxes that spatially enclose the gaze point are offered to annotators as possible gaze targets.
    Supplementary A.1: the candidate pool is filtered to boxes enclosing the ground-truth gaze point before human verification; this structurally limits the benchmark's label space and ties ground truth to a pre-existing object inventory.
  • domain assumption DiSG base/novel split aligns with the standard COCO base/novel split, and the chosen Grounding DINO checkpoint gives no box-level supervision on novel categories.
    Supplementary section 'Clarification of novel classes' claims the checkpoint follows the protocol and novel classes receive no box supervision during pretraining or DiSG training; the 57/23 split is not the common 48/17 COCO split, and GoldG/O365 may contain related supervision, so the assumption is unverified.
  • domain assumption The COCO and GazeFollow intersection is representative of diverse in-the-wild gaze scenes.
    DiSG is entirely assembled from existing collections and never collects new scene imagery; generality to truly unconstrained scenes is assumed rather than demonstrated.
  • domain assumption The off-the-shelf part detector [59] gives reliable head/hand/feet boxes for gaze-target annotation.
    Small-part candidates come from detector outputs with a confidence threshold; errors in the part detector propagate into the benchmark's fine-grained body-part labels.
  • domain assumption The decoupling of semantic candidate discovery and gaze-heatmap selection is sufficient to identify the attended object.
    The framework in Sec. 4.1 assumes that the gaze heatmap plus candidate boxes captures target selection; failure cases in Sec. G (ambiguous gaze, crowded scenes, tiny objects) show this assumption has limits.

pith-pipeline@v1.3.0-alltime-deepseek · 20207 in / 12968 out tokens · 118722 ms · 2026-08-01T14:10:56.312080+00:00 · methodology

0 comments
read the original abstract

Gaze Object Prediction (GOP) aims to localize and recognize the objects humans attend to, a task crucial for understanding human-centric interactions. However, existing methods are typically trained under a closed-vocabulary paradigm with a fixed label space and evaluated on scene-specific datasets, limiting their applicability to real-world scenarios where gaze targets often follow a long-tail distribution or belong to unseen categories. To address this gap, we introduce Diverse Scenes for Gaze object prediction (DiSG), a benchmark containing 86 in-the-wild categories that facilitates the evaluation of Open-Vocabulary GOP (OVGOP). Building on DiSG, we propose a framework that leverages text-driven object discovery to localize potential gaze candidates, with a gaze-guided selection module to pinpoint the intended target from the candidate objects. Furthermore, to better capture semantic knowledge across diverse in-the-wild categories, we introduce Gradient-Informed Selection Tuning (GIST) to selectively update parameters most relevant to a given class vocabulary. Extensive experiments demonstrate that our proposed model performs effectively in open-vocabulary settings and also outperforms existing methods in the conventional closed-vocabulary setting. The benchmark and code is available at https://github.com/sensniu/ovgop.

Figures

Figures reproduced from arXiv: 2607.18827 by Binglu Wang, Guangyu Guo, Sensen Niu, Ying Chen.

Figure 1
Figure 1. Figure 1: (a) Existing GOP methods operate under a fixed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DiSG benchmark and its open-vocabulary split. (a–b) Comparison with existing GOP benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our OVGOP framework. Given an input image, head locations, and a free-form vocabulary, the discovery [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of Gradient-Informed Selection Tuning (GIST). GIST adopts a two-stage, layer-wise tuning strategy for the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of predictions on DiSG. Training setup. We train the model for 60 epochs with batch size 16 and learning rate 1 × 10−4 . GIST is implemented with a two￾stage schedule: we first run a 5-epoch probing stage to estimate parameter sensitivity, then select the top 40% parameters in each transformer layer and retrain the model for 60 epochs with only the selected parameters updated. Vocabulary and … view at source ↗
Figure 6
Figure 6. Figure 6: Layer-wise sensitivity capture of GIST across param [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 1
Figure 1. Figure 1: The annotation pipeline for the DiSG dataset. Our process begins with data initialization from COCO and GazeFollow. [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effect of the probing length and training scheme in [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative OVGOP results of our method on DiSG. Top: predictions on base categories. Bottom: predictions on novel [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 10 linked inside Pith

  1. [1]

    Maria A Bravo, Sudhanshu Mittal, and Thomas Brox. 2022. Localized vision- language matching for open-vocabulary object detection. InDAGM German conference on pattern recognition. Springer, 393–408

    2022

  2. [2]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. InEur. Conf. Comput. Vis.Springer, 213–229

    2020

  3. [3]

    Dazhi Chen and Gang Gou. 2024. Towards Adapting CLIP for Gaze Object Prediction.Proceedings of the ACM on Human-Computer Interaction8, ISS (2024), 155–164

    2024

  4. [4]

    Yihua Cheng and Feng Lu. 2022. Gaze estimation using transformer. InInt. Conf. Pattern Recog.3341–3347

    2022

  5. [5]

    Hojun Choi, Youngsun Lim, Jaeyo Shin, and Hyunjung Shim. 2025. CoT-PL: Visual Chain-of-Thought Reasoning Meets Pseudo-Labeling for Open-Vocabulary Object Detection.arXiv preprint arXiv:2510.14792(2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Eunji Chong, Yongxin Wang, Nataniel Ruiz, and James M Rehg. 2020. Detecting attended visual targets in video. InIEEE Conf. Comput. Vis. Pattern Recog.5396– 5406

    2020

  7. [7]

    Alessio Devoto, Federico Alvetreti, Jary Pomponi, Paolo Di Lorenzo, Pasquale Minervini, and Simone Scardapane. 2024. Adaptive layer selection for efficient vision transformer fine-tuning.arXiv preprint arXiv:2408.08670(2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Yi Fang, Jiapeng Tang, Wang Shen, Wei Shen, Xiao Gu, Li Song, and Guangtao Zhai. 2021. Dual Attention Guided Gaze Target Detection in the Wild. InIEEE Conf. Comput. Vis. Pattern Recog.11390–11399

    2021

  9. [9]

    Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. 2021. Yolox: Exceeding yolo series in 2021.arXiv preprint arXiv:2107.08430(2021)

    Pith/arXiv arXiv 2021

  10. [10]

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.580–587

    2014

  11. [11]

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. 2021. Open-vocabulary Object Detection via Vision and Language Knowledge Distillation. InInt. Conf. Learn. Represent.https://api.semanticscholar.org/CorpusID:238744187

    2021

  12. [12]

    Guangyu Guo, Chenxi Guo, Zhaozhong Wang, and Binglu Wang. 2025. TransGOP- R: Transformer-based Real-World Gaze Object Prediction.IEEE Trans. Multimedia (2025), 1–15

    2025

  13. [13]

    Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019. Lvis: A dataset for large vocabulary instance segmentation. InIEEE Conf. Comput. Vis. Pattern Recog. 5356–5364

    2019

  14. [14]

    Yang Jin, Lei Zhang, Shi Yan, Bin Fan, and Binglu Wang. 2024. Boosting Gaze Object Prediction via Pixel-Level Supervision from Vision Foundation Model. In Eur. Conf. Comput. Vis.Springer, 369–386

    2024

  15. [15]

    Gal Kaplun, Andrey Gurevich, Tal Swisa, Mazor David, Shai Shalev-Shwartz, and Eran Malach. 2023. Less is more: Selective layer finetuning with subtuning.arXiv preprint arXiv:2302.06354(2023)

    Pith/arXiv arXiv 2023

  16. [16]

    Dahun Kim, Anelia Angelova, and Weicheng Kuo. 2023. Region-aware pretraining for open-vocabulary object detection with vision transformers. InIEEE Conf. Comput. Vis. Pattern Recog.11144–11154

    2023

  17. [17]

    Dahun Kim, Anelia Angelova, and Weicheng Kuo. 2024. Detection-oriented image-text pretraining for open-vocabulary detection. (2024). https://openreview. net/forum?id=wHlYlC2TCM

    2024

  18. [18]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.Int. J. Comput. Vis.128, 7 (2020), 1956–1981

    2020

  19. [19]

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-image pre-training. InIEEE Conf. Comput. Vis. Pattern Recog. 10965–10975

    2022

  20. [20]

    Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, and Masashi Sugiyama

  21. [21]

    Dongze Lian, Zehao Yu, and Shenghua Gao. 2018. Believe it or not, we know what you are looking at!. InACCV. Springer, 35–50

    2018

  22. [22]

    Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. 2022. Learning object-language alignments for open-vocabulary object detection.arXiv preprint arXiv:2211.14843(2022)

    Pith/arXiv arXiv 2022

  23. [23]

    Jianghang Lin, Yunhang Shen, Bingquan Wang, Shaohui Lin, Ke Li, and Liujuan Cao. 2024. Weakly supervised open-vocabulary object detection. InAAAI, Vol. 38. 3404–3412

    2024

  24. [24]

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. InInt. Conf. Comput. Vis.2980–2988

    2017

  25. [25]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. InEur. Conf. Comput. Vis.Springer, 740–755

    2014

  26. [26]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEur. Conf. Comput. Vis.Springer, 38–55

    2024

  27. [27]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. InInt. Conf. Comput. Vis.10012–10022

    2021

  28. [28]

    2025.GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding

    Athul M Mathew, Haithem Hermassi, Thariq Khalid, Arshad Ali Khan, and Riad Souissi. 2025.GazeVLM: A Vision-Language Model for Multi-Task Gaze Understanding. arXiv:2511.06348 doi:10.48550/arXiv.2511.06348

    doi:10.48550/arxiv.2511.06348 2025

  29. [29]

    Athul M Mathew, Arshad Ali Khan, Thariq Khalid, and Riad Souissi. 2024. GESCAM: A Dataset and Method on Gaze Estimation for Classroom Attention Measurement. InIEEE Conf. Comput. Vis. Pattern Recog.636–645

    2024

  30. [30]

    Qiaomu Miao, Minh Hoai, and Dimitris Samaras. 2023. Patch-Level Gaze Distri- bution Prediction for Gaze Following. InW ACV. 880–889

    2023

  31. [31]

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs.Adv. Neural Inform. Process. Syst. 24 (2011)

    2011

  32. [32]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. 8748–8763

    2021

  33. [33]

    Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. 2015. Where are they looking?Advances in neural information processing systems28 (2015)

    2015

  34. [35]

    Adria Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. 2017. Fol- lowing gaze in video. InInt. Conf. Comput. Vis.1435–1443

    2017

  35. [36]

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. InIEEE Conf. Comput. Vis. Pattern Recog.779–788

    2016

  36. [37]

    Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767(2018)

    Pith/arXiv arXiv 2018

  37. [38]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks.Advances in neural information processing systems28 (2015)

    2015

  38. [39]

    Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman, and James M Rehg. 2025. Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders. InIEEE Conf. Comput. Vis. Pattern Recog.28874–28884

    2025

  39. [40]

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019. Objects365: A large-scale, high-quality dataset for object detection. InInt. Conf. Comput. Vis.8430–8439

    2019

  40. [41]

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2556–2565

    2018

  41. [42]

    Hwanjun Song and Jihwan Bang. 2023. Prompt-guided transformers for end-to- end open-vocabulary object detection.arXiv preprint arXiv:2303.14386(2023). MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Binglu Wang, Sensen Niu, Ying Chen, and Guangyu Guo

    Pith/arXiv arXiv 2023

  42. [43]

    Amelia Sorrenti, Giovanni Bellitto, Federica Proietto Salanitri, Matteo Pennisi, Concetto Spampinato, and Simone Palazzo. 2023. Selective freezing for efficient continual learning. InInt. Conf. Comput. Vis.3550–3559

    2023

  43. [44]

    Samy Tafasca, Anshul Gupta, Victor Bros, and Jean-Marc Odobez. 2024. Toward semantic gaze target detection.Advances in neural information processing systems 37 (2024), 121422–121448

    2024

  44. [45]

    Samy Tafasca, Anshul Gupta, and Jean-Marc Odobez. 2023. Childplay: A new benchmark for understanding children’s gaze behaviour. InInt. Conf. Comput. Vis.20935–20946

    2023

  45. [46]

    Henri Tomas, Marcus Reyes, Raimarc Dionido, Mark Ty, Jonric Mirando, Joel Casimiro, Rowel Atienza, and Richard Guinto. 2021. Goo: A dataset for gaze object prediction in retail environments. InIEEE Conf. Comput. Vis. Pattern Recog. Worksh.3125–3133

    2021

  46. [47]

    Mao Tomita, Ikuro Sato, Rei Kawakami, Nakamasa Inoue, Satoshi Ikehata, and Masayuki Tanaka. 2024. A Simple Finetuning Strategy Based on Bias-Variance Ratios of Layer-Wise Gradients. InACCV. 471–487

    2024

  47. [48]

    Francesco Tonini, Cigdem Beyan, and Elisa Ricci. 2022. Multimodal across domains gaze target detection. InProceedings of the 2022 International Conference on Multimodal Interaction. 420–431

    2022

  48. [49]

    Francesco Tonini, Nicola Dall’Asen, Cigdem Beyan, and Elisa Ricci. 2023. Object- aware gaze target detection. InInt. Conf. Comput. Vis.21860–21869

    2023

  49. [50]

    Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. 2022. End-to-end human-gaze-target detection with transformers. In IEEE Conf. Comput. Vis. Pattern Recog.2192–2200

    2022

  50. [51]

    Danyang Tu, Wei Shen, Wei Sun, Xiongkuo Min, and Guangtao Zhai. 2023. Joint gaze-location and gaze-object detection.arXiv preprint arXiv:2308.13857(2023)

    Pith/arXiv arXiv 2023

  51. [52]

    Danyang Tu, Wei Shen, Wei Sun, Xiongkuo Min, Guangtao Zhai, and Changwen Chen. 2023. Un-Gaze: A unified transformer for joint gaze-location and gaze- object detection.IEEE Transactions on Circuits and Systems for Video Technology 34, 5 (2023), 3271–3285

    2023

  52. [53]

    Binglu Wang, Chenxi Guo, Yang Jin, Haisheng Xia, and Nian Liu. 2024. Transgop: Transformer-based gaze object prediction. InAAAI, Vol. 38. 10180–10188

    2024

  53. [54]

    Binglu Wang, Tao Hu, Baoshan Li, Xiaojuan Chen, and Zhijie Zhang. 2022. Gatector: A unified framework for gaze object prediction. InIEEE Conf. Comput. Vis. Pattern Recog.19588–19597

    2022

  54. [55]

    Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang, Xiangyuan Lan, and Xiaodan Liang. 2024. OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion.arXiv preprint arXiv:2407.07844(2024)

    Pith/arXiv arXiv 2024

  55. [56]

    Jun Wang, Hao Ruan, Mingjie Wang, Chuanghui Zhang, Huachun Li, and Jun Zhou. 2023. Gazeclip: Towards enhancing gaze estimation via text guidance. arXiv preprint arXiv:2401.00260(2023)

    Pith/arXiv arXiv 2023

  56. [57]

    Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. 2023. V3det: Vast vocabulary visual detection dataset. InInt. Conf. Comput. Vis.19844–19854

    2023

  57. [58]

    Yifan Xu, Mengdan Zhang, Xiaoshan Yang, and Changsheng Xu. 2024. Exploring multi-modal contextual knowledge for open-vocabulary object detection.IEEE Trans. Image Process.(2024)

    2024

  58. [59]

    Lu Yang, Qing Song, Zhihui Wang, Mengjie Hu, and Chun Liu. 2021. Hier R-CNN: Instance-Level Human Parts Detection and A New Benchmark.IEEE Transactions on Image Processing30 (2021), 39–54

    2021

  59. [60]

    Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. 2023. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. InIEEE Conf. Comput. Vis. Pattern Recog.23497– 23506

    2023

  60. [61]

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. 2022. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.Advances in Neural Information Processing Systems35 (2022), 9125–9138

    2022

  61. [62]

    Pengwei Yin, Jingjing Wang, Guanzhong Zeng, Di Xie, and Jiang Zhu. 2024. Lg- gaze: Learning geometry-aware continuous prompts for language-guided gaze estimation. InEur. Conf. Comput. Vis.Springer, 1–17

    2024

  62. [63]

    Pengwei Yin, Guanzhong Zeng, Jingjing Wang, and Di Xie. 2024. Clip-gaze: Towards general gaze estimation via visual-linguistic model. InAAAI, Vol. 38. 6729–6737

    2024

  63. [64]

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. 2022. Open-vocabulary detr with conditional matching. InEur. Conf. Comput. Vis. Springer, 106–122

    2022

  64. [65]

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. 2021. Open- vocabulary object detection using captions. InIEEE Conf. Comput. Vis. Pattern Recog.14393–14402

    2021

  65. [66]

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. 2023. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. InInt. Conf. Learn. Represent

    2023

  66. [67]

    Yi Zhang, Yi-Xuan Deng, Meng-Hao Guo, and Shi-Min Hu. 2025. Adaptive Parameter Selection for Tuning Vision-Language Models. InIEEE Conf. Comput. Vis. Pattern Recog.4280–4290

    2025

  67. [68]

    Henry Hengyuan Zhao, Pichao Wang, Yuyang Zhao, Hao Luo, Fan Wang, and Mike Zheng Shou. 2024. Sct: A simple baseline for parameter-efficient fine-tuning via salient channels.Int. J. Comput. Vis.132, 3 (2024), 731–749

    2024

  68. [69]

    Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anas- tasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas. 2022. Ex- ploiting unlabeled data with vision and language models for object detection. In Eur. Conf. Comput. Vis.Springer, 159–175

    2022

  69. [70]

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liu- nian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. 2022. Regionclip: Region-based language-image pretraining. InIEEE Conf. Comput. Vis. Pattern Recog.16793–16803

    2022

  70. [71]

    {class}

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. Deformable detr: Deformable Transformers for End-to-End Object Detection. In Int. Conf. Learn. Represent. Open-Vocabulary Gaze Object Prediction: Benchmark and Method MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil A Detailed Construction of the DiSG Dataset This supplement...

    2021

  71. [2024]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Vision-language model fine-tuning via simple parameter-efficient modi- fication. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 14394–14410

    2024