Pith. sign in

REVIEW 3 major objections 4 minor 89 references

By training a multimodal language model to read gaze scanpaths as text, this paper claims the first generative gaze-to-text framework, using a 'think-aloud' auxiliary objective to lift decoding performance across visual search, object refer

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:31 UTC pith:VJ6R2CTT

load-bearing objection A genuinely new generative gaze-to-text task with a plausible GPT-4 auxiliary objective, but the core disentanglement claim needs a same-image/different-goal check and the headline numbers need error bars. the 3 major comments →

arxiv 2607.23917 v1 pith:VJ6R2CTT submitted 2026-07-27 cs.CV

Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention

Sounak Mondal , Dimitris Samaras , Gregory Zelinsky , Minh Hoai This is my paper
classification cs.CV
keywords gaze decodingscanpathgoal inferencemultimodal large language modelinstruction tuningthink-aloud transcriptsvisual searchvisual question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that gaze decoding need not be restricted to choosing among pre-set categories: a model can read an image and a scanpath and generate free-form text describing what a person is looking for, whether a search target, a referring expression, or a question. It proposes Gazette, a multimodal large language model fine-tuned for this generative task, and an auxiliary 'think-aloud' objective that trains the model to describe the common attention strategy shared by multiple people doing the same task. The central claim is that this auxiliary objective helps the model separate goal-driven attention from individual scanpath differences, yielding gains on object referral, VQA, and visual search. If true, eye tracking becomes a practical, non-intrusive route to open-ended human intention inference.

Core claim

The paper introduces the task of decoding a gaze scanpath into a natural-language description of the human's cognitive context: a coarse behavior type (visual search, object referral, or VQA) plus a finer stimulus such as a search target, a referring expression, or a question. It claims the first generative gaze-to-text model, Gazette, built as a multimodal large language model that takes an image and a textualized scanpath and autoregressively generates structured text. To handle weak supervision—scanpaths are confounded by individual differences—it generates think-aloud transcripts by prompting a large language model to summarize common spatiotemporal patterns across multiple participants'

What carries the argument

The central object is the think-aloud transcript: a pseudo-label narrative stating the common attention allocation strategy across observers who share the same image and goal, with three idea-units—attention-allocation explanation, target localization, and scanpath length. The load-bearing mechanism is the invariance assumption that, for a fixed image and goal, a scanpath equals a goal-driven component plus participant-specific noise, so the common spatiotemporal pattern across participants isolates the goal signal. A large language model is prompted with multiple scanpaths plus scene object boxes to produce these transcripts, and Gazette is instruction-tuned on both the primary GazeDec task

Load-bearing premise

The claim depends on the assumption that when several people look at the same image with the same goal, the pattern they share is the goal-driven signal, and that a large language model can extract it from scanpath coordinates plus object boxes rather than encoding dataset-specific regularities or majority behavior.

What would settle it

Train the same pipeline but replace the goal labels with randomly shuffled or synthetic goals when prompting the large language model for transcripts; if decoding performance on a held-out goal set stays near the reported gains, the transcripts are encoding dataset regularities rather than goal-driven attention, while a drop to baseline would confirm they carry goal information.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Gaze becomes a language-like input: a scanpath plus an image can map to free-form goals, removing the need for a fixed category set.
  • For object referral, decoding the full scanpath with the auxiliary objective outperforms using only the final fixation, suggesting the temporal path itself carries goal information.
  • The auxiliary think-aloud objective transfers across three task families, indicating the method is not tied to a single gaze behavior.
  • Behavior-type prediction is nearly trivial on the pooled datasets, so the paper's substantive contribution is fine-grained stimulus decoding.
  • Target-absent visual search gains are modest and mixed, consistent with low cross-observer agreement when no target exists.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the shared-pattern assumption holds, the same transcript-generation recipe could be applied to other temporal behavioral signals—mouse movements, touch, or instructional video watching—to infer goals.
  • A direct testable extension would replace the large-language-model transcript writer with human think-aloud protocols from a few observers; similar gains would indicate the method depends on commonality itself, not on the specific generator.
  • The reliance on majority patterns implies the decoded goal is the typical human goal rather than an individual's idiosyncratic goal; applications needing personalized intent may require a different objective.
  • The method could enable gaze-driven assistive systems that name objects or read questions aloud, but only in tasks where observer agreement is high; low-agreement tasks will need an additional signal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Gazette, an MLLM-based framework for decoding human gaze scanpaths into free-form natural-language descriptions of the underlying goal, framed as a generative alternative to prior categorical gaze-decoding methods. The proposed training pipeline combines the primary GazeDec instruction-tuning objective with an auxiliary ThinkAloud objective: GPT-4 is prompted with multiple scanpaths collected under the same image and goal, along with scene-object bounding boxes, to synthesize a common 'attention allocation strategy' transcript, which is then used as pseudo-label supervision. Gazette is evaluated on object referral (RefCOCO-Gaze), VQA (AiR-D), and visual search (COCO-Search18), with comparisons against LLaVA-based baselines and prior discriminative methods. The central claim is that the auxiliary think-aloud objective improves gaze decoding substantially and consistently, enabling open-ended goal inference from a single scanpath at test time.

Significance. If the claims hold, the paper makes a useful contribution: it opens a new problem formulation (generative gaze-to-text) and demonstrates a practical way to inject attention-allocation knowledge into an MLLM without human transcript annotations. The paper has clear strengths: the framework is simple and reproducible (code is promised), the ablations in Tables 4–7 dissect each idea unit of the think-aloud transcript and show task-dependent contributions, the human evaluation of pseudo-label quality in Supplement III.2 is a welcome check, and the use of classic ideal-observer/cognitive-relevance theory as motivation is appropriate. However, the central validation is incomplete. The discriminative power of the pseudo-labels for goals, rather than scene-layout regularities, is not directly tested, and the main empirical claims rest on point estimates without uncertainty quantification. The same LLM family used to generate the auxiliary labels is also used as the judge in a key evaluation paradigm, so the 'external' evaluation is only partially external.

major comments (3)
  1. [Sec. 3.2 and Supplement III.2] The disentanglement assumption S_i = f(I_i,D_i)+U_i is operationalized by prompting GPT-4 to summarize common patterns across scanpaths sharing image and goal. This is load-bearing: if the common component reflects scene layout, dataset artifacts, or majority fixation behavior rather than the goal, the auxiliary objective injects biased supervision. The human evaluation in Supplement III.2 compares positive samples against scanpaths from 'other image-stimulus pairs', which does not control for the critical same-image/different-goal case. Since many images in the three datasets have multiple goals, the authors should add a negative condition with the same image and a different goal, and/or a control experiment measuring whether GPT-4-generated transcripts (without the goal label) can be matched to the correct goal above chance. Without this, the paper's own Limitations admission that tran
  2. [Tables 1, 2, and 3] The paper repeatedly states that Gazette 'significantly outperforms' the w/o ThinkAloud variant and prior baselines, but no error bars, confidence intervals, or significance tests are reported. Differences are point estimates on one test split. Given that scanpath-level variability across participants is likely substantial, and some improvements (e.g., Target-Absent Precision 0.438 vs 0.437, F1 0.424 vs 0.420) are very small, the main comparative claim is not statistically supported. Please provide bootstrap or per-participant confidence intervals for the headline comparisons, or at least multi-seed training runs with variance reporting.
  3. [Sec. 4.1, LLM-as-a-Judge] The evaluation paradigm uses GPT-4 as the judge for object-referral and VQA outputs. The same GPT-4 family generated the think-aloud pseudo-labels used to train Gazette. The paper states this 'avoids any circular dependency via GPT-4 that may bias evaluation' because the judge compares against ground-truth expressions/questions, but the concern is not fully addressed: GPT-4 may systematically favor outputs whose style or content resembles its own generated text, and the judge prompt supplies only scene context, not the scanpath. A human evaluation of the final decoded referring expressions/questions, or an alternative judge from a different model family, would substantially strengthen the 'rigorous evaluation scheme' claim.
minor comments (4)
  1. [Sec. 4.2 / Abstract] The abstract and text claim 'state-of-the-art performance', but in Table 3 Target-Present Accuracy Gazette is 0.773 versus Mondal et al.'s 0.776, which is slightly below, and Target-Absent improvements are modest. Please temper the SOTA claim or clarify the specific metrics on which Gazette is SOTA.
  2. [Supplement III.1] The prompt templates contain formatting artifacts, e.g., 'Given a scanpath }[($x_0, ...' and ' exttt{, which is a list ...'. These should be cleaned for reproducibility.
  3. [References] References [77] and [78] appear to be the same paper (same title, different venue strings). Please deduplicate.
  4. [Table 1] The underline convention ('Results exceeding RefCOCO-IAC') is not visually applied to all qualifying values; please check that all underlined entries actually exceed the IAC row and clarify in the caption.

Circularity Check

0 steps flagged

No circularity: the auxiliary GPT-4 pseudo-labeling is supervised distillation, not a self-fulfilling prediction; evaluations are held-out and grounded in human-annotated targets.

full rationale

The paper's derivation chain is not circular in the defined sense. The primary training objective is standard instruction tuning from human-annotated (image, scanpath, goal) triplets, with held-out test labels for evaluation. The ThinkAloud transcripts are produced from training-set scanpaths and the corresponding ground-truth goals; using them as auxiliary supervision is a distillation/multi-task strategy, not a fitted parameter being renamed as a prediction. The main reported gains (Tables 1 and 3) are measured against held-out human referring expressions/questions and COCO-Search18 target categories, not against the GPT-4-generated transcripts. The LLM-as-a-Judge evaluation (Table 2) is explicitly separated from the ThinkAloud transcripts and is grounded in ground-truth expressions and object bounding boxes; while GPT-4's dual role is a mild evaluation-bias concern, it does not reduce the reported prediction to the input by construction. The cross-observer convergence premise is an empirical/domain assumption supported by external citations (Yarbus; Najemnik and Geisler; Hayhoe and Ballard), not by a self-citation chain. The paper's own Limitations section acknowledges that transcripts inherit GPT-4 priors and degrade under low inter-observer agreement; these are robustness/correctness concerns, not circular steps.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The paper's central claim depends primarily on the domain assumption that shared scanpath patterns across participants encode the goal-driven signal, and on the tooling assumption that GPT-4 can extract that signal from bounding-box-annotated scene context and raw fixations. The training hyperparameters are standard and not load-bearing. No new physical entities are introduced; the only invented construct is the synthetic think-aloud transcript, which has partial independent evidence.

free parameters (4)
  • LoRA rank = 64
    Chosen for MLLM fine-tuning; not fitted to the test set, but not ablated.
  • learning rate = 2e-5
    Standard instruction-tuning LR; no sweep reported, so the result may depend on this choice.
  • training epochs = 3
    Maximum epochs; no early-stopping or epoch sweep reported.
  • batch size = 32
    Chosen for two RTX A6000 GPUs; not central to the claim.
axioms (4)
  • domain assumption For a fixed image and goal, scanpaths from different participants share an invariant goal-driven component that can be separated from individual variation.
    Sec. 3.2 formalizes S_i = f(I_i,D_i)+U_i and assumes group invariance; used to justify GPT-4 transcript generation. Supported by citations to Yarbus, Najemnik & Geisler, and Hayhoe & Ballard, but not proven.
  • ad hoc to paper GPT-4's summary of common spatiotemporal patterns across scanpaths is a valid pseudo-label for top-down attention allocation.
    Sec. 3.2: 'we hypothesize indicates the top-down attentional allocation strategies'; human evaluation shows consistency but not ground truth, and the authors acknowledge GPT-4 training priors may bias transcripts.
  • domain assumption COCO/Visual Genome bounding-box annotations provide sufficient scene context for GPT-4 to relate fixations to objects and for the model to generate goals.
    Sec. 3.2 and prompt templates in Sec. III.2 use scene object bounding boxes as the only scene representation for GPT-4; no ablation tests the sensitivity to this representation.
  • domain assumption MiniLM cosine matching between generated text and a fixed label vocabulary correctly maps free-form outputs to the 18 COCO-Search18 categories and 4 behavior types.
    Sec. 3.3 and Supplement IV describe this matching; no error analysis is provided for cases where the model generates a valid description that does not lexically match a label.
invented entities (1)
  • Think-aloud transcripts independent evidence
    purpose: Auxiliary supervision for instruction tuning to disentangle goal-driven gaze from individual variability.
    Human evaluation (Sec. III.2) shows positive scanpaths rated higher than negative ones (3.28 vs 2.04 on a 1–5 scale), but the transcripts are synthetic, not actual participant verbal protocols, and the ratings are modest.

pith-pipeline@v1.3.0-alltime-deepseek · 22959 in / 11651 out tokens · 119664 ms · 2026-07-31T23:31:12.177354+00:00 · methodology

0 comments
read the original abstract

We introduce a novel learning problem: decoding gaze into natural language descriptions of human goals across diverse visual tasks. Unlike prior work, which frames gaze decoding as a discriminative task over predefined categories, we formulate it as a generative learning problem: training a model to produce free-form descriptions that capture the rich nuances and open-ended nature of human intentions beyond fixed labels. To this end, we introduce Gazette, the first gaze-to-text decoding framework. Based on multimodal large language models (MLLMs), Gazette learns to decode gaze scanpaths into natural language for goals that may extend beyond categorical labels and require articulation in natural language. To help Gazette filter out individual differences in gaze behavior and learn the goal-specific spatiotemporal dynamics crucial for generating accurate natural language goal descriptions, we propose a novel strategy that leverages the encyclopedic knowledge and reasoning abilities of a large language model to synthesize natural language explanations of goal-directed attentional behavior called think-aloud transcripts. Instruction tuning on these synthetic narratives allows Gazette to achieve state-of-the-art performance in gaze decoding across multiple tasks, demonstrating its generalizability and versatility, thereby enabling gaze to serve as a powerful, non-intrusive cue for inferring human goals and intentions in diverse scenarios.

Figures

Figures reproduced from arXiv: 2607.23917 by Dimitris Samaras, Gregory Zelinsky, Minh Hoai, Sounak Mondal.

Figure 1
Figure 1. Figure 1: Gazette: A text-generative decoding framework for top-down attention. For an input of an image and a language instruction conveying scanpath information, a textual response is generated by the Multimodal LLM comprising a Vision Encoder, an LLM, and a linear projection θproj interfacing the Vision Encoder and the LLM. Language instruction can correspond to either the primary gaze decoding task GazeDec, or t… view at source ↗
Figure 2
Figure 2. Figure 2: A novel prompting strategy for GPT-4 to extract top-down attention allocation explanation for think-aloud transcript generation task (ThinkAloud). (a) This strategy instructs GPT-4 to summarize common spatiotemporal patterns in scanpaths of n participants, given the task type and goal, along with scene information via scene object bounding boxes from COCO [41] and Visual Genome [35]. The response is used t… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Results. Comparison of Gazette variants on decoding a gaze scanpath for Object Referral target “black car on right” (blue bounding box). We show predictions from Gazette and its variant not trained on ThinkAloud, along with the attention allocation explanation from the Gazette-generated think-aloud transcript. we show that similar to Object Referral, both target localization and attention alloc… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative Results [1/3]. Comparison of methods on decoding a gaze scanpath corresponding to Visual Question Answering for the question “Is the window behind the chair near the pillows?”. We provide model predictions from full model Gazette, and its variant not trained on ThinkAloud instructions. We also provide the attention allocation explanation idea unit in the Gazette-generated think￾aloud transcript… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative Results [2/3]. Comparison of methods on decoding a gaze scanpath corresponding to Target-Present Search for a “TV”. We provide model pre￾dictions from full model Gazette, and its variant not trained on ThinkAloud instructions. We also provide the attention allocation explanation idea unit in the Gazette-generated think-aloud transcript [PITH_FULL_IMAGE:figures/full_fig_p025_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative Results [3/3]. Comparison of methods on decoding a gaze scan￾path corresponding to Target-Absent Search for a “bowl”. We provide model predic￾tions from full model Gazette, and its variant not trained on ThinkAloud instructions. We also provide the attention allocation explanation idea unit in the Gazette-generated think-aloud transcript [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 11 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Attention, Perception, & Psychophysics79, 1415–1425 (2017)

    Bahle, B., Mills, M., Dodd, M.D.: Human classifier: Observers can deduce task solely from eye movements. Attention, Perception, & Psychophysics79, 1415–1425 (2017)

    2017

  3. [3]

    In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization

    Banerjee, S., Lavie, A.: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics, Ann Arbor, Michigan (2005)

    2005

  4. [4]

    In: ACM Symposium on Eye Tracking Research and Applications (2020)

    Barz, M., Stauden, S., Sonntag, D.: Visual search target inference in natural interaction settings with machine learning. In: ACM Symposium on Eye Tracking Research and Applications (2020)

    2020

  5. [5]

    Computers & Graphics119, 103909 (2024)

    Bektas, K., Strecker, J., Mayer, S., Garcia, K.: Gaze-enabled activity recognition for augmented reality feedback. Computers & Graphics119, 103909 (2024)

    2024

  6. [6]

    arXiv preprint arXiv:2310.19812 (2023)

    Benchetrit, Y., Banville, H., King, J.R.: Brain decoding: toward real-time recon- struction of visual perception. arXiv preprint arXiv:2310.19812 (2023)

    Pith/arXiv arXiv 2023

  7. [7]

    Journal of Vision14(3), 29–29 (2014)

    Borji, A., Itti, L.: Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision14(3), 29–29 (2014)

    2014

  8. [8]

    Neurocomputing149, 788– 799 (2015)

    Borji, A., Lennartz, A., Pomplun, M.: What do eyes reveal about the mind?: Algorithmic inference of search targets from fixations. Neurocomputing149, 788– 799 (2015)

    2015

  9. [9]

    Proceedings of the ACM on Human-Computer Interaction8(ETRA) (2024)

    Bühler, B., Bozkir, E., Deininger, H., Gerjets, P., Trautwein, U., Kasneci, E.: On task and in sync: Examining the relationship between gaze synchrony and self-reported attention during video lecture learning. Proceedings of the ACM on Human-Computer Interaction8(ETRA) (2024)

    2024

  10. [10]

    IEEE transactions on pattern analysis and machine intelligence33(4), 741–753 (2010)

    Bulling, A., Ward, J.A., Gellersen, H., Tröster, G.: Eye movement analysis for activity recognition using electrooculography. IEEE transactions on pattern analysis and machine intelligence33(4), 741–753 (2010)

    2010

  11. [11]

    arXiv preprint arXiv:2503.00034 (2025)

    Cai, H., Fu, Y., Fu, H., Zhao, B.: MergeIT: From selection to merging for efficient instruction tuning. arXiv preprint arXiv:2503.00034 (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    In: European Conference on Computer Vision (2020)

    Chen, S., Jiang, M., Yang, J., Zhao, Q.: AiR: Attention with reasoning capability. In: European Conference on Computer Vision (2020)

    2020

  13. [13]

    European Conference on Computer Vision (2024)

    Chen, X., Jiang, M., Zhao, Q.: GazeXplain: Learning to predict natural language explanations of visual scanpaths. European Conference on Computer Vision (2024)

    2024

  14. [14]

    Scientific reports 11(1), 8776 (2021)

    Chen, Y., Yang, Z., Ahn, S., Samaras, D., Hoai, M., Zelinsky, G.: COCO-Search18 fixation dataset for predicting goal-directed attention control. Scientific reports 11(1), 8776 (2021)

    2021

  15. [15]

    In: Proceedings of CVPR International Workshop on Gaze Estimation and Prediction in the Wild (2022)

    Chen, Y., Yang, Z., Chakraborty, S., Mondal, S., Ahn, S., Samaras, D., Hoai, M., Zelinsky, G.: Characterizing target-absent human attention. In: Proceedings of CVPR International Workshop on Gaze Estimation and Prediction in the Wild (2022)

    2022

  16. [16]

    Journal of neural engineering 10(2), 026002 (2013) 28 Sounak Mondal et al

    Chestek, C.A., Gilja, V., Blabe, C.H., Foster, B.L., Shenoy, K.V., Parvizi, J., Henderson, J.M.: Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas. Journal of neural engineering 10(2), 026002 (2013) 28 Sounak Mondal et al

    2013

  17. [17]

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality (March 2023)

    2023

  18. [18]

    Scientific Reports13(1), 624 (2023)

    Daly, I.: Neural decoding of music from the EEG. Scientific Reports13(1), 624 (2023)

    2023

  19. [19]

    Nature Machine Intelligence5(10), 1097–1107 (2023)

    Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O., King, J.R.: Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence5(10), 1097–1107 (2023)

    2023

  20. [20]

    arXiv preprint arXiv:2303.03378 (2023)

    Driess, D., Xia, F., Sajjadi, M.S.M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., Florence, P.: PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)

    Pith/arXiv arXiv 2023

  21. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Duan, Z., Cheng, H., Xu, D., Wu, X., Zhang, X., Ye, X., Xie, Z.: CityLLaVA: Efficient fine-tuning for VLMs in city scenario. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  22. [22]

    MIT Press, Cambridge, MA (1984)

    Ericsson, K.A., Simon, H.A.: Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge, MA (1984)

    1984

  23. [23]

    Qualitative Health Research3(4), 430–441 (1993)

    Fonteyn, M.E., Kuipers, B., Grobe, S.J.: A description of think aloud method and protocol analysis. Qualitative Health Research3(4), 430–441 (1993)

    1993

  24. [24]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    Hamza, A., Ahn, Y.H., Lee, S., Kim, S.T., et al.: LLaVA needs more knowledge: Retrieval augmented natural language generation with knowledge graph for ex- plaining thoracic pathologies. In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    2025

  25. [25]

    Trends in cognitive sciences9(4), 188–194 (2005)

    Hayhoe, M., Ballard, D.: Eye movements in natural behavior. Trends in cognitive sciences9(4), 188–194 (2005)

    2005

  26. [26]

    In: Eye movements, pp

    Henderson, J.M., Brockmole, J.R., Castelhano, M.S., Mack, M.: Visual saliency does not account for eye movements during visual search in real-world scenes. In: Eye movements, pp. 537–III. Elsevier (2007)

    2007

  27. [27]

    Frontiers in Human Neuroscience15, 659410 (2021)

    Hollenstein, N., Renggli, C., Glaus, B., Barrett, M., Troendle, M., Langer, N., Zhang, C.: Decoding EEG brain activity for multi-modal natural language processing. Frontiers in Human Neuroscience15, 659410 (2021)

    2021

  28. [28]

    arXiv preprint arXiv:2106.09685 (2021)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    Pith/arXiv arXiv 2021

  29. [29]

    IEEE Transactions on Visualization and Computer Graphics29(4), 1992–2004 (2021)

    Hu, Z., Bulling, A., Li, S., Wang, G.: EHTask: Recognizing user tasks from eye and head movements in immersive virtual reality. IEEE Transactions on Visualization and Computer Graphics29(4), 1992–2004 (2021)

    1992

  30. [30]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2019)

    Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2019)

    2019

  31. [31]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11), 1254–1259 (1998)

    Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11), 1254–1259 (1998)

    1998

  32. [32]

    In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR), poster (2019)

    Khokhar, A., Yoshimura, A., Borst, C.: Eye-gaze-triggered visual cues to restore attention in educational vr. In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR), poster (2019)

    2019

  33. [33]

    Koehler, K., Guo, F., Zhang, S., Eckstein, M.P.: What do saliency models predict? Journal of Vision14(3), 14–14 (2014) Gaze-to-text Generation 29

    2014

  34. [34]

    Scientific Reports14(1), 11491 (2024)

    Komeiji, S., Mitsuhashi, T., Iimura, Y., Suzuki, H., Sugano, H., Shinoda, K., Tanaka, T.: Feasibility of decoding covert speech in ECoG with a transformer trained on overt speech. Scientific Reports14(1), 11491 (2024)

    2024

  35. [35]

    International journal of computer vision123, 32–73 (2017)

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision123, 32–73 (2017)

    2017

  36. [36]

    arXiv preprint arXiv:2405.13326 (2024)

    Li, M., Chen, P., Wang, C., Zhao, H., Liang, Y., Hou, Y., Liu, F., Zhou, T.: Mosaic-IT: Free compositional data augmentation improves instruction tuning. arXiv preprint arXiv:2405.13326 (2024)

    Pith/arXiv arXiv 2024

  37. [37]

    IEEE Transactions on Human-Machine Systems 47(4), 437–448 (2017)

    Li, S., Zhang, X.: Implicit intention communication in human–robot interaction through visual behavior studies. IEEE Transactions on Human-Machine Systems 47(4), 437–448 (2017)

    2017

  38. [38]

    International Conference on Learning Representations (2025)

    Li, X., Mata, C., Park, J., Kahatapitiya, K., Jang, Y.S., Shang, J., Ranasinghe, K., Burgert, R., Cai, M., Lee, Y.J., et al.: LLaRA: Supercharging robot learning data for vision-language policy. International Conference on Learning Representations (2025)

    2025

  39. [39]

    Signal Processing: Image Communication94, 116198 (2021)

    Liaqat, S., Wu, C., Duggirala, P.R., Cheung, S.c.S., Chuah, C.N., Ozonoff, S., Young, G.: Predicting ASD diagnosis in children with synthetic and image-based eye gaze data. Signal Processing: Image Communication94, 116198 (2021)

    2021

  40. [40]

    In: Text Summarization Branches Out

    Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain (2004)

    2004

  41. [41]

    In: European Conference on Computer Vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: European Conference on Computer Vision. pp. 740–755. Springer (2014)

    2014

  42. [42]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  43. [43]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  44. [44]

    Journal of Vision9(11), 25–25 (2009)

    Masciocchi, C.M., Mihalas, S., Parkhurst, D., Niebur, E.: Everyone knows what is interesting: Salient locations which should be fixated. Journal of Vision9(11), 25–25 (2009)

    2009

  45. [45]

    arXiv preprint arXiv:2307.04721 (2023)

    Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M.G., Rao, K., Sadigh, D., Zeng, A.: Large language models as general pattern machines. arXiv preprint arXiv:2307.04721 (2023)

    Pith/arXiv arXiv 2023

  46. [46]

    In: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (2024)

    Mohbat, F., Zaki, M.J.: LLaVA-Chef: A multi-modal generative model for food recipes. In: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (2024)

    2024

  47. [47]

    In: European Conference on Computer Vision (2024)

    Mondal, S., Ahn, S., Yang, Z., Balasubramanian, N., Samaras, D., Zelinsky, G., Hoai, M.: Look hear: Gaze prediction for speech-directed human attention. In: European Conference on Computer Vision (2024)

    2024

  48. [48]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (October 2025)

    Mondal, S., Sendhilnathan, N., Zhang, T., Liu, Y., Proulx, M., Iuzzolino, M.L., Qin, C., Jonker, T.R.: Gaze-language alignment for zero-shot prediction of visual search targets from human gaze scanpaths. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (October 2025)

    2025

  49. [49]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 30 Sounak Mondal et al

    Mondal, S., Yang, Z., Ahn, S., Samaras, D., Zelinsky, G., Hoai, M.: Gazeformer: Scalable, effective and fast prediction of goal-directed human attention. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 30 Sounak Mondal et al

    2023

  50. [50]

    Nature434(7031), 387–391 (2005)

    Najemnik, J., Geisler, W.S.: Optimal eye movement strategies in visual search. Nature434(7031), 387–391 (2005)

    2005

  51. [51]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Nishiyasu, T., Sato, Y.: Gaze scanpath transformer: Predicting visual search target by spatiotemporal semantic modeling of gaze scanpath. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  52. [52]

    In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA (2002)

    2002

  53. [53]

    Advances in Neural Information Processing Systems (2019)

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems (2019)

    2019

  54. [54]

    Developmental Neurorehabilitation23(3), 145–159 (2020)

    Perfect, E., Hoskin, E., Noyek, S., Davies, C.T.: Outcome measures and uptake barriers when children and youth with complex disabilities use eye gaze assistive technology. Developmental Neurorehabilitation23(3), 145–159 (2020)

    2020

  55. [55]

    In: International Conference on Machine Learning (2021)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (2021)

    2021

  56. [56]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Ranasinghe, K., Shukla, S.N., Poursaeed, O., Ryoo, M.S., Lin, T.Y.: Learning to localize objects improves spatial reasoning in visual-LLMs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  57. [57]

    In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (2020)

    Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (2020)

    2020

  58. [58]

    IEEE Access10, 70779–70792 (2022)

    Sabab, S.A., Kabir, M.R., Hussain, S.R., Mahmud, H., Rubaiyeat, H.A., Hasan, M.K.: Vis-iTrack: Visual intention through gaze tracking using low-cost webcam. IEEE Access10, 70779–70792 (2022)

    2022

  59. [59]

    In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2017)

    Sattar, H., Bulling, A., Fritz, M.: Predicting the category and attributes of visual search targets using deep gaze pooling. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2017)

    2017

  60. [60]

    Neurocomputing387, 369–382 (2020)

    Sattar, H., Fritz, M., Bulling, A.: Deep gaze pooling: Inferring and visually decoding search intents from human gaze fixations. Neurocomputing387, 369–382 (2020)

    2020

  61. [61]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)

    Sattar, H., Muller, S., Fritz, M., Bulling, A.: Prediction of search targets from fixa- tions in open-world settings. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)

    2015

  62. [62]

    Nature620, 297–302 (2023)

    Singhal, K., Azizi, S., Tu, T., et al.: Large language models encode clinical knowledge. Nature620, 297–302 (2023)

    2023

  63. [63]

    Academic Press, London (1994)

    van Someren, M.W., Barnard, Y.F., Sandberg, J.A.C.: The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. Academic Press, London (1994)

    1994

  64. [64]

    In: KI 2018: Advances in Artificial Intelligence: 41st German Con- ference on AI, Berlin, Germany, September 24–28, 2018, Proceedings 41

    Stauden, S., Barz, M., Sonntag, D.: Visual search target inference using bag of deep visual words. In: KI 2018: Advances in Artificial Intelligence: 41st German Con- ference on AI, Berlin, Germany, September 24–28, 2018, Proceedings 41. Springer (2018)

    2018

  65. [65]

    In: Proceedings of the 2015 acm international joint conference on pervasive and ubiquitous computing (2015) Gaze-to-text Generation 31

    Steil, J., Bulling, A.: Discovery of everyday human activities from long-term visual behaviour using topic models. In: Proceedings of the 2015 acm international joint conference on pervasive and ubiquitous computing (2015) Gaze-to-text Generation 31

    2015

  66. [66]

    In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (2023)

    Strohm, F., Bâce, M., Bulling, A.: Usable and fast interactive mental face recon- struction. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (2023)

    2023

  67. [67]

    In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (2021)

    Strohm, F., Sood, E., Mayer, S., Müller, P., Bâce, M., Bulling, A.: Neural photofit: gaze-based mental image reconstruction. In: Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (2021)

    2021

  68. [68]

    In: Annual Conference on Neural Information Processing Systems

    Strohm, F., Sood, E., Thomas, D., Bâce, M., Bulling, A.: Facial composite generation with iterative human feedback. In: Annual Conference on Neural Information Processing Systems. PMLR (2023)

    2023

  69. [69]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    Takagi, Y., Nishimoto, S.: High-resolution image reconstruction with latent diffusion models from human brain activity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    2023

  70. [70]

    In: 17th International IEEE conference on intelligent transportation systems (ITSC)

    Tawari, A., Chen, K.H., Trivedi, M.M.: Where is the driver looking: Analysis of head, eye and iris for robust gaze zone estimation. In: 17th International IEEE conference on intelligent transportation systems (ITSC). IEEE (2014)

    2014

  71. [71]

    IEEE Transactions on Visualization and Computer Graphics16(6), 953– 962 (2010)

    Tsang, H.Y., Tory, M., Swindells, C.: eseetrack—visualizing sequential fixation patterns. IEEE Transactions on Visualization and Computer Graphics16(6), 953– 962 (2010)

    2010

  72. [72]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Vedantam, R., Zitnick, C.L., Parikh, D.: CIDEr: Consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    2015

  73. [73]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2024)

    Wang, B., Pan, H., Aboah, A., Zhang, Z., Keles, E., Torigian, D., Turkbey, B., Krupinski, E., Udupa, J., Bagci, U.: GazeGNN: A gaze-guided graph neural network for chest x-ray classification. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2024)

    2024

  74. [74]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2024)

    Wang, B., Wu, F., Han, X., Peng, J., Zhong, H., Zhang, P., Dong, X., Li, W., Li, W., Wang, J., et al.: VIGC: Visual instruction generation and correction. In: Proceedings of the AAAI Conference on Artificial Intelligence (2024)

    2024

  75. [75]

    In: Advances in Neural Information Processing Systems (2020)

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In: Advances in Neural Information Processing Systems (2020)

    2020

  76. [76]

    In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019)

    Wang, X., Ley, A., Koch, S., Lindlbauer, D., Hays, J., Holmqvist, K., Alexa, M.: The mental image revealed by gaze tracking. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019)

    2019

  77. [77]

    In: AKWI Jahrestagung 2024 (2024)

    Weber, I.: Large language models are pattern matchers: Editing semi-structured and structured documents with ChatGPT. In: AKWI Jahrestagung 2024 (2024)

    2024

  78. [78]

    arXiv preprint arXiv:2409.07732 (2024)

    Weber, I.: Large language models are pattern matchers: Editing semi-structured and structured documents with ChatGPT. arXiv preprint arXiv:2409.07732 (2024)

    Pith/arXiv arXiv 2024

  79. [79]

    Vision3(3), 33 (2019)

    Williams, C.C., Castelhano, M.S.: The changing landscape: High-level influences on eye movement guidance in scenes. Vision3(3), 33 (2019)

    2019

  80. [80]

    Xia, W., de Charette, R., Oztireli, C., Xue, J.H.: DREAM: Visual decoding from reversinghumanvisualsystem.In:ProceedingsoftheIEEE/CVFWinterConference on Applications of Computer Vision (2024)

    2024

Showing first 80 references.