RECORD
Visual Decoding of Targets During Visual Search From Human Eye Fixations
Computer Vision and Pattern Recognition · Submitted 19 Jun 2017
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We show, for the first time, that visual representations of search targets can indeed be decoded only from human gaze fixations.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
What does human gaze reveal about a users' intents and to which extend can these intents be inferred or even visualized? Gaze was proposed as an implicit source of information to predict the target of visual search and, more recently, to predict the object class and attributes of the search target. In this work, we go one step further and investigate the feasibility of combining recent advances in encoding human gaze information using deep convolutional neural networks with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Such visual decoding is challenging for two reasons: 1) the search target only resides in the user's mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all. We show, for the first time, that visual representations of search targets can indeed be decoded only from human gaze fixations. We propose to first encode fixations into a semantic representation and then decode this representation into an image. We evaluate our method on a recent gaze dataset of 14 participants searching for clothing in image collages and validate the model's predictions using two human studies. Our results show that 62% (Chance level = 10%) of the time users were able to select the categories of the decoded image right. In our second studies we show the importance of a local gaze encoding for decoding visual search targets of user
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | What does human gaze reveal about a users' intents and to which extend can these intents be inferred or even visualized? | background 0.98 | 0.04 | 0.02 | unclear 1.00 |
| s1 | Gaze was proposed as an implicit source of information to predict the target of visual search and, more recently, to predict the object class and attributes of the search target. | background 1.00 | 0.04 | 0.02 | asserted 0.80 |
| s2 | In this work, we go one step further and investigate the feasibility of combining recent advances in encoding human gaze information using deep convolutional neural networks with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. | contribution 1.00 | 0.57 | 0.04 | asserted 0.48 |
| s3 | Such visual decoding is challenging for two reasons: 1) the search target only resides in the user's mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all. | gap 0.93 | 0.08 | 0.02 | asserted 0.80 |
| s4 | We show, for the first time, that visual representations of search targets can indeed be decoded only from human gaze fixations. | result 0.81 | 0.96 | 0.91 | asserted 1.00 |
| s5 | We propose to first encode fixations into a semantic representation and then decode this representation into an image. | method 0.99 | 0.78 | 0.05 | asserted 0.89 |
| s6 | We evaluate our method on a recent gaze dataset of 14 participants searching for clothing in image collages and validate the model's predictions using two human studies. | method 0.97 | 0.95 | 0.03 | asserted 0.93 |
| s7 | Our results show that 62% (Chance level = 10%) of the time users were able to select the categories of the decoded image right. | result 1.00 | 0.96 | 0.29 | asserted 1.00 |
| s8 | In our second studies we show the importance of a local gaze encoding for decoding visual search targets of user | result 0.86 | 0.92 | 0.06 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
What does human gaze reveal about a users' intents and to which extend can these intents be inferred or even visualized? Gaze was proposed as an implicit source of information to predict the target of visual search and, more recently, to predict the object class and attributes of the search target. In this work, we go one step further and investigate the feasibility of combining recent advances in encoding human gaze information using deep convolutional neural networks with the power of generative image models to visually decode, i.e. create a visual representation of, the search target. Such visual decoding is challenging for two reasons: 1) the search target only resides in the user's mind as a subjective visual pattern, and can most often not even be described verbally by the person, and 2) it is, as of yet, unclear if gaze fixations contain sufficient information for this task at all. We show, for the first time, that visual representations of search targets can indeed be decoded only from human gaze fixations. We propose to first encode fixations into a semantic representation and then decode this representation into an image. We evaluate our method on a recent gaze dataset of 14 participants searching for clothing in image collages and validate the model's predictions using two human studies. Our results show that 62% (Chance level = 10%) of the time users were able to select the categories of the decoded image right. In our second studies we show the importance of a local gaze encoding for decoding visual search targets of user
Human review
How the literature uses it
2 arXiv papers cite this in 2 sentences
Papers citing it · 1 sentence without a confident label
Cites it as background 1 paper
-
Besides saliency estimation, eye tracking has been also used in order to improve the performance of various CV tasks such as object classification [ref] , object segmentation [ref] , action recognition [ref] , zero-shot image classification [ref] , or image generation [ref] .
Teacher's Perception in the Classroom · Related Works · 0.96
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.