REVIEW 3 major objections 5 minor 39 references
Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read SemBA-FAST, a semantic-foveal Bayesian model that never trains on human gaze, produces fixation sequences on target-present visual search that match or beat several gaze-trained models on COCO-Search18.
desk verdict Useful benchmark evaluation of a prior gaze-free model, but the central claim is undercut by test-set tuning of the fovea width. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a Dirichlet-compound multinomial belief map over the scene grid: each cell $x$ holds a distribution $P(C_x=k\mid\beta_x)$ over object classes, initialized flat and updated by a subjective-logic classifier fusion rule with either raw detection scores, fovea-calibrated Dirichlet likelihoods, or predicted expected scores. The active-perception step greedily selects the cell maximizing the target class's posterior, adds inhibition of return around the chosen fixation, and repeats until an oracle confirms the target is fixated. This mechanism converts a static object detector into a sequential, goal-directed gaze policy with no gaze training.
What would settle it
Permute the detector's class labels (or replace its score vectors with chance-level values) on the same COCO-Search18 scenes, keep the foveation, oracle, and inhibition-of-return unchanged, and rerun SemBA-FAST; if the scanpath metrics stay near the reported values, then semantic content is not what produces the human-like fixations.
Extended reading notes
Core claim
On the COCO-Search18 target-present benchmark, SemBA-FAST — with no scanpath training — produces fixation sequences closer to human ground truth than random selection and IVSN, and compares favorably with gaze-trained models: it beats FFMs on fixation-sequence similarity, beats IRL on FED and SemFED, and reaches cAUC above 0.90. The best variant, SemBA-FASTPred, simulates the detector's expected score if the gaze moved to each candidate cell and fuses those expectations into the beliefs; the raw-score variant is nearly as good at lower compute, while the explicitly calibrated variant is worst. The paper reads this as evidence that semantic evidence, foveal distortion, and sequential Bayesian fusion can approximate human target-present scanpaths without learning from human gaze.
Load-bearing premise
The load-bearing premise is that the object detector's class scores on artificially foveated inputs are a faithful proxy for the perceptual evidence a human uses to choose the next fixation; if that proxy misrepresents what humans actually see, the whole fixation sequence is driven by the wrong signal.
Editorial extensions
If this is right
- Without ever seeing human fixations, SemBA-FAST surpasses a zero-shot top-down baseline and random gaze on every reported scanpath metric.
- It beats some scanpath-trained models on sequence metrics and matches or exceeds several on conditional next-fixation metrics including cAUC above 0.90.
- Only SemBA-FAST and two gaze-trained models achieve positive information gain over the human-average fixation density baseline, suggesting the semantic-foveal maps carry genuine predictive signal.
- The raw-score variant nearly matches the best predictive variant at lower compute, so the foveal calibration step is not what carries the performance.
- Because the model needs no gaze data, its architecture can be transferred to new target classes or tasks by swapping the detector or target semantics.
Reading between the lines
- If the semantic evidence is indeed the driver, gaze-training may partly rediscover semantic-foveal priors; SemBA-FAST could serve as a training-free baseline for judging how much value human gaze data actually adds.
- The failure of the calibrated variant suggests that explicitly modeling peripheral-distortion uncertainty from the same detector can hurt; a learned noise model from human peripheral sensitivity rather than detector error might do better.
- The same posterior-maximization machinery, with a target-absent termination criterion, is a natural testbed for target-absent search, which the paper leaves for future work.
- The model's conditional saliency metrics are generated without gaze labels, so it may be useful for evaluating whether gaze-trained models generalize beyond their training distribution of targets.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SemBA-FAST, a semantic-foveal Bayesian active-visual-search model that predicts human scanpaths in target-present images without training on human gaze data. The model uses YOLOv5 detections on artificially foveated images, updates Dirichlet-compound semantic maps via a subjective-logic fusion rule, and greedily selects the next fixation as the argmax of the target-class posterior (Eq. 5), with inhibition of return and an oracle-based termination criterion. The authors evaluate three variants (Base, Calib, Pred) on COCO-Search18 and compare them against human and random baselines, IVSN, and gaze-trained models (IRL, Chen et al., FFMs, Gazeformer, HAT, CLIPgaze) using scanpath and next-fixation metrics. The central claim is that this pure semantic model closely mimics human target-present search performance and competes with gaze-trained models on several metrics.
Significance. The strength of the paper is that it proposes a biologically motivated mechanism that does not use human scanpath labels in training, explicitly models foveal distortion, and provides a broad comparison against multiple scanpath-prediction baselines on a standard benchmark. If the reported results were obtained under a clean evaluation protocol, the demonstration that a semantic-foveal Bayesian model can approach gaze-trained performance would be a useful contribution to the attention-modeling literature. The paper also reports per-iteration time costs, which is helpful for real-time applications. However, the main empirical claim is currently undermined by a test-set-based selection of the key free parameter eta, as detailed below, so the significance of the numerical results cannot be fully assessed without re-evaluation under a leakage-free protocol.
major comments (3)
- [Section IV-B] The foveal width parameter is selected directly on the test partition: 'We define eta = 0.156, as it led SemBA-FAST's Pred variant (see Fig. 4) to achieve cumulative performances similar to those attained by humans in COCO-Search18.' Figure 4 is explicitly computed on the test partition, and eta directly controls the degree of peripheral blur applied to YOLOv5 inputs, which in turn determines the detections that feed Eq. (5) and hence the entire fixation sequence. All headline results in Table I and Fig. 5 are evaluated on the same test partition. This is test-set leakage in model selection, not gradient-based training, but it means the claim that SemBA-FAST 'closely mimics human performance' is not an independent empirical finding: the free parameter was chosen to make it so. The authors should select eta on a validation split (or via cross-validation over subjects), report the sensitivity of Table I to eta, and only then evaluate on a held-out test set. Without this, the reported SS/FED/SemSS/SemFED and next-fixation numbers are optimistically biased and the central claim is unvalidated.
- [Section IV-B] The oracle-based termination rule uses ground-truth target bounding boxes to stop the search once the predicted fixation lands on the target. Because the task is target-present by construction, this gives the model privileged information about the target's location that a human searcher does not have (humans must decide when they have found the target). This can affect scanpath length, SR, and subsequent fixation metrics, and it makes comparisons with models that do not use such an oracle difficult to interpret. The authors acknowledge the choice but do not analyze its impact. They should ablate the oracle (e.g., stop after a fixed number of fixations or when the posterior exceeds a threshold) and report how the comparative results change, or at minimum justify that the oracle does not favor SemBA-FAST over the compared models.
- [Table I and Section IV-E] All headline comparisons are reported as single point estimates without variances, confidence intervals, or statistical significance tests. For example, the claimed advantage of SemBA-FAST over FFMs in SS/FED and the statement that SemBA-FAST 'closely mimics human performance' rest on differences that may be within inter-subject or inter-run variability, particularly since the SemBA-FAST variants themselves differ by only small margins (SS 0.409 vs 0.413; FED 2.639 vs 2.616). The authors should provide subject-level or bootstrap confidence intervals for the metrics and, where appropriate, significance tests for the comparisons that support the paper's main claims.
minor comments (5)
- [Section IV-B] The sentence 'We define eta = 0.156...' is presented as an experimental setup choice, but because it is justified by matching test performance, it should be moved to a validation procedure and stated as a selected hyperparameter, not a fixed design choice.
- [Section III-E, Eq. (6)] Equation (6) is difficult to parse because of the nested summations and the lack of brackets around the Dirichlet mean. Adding parentheses around the alpha ratio and the beta ratio would improve readability.
- [Section IV-D] The IVSN 'Picked Target' condition uses one manually selected target image per class, while 'Random Target' uses a random COCO image. The choice of target image can materially affect IVSN's attention maps; the authors should specify how representative these choices are or report sensitivity to the target image selection.
- [Table I] The 'Time Cost' column mixes per-iteration timings from different models that may have been measured on different hardware and input resolutions; the authors should state the hardware/software configuration for their own timings and clarify that cross-model time comparisons are only indicative.
- [Section IV-C] The description of the scanpath metrics would benefit from a small illustrative example of how fixation cluster IDs and semantic labels are converted to strings, since SS, FED, SemSS, and SemFED depend on this encoding and the paper does not define the clustering procedure.
Circularity Check
Foveal width η is selected on the COCO-Search18 test partition using human cumulative performance, so the headline 'closely mimics human performance' is a tuned outcome rather than an independent prediction.
-
fitted input called prediction
[Section IV-B (Experimental Setup); see also Fig. 4 and Table I.]
"We define η = 0.156, as it led SemBA-FAST's Pred variant (see Fig. ́ 4) to achieve cumulative performances similar to those attained by humans in COCO-Search18 [29]."
η is the model's key free parameter controlling foveal width and therefore how strongly peripheral blur degrades YOLOv5 detections, which directly shapes the semantic evidence used in Eq. (5) to generate the entire fixation sequence. The paper selects η using human cumulative performance on COCO-Search18, the same test partition on which all reported metrics in Section IV-A, Table I, and Fig. 5 are evaluated. Consequently, the central claim that SemBA-FAST 'closely mimics human performance' is not an independent prediction: the only free parameter was chosen to make the model's cumulative performance resemble human performance on the evaluation set.
full rationale
Aside from the test-set selection of η, the model does not exhibit definitional circularity: SemBA-FAST never trains on human scanpaths, YOLOv5 is trained on COCO 2017 with COCO-Search18 images excluded, and the Dirichlet calibration is fit on foveated COCO images rather than on human gaze data. The Bayesian update rules and the greedy next-fixation rule in Eq. (5) are not constructed from the human test scanpaths they predict. The self-citations to the authors' prior work [7] and to Dias et al. [8] are methodology attribution, not load-bearing circularity: the equations and pipeline are described in the present paper and no uniqueness or impossibility theorem is imported from the self-citations. However, the single most important parameter, η, is explicitly chosen to match human cumulative performance on the test partition, and the strongest qualitative claim is exactly that the method mimics human performance. This makes part of the reported 'prediction' a fitted input rather than a free prediction. The score is therefore 6 rather than higher because only this one parameter is tuned on the test set, while the other architectural choices and the external benchmark comparisons retain independent content.
Assumptions & free parameters
free parameters (6)
- eta (fovea size fraction) =
0.156
- D (peripheral distortion levels) =
7
- Attention map grid resolution =
20x32
- Maximum scanpath length =
7 fixations
- Inhibition-of-return radius =
3x3 grid
- Dirichlet calibration hyperparameters alpha_{k,d} =
K x D sets, not listed
assumptions (6)
- domain assumption YOLOv5 detections under artificial foveation are a sufficient proxy for the perceptual evidence humans use during visual search.
- domain assumption The next human fixation is the grid cell maximizing the posterior P(C_x=k|beta_x).
- domain assumption Almeida's artificial foveal system is a valid model of human peripheral blur.
- standard math The Dirichlet-compound multinomial and Kaplan fusion rule correctly combine sequential detections.
- ad hoc to paper An oracle knows when the gaze lands on the target and terminates search.
- domain assumption The target class is known a priori and only target-present images are considered.
Cite this review
Pith. "Pith review of Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention." pith.science (2026). https://pith.science/paper/WEZMD33F
@misc{pith2026250718503,
author = {Pith},
title = {Pith review of: Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention},
year = {2026},
howpublished = {\url{https://pith.science/paper/WEZMD33F}},
note = {Machine review of arXiv:2507.18503}
}
read the original abstract
In goal-directed visual tasks, human perception is guided by both top-down and bottom-up cues. At the same time, foveal vision plays a crucial role in directing attention efficiently. Modern research on bio-inspired computational attention models has taken advantage of advancements in deep learning by utilizing human scanpath data to achieve new state-of-the-art performance. In this work, we assess the performance of SemBA-FAST, i.e. Semantic-based Bayesian Attention for Foveal Active visual Search Tasks, a top-down framework designed for predicting human visual attention in target-present visual search. SemBA-FAST integrates deep object detection with a probabilistic semantic fusion mechanism to generate attention maps dynamically, leveraging pre-trained detectors and artificial foveation to update top-down knowledge and improve fixation prediction sequentially. We evaluate SemBA-FAST on the COCO-Search18 benchmark dataset, comparing its performance against other scanpath prediction models. Our methodology achieves fixation sequences that closely match human ground-truth scanpaths. Notably, it surpasses baseline and other top-down approaches and competes, in some cases, with scanpath-informed models. These findings provide valuable insights into the capabilities of semantic-foveal probabilistic frameworks for human-like attention modelling, with implications for real-time cognitive computing and robotics.
Figures
Reference graph
Works this paper leans on
-
[7]
SemBA-FAST: Semantic- Based Bayesian Attention Applied to Foveal Active Visual Search Tasks,
J. Luzio, A. Bernardino, and P. Moreno, “SemBA-FAST: Semantic- Based Bayesian Attention Applied to Foveal Active Visual Search Tasks,” Available at SSRN 5143184
-
[1]
W. S. Tuten and W. M. Harmening, “Foveal vision,” Current Biology, vol. 31, no. 11, pp. R701–R703, 2021, doi: 10.1016/j.cub.2021.03.097
-
[2]
Revisiting active per- ception,
R. Bajcsy, Y . Aloimonos, and J. K. Tsotsos, “Revisiting active per- ception,” Autonomous Robots, vol. 42, pp. 177–196, 2018
work page 2018
-
[3]
An overview of space-variant and active vision mechanisms for resource-constrained human inspired robotic vision,
R. P. de Figueiredo and A. Bernardino, “An overview of space-variant and active vision mechanisms for resource-constrained human inspired robotic vision,” Autonomous Robots, pp. 1–17, 2023
2023
-
[4]
Foveated rendering: A state-of-the-art survey,
L. Wang, X. Shi, and Y . Liu, “Foveated rendering: A state-of-the-art survey,” Computational Visual Media, vol. 9, no. 2, pp. 195–228, 2023
work page 2023
-
[5]
Deep learning for object recognition: A compre- hensive review of models and algorithms,
P. Tsirtsakis et al., “Deep learning for object recognition: A compre- hensive review of models and algorithms,” International Journal of Cognitive Computing in Engineering, vol. 6, pp. 298–312, 2025
work page 2025
-
[6]
L. Itti, C. Koch, et al., ”A model of saliency-based visual attention for rapid scene analysis,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no 11, pp. 1254-1259, 1998
1998
-
[8]
A. Dias, L. Sim ˜oes, et al., ”Active Gaze Control for Foveal Scene Exploration,” 2022 IEEE International Conference on Development and Learning (ICDL), London, UK, pp. 115-120, 2022
work page 2022
Show all 39 references
-
[9]
Predicting visual fixations,
M. K ¨ummerer and M. Bethge, “Predicting visual fixations,” Annual Review of Vision Science, vol. 9, pp. 269–291, 2023
2023
-
[10]
Eye movement characteristics in schizophrenia: A recent update with clinical implica- tions,
K. Morita, K. Miura, K. Kasai, and R. Hashimoto, “Eye movement characteristics in schizophrenia: A recent update with clinical implica- tions,” Neuropsychopharmacology Reports, vol. 40:1, pp. 2–9, 2020
2020
-
[11]
Eye-movements reveal semantic interference ef- fects during the encoding of naturalistic scenes in long-term memory,
A. Mikhailova, et al., “Eye-movements reveal semantic interference ef- fects during the encoding of naturalistic scenes in long-term memory,” Psychonomic Bulletin & Review, vol. 28:5, pp. 1601–1614, 2021
2021
-
[12]
The neural basis of visual attention,
J. W. Bisley, “The neural basis of visual attention,” The Journal of physiology, vol. 589, no. 1, pp. 49–57, 2011
2011
-
[13]
Visual search: How do we find what we are looking for?,
J. M. Wolfe, “Visual search: How do we find what we are looking for?,” Annual review of vision science, vol. 6, pp. 539–562, 2020
2020
-
[14]
Dirichlet-based Uncertainty Calibration for Active Domain Adaptation,
M. Xie, S. Li, R. Zhang, and C. H. Liu, “Dirichlet-based Uncertainty Calibration for Active Domain Adaptation,” The Eleventh International Conference on Learning Representations, 2023
2023
-
[15]
L. M. Kaplan, S. Chakraborty and C. Bisdikian, ”Fusion of classifiers: A subjective logic perspective,” 2012 IEEE Aerospace Conference, Big Sky, MT, USA, pp. 1-13, 2012
2012
-
[16]
The Mahalanobis distance,
R. De Maesschalck, D. Jouan-Rimbaud, and D. L. Massart, “The Mahalanobis distance,” Chemometrics and Intelligent Laboratory Sys- tems, vol. 50, no. 1, pp. 1–18, 2000
2000
-
[17]
Deep networks for human visual attention: A hybrid model using foveal vision,
A. F. Almeida et al., “Deep networks for human visual attention: A hybrid model using foveal vision,” in ROBOT 2017: Third Iberian Robotics Conference: V olume 2, pp. 117–128, 2018
2017
-
[18]
Rich feature hierarchies for accurate object detection and semantic segmentation,
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587, 2014
2014
-
[19]
You only look once: Unified, real-time object detection,
Joseph Redmon et al., “You only look once: Unified, real-time object detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016
2016
-
[20]
A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas,
J. Terven et al., “A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas,” Machine Learning and Knowledge Extraction, vol. 5:4, pp. 1680–1716, 2023
2023
-
[21]
A Review of Yolo algorithm developments,
P. Jiang, D. Ergu, et al., “A Review of Yolo algorithm developments,” Procedia computer science, vol. 199, pp. 1066–1073, 2022
2022
-
[22]
End-to-end object detection with transformers,
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision, pp. 213–229, 2020
2020
-
[23]
Learning to predict where humans look,
T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learning to predict where humans look,” in 2009 IEEE 12th international conference on computer vision, 2009, pp. 2106–2113
2009
-
[24]
Estimating a Dirichlet distribution,
T. Minka, “Estimating a Dirichlet distribution,” Massachusetts Institute of Technology, Tech Report, 2000
2000
-
[25]
C. M. Bishop and N. M. Nasrabadi, Pattern recognition and machine learning, vol. 4. Springer, 2006
2006
-
[26]
Microsoft COCO: Common Objects in Context,
T.Y . Lin et al., “Microsoft COCO: Common Objects in Context,” in Computer Vision – ECCV 2014, pp. 740–755, 2014
2014
-
[27]
Saliency benchmarking made easy: Separating models, maps and metrics,
M. Kummerer, T. S. Wallis, and M. Bethge, “Saliency benchmarking made easy: Separating models, maps and metrics,” in Proceedings of the European Conference on Computer Vision, pp. 770–787, 2018
2018
- [28]
-
[29]
Coco-search18 fixation dataset for predicting goal-directed attention control,
Y . Chen, Z. Yang, S. Ahn, D. Samaras, M. Hoai, and G. Zelinsky, “Coco-search18 fixation dataset for predicting goal-directed attention control,” Scientific reports, vol. 11, no. 1, p. 8776, 2021
2021
-
[30]
A general method applicable to the search for similarities in the amino acid sequence of two proteins,
S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” Journal of molecular biology, vol. 48, no. 3, pp. 443–453, 1970
1970
-
[31]
Binary codes capable of correcting deletions, inser- tions, and reversals,
V . Levenshtein, “Binary codes capable of correcting deletions, inser- tions, and reversals,” Proceedings of the Soviet physics doklady, 1966
1966
-
[32]
DeepGaze III: Modeling free-viewing human scanpaths with deep learning,
M. K ¨ummerer et al., “DeepGaze III: Modeling free-viewing human scanpaths with deep learning,” Journal of Vision, vol. 22:5, pp. 7, 2022
2022
-
[33]
Finding any Waldo with zero-shot invariant and efficient visual search,
M. Zhang, J. Feng, K. T. Ma, J. H. Lim, Q. Zhao, and G. Kreiman, “Finding any Waldo with zero-shot invariant and efficient visual search,” Nature Communications, vol. 9, no. 1, p. 3730, 2018
2018
-
[34]
Predicting goal-directed human attention using inverse reinforcement learning,
Z. Yang et al., “Predicting goal-directed human attention using inverse reinforcement learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 193–202, 2020
2020
-
[35]
Predicting human scanpaths in visual question answering,
X. Chen, M. Jiang, and Q. Zhao, “Predicting human scanpaths in visual question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10876–10885, 2021
2021
-
[36]
Target-absent human attention,
Z. Yang, S. Mondal, S. Ahn, et al., “Target-absent human attention,” in European Conference on Computer Vision, 2022, pp. 52–68
2022
-
[37]
Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention,
S. Mondal, Z. Yang, S. Ahn, et al., “Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2023, pp. 1441–1450
2023
-
[38]
Unifying Top-down and Bottom-up Scanpath Predic- tion Using Transformers,
Z. Yang, et al., “Unifying Top-down and Bottom-up Scanpath Predic- tion Using Transformers,” The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2024
2024
-
[39]
CLIPGaze: Zero-Shot Goal-Directed Scanpath Predic- tion Using CLIP,
Y . Lai et al., “CLIPGaze: Zero-Shot Goal-Directed Scanpath Predic- tion Using CLIP,” in ICASSP 2025 - IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1–5, 2025
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.