Pith. sign in

REVIEW 3 major objections 6 minor 90 references

IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims people can grade vision-language models by reading green-red heatmaps, with grades tracking true F1 scores even without ground truth.

desk verdict A useful, honest HCI study on letting users audit VLMs via binary heatmaps, but the headline correlation rests on three medians and the ground-truth baseline lacks inter-rater reliability; still deserves serious review. read the letter →

arxiv 2505.22305 v1 pith:KXDKYEJ2 submitted 2025-05-28 cs.CV

Md Touhidul Islam , Imran Kabir , Md Alimoor Reza , Syed Masum Billah This is my paper
classification cs.CV
keywords vision-languagemodelslargemultimodalbinaryheatmapvisualizationground-truth-freeevaluationhumanperceptionvideoobjectrecognitionspyobjectsF1-scorecorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models are increasingly deployed in situations where no labeled ground truth exists, so users have no way to check whether the model is really seeing the objects it claims to see. This paper claims that people can perform that check themselves: when the model's yes/no object predictions are rendered as a binary heatmap—green for 'model sees it,' red for 'model does not see it'—readers can spot patterns of agreement and disagreement with their own perception and rate the model's reliability. In a study with 15 participants, those ratings tracked the models' true $F_1$ scores ($R^2 = 0.90$ across the three real models, and $R^2 = 0.83$ on the subset of objects each user chose), and users rated a random model lowest and an oracle model highest. If the claim holds, ordinary users can audit AI systems in context by looking, without waiting for a benchmark.

What carries the argument

The load-bearing object is the binary heat map: a matrix whose columns are video keyframes and whose rows are user-chosen objects, with each cell colored green when the model says the object is present and red when the model says it is absent. IKIWISI adds two mechanisms that make the heat map diagnostic: interactive cell toggles that let users correct what they believe are errors while a bar graph records those corrections, and 'spy objects'—items that are almost certainly absent—which force an all-red row unless the model is hallucinating. The heat map works by compressing thousands of predictions into a small number of visual patterns, named in the paper as uni-color rows, single outliers, outlier islands, and checkered patterns, that users can interpret at a glance and verify by inspecting only the few cells that break the pattern.

What would settle it

Present a new group of participants with the same heatmaps but reverse the color-to-meaning mapping for selected rows (green meaning 'model says absent' and red meaning 'model says present'), and compare their reliability ratings with the original study; if the ratings still reproduce the $F_1$ ordering, the signal is in the pattern statistics rather than in users' perceptual verification, and if they reverse, the result depends on the color convention. A complementary check is to measure agreement between each participant's cell corrections and the authors' ground-truth labels, since low agreement would show that users and annotators do not share the same perception baseline.

Watch

Extended reading notes

Core claim

The central discovery is that a user's own visual judgment can substitute for ground-truth labels in a restricted but practical setting: multi-object recognition in video. IKIWISI renders a model's outputs as a grid with user-selected objects as rows and video keyframes as columns; a green cell means the model reports the object present, and a red cell means it reports it absent. Users evaluate the model by reading the resulting texture—solid rows, isolated mismatched cells, and flickering checkerboards—rather than by checking every prediction. The paper reports that these judgments correlate with the models' objective $F_1^O$ scores ($R^2 = 0.90$ for GPV-1, BLIP, and GPT-4V) and with the per-user $F_1^{O*}$ scores on the objects actually selected ($R^2 = 0.83$), while remaining uncorrelated with the random model's inflated $F_1^{O*}$ values. The tool also introduces 'spy objects,' items users know to be absent, so a green cell on a spy row immediately exposes a model that hallucinates nonexistent objects.

Load-bearing premise

The load-bearing assumption is that the user's own perception is the correct reference, so a cell that contradicts what the user believes is a model error; if users misread frames or disagree with the authors' ground-truth labels, the reported correlations could reflect disagreement between two human judgments rather than between human and machine.

Editorial extensions

If this is right

  • End users can screen a vision-language model in their own deployment context—say, a navigation aid for a blind relative—by picking the objects they care about and reading the resulting heatmap, with no labeled test set required.
  • Spy objects give a cheap adversarial check: a green cell on a known-absent object reveals hallucination faster than scanning all predictions.
  • Users converge on the same ranking as $F_1$-based evaluation: the ground-truth oracle is rated highest, the random model lowest, and among the real models GPT-4V is rated near the top, with BLIP statistically indistinguishable from it.
  • Machine-learning expertise is not a prerequisite: expert and non-expert participants did not differ significantly in task completion time, so caretakers, accessibility specialists, and other non-technical users can apply the same judgment.
  • Because users inspect only a small fraction of cells (with the exception of single outliers), the evaluation can be repeated across many videos and object sets cheaply, supporting iterative trust-building instead of a one-time benchmark score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the perception-as-reference premise holds, the heatmap could also serve as a pure alignment instrument: mismatches between user and model are findings in themselves, independent of any $F_1$ calculation, and could be mined to show where human commonsense and machine vision systematically diverge.
  • A direct test of the premise would be to have an independent group of raters correct the same heatmaps and measure agreement with the paper's ground-truth labels; high agreement would show the perception baseline is a shared standard, while low agreement would mean the reported correlations are user-specific.
  • The spy-object mechanism could be standardized into a label-free hallucination probe: seed fixed lists of known-absent objects into arbitrary videos and compare green-cell rates across models; the paper reports qualitative evidence for this, but a quantitative benchmark is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents IKIWISI, an interactive tool that converts vision-language model outputs in video object recognition into a binary heatmap, where green cells indicate detected objects and red cells indicate absent objects. Users select objects and video segments, inspect the heatmap, optionally correct cells, and rate model reliability on a 0-100 percent slider. The central claim is that users can evaluate model reliability without ground truth, and that their ratings correlate with objective F1 scores when those are available: Section 6.1 reports R2 = 0.90 for ratings versus F1^O across non-random models (GPV-1, BLIP, GPT4V) and R2 = 0.83 for ratings versus F1^{O*} across all non-random models. The paper also reports qualitative pattern analyses (uni-color rows, outliers, checkered patterns), task completion times, object counts, expertise comparisons via Mann-Whitney tests, and NASA-TLX results. The tool includes 'spy objects' to probe hallucination on absent objects. The study involved 15 sighted participants, primarily graduate students from the authors' university community.

Significance. If the central claim holds, IKIWISI offers a useful complement to automated metrics for evaluating vision-language models in settings without ground truth, a genuinely practical problem for accessibility and navigation applications. The paper has several strengths: the tool itself is thoughtfully designed and iteratively refined through three pilot studies; the study uses hidden F1 scores as an objective benchmark; the authors report Shapiro-Wilk, Kruskal-Wallis, Tukey, and Mann-Whitney tests; the dataset is publicly available; and the qualitative observations about visual patterns (Section 6.2) are concrete and falsifiable. The spy-object idea is a simple and potentially effective probe for hallucination. The main weakness is that the ground-truth reference used to compute F1 is the authors' own manual labeling, and the paper provides no inter-annotator agreement statistic and no direct measurement of whether participants' perceptual judgments agree with those labels. This gap is load-bearing for the 'without ground truth' claim because the correlation evidence is only as objective as the reference labels.

major comments (3)
  1. [Section 3.2.2 and Appendix A.2] The ground truth labels used to compute F1^O and F1^{O*} were produced by the authors themselves, with 'at least two authors independently annotated each segment, and discrepancies were resolved collaboratively' (Appendix A.2), but the paper reports no inter-annotator agreement statistic. Because the central quantitative claims (R2 = 0.90 in Section 6.1.1 and R2 = 0.83 in Section 6.1.2) are correlations against this reference, the absence of an IRR metric leaves open the possibility that the labels reflect the authors' own perceptual judgments rather than an objective standard. Please report Cohen's kappa or Fleiss' kappa on a held-out sample of frames, and ideally have an independent annotator label a subset that is not involved in the study.
  2. [Section 5.2 and Section 6.1] The study never measures agreement between participants' perceptions and the authors' ground-truth labels. The participants are a convenience sample of 15 sighted individuals, mostly graduate students from the same university community (Table 3), and the heatmap-based evaluation is inherently comparative: a user judges a cell as correct or incorrect by checking it against what they themselves see in the keyframe. F1 scores, however, are computed against the authors' labels. If participant perception diverges systematically from the authors' labels, the reported correlations could reflect shared perception between participants and authors rather than an objective measure of model performance. Please add a validation step in which participants mark object presence/absence on a random sample of keyframes, report agreement with the ground-truth labels, and demonstrate that the rating-F1 correlations are robust when participant-corrected labels are used.
  3. [Section 6.1.1, Fig. 10] The claim that user ratings 'strongly and positively correlated' with F1^O (R2 = 0.90) is based on only three non-random, non-GT models (GPV-1, BLIP, GPT4V). With three aggregate points, R2 is extremely sensitive to the choice of those points and does not provide a stable estimate of correlation strength. Please report the Pearson or Spearman correlation with a confidence interval, or model the trial-level data (e.g., mixed-effects regression with participant and model as random effects) rather than relying on a three-point regression. This concern does not invalidate the direction of the finding, but it overstates the precision of the quantitative claim.
minor comments (6)
  1. [Section 6.1.1] R2 is a coefficient of determination, not a correlation coefficient; when the text says 'strongly and positively correlated,' please report r or Spearman's rho in addition to R2.
  2. [Section 5.7.1] The normalization procedure is described as mean-centering followed by min-max normalization; please specify how degenerate cases are handled, for example if a participant gives the same rating for all trials and the range becomes zero.
  3. [Section 3.2.3] The text states that GPT4V took an average of 27 seconds per keyframe while being prompted once for all 90 objects; please clarify whether this is wall-clock time per keyframe and whether it includes network latency.
  4. [Fig. 10 caption] The caption says the models are sorted on the x-axis by F1^O score, but the x-axis appears to be a categorical model list; please make the ordering explicit in the caption or in the figure itself.
  5. [Section 6.5.3 and Fig. 17] The 'spy column' described as a frame used as a probe is an interesting observation, but the figure caption does not explain the yellow arrow or the empty-frame nature of the spy column; please expand the caption for accessibility.
  6. [Notation throughout] The notation F1^O and F1^{O*} is used inconsistently in the text (sometimes written as 'F1-score', sometimes as 'F_1^O'); please standardize the notation in equations and prose.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: user ratings and objective F1 scores are separate measurements; author-generated ground truth is a validity limitation, not a circular input.

full rationale

I checked the derivation chain that connects IKIWISI's heatmap display to the claim that users can evaluate models without ground truth. The heatmap (Sec. 3.1.4) shows raw model output (green/red for model sees/does not see), not correctness labels, and the F1 metrics (Sec. 5.4) are computed from manual ground-truth annotations (Sec. 3.2.2, App. A.2) rather than from user ratings. The rating slider collects a separate measurement (Sec. 5.6.2), and the paper does not define reliability as agreement with the user's corrected heatmap. Therefore correlations such as R3 (R2=0.90 with F1^O) and R5 (R2=0.83 with F1^{O*}) are not forced by construction: the objective score is hidden from participants and is not a function of their ratings. The author-generated labels published in the self-cited dataset papers [32,33] are a data-provenance and external-validity concern (no inter-rater reliability between participants and authors is reported), but the central claim does not reduce to that citation, and the current paper independently describes the labeling procedure in Appendix A.2. I find no step where an equation or claimed result is equivalent to its inputs by definition.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The tool introduces spy objects as the one genuinely new entity, with a falsifiable handle. The analysis relies on free parameters only in the mild sense of rating normalization and experiment design choices, not fitted constants. The axioms are standard HCI and ML assumptions plus the unverified correctness of the authors' ground-truth labels.

free parameters (3)
  • Mean-centering then min-max normalization of user ratings = no numeric value; a normalization transform applied to all ratings
    Section 5.7.1 applies mean-centering per participant followed by min-max normalization to remove individual bias. This transform changes the absolute rating values and is a modeling choice that could inflate or deflate the reported correlations and medians, though it is a standard and openly disclosed preprocessing step.
  • Model IDs randomly initialized per participant = five model identities (Random, GPV-1, BLIP, GPT4V, GT) assigned per participant
    Section 5.6.2 states that model IDs were randomly initialized for each participant so users did not know the underlying model name. This is a deliberately chosen experimental design parameter that controls for name bias.
  • Suggested object count of 4-6 to start each trial = 4 to 6 objects
    Section 6.3.2 says participants were advised to start each trial with 4-6 objects. The paper reports the number of objects used as a proxy for task difficulty, but the instructed starting range influences that proxy.
assumptions (5)
  • domain assumption A green or red cell in the heatmap is interpretable as the model's output for that object and frame, independent of confidence or calibration.
    Section 3.1.4 defines the heatmap as green when the model recognizes an object and red when it does not. The visualization collapses all confidence levels into two colors, and the paper's design argument is that this simplification is acceptable because the tool aims at pattern-based assessment rather than confidence-based interpretation.
  • domain assumption Human visual pattern recognition transfers to a grid of colored cells and serves as a reliable baseline for evaluating model correctness.
    Section 2.4 argues from pre-attentive vision and anomaly detection that users can spot colored-cell anomalies quickly. The entire design depends on this assumption, and the study's R2 values are the empirical test of it.
  • domain assumption The ground-truth labels in the dataset are correct.
    Appendix A.2 reports that the authors labeled the presence or absence of 90 objects in 31 video segments, with at least two authors per segment and discrepancies resolved collaboratively. The F1 scores used as objective metrics assume these labels are accurate, and no inter-annotator agreement statistic is reported.
  • standard math Micro-averaged F1 score is an appropriate summary of model performance for this object-detection evaluation.
    Section 5.4 defends the choice of F1 with micro averaging, arguing that precision and recall are equally important for navigation assistance. The choice is a standard metric for the subfield, which the paper cites directly.
  • domain assumption User ratings can be treated as continuous enough for correlation, regression, and normalization analysis despite the discrete 10-percent slider.
    Section 5.6.2 describes the slider as selectable in 10-point increments, a discrete Likert-like scale. Section 5.7.2 then uses non-parametric tests because ratings are not normally distributed, but the reported R2 values treat the ratings as continuous.
invented entities (2)
  • Spy objects independent evidence
    purpose: Adversarial objects that users know are absent, added to the model's object list to reveal hallucination: any green cell in a spy row indicates the model falsely claims to see an object.
    Spy objects are testable by anyone who runs the tool on the released dataset: if a model reports 'yes' for a spy object in a frame where the object is visibly absent, the tool flags a hallucination. The paper reports that participants used this mechanism (Section 6.5.3), and the mechanism is falsifiable in the sense that a competent third party could apply it to any video and model.
  • Spy column (a frame used as a probe)
    purpose: A near-informationaless frame between keyframes that the user treats as a spy: any green cell in that column reveals a model weakness.
    Introduced in Section 6.5.3 as a participant invention. Only one participant (P11) used it, and the paper presents it as an anecdotal observation rather than as a tested, falsifiable feature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth." pith.science (2026). https://pith.science/paper/KXDKYEJ2

@misc{pith2026250522305,
  author       = {Pith},
  title        = {Pith review of: IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KXDKYEJ2}},
  note         = {Machine review of arXiv:2505.22305}
}
read the original abstract

We present IKIWISI ("I Know It When I See It"), an interactive visual pattern generator for assessing vision-language models in video object recognition when ground truth is unavailable. IKIWISI transforms model outputs into a binary heatmap where green cells indicate object presence and red cells indicate object absence. This visualization leverages humans' innate pattern recognition abilities to evaluate model reliability. IKIWISI introduces "spy objects": adversarial instances users know are absent, to discern models hallucinating on nonexistent items. The tool functions as a cognitive audit mechanism, surfacing mismatches between human and machine perception by visualizing where models diverge from human understanding. Our study with 15 participants found that users considered IKIWISI easy to use, made assessments that correlated with objective metrics when available, and reached informed conclusions by examining only a small fraction of heatmap cells. This approach not only complements traditional evaluation methods through visual assessment of model behavior with custom object sets, but also reveals opportunities for improving alignment between human perception and machine understanding in vision-language systems.

Figures

Figures reproduced from arXiv: 2505.22305 by the authors.

Figure 1
Figure 1. IKIWISI’s interactive interface for evaluating vision-language models (best viewed in color). The interface shows [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Object Selection Panel (E) enlarged from Fig. 1. Ob [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. IKIWISI’s click-to-zoom feature in action. When a user clicks keyframe 2 (highlighted in Fig. 1), the system opens an [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Example prompts to GPT4V (left column), GPV-1 (center column), and BLIP (right column), and the model generated [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Early design 1: Two heat maps, two models, and the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Default and accessible color codes in the heat map of IKIWISI. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: BLIP models’ outputs for the first 15 frames from Fig. 1. The left heat map (in red and green) displays the models’ raw [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 11
Figure 11. Figure 11: Normalized user ratings plotted against different [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Different predominant visual patterns that affect [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: a. These cells drew attention from all participants throughout [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 15
Figure 15. Figure 15: Boxplots showing the deviation of user ratings [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Boxplots showing the trial completion times for [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Spy column (i.e., frame), shown using a yellow arrow. [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: NASA-TLX load indices for using IKIWISI. MD [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 51 canonical work pages

  1. [1]

    [n.d.]. Aira. https://aira.io/. IKIWISI: An Interactive Visual Pattern Generator DIS ’25, July 5–9, 2025, Funchal, Portugal

    2025

  2. [2]

    Bilal Alsallakh, Allan Hanbury, Helwig Hauser, Silvia Miksch, and Andreas Rauber. 2014. Visual methods for analyzing probabilistic classification data. IEEE transactions on visualization and computer graphics 20, 12 (2014), 1703–1712

    2014

  3. [3]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In Proceedings of the IEEE International Conference on computer vision

    2015

  4. [4]

    Behley, M

    J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall

  5. [5]

    BeMyEyes. 2021. Be My Eyes. https://www.bemyeyes.com/

    2021

  6. [6]

    Porter, and I.V

    Syed Masum Billah, Vikas Ashok, Donald E. Porter, and I.V. Ramakrishnan. 2017. Speed-Dial: A Surrogate Mouse for Non-Visual Web Browsing. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 3132531, 110–119. https://doi.org/10.1145/3132525.3132531

    arXiv 2017

  7. [7]

    Syed Masum Billah and Susan Gauch. 2015. Social network analysis for predicting emerging researchers. In 2015 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K) , Vol. 1. IEEE, 27–35

    2015

  8. [8]

    Porter, and I

    Syed Masum Billah, Donald E. Porter, and I. V. Ramakrishnan. 2016. Sinter: low-bandwidth remote access for the visually-impaired. In Proceedings of the Eleventh European Conference on Computer Systems . ACM, 2901335, 1–16. https: //doi.org/10.1145/2901318.2901335

    arXiv 2016

Show all 90 references
  1. [9]

    Miquel Romero Blanch, Zenjie Li, Sergio Escalera, and Kamal Nasrollahi. 2024. LiDAR-Assisted 3D Human Detection for Video Surveillance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 123–131

    2024

  2. [10]

    Ronald J Brachman and Hector J Levesque. 2023. Machines like us: toward AI with common sense. MIT Press

    2023

  3. [11]

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)

    2023 arXiv

  4. [12]

    John M Carroll and Judith Reitman Olson. 1988. Mental models in human- computer interaction. Handbook of human-computer interaction (1988), 45–65

    1988

  5. [13]

    Sonia Castelo, Joao Rulff, Erin McGowan, Bea Steers, Guande Wu, Shaoyu Chen, Iran Roman, Roque Lopez, Ethan Brewer, Chen Zhao, et al. 2023. Argus: Visual- ization of ai-assisted task guidance in ar. IEEE Transactions on Visualization and Computer Graphics (2023)

    2023

  6. [14]

    David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny

  7. [15]

    Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. 2023. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. arXiv preprint arXiv:2310.01957 (2023)

    2023 arXiv

  8. [16]

    Teresa Datta and John P Dickerson. 2023. Who’s Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbook. arXiv preprint arXiv:2303.06223 (2023)

    2023 arXiv

  9. [17]

    Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. 2020. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions ...

    2020

  10. [18]

    Baltasar Fernandez-Manjon and Alfredo Fernandez-Valmayor. 1998. Building educational tools based on formal concept analysis. Education and Information Technologies 3, 3 (1998), 187–201

    1998

  11. [19]

    Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language repre- sentations learn physical commonsense? arXiv preprint arXiv:1908.02899 (2019)

    2019 arXiv

  12. [20]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems , Vol. 27. Curran Associates, Inc., 2672–2680

    2014

  13. [21]

    Rebecca A Grier. 2015. How high is high? A meta-analysis of NASA-TLX global workload scores. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 59. Sage Publications Sage CA: Los Angeles, CA, 1727–1731

    2015

  14. [22]

    Kamath, Aniruddha Kembhavi, and Derek Hoiem

    Tanmay Gupta, A. Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2021. To- wards General Purpose Vision Systems. ArXiv abs/2104.00743 (2021)

    2021 arXiv

  15. [23]

    Kamath, Aniruddha Kembhavi, and Derek Hoiem

    Tanmay Gupta, A. Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2022. To- wards General Purpose Vision Systems. Conference of Computer Vision and Pattern Recognition (CVPR) (2022)

    2022

  16. [24]

    Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2022. Towards general purpose vision systems: An end-to-end task-agnostic vision- language architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16399–16409

    2022

  17. [25]

    Chaeeun Han, Prasenjit Mitra, and Syed Masum Billah. 2024. Uncovering Human Traits in Determining Real and Spoofed Audio: Insights from Blind and Sighted Individuals. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–14

    2024

  18. [26]

    David Hand and Peter Christen. 2018. A note on using the F-measure for evalu- ating record linkage algorithms. Statistics and Computing 28 (2018), 539–547

    2018

  19. [27]

    Andreas Hinterreiter, Peter Ruch, Holger Stitz, Martin Ennemoser, Jürgen Bernard, Hendrik Strobelt, and Marc Streit. 2020. ConfusionFlow: A model- agnostic visualization for temporal analysis of classifier confusion. IEEE Transac- tions on Visualization and Computer Graphics 2...

    2020

  20. [28]

    Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE transactions on visualization and computer graphics 25, 8 (2018), 2674–2693

    2018

  21. [29]

    Md Naimul Hoque, Nazmus Saquib, Syed Masum Billah, and Klaus Mueller. 2020. Toward Interactively Balancing the Screen Time of Actors Based on Observable Phenotypic Traits in Live Telecast. 4, CSCW2, Article 154 (oct 2020), 18 pages. https://doi.org/10.1145/3415225

    2020 doi

  22. [30]

    Adele E Howe and Ryan D Forbes. 2008. Re-considering neighborhood-based collaborative filtering parameters in the context of new data. In Proceedings of the 17th ACM conference on Information and knowledge management . 1481–1482

    2008

  23. [31]

    Md Touhidul Islam and Syed Masum Billah. 2023. SpaceX Mag: An Automatic, Scalable, and Rapid Space Compactor for Optimizing Smartphone App Interfaces for Low-Vision Users. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 2 (2023), 1–36

    2023

  24. [32]

    Md Touhidul Islam, Imran Kabir, Elena Ariel Pearce, Md Alimoor Reza, and Syed Masum Billah. 2024. A Dataset for Crucial Object Recognition in Blind and Low-Vision Individuals’ Navigation. arXiv:2407.16777 [cs.CV] https://arxiv.org/ abs/2407.16777

    2024

  25. [33]

    Md Touhidul Islam, Imran Kabir, Elene Ariel Pearce, Md Alimoor Reza, and Syed Masum Billah. 2024. Identifying Crucial Objects in Blind and Low-Vision Individuals’ Navigation. In The 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS’24). ACM. ht...

    2024 doi

  26. [34]

    Md Touhidul Islam, Noushad Sojib, Imran Kabir, Ashiqur Rahman Amit, Moham- mad Ruhul Amin, and Syed Masum Billah. 2024. Demonstration of Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interaction. In The 37th Annual ACM Symposium on User ...

    2024

  27. [35]

    Md Touhidul Islam, Noushad Sojib, Imran Kabir, Ashiqur Rahman Amit, Mo- hammad Ruhul Amin, and Syed Masum Billah. 2024. Wheeler: A Three- Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interac- tion. In The 37th Annual ACM Symposium on User Interface Soft...

    2024

  28. [36]

    Zhaoyin Jia, Andy Gallagher, Ashutosh Saxena, and Tsuhan Chen. 2014. 3D Reasoning from Blocks to Stability. IEEE Trans PAMI (2014)

    2014

  29. [37]

    Rong Jin and Luo Si. 2004. A study of methods for normalizing user ratings in collaborative filtering. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval . 568–569

    2004

  30. [38]

    Roger T Johnson and David W Johnson. 1986. Cooperative learning in the science classroom. Science and children 24, 2 (1986), 31–32

    1986

  31. [39]

    Imran Kabir, Md Alimoor Reza, and Syed Billah. 2025. Logic-RAG: Augmenting Large Multimodal Models with Visual-Spatial Knowledge for Road Scene Un- derstanding. In 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE

    2025

  32. [40]

    Minsuk Kahng, Pierre Y Andrews, Aditya Kalro, and Duen Horng Chau. 2017. A cti v is: Visual exploration of industry-scale deep neural network models. IEEE transactions on visualization and computer graphics 24, 1 (2017), 88–97

    2017

  33. [41]

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s" up" with vision- language models? Investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19785 (2023)

    2023 arXiv

  34. [42]

    Hakan Karaoguz and Patric Jensfelt. 2019. Object detection approach for robot grasp detection. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 4953–4959

    2019

  35. [43]

    Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiaogang Wang. 2019. Gs3d: An efficient 3d object detection framework for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1019–1028

    2019

  36. [44]

    Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven CH Hoi. 2022. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.09019 (2022)

    2022 arXiv

  37. [45]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. PMLR, 12888–12900

    2022

  38. [46]

    Fangyu Liu, Guy Emerson, and Nigel Collier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11 (2023), 635–651

    2023

  39. [47]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruc- tion tuning. arXiv preprint arXiv:2304.08485 (2023)

    2023 arXiv

  40. [48]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

    2024

  41. [49]

    Mengchen Liu, Jiaxin Shi, Kelei Cao, Jun Zhu, and Shixia Liu. 2017. Analyzing the training processes of deep generative models. IEEE transactions on visualization DIS ’25, July 5–9, 2025, Funchal, Portugal MT Islam, I Kabir, MA Reza, and SM Billah and computer graphics 24, 1 (...

    2017

  42. [50]

    Shixia Liu, Xiting Wang, Mengchen Liu, and Jun Zhu. 2017. Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics 1, 1 (2017), 48–56

    2017

  43. [51]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan

  44. [52]

    Microsoft. 2017. Surface Dial. https://www.microsoft.com/en-us/surface/ accessories/surface-dial

    2017

  45. [53]

    Mark EJ Newman. 2004. Coauthorship networks and patterns of scientific col- laboration. Proceedings of the national academy of sciences 101, suppl_1 (2004), 5200–5205

    2004

  46. [54]

    arXiv preprint arXiv:2306.05424 (2023)

    Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

    2023 arXiv

  47. [55]

    Donald A Norman and Stephen W Draper. 1986. User centered system design; new perspectives on human-computer interaction . L. Erlbaum Associates Inc

    1986

  48. [56]

    OpenAI. [n.d.]. GPTV Sysmtem Card. https://cdn.openai.com/papers/GPTV_ System_Card.pdf

  49. [57]

    Steve Nison. 2001. Japanese candlestick charting techniques: a contemporary guide to the ancient investment techniques of the Far East . Penguin

    2001

  50. [58]

    OpenAI. 2023. GPT-4V(ision) System Card. https://cdn.openai.com/papers/GPTV_ System_Card.pdf

    2023

  51. [59]

    OpenAI. 2023. GPT-4V(ision) technical work and authors . https://openai.com/ contributions/gpt-4v

    2023

  52. [60]

    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774v2 https://arxiv.org/abs/ 2303.08774v2

    2023 arXiv

  53. [61]

    Shuvo Kumar Paul, Muhammed Tawfiq Chowdhury, Mircea Nicolescu, Monica Nicolescu, and David Feil-Seifer. 2021. Object detection and pose estimation from rgb and depth data for real-time, adaptive robotic grasping. In Advances in Computer Vision and Computational Biology: Procee...

    2021

  54. [62]

    David MW Powers. 2020. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061 (2020)

    2020 arXiv

  55. [63]

    Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the web. Technical Report. Stanford infolab

    1999

  56. [64]

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. 2023. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 (2023)

    2023 arXiv

  57. [65]

    Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D Williams

  58. [66]

    Junaid Qadir, Mohammad Qamar Islam, and Ala Al-Fuqaha. 2022. Toward ac- countable human-centered AI: rationale and promising directions. Journal of Information, Communication and Ethics in Society 20, 2 (2022), 329–342

    2022

  59. [67]

    Christopher A Sanchez and Jennifer Wiley. 2009. To scroll or not to scroll: Scrolling, working memory capacity, and comprehending complex texts. Human Factors 51, 5 (2009), 730–738

    2009

  60. [68]

    Daniel J Simons and Daniel T Levin. 1997. Change blindness. Trends in cognitive sciences 1, 7 (1997), 261–267

    1997

  61. [69]

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2022

  62. [70]

    Mark O Riedl. 2019. Human-centered artificial intelligence and machine learning. Human behavior and emerging technologies 1, 1 (2019), 33–36

    2019

  63. [71]

    Anne Treisman. 1985. Preattentive processing in vision.Computer vision, graphics, and image processing 31, 2 (1985), 156–177

    1985

  64. [72]

    Anne M Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive psychology 12, 1 (1980), 97–136

    1980

  65. [73]

    Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2023. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. 7464–7475

    2023

  66. [74]

    Dejan Todorovic. 2008. Gestalt principles. Scholarpedia 3, 12 (2008), 5345

    2008

  67. [75]

    Matthew O Ward, Georges Grinstein, and Daniel Keim. 2010. Interactive Data Disualization: Foundations, Techniques, and Applications. CRC Press

    2010

  68. [76]

    What It Can Create, It May Not Understand

    Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et al. 2023. THE GENERATIVE AI PARADOX:“What It Can Create, It May Not Understand”. In The Twelfth International Conference on Learni...

    2023

  69. [77]

    Jingyi Xie, Rui Yu, He Zhang, Syed Masum Billah, Sooyeon Lee, and John M Carroll. 2025. Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models. In Proceedings of the 2025 CHI Conference on Human Factors in Computi...

    2025

  70. [78]

    Liuping Wang, Zhan Zhang, Dakuo Wang, Weidan Cao, Xiaomu Zhou, Ping Zhang, Jianxing Liu, Xiangmin Fan, and Feng Tian. 2023. Human-centered design and evaluation of AI-empowered clinical decision support systems: a systematic review. Frontiers in Computer Science 5 (2023), 1187299

    2023

  71. [79]

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. 2023. A survey of large language models for autonomous driving. arXiv preprint arXiv:2311.01043 (2023)

    2023 arXiv

  72. [80]

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bag-of-words models, and what to do about it. arXiv preprint arXiv:2210.01936 5 (2022)

    2022 arXiv

  73. [81]

    He Zhang, Nicholas J Falletta, Jingyi Xie, Rui Yu, Sooyeon Lee, Syed Masum Billah, and John M Carroll. 2025. Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System. In Companion...

    2025

  74. [82]

    Jingyi Xie, Rui Yu, He Zhang, Sooyeon Lee, Syed Masum Billah, and John M Carroll. 2024. Emerging practices for large multimodal model (lmm) assistance for people with visual impairments: Implications for design. arXiv preprint arXiv:2407.08882 (2024)

    2024 arXiv

  75. [83]

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang

  76. [84]

    Xingcheng Zhou, Mingyu Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C Knoll. 2023. Vision language models in autonomous driving and intelligent transportation systems. arXiv preprint arXiv:2310.14414 (2023)

    2023 arXiv

  77. [85]

    change blindness,

    John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through Design as a Method for Interaction Design Research in HCI . Association for Com- puting Machinery, New York, NY, USA, 493–502. https://doi.org/10.1145/1240624. 1240704 A APPENDIX: SUMMARY OF DATASET VID...

    2007

  78. [86]

    Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, and David S Ebert. 2018. Man- ifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE transactions on visualization and computer graphics 25, 1 (2018), 364–373

    2018

  79. [2016]

    IEEE transactions on visualization and computer graphics 23, 1 (2016), 61–70

    Squares: Supporting interactive performance analysis for multiclass clas- sifiers. IEEE transactions on visualization and computer graphics 23, 1 (2016), 61–70

    2016

  80. [2018]

    In Proceedings of the IEEE conference on computer vision and pattern recognition

    The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition . 586–595

  81. [2019]

    SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV)

  82. [2023]

    Clair: Evaluating image captions with large language models.arXiv preprint arXiv:2310.12971 (2023)

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.