pith. machine review for the scientific record. sign in

arxiv: 2602.16138 · v2 · submitted 2026-02-18 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models

Parsa Madinei , Srijita Karmakar , Russell Cohen Hoffing , Felix Gervitz , Miguel P. Eckstein
Authors on Pith no claims yet

Pith reviewed 2026-05-15 21:36 UTC · model grok-4.3

classification 💻 cs.CV
keywords eye-trackingvisual question answeringvision-language modelsambiguity resolutiongaze datareal-time inferenceopen-ended VQA
0
0 comments X

The pith

Eye fixations right at the start of a spoken question more than double VLM accuracy on ambiguous visual queries.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents IRIS, a training-free method that feeds real-time eye-tracking data into large vision-language models to resolve which object a user means in open-ended visual questions. It finds that fixations captured closest to the moment a person begins speaking their question supply the strongest disambiguation signal. Across 500 image-question pairs, adding these fixations raises accuracy on ambiguous cases from 35.2 percent to 77.2 percent while leaving performance on clear queries unchanged. The gains hold across multiple current VLMs, and the authors release the gaze data, a real-time protocol, and an evaluation suite as a new benchmark.

Core claim

IRIS shows that eye fixations nearest to verbal question onset encode the user's intended visual referent most reliably, allowing large vision-language models to produce correct answers on ambiguous image-question pairs more than twice as often (35.2 percent to 77.2 percent) without any model retraining or fine-tuning.

What carries the argument

Real-time integration of eye fixations at verbal question onset, which selects and emphasizes the image region the model should attend to during inference.

If this is right

  • Accuracy on ambiguous open-ended VQA rises consistently across current large vision-language models regardless of their internal architecture.
  • Performance on unambiguous image-question pairs stays at the original high level when gaze data is added.
  • A new public benchmark of 500 gaze-annotated image-question pairs becomes available for testing future disambiguation methods.
  • Real-time interactive VQA becomes possible without retraining any model weights.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same fixation timing cue could be tested in other interactive vision tasks such as referring-expression generation or image editing driven by spoken intent.
  • Wearable eye-tracking hardware paired with voice assistants might allow everyday users to point at objects simply by looking while speaking.
  • Robustness checks with lower-quality consumer eye trackers or multilingual questions would reveal how sensitive the timing cue is to real-world noise.

Load-bearing premise

That the eye fixations recorded exactly when a user starts speaking reliably mark the intended object and can be delivered to the model with negligible timing error.

What would settle it

Run the same ambiguous questions while adding random timing jitter or spatial noise to the eye data at spoken onset and check whether the accuracy gain over the no-gaze baseline shrinks below statistical significance.

read the original abstract

We introduce IRIS (Intent Resolution via Inference-time Saccades), a novel training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA. Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs, more than doubling the accuracy of responses on ambiguous questions (from 35.2% to 77.2%) while maintaining performance on unambiguous queries. We evaluate our approach across state-of-the-art VLMs, showing consistent improvements when gaze data is incorporated in ambiguous image-question pairs, regardless of architectural differences. We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces IRIS, a training-free method that leverages real-time eye-tracking fixations to resolve ambiguity in open-ended visual question answering (VQA) for large vision-language models (VLMs). A user study with 500 unique image-question pairs shows that fixations nearest the onset of verbal questions are most informative, more than doubling accuracy on ambiguous queries from 35.2% to 77.2% while preserving performance on unambiguous ones. The approach is evaluated across multiple state-of-the-art VLMs, and the paper releases a new benchmark dataset, real-time interactive protocol, and evaluation suite.

Significance. If the reported accuracy gains hold under realistic deployment conditions, IRIS would represent a meaningful advance in interactive multimodal systems by enabling training-free intent resolution via natural gaze behavior. The release of the benchmark dataset and protocol is a clear strength that supports reproducibility and follow-on work. However, the absence of statistical validation details and robustness analysis for real-time onset detection substantially weakens the immediate significance of the central empirical claim.

major comments (2)
  1. [Abstract] Abstract: the headline result (fixations at verbal onset doubling accuracy from 35.2% to 77.2% on ambiguous questions) is presented without evidence that the same fixations would be selected under real-time voice activity detection or ASR, which introduces latency and boundary errors; this directly undermines the claim that the method works in real time as stated.
  2. [User Study] User study description: no details are given on statistical tests for the accuracy improvement, controls for how ambiguous questions were labeled, or the exact processing pipeline for selecting and aligning fixations to verbal onset, leaving the reliability of the 77.2% figure difficult to evaluate.
minor comments (2)
  1. The real-time protocol section would benefit from a diagram or pseudocode showing the end-to-end pipeline from gaze capture to VLM input augmentation.
  2. Clarify whether the 500-pair study used post-hoc audio annotation for onset timing or an online detection method, as this distinction is central to the real-time applicability claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback on our manuscript. We address each major comment below and have revised the paper to improve clarity on real-time aspects and to provide the requested methodological details.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline result (fixations at verbal onset doubling accuracy from 35.2% to 77.2% on ambiguous questions) is presented without evidence that the same fixations would be selected under real-time voice activity detection or ASR, which introduces latency and boundary errors; this directly undermines the claim that the method works in real time as stated.

    Authors: The headline result is based on fixations aligned to precise verbal onsets from synchronized audio recordings collected in the controlled user study. We agree that the abstract and main text should better address real-time deployment. In the revised manuscript we have updated the abstract to qualify the result as relying on accurate onset detection and added a new subsection on real-time implementation. This subsection discusses compatibility with standard VAD systems, notes typical detection latencies, and reports a sensitivity analysis in which onset timing is deliberately shifted by up to 300 ms; performance remains well above the no-gaze baseline. Full end-to-end simulation of ASR boundary errors is acknowledged as future work. revision: partial

  2. Referee: [User Study] User study description: no details are given on statistical tests for the accuracy improvement, controls for how ambiguous questions were labeled, or the exact processing pipeline for selecting and aligning fixations to verbal onset, leaving the reliability of the 77.2% figure difficult to evaluate.

    Authors: We have substantially expanded the user-study section. The revision now reports a paired t-test (t = 14.8, p < 0.001) confirming the statistical significance of the accuracy gain. Ambiguity labeling was performed by three independent annotators who classified each image-question pair according to whether multiple plausible answers existed; inter-annotator agreement reached Fleiss' kappa = 0.81. The fixation-selection pipeline is described in detail: eye-tracking streams (120 Hz) were time-synchronized to audio via a shared hardware clock, verbal onset was defined as the timestamp of the first word in the transcribed question, and the single fixation whose center was temporally closest to that onset was retained. These additions allow direct evaluation of the reported 77.2 % figure. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical user-study result with independent data support

full rationale

The paper reports an empirical user study on 500 image-question pairs demonstrating that fixations nearest verbal question onset improve VLM accuracy on ambiguous VQA (35.2% to 77.2%). No equations, derivations, fitted parameters, or self-citations are used to derive this result; the claim rests directly on observed human gaze data collected in the study. The approach is explicitly training-free and does not invoke uniqueness theorems, ansatzes, or prior author work to justify core measurements. The skeptic concern about real-time onset detection is a deployment limitation, not evidence that the reported empirical finding reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical derivations or free parameters are described; the approach is an empirical protocol relying on standard eye-tracking hardware and VLM inference.

pith-pipeline@v0.9.0 · 5462 in / 1037 out tokens · 22932 ms · 2026-05-15T21:36:26.891630+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.