See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

Ding Xia; Dongyuan Li; Fan Gao; Mark Colley; Renhe Jiang; Takeo Igarashi; Xinyue Gui; Zhongyi Zhou

arxiv: 2602.02063 · v2 · submitted 2026-02-02 · 💻 cs.HC · cs.AI

See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

Ding Xia , Xinyue Gui , Mark Colley , Fan Gao , Zhongyi Zhou , Dongyuan Li , Renhe Jiang , Takeo Igarashi This is my paper

Pith reviewed 2026-05-16 08:23 UTC · model grok-4.3

classification 💻 cs.HC cs.AI

keywords eHMILLM action designvision-language feedbackautonomous vehiclesclosed-loop refinementexternal interfaceshuman-machine communication

0 comments

The pith

Vision-language models refine LLM outputs for vehicle external interfaces by evaluating action appropriateness in context.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents See2Refine, a closed-loop system that feeds vision-language model assessments back to large language models to iteratively improve generated external human-machine interface actions for automated vehicles. Most prior eHMI work uses static developer-specified designs that cannot scale to changing traffic scenes, while prompt-only LLMs lack perceptual checks. The framework treats VLM judgments of visual appropriateness as training signals for revision, producing actions that score higher on both automated metrics and human ratings. Results hold across light-bar, eye, and arm modalities and multiple model scales, showing the refinement step adds value beyond the initial prompt.

Core claim

See2Refine closes the loop by having a vision-language model score the perceived appropriateness of an LLM-generated eHMI action in a given driving scene and then using that score to prompt the LLM to revise its output, yielding designs that outperform both fixed prompt baselines and manually authored references on VLM metrics and in human-subject tests.

What carries the argument

The iterative revision step that converts VLM perceptual evaluations into targeted prompts for the LLM eHMI designer.

If this is right

The method scales to new driving contexts without requiring fresh human annotations for each scenario.
Performance gains appear across three distinct eHMI output modalities.
Alignment between VLM and human ratings supports using the automated loop as a proxy during development.
Larger and smaller LLMs both benefit from the same refinement procedure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Similar visual-feedback loops could be applied to other LLM-generated interfaces that require real-time human readability.
If VLM-human alignment holds in more complex scenes, the framework could support on-vehicle adaptation rather than offline design only.
The approach reduces reliance on costly human feedback loops that currently limit eHMI customization.

Load-bearing premise

VLM scores of action appropriateness serve as reliable, unbiased stand-ins for human judgment across traffic contexts.

What would settle it

A controlled human rating study in which participants consistently prefer the original prompt-only LLM actions over the VLM-refined versions on the same scenes.

read the original abstract

Automated vehicles lack natural communication channels with other road users, making external Human-Machine Interfaces (eHMIs) essential for conveying intent and maintaining trust in shared environments. However, most eHMI studies rely on developer-crafted message-action pairs, which are difficult to adapt to diverse and dynamic traffic contexts. A promising alternative is to use Large Language Models (LLMs) as action designers that generate context-conditioned eHMI actions, yet such designers lack perceptual verification and typically depend on fixed prompts or costly human-annotated feedback for improvement. We present See2Refine, a human-free, closed-loop framework that uses vision-language model (VLM) perceptual evaluation as automated visual feedback to improve an LLM-based eHMI action designer. Given a driving context and a candidate eHMI action, the VLM evaluates the perceived appropriateness of the action, and this feedback is used to iteratively revise the designer's outputs, enabling systematic refinement without human supervision. We evaluate our framework across three eHMI modalities (lightbar, eyes, and arm) and multiple LLM model sizes. Across settings, our framework consistently outperforms prompt-only LLM designers and manually specified baselines in both VLM-based metrics and human-subject evaluations. Results further indicate that the improvements generalize across modalities and that VLM evaluations are well aligned with human preferences, supporting the robustness and effectiveness of See2Refine for scalable action design.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

See2Refine gives a clean closed-loop VLM feedback method for refining LLM eHMI actions, but the evaluation details stay too thin to fully back the outperformance claims.

read the letter

The paper's main move is See2Refine: an LLM proposes eHMI actions for a driving scene, a VLM scores how appropriate they look, and that score drives iterative revisions without further human input. This is presented as a step beyond fixed prompts or manual baselines, and they test it on lightbar, eyes, and arm modalities plus different LLM sizes. The abstract says the loop beats the baselines on both VLM metrics and human ratings, and that the VLM scores line up with people well enough to generalize across settings. That framework description is the clearest part of the work and the part that could be useful to others trying to automate perceptual design tasks in HCI. The tests across modalities and model scales are a reasonable start for showing the idea is not tied to one narrow case. The soft spots sit in the evaluation. The abstract states consistent gains and good VLM-human alignment but gives no numbers on the metrics, no statistical tests, no controls for scene dynamics, and no checks on whether the iteration step introduces new biases instead of removing them. The stress-test point about VLM failure modes in edge cases is not addressed in the provided summary, so the claim that the loop reliably improves outputs rests on unshown evidence. If the full paper has solid ablations and agreement rates after controlling for modality-specific errors, that would strengthen it; right now it reads as preliminary. This is for people working on external interfaces for autonomous vehicles or on LLM pipelines that need perceptual feedback loops. A reader already familiar with eHMI studies would see the practical angle quickly. It deserves peer review because the core loop is straightforward to implement and the problem it targets is real, even if the current results need more transparent validation before anyone treats the alignment as settled.

Referee Report

3 major / 2 minor

Summary. The paper introduces See2Refine, a closed-loop framework that uses a vision-language model (VLM) to evaluate the perceptual appropriateness of candidate eHMI actions generated by an LLM and then iteratively revises the LLM outputs based on that feedback. The central claim is that this human-free refinement process yields eHMI actions that outperform both prompt-only LLM baselines and manually specified designs across three modalities (lightbar, eyes, arm), with the improvements holding in both VLM-based metrics and direct human-subject evaluations, and with VLM scores aligning well with human preferences.

Significance. If the VLM-human alignment and iterative improvement claims hold under rigorous controls, the work would provide a scalable method for context-adaptive eHMI design without costly human annotation loops, addressing a practical bottleneck in AV communication research. The multi-modality evaluation and inclusion of human validation are strengths; however, the load-bearing assumption that VLM feedback reliably proxies human perception without amplifying modality-specific biases remains under-supported by the reported evidence.

major comments (3)

[Abstract and Evaluation] Abstract and Evaluation section: the claim of 'consistent outperformance' and 'well aligned' VLM-human evaluations is stated without accompanying correlation coefficients, agreement rates, or statistical tests; this is load-bearing because the closed-loop refinement rests on VLM scores being an unbiased proxy.
[Results and Human Evaluation] Results and Human Evaluation subsections: no description of participant count, trial structure, or controls for order effects and modality-specific failure modes (e.g., dynamic scenes) is supplied, preventing verification that human judgments confirm the VLM-driven gains rather than reflecting the same biases.
[Framework description] Framework description (likely §3 or §4): the precise mechanism for converting VLM textual feedback into prompt revisions, the number of refinement iterations, and stopping criteria are not specified, making it impossible to assess whether the reported gains are due to systematic improvement or prompt engineering artifacts.

minor comments (2)

[Evaluation] Notation for the three modalities is introduced without a summary table; a compact comparison table of VLM vs. human scores per modality would improve readability.
[Results] The abstract mentions 'multiple LLM model sizes' but the results do not tabulate performance by model scale; adding this breakdown would strengthen the generalization claim.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We agree that additional quantitative support, methodological details, and framework clarifications are needed to strengthen the manuscript. We will revise accordingly to address each point.

read point-by-point responses

Referee: [Abstract and Evaluation] Abstract and Evaluation section: the claim of 'consistent outperformance' and 'well aligned' VLM-human evaluations is stated without accompanying correlation coefficients, agreement rates, or statistical tests; this is load-bearing because the closed-loop refinement rests on VLM scores being an unbiased proxy.

Authors: We agree that explicit quantitative measures of alignment are required to support the proxy assumption. In the revised manuscript we will add Pearson correlation coefficients, percentage agreement, Cohen's kappa, and statistical significance tests (paired t-tests or Wilcoxon signed-rank) between VLM scores and human ratings. These will appear in the Evaluation section and be summarized in the abstract. revision: yes
Referee: [Results and Human Evaluation] Results and Human Evaluation subsections: no description of participant count, trial structure, or controls for order effects and modality-specific failure modes (e.g., dynamic scenes) is supplied, preventing verification that human judgments confirm the VLM-driven gains rather than reflecting the same biases.

Authors: We will expand the Human Evaluation subsection with the omitted details: participant count and demographics, within-subjects trial structure with randomized order presentation, attention checks, and separate reporting for static versus dynamic scenes. Inter-rater reliability will also be reported to demonstrate that human judgments support the VLM-driven improvements rather than shared biases. revision: yes
Referee: [Framework description] Framework description (likely §3 or §4): the precise mechanism for converting VLM textual feedback into prompt revisions, the number of refinement iterations, and stopping criteria are not specified, making it impossible to assess whether the reported gains are due to systematic improvement or prompt engineering artifacts.

Authors: We will add a precise description, pseudocode, and example transformations in Section 3. The mechanism parses VLM textual feedback into appended instructions for the LLM prompt; we use a fixed maximum of four iterations with an early-stopping rule based on VLM score plateau. This will clarify that gains arise from the iterative process rather than ad-hoc prompting. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework with independent VLM and human evaluations

full rationale

The paper presents See2Refine as an iterative, closed-loop system in which an LLM generates candidate eHMI actions and a separate VLM supplies perceptual feedback for refinement. Evaluation proceeds via two distinct channels: VLM-based appropriateness metrics and independent human-subject studies. No equations, fitted parameters, or uniqueness theorems are invoked; the central claims rest on comparative performance numbers across modalities and model sizes rather than any derivation that reduces outputs to inputs by construction. Self-citations, if present, are not load-bearing for the reported outperformance or alignment results. The derivation chain is therefore self-contained and non-circular.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that VLMs can serve as accurate automated proxies for human perceptual judgment of eHMI appropriateness; no free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)

domain assumption Vision-language models can reliably evaluate the perceived appropriateness of eHMI actions in driving contexts
This assumption underpins the entire feedback loop and is invoked without detailed justification or validation in the abstract.

pith-pipeline@v0.9.0 · 5573 in / 1155 out tokens · 56985 ms · 2026-05-16T08:23:14.358418+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We propose SEE2REFINE, a closed-loop framework that integrates VLM-based perceptual evaluation as automated visual feedback to improve LLM-based eHMI action designers... kernel score K = (κ×s) + t + τ + u + c
IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

iterative preference-based learning... DPO... importance sampling... diverse beam search

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.