MELLON inserts a fine-tuned visual projection layer into a frozen CodeLlama agent and reports higher step-wise action accuracy on a small, nonstandard WebShop test set.
Mapping Natural Language Commands to Web Elements
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.
fields
cs.AI 1years
2026 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
MELLON - Multimodal Enhanced LLM for Online Navigation
MELLON inserts a fine-tuned visual projection layer into a frozen CodeLlama agent and reports higher step-wise action accuracy on a small, nonstandard WebShop test set.