{"id":"4e4dbe01-8d29-417b-b664-4b27deb8bf3d","arxiv_id":"2608.09121","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MELLON inserts a fine-tuned visual projection layer into a frozen CodeLlama agent and reports higher step-wise action accuracy on a small, nonstandard WebShop test set.","lead":"This paper introduces MELLON, a shopping agent that adds product image understanding to a frozen text model by training a small projection layer between a vision encoder and the language model. The authors report a 9.26 percentage point improvement in action-prediction accuracy on a curated subset of WebShop, but the absolute accuracy is low and their two other proposed methods performed worse than the baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 9.26% gain is step-wise accuracy on a nonstandard 338-instance test set; no task-level metric supports the 'task completion' claim.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the paper equates step-wise action accuracy with task completion accuracy and evaluates on a nonstandard, deliberately selected test set. I examined whether the central claim could still hold despite this, and it cannot. The paper's own Table 3 is the decisive internal evidence: the ViT+Trainable QFormer architecture, which is the visual alignment path used in MELLON, matches the ResNet baseline exactly on reward score (39.92) and is slightly worse on success rate (9.6% vs 9.8%). If the visual projection layer and Q-Former produced a genuine task-level improvement, it should appear here. Instead, the only reported improvement is in step-wise accuracy on a different, smaller test set. The standard WebShop evaluation would also be necessary to compare against the paper's own cited baselines, such as ReAct with 40.0% success rate and 66.6 score. Because the positive result in the abstract is the paper's main contribution, and because the other two proposed methods are honestly reported as failures, the overclaim is not peripheral. The verdict should remain REJECT, as the reader concluded. The proposed concrete test would settle the question by forcing the comparison onto the standard metric and test set.","tokens_in":11225,"tokens_out":3203,"duration_ms":33812,"concrete_test":"Run the trained MELLON model and the ViT+CodeLlama inference baseline on the standard WebShop 500-instance test set, reporting success rate and average reward for both. If MELLON's success rate and reward score are not materially higher than the baseline's (or are comparable to the 9.8% success rate and 39.92 reward shown in Table 3), the headline claim of improved task completion is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that MELLON improves 'task completion accuracy' by 9.26%, but the metric actually reported in Section 4.5 is step-wise action accuracy, not task completion. The paper never reports MELLON's success rate or WebShop reward score on the standard 500-instance test set. The only task-level comparison in Section 5.1.1, Table 3, shows that the ViT+Trainable QFormer variant achieves the same reward score as the ResNet baseline (39.92) and a nearly identical success rate (9.6% vs 9.8%), directly undercutting the claim of task-level improvement. Moreover, MELLON's test set is 338 instances 'curated from the midterm report' and 'particularly challenging for baseline models' (Section 4.3), so the reported step-wise accuracy is not comparable to existing WebShop baselines such as ReAct, IL, or IL+RL. The 9.26% improvement is therefore at best a step-wise accuracy gain on a nonstandard, non-public test set; it does not establish improved task completion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three multimodal enhancements for the WebShop web-navigation benchmark: MELLON, which projects aligned ViT/Q-Former visual embeddings into a frozen CodeLlama model with redesigned ReAct prompts; VQAgent, which frames WebShop as a multiple-choice VQA task using BEiT3; and Multimodal Ranker, which augments action selection with BERT and CLIP similarity scores. The authors report that one epoch of MELLON training improves action accuracy from 5.11% to 14.37% on a 338-instance test set, while VQAgent and Multimodal Ranker both underperform the IL baseline. The abstract interprets the MELLON gain as a 9.26% improvement in 'task completion accuracy,' but Section 4.5 defines the reported metric as step-wise action accuracy, and no task-level success rate or reward score is reported for MELLON on the standard 500-instance WebShop test set.","tokens_in":11395,"tokens_out":6569,"duration_ms":64020,"significance":"If the step-wise accuracy gain were accompanied by task-level improvements on the standard WebShop test set, the parameter-efficient recipe of freezing the LLM and ViT while training only the projection layer and Q-Former would be a useful contribution, and the paper is candid in reporting that VQAgent and Multimodal Ranker fail to beat the baseline. However, the central positive claim is currently supported only by a non-public, nonstandard test set and a metric that does not measure task completion. The paper's only task-level evidence involving the visual architecture shows no improvement over the ResNet baseline, so the significance as stated is not established.","major_comments":[{"comment":"The abstract and Section 5.1.1 claim a 9.26% improvement in 'task completion accuracy,' but Section 4.5 explicitly defines the MELLON metric as step-wise accuracy, i.e., the fraction of steps where the predicted action matches the human action. No success rate or WebShop reward score for MELLON is reported on the standard 500-instance test set. The 9.26% gain is therefore a step-wise accuracy increase on the paper's own test set, not evidence of improved task completion; the conclusion in Section 6 itself reverts to 'step-wise accuracy,' confirming the discrepancy.","section":"Abstract; §4.5; §5.1.1"},{"comment":"The MELLON test set consists of 338 instances 'curated from the midterm report' and described as 'particularly challenging for baseline models,' but the curation procedure and the set itself are not described, justified, or released. Reporting results only on this nonstandard set makes the 9.26% gain non-comparable to existing WebShop results such as ReAct, IL, and IL+RL, and selecting instances specifically hard for baselines can inflate the apparent improvement. The paper should evaluate MELLON on the standard 500-instance test set or provide a clear argument that the curated set is representative.","section":"§4.3"},{"comment":"The only task-level results connected to MELLON's visual architecture show that ViT+Trainable QFormer achieves a score of 39.92, identical to the ResNet baseline, and a success rate of 9.6% versus 9.8%. This is direct evidence that the improved step-wise action accuracy does not translate into improved task completion, and it contradicts the abstract's claim. The authors should report full MELLON success rate and score on the standard test set and reconcile these numbers with the claimed task-completion improvement.","section":"§5.1.1, Table 3"},{"comment":"This section states that 'we integrated the WebShop completion score into the loss function as a contributing factor,' but the loss function defined immediately afterward is a pure cross-entropy loss over action tokens with no completion-score term. Either the score integration should be described precisely, for example as a weighted auxiliary term, or the claim should be removed; as written, the training objective is not actually specified.","section":"§3.1.2"},{"comment":"Section 2.2 reports that ReAct achieves a score of 66.6 and a success rate of 40.0 on WebShop, but Table 4 lists ReAct (CodeLlama-7b) with a score of 24.98. These numbers are inconsistent, and the discrepancy is not explained. Because ReAct is a key comparison baseline, the paper must clarify the configuration and evaluation conditions used in each table.","section":"§2.2 versus §5.1.2"}],"minor_comments":[{"comment":"There are several typos and formatting errors, including 'backbond' in §3.1.1, 'batsh size' in §4.4, 'we doesn't train' in §5.1.1, and 'MELON' instead of 'MELLON' in Table 7.","section":"Throughout"},{"comment":"The MELLON training set of 8,237 instances is not related to the standard WebShop training split of 10,587 instances; please state explicitly how the training instances were filtered or subsampled.","section":"§4.3"},{"comment":"The phrase '9.26% increase' is ambiguous between an absolute percentage-point gain and a relative gain; from Table 2 it is an absolute increase from 5.11% to 14.37%, so the wording should be clarified.","section":"§5.1.1, Table 2"},{"comment":"The transition from the initially trained projection layer to the 'dual-focused' training strategy is described only qualitatively; a precise description of the data collection, loss weighting, and training schedule would be needed for reproducibility.","section":"§3.1.2"},{"comment":"Several textual references such as 'example 7', '11', '12', and '13' should be replaced with consistent figure or appendix numbers.","section":"§5.1.3 and Appendix"}],"recommendation":"reject","confidential_remarks":"The manuscript has the character of a course project report, and the central claim in the abstract is not supported by the reported metric or test set. The most serious issue is that the paper's own task-level evidence in Table 3 shows no task-completion improvement. I would not recommend acceptance unless the authors can provide standard WebShop success-rate and reward-score results for MELLON and clarify the relationship between the step-wise accuracy gain and task-level performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked about MELLON. The short version: it's a course project that puts together BLIP-2-style visual alignment with a frozen CodeLlama for WebShop, and it reports a 9.26% accuracy gain after one epoch of training. What's actually new is modest but real: applying a ViT/Q-Former/projection pipeline to WebShop with a frozen LLM and ReAct-style prompting, and showing it can improve step-wise action prediction on a small curated set. The paper also deserves credit for honest reporting of two failed methods (VQAgent and Multimodal Ranker) with genuine error analysis, and for clearly stating its limitations.\n\nThe soft spot is load-bearing: the abstract and Section 5.1.1 call the gain \"task completion accuracy,\" but the metric defined in Section 4.5 is step-wise action accuracy — whether the model picks the human's action at each step given the same observation. That is not task completion. The test set is 338 instances \"curated from the midterm report\" and chosen because they are \"particularly challenging for baseline models,\" so the result is not comparable to the standard WebShop 500-instance test set. The paper never reports success rate or reward score for MELLON on the standard split. Worse, the ablation in Table 3 shows ViT + Trainable Q-Former achieves the same reward score as the ResNet baseline (39.92) and nearly the same success rate (9.6% vs 9.8%), which undercuts any task-level improvement claim. The loss function is also under-specified (the \"WebShop completion score\" mentioned in Section 3.1.2 never appears in the formula), and no code is released.\n\nThese are not tiny quibbles: they break the central claim as stated. If the authors had reported success rate and reward on the standard test split, the result might be a modest but credible step. As it stands, the positive result is a step-wise accuracy gain on a non-public, cherry-picked set.\n\nThat said, the paper is not a mess. The writing is clear, the negative results are analyzed thoughtfully, and the engineering effort (e.g., dealing with WebShop's environment) is real. For a course project, this is solid work. For a paper claiming a task-level improvement, it needs a different evaluation.\n\nMy recommendation: don't send this to a serious venue as-is. If the authors fix the evaluation and report task-level metrics on standard splits, it could become a decent workshop paper. For now, it's a REJECT with a clear path to revision.\n\nBest.","headline":"A course project with a real architecture and honest negative results, but the headline claim about task completion accuracy is not supported by the reported step-wise metric on a curated test set.","tokens_in":11991,"tokens_out":3311,"would_cite":false,"duration_ms":30872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MELLON, a multimodal web navigation agent, raises step-wise action accuracy on WebShop from 5.11% to 14.37% after one epoch of training.","keywords":["MELLON","WebShop","multimodal web navigation","visual-text alignment","frozen LLM","parameter-efficient training","ReAct prompting","imitation learning"],"falsifier":"Train MELLON for one epoch as described and evaluate it on the standard 500-instance WebShop test set, reporting success rate and reward score. If the score and success rate do not exceed the ReAct baselines of 66.6 and 40.0 percent, the claimed 9.26-point improvement does not carry over to the benchmark's standard metric.","tokens_in":10960,"feed_emoji":"🛒","tokens_out":9811,"duration_ms":84154,"temperature":0.7,"pith_summary":"The paper sets out to show that web navigation agents benefit from seeing product images as well as reading page text, and that this can be achieved without retraining the language model. Its main proposal, MELLON, feeds aligned visual embeddings from a ViT and Q-Former through a single trainable projection layer into a frozen CodeLlama, trained with a cross-entropy action-prediction loss. On the WebShop benchmark the authors report step-wise action accuracy climbing from 5.11% to 14.37% after one training epoch, an average gain of 9.26 percentage points. Two alternative proposals, VQAgent and Multimodal Ranker, perform worse than their baselines, which the paper interprets as evidence about when multimodal input helps. The core claim is that parameter-efficient visual alignment can give a frozen LLM meaningful web-navigation ability.","feed_headline":"One training epoch lifts web-agent accuracy by 9.26 points","feed_subtitle":"MELLON adds aligned image embeddings to a frozen CodeLlama and beats its text-only WebShop baseline.","key_machinery":"The load-bearing mechanism is the projection layer: a single trainable linear mapping placed between the Q-Former's visual output and the frozen LLM's embedding space. It converts visual information into prompt tokens inserted at an <ImageHere> slot in a shortened ReAct prompt, allowing the LLM to reason over text and image together. Because only the projection layer and Q-Former are trained, the whole adaptation fits in one epoch on a 24 GB GPU.","core_discovery":"The paper's central claim is that a frozen large language model can become a multimodal web navigation agent through a small trainable bridge. MELLON uses an EVA ViT to encode product images, a Q-Former to compress them, and a projection layer to turn the result into soft prompt tokens that CodeLlama can read alongside the WebShop accessibility tree and a ReAct-style reasoning prompt. After one epoch of training the projection layer and Q-Former with a cross-entropy generative loss, step-wise action accuracy on the paper's 338-instance test set rises from 5.11% to 14.37%, which the abstract reports as a 9.26% increase in task completion accuracy. The authors also find that a ViT-based encoder with a trainable Q-Former matches the ResNet baseline on score and success rate, while a frozen Q-Former does not improve over ResNet. On their own terms, the result is that multimodal alignment improves an LLM's action selection on WebShop without unfreezing the LLM.","pith_inferences":["Editorial inference: the reported gain is measured on step-wise action accuracy, not success rate or reward score; on the standard 500-instance WebShop test the improvement could be smaller or absent, and the paper does not provide that number.","Editorial inference: because the test set was curated to be challenging for baselines, the 9.26-point gain may overstate average-case benefit if hard cases are over-represented.","Editorial inference: the failure of VQAgent and Multimodal Ranker suggests that in WebShop, text usually carries more information than images; a testable implication is that MELLON's gains concentrate on attributes that appear only in images, such as color or material.","Editorial inference: a direct comparison on standard WebShop success and reward metrics against ReAct and the WebShop IL baseline would settle whether the step-wise gain translates into better task completion."],"forward_implications":["MELLON can be adapted to new frozen LLMs without retraining the model, so stronger base models should translate directly into better web agents.","The one-epoch training budget makes the approach practical for environments where collecting large trajectory datasets is expensive.","The authors' error analysis suggests frozen CodeLlama's empty or invalid outputs are the main remaining bottleneck, so better prompting or light LLM tuning is the next lever.","The architecture is defined on WebShop but is not benchmark-specific; the same projection-layer recipe could be applied to other multimodal web environments."],"supporting_citations":[{"why":"Supplies the WebShop environment, action space, IL baseline, and reward and success definitions used in all experiments.","marker":"(Yao et al., 2023a)"},{"why":"Provides the ReAct reasoning-and-acting prompt design that MELLON extends with visual prompt tokens.","marker":"(Yao et al., 2023b)"},{"why":"Supplies the Q-Former visual-language module used to compress image features before the projection layer.","marker":"(Li et al., 2023)"},{"why":"Supplies the EVA ViT backbone used to encode product images.","marker":"(Fang et al., 2022)"},{"why":"Inspires the single projection layer that aligns visual encoder outputs with the LLM.","marker":"(Su et al., 2023)"},{"why":"The other projection-layer alignment reference used to justify the trainable visual bridge.","marker":"(Zhu et al., 2023)"},{"why":"Prior multimodal web agent that the paper contrasts with, noting it removed the image encoder for WebShop.","marker":"(Furuta et al., 2023)"}],"fun_headline_variants":["Frozen LLM learns multimodal web navigation in one epoch","Multimodal alignment lifts WebShop agent accuracy by 9.26%","Trainable vision bridge makes LLM a better web navigator","One epoch of multimodal training boosts web agent accuracy by 9.26%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on treating step-wise action accuracy on a 338-instance test set, which the authors curated because it was hard for baselines, as task completion accuracy; if that metric and test set do not represent the standard WebShop benchmark, the headline improvement may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLM learns multimodal web navigation in one epoch","Multimodal alignment lifts WebShop agent accuracy by 9.26%","Trainable vision bridge makes LLM a better web navigator","One epoch of multimodal training boosts web agent accuracy by 9.26%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":2982,"prompt_tokens":890,"completion_tokens":2092,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2016}},"tokens_in":506,"tokens_out":2092,"duration_ms":16334,"temperature":1.0,"reasoning_tokens":2016,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:05:49.021465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MELLON for one epoch as described and evaluate it on the standard 500-instance WebShop test set, reporting success rate and reward score. If the score and success rate do not exceed the ReAct baselines of 66.6 and 40.0 percent, the claimed 9.26-point improvement does not carry over to the benchmark's standard metric.","supporting_citations":[],"review_version":1}