Pith. sign in

REVIEW 3 major objections 5 minor 39 references

MMShopBench tests whether shopping agents can pull a full request out of images and multi-turn dialogue, then verify each candidate against every requirement before recommending it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:34 UTC pith:6P4UTLON

load-bearing objection A genuinely useful new shopping-agent benchmark with a real gap that needs closing: the paper never shows the SFT corpus is disjoint from the evaluation set, so the headline SFT gains are uninterpretable until that's resolved. the 3 major comments →

arxiv 2607.29002 v1 pith:6P4UTLON submitted 2026-07-31 cs.AI

MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents

Zeying Hao , Hao Guo , Mengtao Xu , Yimin Hu , Yuheng Song , Zesheng Zhou , Jinsong Lan , Xiaoyong Zhu This is my paper
classification cs.AI
keywords multimodal shopping agentmulti-turn dialoguebenchmarkreal shopping logsmandatory requirementsevidence-grounded verificationregion-aware image retrievalsupervised fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MMShopBench introduces the first real-log benchmark for multimodal, multi-turn shopping agents: 289 real interactions from an online shopping assistant, each annotated with the purchase intent and the non-negotiable mandatory attributes a target product must satisfy. Agents must infer that specification from user images and dialogue, retrieve candidates through text and image search, and verify each candidate against product images, structured attributes, and OCR text from a frozen 100,000-product catalog. The paper's central empirical claim is that fine-tuning an open-source model on 900 teacher-generated shopping trajectories lifts its top-one success rate from 5.5% to 52.9%, substantially narrowing the gap to leading proprietary assistants. The authors use this to argue that real multimodal dialogue, not just text queries, is a distinct and trainable shopping-agent capability.

Core claim

The paper claims that real shopping requests are multimodal and multi-turn in a way no prior benchmark captures: users express requirements through uploaded images, ambiguous phrases like 'this kind', and later refinements, so the operative request is an accumulated specification rather than the latest utterance. MMShopBench operationalizes that claim by preserving full conversation histories, annotating intent and mandatory requirements, and scoring agents with a strict conjunctive rule: a product is valid only if evidence supports the intent and every mandatory requirement. Its evaluation shows that supervised fine-tuning on teacher trajectories materially improves open models, with the be

What carries the argument

The load-bearing mechanism is Evidence-Grounded Verification and Selection (EGVS), a post-retrieval stage that reconstructs the full candidate pool from every text and image search call, assembles per-product evidence including title, primary image, structured attributes, SKU labels, and product-detail OCR, then asks the policy model to confirm a candidate only if that evidence supports the inferred intent and every inferred mandatory requirement. Two interface choices make retrieval work in multi-turn settings: persistent image identities, where each user image gets a session-global img_idx that later tool calls can reference, and requirement-conditioned region retrieval, where the agent pr

Load-bearing premise

All headline numbers rest on a single frozen LLM judge being correct when it decides whether product evidence supports the annotated intent and every mandatory requirement; no human-agreement or error analysis is reported, so a systematic judge bias would shift every metric.

What would settle it

Take a random sample of cases, have several human annotators independently apply the same conjunctive rule (intent plus all mandatory requirements) to the same candidate pools, and compare their binary decisions with the frozen judge's. If human-judge agreement is low on visual attributes, OCR text, or multi-requirement logic, the reported Judge@1 and Judge@3 gains are not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim is correct, text-only shopping benchmarks omit a core capability: removing user images degrades Judge@3 most on image-required and mixed cases, while text-sufficient cases change little.
  • Supervised fine-tuning on only 900 teacher trajectories produces large gains for open models across 9B, 27B, and 122B scales, suggesting the bottleneck is operational policy rather than model size alone.
  • EGVS raises Judge@3 for every configuration tested, and ID@3 also rises, indicating verification recovers products the agent's own final selection dropped.
  • Because every returned product ID must come from a tool result, the benchmark keeps each recommendation traceable to a retrieval action, which supports reproducible offline evaluation.
  • Preserving turn order and persistent image identities means the benchmark measures accumulated specification inference, not single-turn retrieval.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to audit the frozen LLM judge against human agreement on a subset of cases; all headline numbers depend on its binary decisions, and no human-agreement or error analysis is reported in the paper.
  • The training recipe suggested by the paper - teacher trajectories plus an explicit verification stage - could transfer to other multimodal agent domains where final-answer supervision alone underperforms.
  • Since the exact-identifier ID@k metric also improves with fine-tuning, the gains are unlikely to be purely an artifact of judge leniency; a stronger test would report per-intent-subtype Judge@1 scores.
  • The sandbox's frozen catalog and traceable product IDs could support future studies of how agents resolve ambiguous referents across turns, such as whether region-retrieval accuracy predicts end-to-end success.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MMShopBench, a benchmark of 289 real multimodal, multi-turn shopping-assistant conversations, each annotated with purchase intent, mandatory requirements, and verified target products. Agents operate in a frozen 100,000-product sandbox with BM25 text search and region-conditioned image retrieval, followed by an Evidence-Grounded Verification and Selection (EGVS) step. The authors evaluate proprietary and open models using a GPT-5.5 evidence-grounded judge, and construct a 900-conversation SFT corpus from Gemini-3.1-Pro-Preview teacher trajectories. Fine-tuning Qwen3.5-122B-A10B raises Judge@1 from 5.5% to 52.9% and ID@1 from 4.8% to 49.8%, narrowing the gap to proprietary models. The paper also reports an EGVS ablation and a modality-dependence diagnostic showing that removing images degrades performance on image-required and mixed cases.

Significance. If the results hold, MMShopBench is a valuable resource: it is built from real shopper-assistant logs, provides human annotations of intent and hard requirements, and includes a reproducible offline sandbox. The use of a separate LLM judge alongside a deterministic ID@k metric is a strength, and the EGVS ablation is informative. However, the benchmark's validity rests on the evaluation set being properly disjoint from the SFT training data; the current manuscript does not establish this, and the reported SFT gains are so large that contamination would be a major confound. The lack of confidence intervals and of any human-validation of the LLM judge further limits the strength of the quantitative claims, though ID@k partially mitigates the judge concern.

major comments (3)
  1. [Supervised Fine-Tuning / Data Collection, Curation, and Annotation] The SFT corpus is described as 'screen[ing] 900 multimodal, multi-turn conversations from online shopping-assistant logs' with no statement that these 900 conversations are disjoint from the 10,000 sampled for MMShopBench or from the final 289 evaluation cases. Since both draw from the same deployed-assistant logs (June–July 2026 for the eval pool; no time window given for SFT), overlap is plausible. If even a small fraction of the 289 evaluation cases appear in the SFT trajectories, the headline 5.5%→52.9% Judge@1 and 4.8%→49.8% ID@1 gains would reflect memorization rather than generalization. ID@k is judge-independent, so this concern is prior to any LLM-judge bias. Please provide an explicit disjointness statement or an overlap analysis (e.g., conversation-hash deduplication) and, if necessary, re-run the SFT experiments on a strictly disjoint training set.
  2. [Evaluation Metrics, paragraph 2] GPT-5.5 is the sole arbiter of the binary decision y_{n,j} on whether product evidence supports the annotated intent and every mandatory requirement. No human-agreement analysis, error analysis, or calibration subset is reported. Freezing the prompt (citing Zheng et al. 2023) controls prompt drift but not systematic judge bias on visual attributes, OCR, or conjunctive requirement logic. Because Judge@k is a primary reported metric, this is load-bearing for the absolute results. Please report agreement between GPT-5.5 and human annotators on a sample, ideally stratified by the Figure 3 modality regimes, and discuss failure modes. ID@k provides a partial complement but does not validate the judge-based numbers.
  3. [Table 3 and Table 4] With N=289, the reported percentages have no confidence intervals or significance testing. Single-point differences in Table 3 are within sampling noise; e.g., Qwen3.5-9B+SFT and Qwen3.5-122B-A10B+SFT both report Judge@1=52.9, and the ID@1 difference of 0.7 points is not meaningful. Similarly, Table 4 EGVS deltas such as +0.7 (MiniMax-M2.7) are indistinguishable from zero. The headline 47-point SFT gain is likely significant, but the paper should still report bootstrap CIs or exact binomial intervals for all main metrics, and at least for the SFT gains and EGVS ablations, so readers can assess the precision of the comparative claims.
minor comments (5)
  1. [Table 3] Table 3 omits Kimi-K2.6 and Qwen3.5-27B, although the Experiments section says these models were evaluated and Table 4 includes them. Either add these rows to the main table or state explicitly that Table 3 reports only a subset.
  2. [Table 4 caption] The caption uses 'Orc.' as a column header without defining it on first use. Define 'Oracle' explicitly, and clarify whether Judge@Pool and ID@Pool are computed over the union of top-20 retrieved candidates and final selections as stated.
  3. [Figure 3] The three modality strata in Figure 3 are reported without sample sizes or error bars. Since the diagnostic set has 300 cases total (100 per stratum), the 11.2% vs 13.3% difference for text-sufficient cases may be noise. Report CIs or at least the per-stratum counts.
  4. [SFT implementation] The SFT training setup is described as 'four epochs with a learning rate of 5×10−6 on 32 NVIDIA A100 GPUs,' but no batch size, sequence length, optimizer, or warmup schedule is given. Add these details for reproducibility.
  5. [Abstract and Positioning] The claim of being the 'first real-log benchmark' should be qualified in light of earlier real-log multimodal dialogue datasets such as JDDC 2.1; the positioning table helpfully clarifies the difference, but the abstract alone may overstate novelty.

Circularity Check

0 steps flagged

No significant circularity: the benchmark and its evaluation are grounded in human-annotated logs, a frozen catalog, and externally judged metrics rather than in fitted constants or self-referential derivations.

full rationale

MMShopBench's central derivation is empirical and self-contained. The 289 evaluation cases are built from real logs, manually annotated with purchase intent, mandatory requirements, and verified target products, and evaluated with two metric families: Judge@k, decided by an external GPT-5.5 judge that sees the human annotations and frozen product evidence, and ID@k, which directly matches returned identifiers against the human-verified set. Neither metric is defined in terms of the model being evaluated, and no fitted parameter is later renamed as a prediction. The SFT experiment is a standard distillation/tuning procedure: 900 teacher-generated trajectories optimize an autoregressive cross-entropy objective, and later gains are measured on separate evaluation cases under a frozen judge and fixed sandbox. No load-bearing claim rests on a self-citation; the only internal references are to the paper's own construction steps, not to an unverified prior result that is used to force an outcome. The most plausible concern—that the 900 SFT conversations and the 10,000 sampled evaluation logs are both drawn from the same deployed-assistant logs without an explicit disjointness statement—is a data-contamination/validity risk, not a circular derivation: the paper provides no equation or construction that makes the reported 5.5%→52.9% gain equal to the training signal by definition. Similarly, reliance on an LLM judge is an external-validity concern, not circularity, because the judge receives human-annotated I_t and R_t and frozen product evidence. Under the stated circularity patterns, the derivation chain is not circular.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper's central claims rest on the correctness of expert annotations, the reliability of an LLM judge, the representativeness of the proprietary sandbox, and the clean separation of SFT and evaluation data. None of these are independently verified with human agreement studies, catalog release, or an explicit disjointness statement.

axioms (4)
  • domain assumption The expert-annotated purchase intent and mandatory requirements for the 289 cases are correct and complete; the verified target products genuinely satisfy them.
    This is the ground truth on which all metrics are built; no inter-annotator agreement or adjudication statistics are reported.
  • domain assumption GPT-5.5's binary validity judgment y_{n,j} is a correct proxy for whether product evidence supports the annotated intent and every mandatory requirement.
    Every Judge@k result depends on this; the paper freezes the prompt but does not validate the judge against human labels.
  • domain assumption The offline 100,000-product sandbox faithfully represents the online platform's catalog and retrieval conditions for the purpose of evaluating shopping agents.
    The catalog is target-grounded and stratified, not a full copy of the online catalog; retrieval tools (BM25, Marqo embeddings) may rank differently than the production assistant.
  • domain assumption The 900 SFT conversations are drawn from a distribution sufficiently close to (and disjoint from) the 289 evaluation cases for the measured transfer to be meaningful.
    The paper says SFT conversations are screened from online logs but does not state explicit disjointness from evaluation cases.

pith-pipeline@v1.3.0-daily-deepseek · 13252 in / 15359 out tokens · 147169 ms · 2026-08-03T15:34:39.520508+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents." pith.science (2026). https://pith.science/paper/6P4UTLON

@misc{pith2026260729002,
  author       = {Pith},
  title        = {Pith review of: MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6P4UTLON}},
  note         = {Machine review of arXiv:2607.29002}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Online shoppers increasingly turn to AI shopping assistants, using images and multi-turn dialogue to express and refine product needs that are difficult to articulate in text alone. However, existing benchmarks largely rely on text-only or synthetic requests, underrepresenting complex real-world shopping requirements jointly expressed through images and language. We introduce MMShopBench, the first real-log benchmark for multimodal, multi-turn shopping agents. Built from carefully cleaned and manually annotated shopping logs, MMShopBench provides ground-truth annotations of each request's purchase intent and mandatory product requirements. Agents must infer these requirements jointly from user images and multi-turn dialogue, retrieve candidate products through image and text search, and verify that each candidate satisfies all requirements using its product images and structured attributes. We evaluate representative open-source and proprietary models using an evidence-grounded multimodal protocol and construct a companion training set for fine-tuning an open-source model. To ensure reproducible experimentation, we build an offline shopping sandbox, where fine-tuning substantially narrows the performance gap between our open-source model and leading proprietary models, demonstrating the effectiveness of our training data.

Figures

Figures reproduced from arXiv: 2607.29002 by Hao Guo, Jinsong Lan, Mengtao Xu, Xiaoyong Zhu, Yimin Hu, Yuheng Song, Zesheng Zhou, Zeying Hao.

Figure 1
Figure 1. Figure 1: Overview of MMShopBench and its offline shopping-agent workflow. Real multimodal, multi-turn logs are curated into [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Hierarchical intent distribution of MMShopBench. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of removing dialogue images across modality [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 8 linked inside Pith

  1. [1]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Shoppingbench: A real-world intent-grounded shopping benchmark for llm-based agents , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  2. [3]

    Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  3. [7]

    2026 , url=

    Xuying Ning and Dongqi Fu and Tianxin Wei and Mengting Ai and Jiaru Zou and Ting-Wei Li and Hanghang Tong and Yada Zhu and Hendrik Hamann and Jingrui He , booktitle=. 2026 , url=

    2026

  4. [10]

    Advances in Neural Information Processing Systems , volume =

    Shunyu Yao and Howard Chen and John Yang and Karthik Narasimhan , title =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    2022

  5. [12]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =

    Satwik Kottur and Seungwhan Moon and Alborz Geramifard and Babak Damavandi , title =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =. 2021 , doi =

    2021

  6. [13]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =

    Nan Zhao and Haoran Li and Youzheng Wu and Xiaodong He , title =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages =. 2022 , doi =

    2022

  7. [14]

    International Conference on Learning Representations , year =

    Dongzhi Jiang and Renrui Zhang and Ziyu Guo and Yanmin Wu and Jiayi Lei and Pengshuo Qiu and Pan Lu and Zehui Chen and Guanglu Song and Peng Gao and Yu Liu and Chunyuan Li and Hongsheng Li , title =. International Conference on Learning Representations , year =

  8. [16]

    International Conference on Learning Representations , year =

    Xijia Tao and Yihua Teng and Xinxing Su and Xinyu Fu and Jihao Wu and Chaofan Tao and Ziru Liu and Haoli Bai and Rui Liu and Lingpeng Kong , title =. International Conference on Learning Representations , year =

  9. [18]

    International Conference on Learning Representations , year =

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik Narasimhan and Yuan Cao , title =. International Conference on Learning Representations , year =

  10. [19]

    Companion Proceedings of the ACM on Web Conference 2025 , pages =

    Tianyu Zhu and Myong Chol Jung and Jesse Clark , title =. Companion Proceedings of the ACM on Web Conference 2025 , pages =. 2025 , doi =

    2025

  11. [20]

    Foundations and Trends in Information Retrieval , volume =

    Stephen Robertson and Hugo Zaragoza , title =. Foundations and Trends in Information Retrieval , volume =. 2009 , doi =

    2009

  12. [21]

    Xing and Hao Zhang and Joseph E

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , title =. Advances in Neural Information Processing Systems , volume =. 2023 , pages =

    2023

  13. [23]

    Anthropic . 2026. Claude Opus 4.8 System Card. https://www.anthropic.com/claude-opus-4-8-system-card. Accessed: 2026-07-28

    2026

  14. [24]

    Du, Z.; Li, T.; and Zhang, H. 2026. EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent. arXiv preprint arXiv:2606.17698

    Pith/arXiv arXiv 2026

  15. [25]

    Fan, S.; Min, S.; Hu, Y.; Xia, B.; Unnikrishnan, J.; Musselmann, R.; Gao, Y.; Yin, Q.; Nigam, P.; and Yin, B. 2026. Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants. arXiv preprint arXiv:2606.12608

    Pith/arXiv arXiv 2026

  16. [26]

    Google DeepMind . 2026. Gemini 3.1 Pro Model Card. https://deepmind.google/models/model-cards/gemini-3-1-pro. Accessed: 2026-07-28

    2026

  17. [27]

    Hou, B.; Gu, J.; Guo, J.; Dang, R.; Leng, S.; Li, X.; Song, X.; and Yang, J. 2026. InterLV-Search : Benchmarking Interleaved Multimodal Agentic Search. arXiv preprint arXiv:2605.07510

    Pith/arXiv arXiv 2026

  18. [28]

    Jiang, D.; Zhang, R.; Guo, Z.; Wu, Y.; Lei, J.; Qiu, P.; Lu, P.; Chen, Z.; Song, G.; Gao, P.; Liu, Y.; Li, C.; and Li, H. 2025. MMSearch : Unveiling the Potential of Large Models as Multi-Modal Search Engines. In International Conference on Learning Representations

    2025

  19. [29]

    Kottur, S.; Moon, S.; Geramifard, A.; and Damavandi, B. 2021. SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 4903--4912. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics

    2021

  20. [30]

    Li, S.; Bu, X.; Wang, W.; Liu, J.; Dong, J.; He, H.; Lu, H.; Zhang, H.; Jing, C.; Li, Z.; Li, C.; Tian, J.; Zhang, C.; Peng, T.; He, Y.; Gu, J.; Zhang, Y.; Yang, J.; Zhang, G.; Huang, W.; Zhou, W.; Zhang, Z.; Ding, R.; and Wen, S. 2025. MM-BrowseComp : A Comprehensive Benchmark for Multimodal Browsing Agents. arXiv preprint arXiv:2508.13186

    Pith/arXiv arXiv 2025

  21. [31]

    MiniMax . 2026. The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence. arXiv preprint arXiv:2605.26494

    Pith/arXiv arXiv 2026

  22. [32]

    Moonshot AI . 2026. Kimi K2.6 : Advancing Open-Source Coding. https://www.kimi.com/blog/kimi-k2-6. Accessed: 2026-07-28

    2026

  23. [33]

    Ning, X.; Fu, D.; Wei, T.; Ai, M.; Zou, J.; Li, T.-W.; Tong, H.; Zhu, Y.; Hamann, H.; and He, J. 2026. MC -Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains. In The Fourteenth International Conference on Learning Representations

    2026

  24. [34]

    OpenAI . 2026. GPT-5.5 System Card. https://openai.com/index/gpt-5-5-system-card/. Accessed: 2026-07-28

    2026

  25. [35]

    Qwen Team . 2026. Qwen3.5 : Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5. Accessed: 2026-07-28

    2026

  26. [36]

    K.; M \`a rquez, L.; Valero, F.; Rao, N.; Zaragoza, H.; Bandyopadhyay, S.; Biswas, A.; Xing, A.; and Subbian, K

    Reddy, C. K.; M \`a rquez, L.; Valero, F.; Rao, N.; Zaragoza, H.; Bandyopadhyay, S.; Biswas, A.; Xing, A.; and Subbian, K. 2022. Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search. arXiv preprint arXiv:2206.06588

    Pith/arXiv arXiv 2022

  27. [37]

    Robertson, S.; and Zaragoza, H. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4): 333--389

    2009

  28. [38]

    Tao, X.; Teng, Y.; Su, X.; Fu, X.; Wu, J.; Tao, C.; Liu, Z.; Bai, H.; Liu, R.; and Kong, L. 2026. MMSearch-Plus : Benchmarking Provenance-Aware Search for Multimodal Browsing Agents. In International Conference on Learning Representations

    2026

  29. [39]

    Wang, H.; Wan, D.; Lee, H.; Pham, T.; Cankosyan, M.; Chen, W.; Stengel-Eskin, E.; Vu, T.; and Bansal, M. 2026 a . MERRIN : A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments. arXiv preprint arXiv:2604.13418

    Pith/arXiv arXiv 2026

  30. [40]

    D.; and Zeng, X

    Wang, J.; Xiao, K.; Sun, Q.; Zhao, H.; Luo, T.; Zhang, J. D.; and Zeng, X. 2026 b . Shoppingbench: A real-world intent-grounded shopping benchmark for llm-based agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 33521--33529

    2026

  31. [41]

    Wang, P.; Wu, Y.; Song, X.; Wang, W.; Chen, G.; Li, Z.; Yan, K.; Liu, Q.; Deng, K.; Zhao, S.; Xiong, S.; Liu, X.; Chen, X.; Deng, W.; Su, W.; and Zheng, B. 2026 c . ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2026

  32. [42]

    Yao, S.; Chen, H.; Yang, J.; and Narasimhan, K. 2022. WebShop : Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems, volume 35, 20744--20757

    2022

  33. [43]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct : Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations

    2023

  34. [44]

    Yu, Z.; Xiao, K.; Zhao, H.; Luo, T.; and Zeng, X. 2026. Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks. arXiv preprint arXiv:2603.14864

    Pith/arXiv arXiv 2026

  35. [45]

    Zhang, H.; Zhou, J.; Li, B.; Zhou, B.; Shan, Y.; Lu, H.; Cao, Z.; Chen, J.; Han, Y.; Sheng, Z.; Tao, Z.; Liang, H.; Wu, J.; Shi, Y.; He, Y.; Lin, J.; Zhang, Q.; Yan, G.; Zhao, R.; Li, Z.; Yu, X.; Mei, L.; Chen, C.; Zhang, W.; and Cui, B. 2026 a . BrowseComp- V^3 : A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents. arXiv preprint ...

    arXiv 2026

  36. [46]

    Zhang, Z.; Miao, C.; Su, J.; Zhou, Z.; Zhang, C.; Wang, X.; Liu, R.; Zheng, K.; Cai, J.; Zhang, B.; Li, Z.; Xiang, S.; and Yan, Y. 2026 b . Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning. arXiv preprint arXiv:2606.15231

    arXiv 2026

  37. [47]

    Zhao, N.; Li, H.; Wu, Y.; and He, X. 2022. JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 12037--12051. Abu Dhabi, United Arab Emirates: Association for Computational Li...

    2022

  38. [48]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM -as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, 46595--46623

    2023

  39. [49]

    C.; and Clark, J

    Zhu, T.; Jung, M. C.; and Clark, J. 2025. Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking. In Companion Proceedings of the ACM on Web Conference 2025, 661--670

    2025