Pith.

Computation and Language · arXiv:2611.05862 v1 · submitted 4 Nov 2026 · 32 references

CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei

show 4 more authors

Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen

The paper says

“Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines).”

more of the abstract

For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. Code and data are available at https://github.com/Da1yuqin/CARTBench.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: minor revision.

A well-designed benchmark that reveals a plausible understanding-to-execution gap in shopping assistants, but its reliance on undisclosed source replies for the core evaluation principles is a transparency issue that needs addressing. This line is wrong

2 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 32 references say.

The paper says, at Section 3.2 Step 3 (Ideal Response Principles)

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The Ideal Response Principles are described as 'distilled from excellent replies manually selected from successfully completed orders' and the authors state that 'to protect privacy, we do not share these replies.' Because these principles anchor both the case-specific rubrics (evaluation) and the Principle-Guided CART Prompting (generation), the inability to examine the source replies or the distillation process makes it impossible for readers to assess whether the principles comprehensively capture high-quality advice or inadvertently encode a narrow standard.

more of objection 1

The paper does not discuss this as a limitation. I recommend that the authors either provide a redacted sample of the source replies (with all personal and commercial identifiers removed) or describe the distillation process in sufficient detail (e.g., number of replies used, demographics of experts, inter-annotator agreement on principle extraction) to allow independent evaluation of the principles' validity.

The paper says, at Section 4.1 (Evaluation of Mainstream Models)

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The evaluation uses DeepSeek-V4-Pro as the sole judge model to assign initial labels, which are then expert-reviewed. While the expert review (99.69% agreement) validates the final labels, the paper does not report the agreement on individual tiers or the distribution of judge-expert disagreements.

more of objection 2

It would be informative to know, for example, whether disagreements cluster in Excellence or Safety criteria, as this could indicate systematic difficulties in judging those aspects. I recommend reporting agreement rates broken down by Core, Excellence, and Safety, and briefly discussing any patterns.

4 minor objections

The paper says, at Section 3.2 Step 3

Pith says: minor objection 1 of 4

The paper should explicitly list the non-disclosure of source replies as a limitation in the Limitations section, even if only to note that the principles are made public and can be evaluated on their face.

The paper says, at Section 4.2.1, Figure 5

Pith says: minor objection 2 of 4

The failure mode labels (F1–F10) are defined only in the figure text; consider providing a brief definition in the body text or a table for easier reference.

show 2 more

The paper says, at Table 2 caption

Pith says: minor objection 3 of 4

The caption mentions 'sample SD' but the text notes that SD is computed across three evaluations. Clarify that this is the standard deviation of the mean over three runs, not a per-instance standard deviation.

The paper says, at Section 4.2.3

Pith says: minor objection 4 of 4

The 13.0% plan failure rate among replies passing all other checks is a striking finding. Please specify how many replies entered this analysis (the denominator) and whether the 95% interval accounts for model-level clustering.

What would settle it

Request that the authors release an anonymized subset of the source replies (e.g., 10–20 diverse examples) together with the corresponding excerpted principles, and ask two independent expert annotators to independently derive and list the principles they see reflected in those replies. If the independently derived principles converge on the same set of four (or a superset that contains them), the concern is substantially mitigated. If the authors cannot release any source replies, commission an independent team to create a separate set of case-specific rubrics for a random subset of the 169 cases using only the four broad principle statements (Address the current purchase decision; Follow active needs; Ground claims; Give decision help) and then re-run the evaluation on 2–3 models.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Nov 2026

  1. Reader holds up confidence moderate
  2. Skeptic agrees: says it holds up only with conditions 42 s · 13,978 tokens read, 3,706 written The Ideal Response Principles, which anchor both the rubrics and the prompting strategy, are distilled from undisclosed source replies; if these principles are incomplete or biased, the observed understanding-to-execution gap may be an artifact of the rubric design rather than a genuine model…
  3. Editor wrote the verdict above 21 s · 14,857 tokens read, 1,742 written
  4. Referee minor revision: 2 major objections, 4 minor 27 s · 14,353 tokens read, 2,234 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper introduces CARTBench, a benchmark that fills a clear gap: evaluating LLM shopping advice under conditions that combine timelines, complex interdependent constraints, and ambiguity. No prior benchmark covers all three. The evaluation design is careful: 1,328 expert-annotated criteria with three tiers (Core, Excellence, Safety), high inter-rater agreement (99.69%), and a structured failure-mode analysis. The central finding—models hit 90%+ on Core but stay below 45% on Excellence—is well supported and practically important. The diagnosis of failure modes like unsupported product claims and unjustified suitability inference is actionable.

The main soft spot is the reliance on the Ideal Response Principles, which are distilled from excellent replies selected from successfully completed orders. The authors state they cannot share these source replies for privacy reasons. This means the community cannot verify whether the principles are complete, representative, or free of bias. Since the same principles guide both rubric design and prompting, there's a risk of circularity: the evaluation framework and the model behavior are shaped by the same possibly idiosyncratic standard. The authors acknowledge the privacy constraint but do not list this as a limitation in the Limitations section, which is an oversight. That said, the principles themselves (four bullet points) are broad and reasonable, and the rubrics are reviewed by experts, so the concern is real but not necessarily fatal. A secondary soft spot is the modest sample size (169 cases), though stratified sampling helps.

This paper is for researchers in conversational recommender systems and LLM evaluation. It deserves a serious referee because the contribution is solid and the practical gap is real. The transparency issue should be addressed in revision: the authors could release redacted samples of the source replies or provide a more detailed validation of the principles against an independent standard. I would accept with minor revisions. For my own work, I'd cite this if I were active in the subfield, and I'd consider it for a reading group if the topic fits.

The strongest objection

The paper's central claim—an understanding-to-execution gap—rests entirely on case-specific rubrics derived from the four Ideal Response Principles (Section 3.2 Step 3). These principles are said to be 'distilled from excellent replies manually selected from successfully completed orders'. The authors explicitly state: 'To protect privacy, we do not share these replies.' Without access to the source replies, the community cannot verify whether the principles capture all relevant facets of high-quality advice, whether they inadvertently encode a narrow or idiosyncratic standard, or whether the instantiation into case-specific rubrics is faithful. This is not a minor documentation gap: the principles guide both the evaluation framework (rubric design) and the prompting strategy (Principle-Guided CART Prompting). If the principles are off-target, both the benchmark scores and the model outputs are shaped by the same possibly distorted standard, creating a circular validation. The reader flagged this as the weakest assumption; it remains unresolved. The paper does not list this as a limitation in the Limitations section, nor does it offer any plan to release even a redacted sample of the source replies for independent scrutiny. Given that all 1,328 criteria and the prompt given to models derive from these non-public principles, the entire evaluation is opaque at its foundation.

The reader's weakest_assumption explicitly identifies the non-public source replies for the Ideal Response Principles as the foundation that could collapse the framework. The stress-tester agrees that this is the single most load-bearing concern. However, the reader still issued an ACCEPT verdict. Given that the paper does not acknowledge this as a limitation and provides no path for external validation, the stress-tester recommends a CONDITIONAL verdict: acceptance conditional on the authors either releasing a verifiable sample of source replies or demonstrating that an independent derivation of the principles produces equivalent results. This does not imply that the work is unsound; it merely acknowledges that the central claim's reproducibility is currently blocked by an opaque design step. The concrete test proposed can be performed by the authors without compromising privacy (anonymized sample) or by third parties (independent rubric design), offering a clear path to resolution.

The referee's report

If the claimed understanding-to-execution gap holds, CARTBench provides a meaningful measurement that distinguishes basic correctness from genuinely helpful purchase advice—a distinction absent from prior shopping benchmarks that do not explicitly credit conditional answers or track timeline changes. The finding that strong Core performance coexists with weak Excellence across all tested models suggests a systematic limitation in current LLM shopping assistants that merits further investigation. The benchmark's design, with expert-reviewed rubrics and high inter-rater agreement (99.69%), offers a practical evaluation protocol that could be extended to other conversational recommendation domains. However, the full significance depends on the reproducibility and independence of the rubric design from the source replies used to derive the Ideal Response Principles.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces CARTBench, a benchmark for evaluating how well large language models handle complex, ambiguous shopping requests with timelines. It finds that while models pass over 90% of core checks—goal alignment, requirement adherence, and uncertainty preservation—none exceed 45% on excellence checks such as product comparisons, trade-off explanations, and conditional advice. This gap, which the paper calls an understanding-to-execution gap, persists even when models are explicitly prompted with ideal response principles. The result matters because it reveals a systematic failure: models understand what is asked but cannot turn that understanding into actionable, well-reasoned purchase guidance.

What carries the argument. The key machinery is a two-part framework built on Ideal Response Principles and case-specific rubrics. The four principles (address the current purchase decision; follow active needs and preserve unknowns; ground claims and explain trade-offs; give clear decision help) are distilled from expert-selected excellent replies and serve both to design consistent per-case rubrics and to prompt the shopping agent. The rubrics contain 1,328 criteria across three tiers (Core, Excellence, Safety), and each criterion checks one observable behavior supported by product evidence. This framework enables systematic evaluation that aligns common standards with case-specific needs, something previous benchmarks lacked.

Core claim. The paper's central discovery is that LLM shopping assistants exhibit an understanding-to-execution gap: they perform well on core obligations (understanding the request, following stated constraints, preserving unknowns) but consistently fail to deliver the high-quality decision help that makes advice truly useful. Across ten flagship models, core pass rates exceed 90% while excellence rates stay below 45%. This gap is not an artifact of poor prompting—it persists despite explicit Principle-Guided CART Prompting—and is driven by shared failure modes: unsupported product claims, unjustified suitability inferences, and premature resolution of unknown preferences or facts.

The premise it rests on. The entire evaluation framework rests on the Ideal Response Principles, which are distilled from expert-selected excellent replies that the paper does not disclose; if those source replies are incomplete or biased, then both the rubrics and the prompted behavior they measure are unfounded.

If the paper is right (Pith's inferences, not the authors')

Shopping assistant developers can use the failure mode taxonomy (unsupported claims, unjustified fit, premature need resolution) to target specific training or fine-tuning interventions.

The observed gap suggests that current evaluation methods focusing on task completion rates miss the quality dimension of purchase advice; future leaderboards should include separate excellence scores.

Because the gap persists across all evaluated models regardless of architecture or scale, it may reflect a fundamental limitation of current LLM training objectives rather than a simple prompt-engineering fix.

The CARTBench protocol provides a reusable template for building case-specific rubrics in other advisory domains (e.g., travel planning, financial advice) where complex, ambiguous, and time-sensitive requests arise.

The finding that 13% of replies passing all need and evidence checks still fail plan checks implies that even grounded reasoning about feasibility is fragile, pointing to a need for explicit plan-validation modules.

The paper's reliance on expert-selected source replies for the Ideal Response Principles means the principles themselves cannot be independently validated or challenged; if the source replies are biased, the whole evaluation framework inherits that bias.

The evaluation cost of under US$24 per model makes CARTBench practical for routine benchmarking, but the 169-case sample—though stratified—may still miss long-tail scenarios that would test model robustness further.

The failure modes identified (especially unsupported product claims) suggest that grounding in actual product records is necessary but not sufficient; models also need to explicitly justify the link from evidence to user suitability, a capability that current prompting strategies do not reliably elicit.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. Code and data are available at https://github.com/Da1yuqin/CARTBench.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-05

The authors' main claim, as the classifier picked it out 1.00

“Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.62 · states limitation 0.54 · human participants 0.35 · controlled comparison 0.10 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.39 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. background 1.00 0.06 0.02 asserted 1.00
However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. gap 0.60 0.10 0.03 asserted 1.00
To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). contribution 1.00 0.87 0.03 asserted 0.99
For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. method 0.90 0.96 0.03 asserted 1.00
For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. method 0.87 0.96 0.03 asserted 1.00
These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. method 0.70 0.88 0.03 asserted 1.00
The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. method 0.74 0.91 0.03 asserted 1.00
To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. method 0.95 0.89 0.03 asserted 0.98
In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. result 1.00 0.95 0.19 asserted 1.00
Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. interpretation 0.98 0.94 0.65 asserted 1.00
To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. interpretation 0.44 0.92 0.08 asserted 1.00
Code and data are available at https://github.com/Da1yuqin/CARTBench. contribution 0.81 0.47 0.02 asserted 0.98

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-05: 1 request, 8,753 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. A CART case: an expired gift deadline (Time
Figure 2 of the paper
Figure 2. CART workflow with selected travel-case turns. An undecided destination (Ambiguity) and linked power
Figure 3 of the paper
Figure 3. Case-specific rubrics distinguish stronger and weaker replies to the same warmth and camera-control
Figure 4 of the paper
Figure 4. Case coverage across shopping targets (inner)
Figure 5 of the paper
Figure 5. Core failure composition. Unsupported prod
Figure 6 of the paper
Figure 6. Scenario reply rates for context-state drift (F1),

Figures are the authors', reproduced from the paper.

References (32)
  1. [1] Alibaba Cloud . 2026. https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max qwen3.8-max model info . Documentation last updated September 28, 2026. Accessed October 7, 2026 on Pith
  2. [2] Anthropic . 2026. https://www.anthropic.com/news/claude-opus-5 Introducing Claude Opus 5 . July 24, 2026. Accessed October 7, 2026 on Pith
  3. [3] Chih-Yu Chien, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2026. https://doi.org/10.63317/5csm2qecy6jh User profiling for specification-sensitive recommendations with large language model prompting . In Proceedings of the Fifteenth Language Resources and Evaluation Conferenc... on Pithdoi:10.63317/5csm2qecy6jh
  4. [4] Preetam Prabhu Srikar Dammu, Omar Alonso, and Barbara Poblete. 2025. https://doi.org/10.1145/3701551.3704124 A shopping agent for addressing subjective product needs . In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pages 1032--1035 on Pithdoi:10.1145/3701551.3704124
  5. [5] DeepSeek . 2026. https://api-docs.deepseek.com/news/news260813/ DeepSeek-V4-Pro GA release . August 13, 2026. Accessed October 7, 2026 on Pith
  6. [6] Kaustubh Dhole, Nikhita Vedula, Saar Kuzi, Giuseppe Castellucci, Eugene Agichtein, and Shervin Malmasi. 2025. https://doi.org/10.18653/v1/2025.naacl-srw.8 Generative product recommendations for implicit superlative queries . In Proceedings of the 2025 Conference of the Nations... on Pithdoi:10.18653/v1/2025.naacl-srw.8
  7. [7] Zeyao Du, Tong Li, and Haibo Zhang. 2026. https://arxiv.org/abs/2606.17698 EComAgentBench : Benchmarking shopping agents on long-horizon tasks with distributed hidden intent . arXiv preprint arXiv:2606.17698 on PithPith's reviewarXiv
  8. [8] Shuxian Fan, Seonwoo Min, Youna Hu, Botao Xia, Jayakrishnan Unnikrishnan, Rowan Musselmann, Yifan Gao, Qingyu Yin, Priyanka Nigam, and Bing Yin. 2026. https://arxiv.org/abs/2606.12608 Shopping reasoning bench: An expert-authored benchmark for multi-turn conversational shopping... on PithPith's reviewarXiv
  9. [9] Zeying Hao, Hao Guo, Mengtao Xu, Yimin Hu, Yuheng Song, Zesheng Zhou, Jinsong Lan, and Xiaoyong Zhu. 2026. https://arxiv.org/abs/2607.29002 MMShopBench : A real-log benchmark for multimodal, multi-turn shopping agents . arXiv preprint arXiv:2607.29002 on PithPith's reviewarXiv
  10. [10] Heejin Kook, Junyoung Kim, Seongmin Park, and Jongwuk Lee. 2025. https://doi.org/10.18653/v1/2025.naacl-long.392 Empowering retrieval-based conversational recommendation with contrasting user preferences . In Proceedings of the 2025 Conference of the Nations of the Americas Ch... on Pithdoi:10.18653/v1/2025.naacl-long.392
  11. [11] Ivica Kostric, Krisztian Balog, and Ujwal Gadiraju. 2025. https://doi.org/10.1145/3699682.3728353 Should we tailor the talk? understanding the impact of conversational styles on preference elicitation in conversational recommender systems . In Proceedings of the 33rd ACM Confe... on Pithdoi:10.1145/3699682.3728353
  12. [12] Chuang Li, Yang Deng, Hengchang Hu, Min-Yen Kan, and Haizhou Li. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.17 ChatCRS : Incorporating external knowledge and goal guidance for LLM -based conversational recommender systems . In Findings of the Association for Computa... on Pithdoi:10.18653/v1/2025.findings-naacl.17
  13. [13] Mengfan Li, Xuanhua Shi, and Yang Deng. 2026. https://doi.org/10.1609/aaai.v40i37.40430 RecToM : A benchmark for evaluating machine theory of mind in LLM -based conversational recommender systems . Proceedings of the AAAI Conference on Artificial Intelligence, 40(37):31636--31644 on Pithdoi:10.1609/aaai.v40i37.40430
  14. [14] Gustavo Adolpho Lucas de Carvalho, Simon Benigeri, Jennifer Healey, Victor Bursztyn, David Demeter, and Lawrence Birnbaum. 2025. https://aclanthology.org/2025.coling-main.561/ A flash in the pan: Better prompting strategies to deploy out-of-the-box LLM s as conversational reco... on Pith
  15. [15] Weimin Lyu, Chen Luo, Guangrui Li, Yaochen Xie, Dhineshkumar Ramasubbu, Arief Koesdwiady, Wanqiu Long, Hansu Gu, Yutong Chen, Zheshen Wang, Dakuo Wang, and Yi Liu. 2026. https://arxiv.org/abs/2608.26180 PACEShop : Evaluating personalized, actionable, compositional, and evidenc... on PithPith's reviewarXiv
  16. [16] Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, and Xiuying Chen. 2025. https://arxiv.org/abs/2506.02839 DeepShop : A benchmark for deep research shopping agents . arXiv preprint arXiv:2506.02839 on PithPith's reviewarXiv
  17. [17] Wangyi Mei, Zhouhong Gu, Zhenhan Bai, Yin Cai, Lefan Zhang, Zhenxin Ding, Bo Chen, Yan Gao, Yi Wu, Yao Hu, Jiaqing Liang, and Deqing Yang. 2026. https://arxiv.org/abs/2606.01091 Deep research as rubric for reinforcement learning . Preprint, arXiv:2606.01091 on PithPith's reviewarXiv
  18. [18] MiniMax . 2026. https://www.minimax.io/blog/minimax-m3 MiniMax M3 : Frontier coding, 1m context, native multimodality---all in one model . June 1, 2026. Accessed October 7, 2026 on Pith
  19. [19] Moonshot AI . 2026. https://www.kimi.com/en/blog/kimi-k3 Kimi K3 : Open frontier intelligence . Official research index dated July 16, 2026. Accessed October 7, 2026 on Pith
  20. [20] Rajdeep Mukherjee, Sonali Singh, and Sachin Farfade. 2025. https://doi.org/10.1145/3690624.3709411 Using instruction-tuned LMs for scalable use case-based shopping---where customers meet their needs . In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and ... on Pithdoi:10.1145/3690624.3709411
show 12 more
  1. [21] Bharath Sivaram Narasimhan and Karthik R. Narasimhan. 2026. https://doi.org/10.1145/3773078.3831847 \( \) -rec: A verifiable benchmark for agentic recommender systems . In Proceedings of the 20th ACM Conference on Recommender Systems, pages 915--919 on Pithdoi:10.1145/3773078.3831847
  2. [22] OpenAI . 2026. https://openai.com/index/introducing-gpt-6-1-sol/ GPT-6.1 Sol . Accessed October 7, 2026 on Pith
  3. [23] Qwen Team . 2026. https://qwen.ai/blog?id=qwen3.5 Qwen3.5 : Towards native multimodal agents on Pith
  4. [24] Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. 2025. https://arxiv.org/abs/2501.17399 MultiChallenge : A realistic multi-turn conversation evaluation benchmark challenging t... on PithPith's reviewarXiv
  5. [25] Huaixiao Tou, Ying Zeng, Yuemeng Li, Cong Ma, Muzhi Li, Minghao Li, Weijie Yuan, He Zhang, and Kai Jia. 2025. https://arxiv.org/abs/2511.22978v2 ShoppingComp : Are LLMs really ready for your shopping cart? arXiv preprint arXiv:2511.22978 on PitharXiv
  6. [26] Nikhita Vedula, Marcus Collins, Eugene Agichtein, and Oleg Rokhlenko. 2023. https://doi.org/10.1145/3539597.3570489 Generating explainable product comparisons for online shopping . In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, page... on Pithdoi:10.1145/3539597.3570489
  7. [27] Jiangyuan Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jian Dong Zhang, and Xiaoyi Zeng. 2026. https://doi.org/10.1609/aaai.v40i39.40640 ShoppingBench : A real-world intent-grounded shopping benchmark for LLM -based agents . Proceedings of the AAAI Conference on Artificia... on Pithdoi:10.1609/aaai.v40i39.40640
  8. [28] Xiaomi . n.d. https://mimo.mi.com/models/en-US/mimo-v2.6-pro MiMo-V2.6-Pro . Accessed October 7, 2026 on Pith
  9. [29] Xinwei Yang, Kelong Mao, Yudong Guo, Sulong Xu, Simiu Gu, Chen Huang, and Wenqiang Lei. 2026. https://arxiv.org/abs/2609.38974 RealWorldShop : Benchmarking and improving conversational shopping agents in real-world e-commerce . arXiv preprint arXiv:2609.38974 on PithPith's reviewarXiv
  10. [30] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. https://arxiv.org/abs/2207.01206 WebShop : Towards scalable real-world web interaction with grounded language agents . In Advances in Neural Information Processing Systems, volume 35 on PithPith's reviewarXiv
  11. [31] Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. 2025 a . https://proceedings.iclr.cc/paper_files/paper/2025/hash/28a46044775d97a4efcbcf14e7f13209-Abstract-Conference.html Do LLMs recognize your preferences? evaluating personalized preference following ... on Pith
  12. [32] Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B. Cohen, and Emine Yilmaz. 2025 b . https://doi.org/10.18653/v1/2025.findings-acl.927 PersonaLens : A benchmark for personalization evaluation in conversational AI assistants . In Findings of the Association for Comp... on Pithdoi:10.18653/v1/2025.findings-acl.927

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment