REVIEW 2 major objections 4 minor 28 references
ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons
T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper argues that current LLM shopping agents cannot reliably assemble multi-item baskets that satisfy compatibility, coupon, and budget constraints, with the best agent passing all checks on only 61.2% of tasks.
desk verdict A solid, genuinely useful shopping-agent benchmark whose witness-based construction is the standout idea; the headline failure rate is provisional until LLM-judge validity is shown across configurations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is solution-first task construction paired with hybrid evaluation. An exploration agent first finds and validates a purchasable basket in a simulated commerce-and-takeout environment; this hidden witness guarantees the task is solvable but is never used as a reference answer. The witness's costs then drive the synthesis of coupons, an optimal-payable budget interval, and a user query with semantic rubrics. At evaluation, deterministic code checks SKU validity, order feasibility, coupon legality and optimality, and budget compliance, while LLM judges assess semantic satisfaction, response quality, and claim faithfulness. Overall success is the intersection of all four dimensions.
What would settle it
Take a random sample of 100 outputs from GPT-5.5 with thinking across the 291 tasks, have two expert annotators judge semantic satisfaction, response quality, and claim faithfulness against the same rubrics, and compare with the Gemini-3.1-Pro judge. If agreement on that sample falls below the 97% level measured on Qwen3.6-27B outputs, then the reported 61.2% overall success rate and the per-dimension pass rates are not reliable.
Extended reading notes
Core claim
The paper's central discovery is that combo shopping — selecting multiple products that jointly satisfy compatibility, availability, store-level rules, coupon legality and optimality, and budget — remains largely unsolved by current LLM agents, even though each individual dimension looks passable. The best agent reaches 83.8% semantic satisfaction and 83.8% rule-based validation, yet only 61.2% of its tasks pass every dimension simultaneously; the gap shows that failures accumulate across the pipeline. The benchmark also shows that thinking configurations help most agents but not uniformly, and that cross-item compatibility requirements and choosing among mutually exclusive coupons are the hardest parts. The paper further establishes that its evaluation is not tied to a single hidden answer: among 1,729 accepted outputs, only 7.2% reproduce the witness basket exactly, and 33.1% use a completely different set of products.
Load-bearing premise
The benchmark's headline numbers assume that the three LLM judges, validated against only 700 human decisions from 30 outputs of a single weaker agent, stay at least 97% accurate on all 291 tasks and on outputs from stronger agents with different error patterns.
Editorial extensions
If this is right
- If the benchmark measures what it claims, then any agent that approaches 100% overall success would have to jointly handle compatibility reasoning, arithmetic, and faithful reporting, not just retrieve relevant items.
- The failure analysis implies that improving individual capabilities, such as semantic matching or coupon ID validity, is insufficient because agents need to avoid constraint accumulation across ten or more simultaneous criteria.
- The observation that approximate budgets are under-spent (only 49% pass rate versus 85–87% for caps or explicit ranges) suggests agents need explicit prompting or training to treat 'around N' as a two-sided spending target.
- The finding that thinking helps most agents but hurts some (for example, Qwen uses fewer calculator calls and loses tasks) implies that enabling a reasoning mode is not a reliable fix by itself.
- Because the benchmark accepts many alternative baskets, it can support future agent development without overfitting to a single reference answer.
Reading between the lines
- The hybrid design could be adapted to other constraint-satisfaction agent tasks where multiple valid outputs exist, such as travel itineraries or multi-step procurement, by using a witness only to certify feasibility.
- The paper's 97% judge agreement may not transfer to outputs from stronger or different models; a stress-test extension would re-run the human agreement study on GPT-5.5 Think failures before relying on precise rankings.
- The budget behavior result suggests a cheap intervention: rewriting approximate budgets as explicit ranges could close much of the observed gap without changing the underlying model.
- If judge accuracy degrades on rare but important failure modes, such as subtle coupon miscalculations, the reported 61.2% is an upper bound on true capability; a deterministic-only audit on a subset would quantify that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ComboShoppingBench, a benchmark of 291 tasks in a simulated e-commerce and takeout environment. Each task is constructed solution-first: an exploration agent finds a feasible hidden witness basket, and from that witness the pipeline synthesizes a coupon pack, a budget expression, a user query, and semantic rubrics. Evaluation combines three LLM judges (semantic satisfaction, response quality, claim faithfulness) with deterministic validation of SKU validity, coupon legality and optimality, and budget compliance. The authors evaluate 22 configurations of 11 agents and report that the strongest configuration, GPT-5.5 with thinking, achieves 61.2% overall success, defined as the strict conjunction of the four dimensions. Reliability of the LLM judges is supported by a human agreement study on 700 rubric decisions from 30 Qwen3.6-27B No-think outputs, and a witness-independence audit shows that successful outputs rarely reproduce the hidden witness.
Significance. If the results hold, ComboShoppingBench is a meaningful contribution to agent evaluation: it addresses open-ended basket construction while retaining objective verification of transactional constraints, avoiding both exact-match rigidity and purely semantic scoring. The strongest parts of the paper are the deterministic validator design, the basket-specific coupon optimality check, and the witness-independence audit, which together provide credible evidence against answer-key leakage. The human-agreement study is a genuine attempt to validate LLM judging, and the reported 97%+ raw agreement on the sampled subset is encouraging. The main risk is that the headline success rates depend on LLM judges whose reliability is demonstrated on only one agent's outputs; if judge accuracy differs on other configurations, the comparative and headline claims could shift. The benchmark itself and the deterministic pipeline are valuable regardless, and the paper is transparent about several of its design choices and limitations.
major comments (2)
- [§4, 'Are the LLM-Based Evaluators Reliable?' and Table 3] The reliability study is the load-bearing support for the LLM-judged dimensions, but it is confined to 30 outputs of a single agent (Qwen3.6-27B No-think) and 700 rubric decisions. The full evaluation applies the same three judges to approximately 6,402 outputs across 22 configurations. Response Quality is the weakest point: human agreement for Gemini-3.1-Pro is 94.67% (N=150), and the judge reports strikingly low Q pass rates for configurations never included in the human sample (e.g., 19.6% for Claude-Opus-4.6 No-think and 14.1% for DeepSeek-V4-Pro No-think in Table 2). Since Overall Success is the strict intersection S ∧ V ∧ Q ∧ F (Eq. (1)), even a small drop in judge accuracy on those outputs can move the headline rate by several points. I therefore ask for either human validation stratified across all or a representative subset of configurations, or a sensitivity analysis that recomputes the main table under conservative assumptions about judge error on low-Q configurations.
- [Appendix D and Table D.1] Appendix D operationalizes the 'around N' budget as a two-sided target band: L ≤ P ≤ U, with L derived from the witness payable p* and a symmetric 15% tolerance (δ(p*) = max(0.15p*, 5)). The questionnaire of 35 participants gives median acceptable deviations of 17% below and 13% above, but the paper does not report the distribution or justify why a symmetric 15% band, rather than the observed median interval or a one-sided interpretation, is the correct formalization. This is consequential because Appendix H.2 reports that 36% of approximate-target outputs fall below L and treats this as underspending, and budget compliance is a deterministic V check. A reader who reads 'around N' as a soft target would not count those as failures. Please report the full questionnaire distribution and provide a robustness check showing how budget compliance and Overall success change under a one-sided or asymmetric band.
minor comments (4)
- [Appendix A.2 and Table A.1] The text contains the Unicode ligature typo 'difficulty' in several places; please replace it with 'difficulty'.
- [Figure 6B] The labels such as '412' and '1712' are not explained in the caption; they appear to be task counts for recovered and lost tasks, and the caption should state this explicitly.
- [§4 and Table G.2] The main text reports only raw agreement percentages; since the label distribution is heavily skewed toward Pass, consider also reporting balanced accuracy or Cohen's kappa in the main text, as Appendix G already does.
- [§5] The sentence 'Our analysis identify cross-item constraint accumulation...' contains a subject-verb agreement error; it should be 'Our analysis identifies...'.
Circularity Check
No significant circularity: benchmark construction is an explicit design guarantee, and the witness is audited not to act as a hidden answer key.
full rationale
ComboShoppingBench is a benchmark-construction paper rather than a derivation or prediction chain. The 'every task is solvable' property is asserted as a design consequence of solution-first construction ("This solution-first construction ensures that every task is solvable by design"), not as an empirical discovery, so it is not circular. The evaluation does not compare against the witness as a reference answer: the rubric writer is instructed that the witness is "only one witness solution... not the only valid answer" and must write criteria applicable to any valid basket; rule-based validation checks the agent's own basket and coupon set deterministically (Eqs. C.1–C.3); and Appendix F provides a direct independence audit showing only 7.2% of Overall-success outputs exactly match the witness while 33.1% are disjoint, and 97.0% of tasks with at least one success have a non-exact successful basket. The LLM-judge reliability claim is an empirical generalization from a 30-output human study (Table 3), not a parameter fitted to the evaluated agents; its limited coverage is an acknowledged validity limitation (Appendix G: "This study establishes benchmark-specific evaluator validity rather than universal correctness of LLM judging"), but it is not a reduction of the headline 61.2% to its own inputs. Budget and coupon tolerances are calibrated from an external questionnaire (Appendix D) and deterministic settlement rules, not from agent outputs. There are no load-bearing self-citations or imported uniqueness theorems. Extrapolating judge agreement across the 22 configurations is a correctness and robustness concern, not circularity.
Assumptions & free parameters
free parameters (2)
- budget tolerance radius delta(p*) =
delta = max(0.15 p*, 5)
- rounding granularity s(p*) =
piecewise: 1 for p*<50, 5 for 50<=p*<200, 10 for 200<=p*<1000, 50 for p*>=1000
assumptions (4)
- domain assumption The frozen catalog snapshot is fixed and read-only; no real-time inventory, pricing, or fulfillment.
- domain assumption Coupon settlement rules (subtraction before percentage, proportional allocation, stack groups) are fixed by the benchmark.
- domain assumption LLM judges' rubric decisions approximate expert human judgment on the full benchmark.
- ad hoc to paper The 15% tolerance and rounding granularity are appropriate interpretations of natural-language budgets.
Cite this review
Pith. "Pith review of ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons." pith.science (2026). https://pith.science/paper/J5E3HVVO
@misc{pith2026260809282,
author = {Pith},
title = {Pith review of: ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons},
year = {2026},
howpublished = {\url{https://pith.science/paper/J5E3HVVO}},
note = {Machine review of arXiv:2608.09282}
}
read the original abstract
Real-world shopping often requires constructing a basket of complementary items rather than retrieving a single product. Such combo-shopping tasks arise in device setup, meal preparation, event planning, and group takeout ordering, requiring joint reasoning about item compatibility, availability, store-level requirements, delivery fees, coupons, and budgets. Evaluation is challenging because multiple baskets may satisfy the same request, making exact-match metrics unsuitable, whereas semantic evaluation alone cannot detect infeasible orders, invalid coupon combinations, or incorrect payments. We introduce ComboShoppingBench, an agentic shopping benchmark for open-ended yet verifiable basket construction in a simulated commerce and takeout environment. During task synthesis, an exploration agent constructs a feasible and semantically coherent basket of purchasable products; this witness guides the generation of coupons, budget constraints, user queries, and aligned evaluation rubrics. During evaluation, LLM judges assess semantic satisfaction, response quality, and claim faithfulness, while deterministic validation checks product-ID validity, budget compliance, and coupon optimality. Experiments with diverse LLM agents demonstrate that even strong agents struggle on ComboShoppingBench, highlighting substantial room for improvement in reliable, constraint-aware combo shopping.
Reference graph
Works this paper leans on
-
[1]
Webshop: Towards scalable real-world web interaction with grounded language agents, 2023
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. URL https://arxiv.org/abs/2207.01206
arXiv 2023
-
[2]
Deepshop: A benchmark for deep research shopping agents, 2025
Yougang Lyu, Xiaoyu Zhang, Lingyong Yan, Maarten de Rijke, Zhaochun Ren, and Xiuying Chen. Deepshop: A benchmark for deep research shopping agents, 2025. URL https://arxiv.org/abs/2506.02839
arXiv 2025
-
[3]
Shoppingcomp: Are llms really ready for your shopping cart?, 2026
Huaixiao Tou, Ying Zeng, Yuemeng Li, Cong Ma, Muzhi Li, Minghao Li, Weijie Yuan, He Zhang, and Kai Jia. Shoppingcomp: Are llms really ready for your shopping cart?, 2026. URL https://arxiv.org/abs/2511.22978
arXiv 2026
-
[4]
Webmall – a multi-shop benchmark for evaluating web agents, 2026
Ralph Peeters, Aaron Steiner, Luca Schwarz, Julian Yuya Caspary, and Christian Bizer. Webmall – a multi-shop benchmark for evaluating web agents, 2026. URL https://arxiv.org/abs/2508.13024
arXiv 2026
-
[5]
Shopping- bench: A real-world intent-grounded shopping benchmark for llm-based agents, 2026
Jiangyuan Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jian Dong Zhang, and Xiaoyi Zeng. Shopping- bench: A real-world intent-grounded shopping benchmark for llm-based agents, 2026. URL https://arxiv.org/ abs/2508.04266
arXiv 2026
-
[6]
Zijian Yu, Kejun Xiao, Huaipeng Zhao, Tao Luo, and Xiaoyi Zeng. Shopping companion: Benchmarking and training llm agents for long-horizon preference-grounded e-commerce tasks, 2026. URL https://arxiv.org/abs/ 2603.14864
arXiv 2026
-
[7]
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Zeyao Du, Tong Li, Yanci Zhang, and Haibo Zhang. Ecomagentbench: Benchmarking shopping agents on long- horizon tasks with distributed hidden intent, 2026. URL https://arxiv.org/abs/2606.17698
work page Pith review arXiv 2026
-
[8]
Mind2web: Towards a generalist agent for the web, 2023
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URL https://arxiv.org/abs/2306.06070
arXiv 2023
Show all 28 references
-
[9]
Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL https://arxiv.org/abs/2307.13854
2024 arXiv
-
[10]
Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URL https://arxiv.org/abs/2401.13649
2024 arXiv
-
[11]
WebLINX: Real-world website navigation with multi-turn dialogue,
Xing Han Lù, Zdeněk Kasner, and Siva Reddy. WebLINX: Real-world website navigation with multi-turn dialogue,
-
[12]
Shopsimulator: Evaluating and exploring rl-driven llm agent for shopping assistants, 2026
Pei Wang, Yanan Wu, Xiaoshuai Song, Weixun Wang, Gengru Chen, Zhongwen Li, Kezhong Yan, Ken Deng, Qi Liu, Shuaibing Zhao, Shaopan Xiong, Xuepeng Liu, Xuefeng Chen, Wanxi Deng, Wenbo Su, and Bo Zheng. Shopsimulator: Evaluating and exploring rl-driven llm agent for shopping assi...
2026
-
[13]
Shopgym: An integrated framework for realistic simulation and scalable benchmarking of e- commerce web agents, 2026
Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, and Lingyun Wang. Shopgym: An integrated framework for realistic simulation and scalable benchmarking of e- commerce web agents, 2026. URL https://arxiv.org/abs/2605.16116. 10
2026 arXiv
-
[14]
McAuley, Rahul Pandey, and Jure Leskovec
Julian J. McAuley, Rahul Pandey, and Jure Leskovec. Inferring networks of substitutable and complementary products. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM, 2015. doi: 10.1145/2783258.2783381. UR...
2015
-
[15]
Apurva Pathak, Kshitiz Gupta, and Julian J. McAuley. Generating and personalizing bundle recommendations on steam. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1073–1076. ACM, 2017. doi: 10.1145/30771...
2017
-
[16]
Matching user with item set: Collaborative bundle recommendation with deep attention network
Liang Chen, Yang Liu, Xiangnan He, Lianli Gao, and Zibin Zheng. Matching user with item set: Collaborative bundle recommendation with deep attention network. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 2095–2101. ijcai.o...
2019 doi
-
[17]
Bundle recommendation with graph convolutional networks
Jianxin Chang, Chen Gao, Xiangnan He, Depeng Jin, and Yong Li. Bundle recommendation with graph convolutional networks. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1673–1676. ACM, 2020. doi: 10.1145/...
2020
-
[18]
Pack- age recommendation with intra- and inter-package attention networks
Chen Li, Yuanfu Lu, Wei Wang, Chuan Shi, Ruobing Xie, Haili Yang, Cheng Yang, Xu Zhang, and Leyu Lin. Pack- age recommendation with intra- and inter-package attention networks. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Informa...
2021
-
[19]
Multi-view intent disentangle graph networks for bundle recommendation
Sen Zhao, Wei Wei, Ding Zou, and Xianling Mao. Multi-view intent disentangle graph networks for bundle recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4379–4387. AAAI Press, 2022. doi: 10.1609/AAAI.V36I4.20359. URL https://doi.org/10.160...
2022 doi
-
[20]
Crosscbr: Cross-view contrastive learning for bundle recommendation
Yunshan Ma, Yingzhi He, An Zhang, Xiang Wang, and Tat-Seng Chua. Crosscbr: Cross-view contrastive learning for bundle recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1233–1241. ACM, 2022. doi: 10.1145/3534678.35392...
2022
-
[21]
Revisiting bundle recommendation for intent-aware product bundling
Zhu Sun, Kaidong Feng, Jie Yang, Hui Fang, Xinghua Qu, Yew-Soon Ong, and Wenyuan Liu. Revisiting bundle recommendation for intent-aware product bundling. ACM Transactions on Recommender Systems, 2(3):24:1– 24:34, 2024. doi: 10.1145/3652865. URL https://doi.org/10.1145/3652865
2024 doi
-
[22]
A survey on bundle recommendation: Methods, applications, and challenges, 2024
Meng Sun, Lin Li, Ming Li, Xiaohui Tao, Dong Zhang, Qing Xie, Peipei Wang, and Jimmy Xiangji Huang. A survey on bundle recommendation: Methods, applications, and challenges, 2024. URL https://arxiv.org/abs/ 2411.00341
2024
-
[23]
BRIDGE: Bundle recommendation via instruction-driven generation, 2024
Tuan-Nghia Bui, Huy-Son Nguyen, Cam-Van Nguyen Thi, Hoang-Quynh Le, and Duc-Trong Le. BRIDGE: Bundle recommendation via instruction-driven generation, 2024. URL https://arxiv.org/abs/2412.18092
2024 arXiv
-
[24]
Modeling item-level dynamic variability with residual diffusion for bundle recommendation, 2025
Dong Zhang, Lin Li, Ming Li, Amran Bhuiyan, Meng Sun, Xiaohui Tao, and Jimmy Xiangji Huang. Modeling item-level dynamic variability with residual diffusion for bundle recommendation, 2025. URL https://arxiv.org/ abs/2507.03280
2025
-
[25]
EpicCBR: Item-relation-enhanced dual-scenario contrastive learning for cold-start bundle recommendation, 2026
Yihang Li, Zhuo Liu, and Wei Wei. EpicCBR: Item-relation-enhanced dual-scenario contrastive learning for cold-start bundle recommendation, 2026. URL https://arxiv.org/abs/2602.11680
2026
-
[26]
CASE: Cadence-aware set encoding for large-scale next basket repurchase recommendation, 2026
Yanan Cao, Ashish Ranjan, Sinduja Subramaniam, Evren Korpeoglu, Kaushiki Nag, and Kannan Achan. CASE: Cadence-aware set encoding for large-scale next basket repurchase recommendation, 2026. URL https://arxiv. org/abs/2604.06718
2026 arXiv
-
[27]
about N yuan,
Zhiying Deng, Yuan Fu, Usman Farooq, Ziwei Tian, Wei Liu, and Jianjun Li. Time-interval-aware disentangled expert modeling for next-basket recommendation, 2026. URL https://arxiv.org/abs/2605.00499. 11 ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Sho...
2026 arXiv
-
[2024]
URL https://arxiv.org/abs/2402.05930
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.