{"id":"d830620d-5c1f-42db-a85f-6995cda5087d","arxiv_id":"2607.03162","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LLM agents succeed on refined product queries but fail on early underspecified intents mainly because they underuse noisy histories; APeB measures this gap and VQRA partially closes it.","lead":"APeB is a new benchmark that tests whether LLM agents can personalize product choices from vague shopping queries and messy real user histories. It shows current models handle clear queries but fail when they must discover intent and preferences, and that a simple history-grounded rewrite step helps.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Purchase-as-preference remains the softest load-bearing label for the intent+hard-candidate claim, even after the paper's own confound diagnostics.","rationale":"I agree with the reader: the manuscript is a solid empirical/resource contribution with multi-model tables, history/candidate ablations, ReAct diagnostics, and VQRA. The distinctive claim—that the hard gap is personalization under vague intent via weak history use—depends on treating observed purchases as preference ground truth among user-viewed near-misses. The paper already documents this caveat and provides partial mitigations (semantic alignment filters, hard-candidate controls, pairwise win rates, counterfactual-history checks). Those reduce but do not eliminate the concern, so the right verdict remains CONDITIONAL rather than REJECT or ACCEPT. No stronger internal inconsistency appears: refined vs intent, random vs hard, and VQRA/rubric results are mutually coherent under the stated label. The single most useful stress test is therefore a purchase-validity re-evaluation of the intent+hard Hit@1/VQRA gap, not a novelty or agent-architecture dispute. Controlled data release is a practical limitation but secondary to this measurement assumption for the scientific claim.","tokens_in":33622,"tokens_out":690,"duration_ms":7269,"concrete_test":"On a stratified sample of ~500 intent+hard sessions, recompute Hit@1 after (i) restricting to purchases that win ≥80% of history-conditioned pairwise judgments without price (Table 8 third row), and (ii) reweighting or dropping sessions where the purchased item had top exposure/rank among Chard if exposure logs exist (or a proxy: earliest/longest-viewed item). If intent Hit@1 rises toward refined levels or the VQRA gain shrinks by >~30% relative, the history-use diagnosis is partly label-confounded; if gaps and VQRA gains hold, the claim is robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that LLMs/agents struggle specifically on early-stage intent + hard user-viewed candidates because of weak history use, not mere semantic matching. That claim is measured almost entirely by Hit@1/5 against the observed purchase among Chard (Eq. 8; §2.2.3, §4.1.3). The paper itself flags that purchases are confounded by ranking exposure, stock, promotions, and platform mediation (Limitations), and residual commercial confounders remain after price/history diagnostics (Appendix B.5.1). Tables 8–9 show history improves purchase distinguishability (e.g., win-rate>60% from 79% query-only to 93% with history), but they do not show that the purchased item is the unique preference-aligned choice among near-miss viewed alternatives under fixed exposure. If a non-trivial fraction of 'correct' labels are exposure/availability-driven rather than preference-driven, then low intent Hit@1 and the VQRA/rubric story partly measure recovery of platform-mediated outcomes, not pure personalization failure. This is the same soft spot the reader flags; it is load-bearing because refined/random settings already show strong semantic competence, so the distinctive personalization gap lives almost entirely on this label.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces APeB, a 5,648-case benchmark for agentic personalization via personalized product search (PPS), built from real content-commerce logs. Each case pairs an underspecified early-stage intent query (and a later refined query) with heterogeneous long-term product/media history and a hard candidate set of user-viewed near-miss items plus the observed purchase. Evaluating strong LLMs under single-prompt, ReAct, and Deerflow workflows against UniSAR, the authors report that models perform well under refined queries or random candidates but drop to near-baseline Hit@1 on intent + hard candidates; multi-step agents help mainly when goals are explicit. Rubric diagnostics (QI/PQ/RQ/AR) attribute the gap largely to weak history-grounded preference extraction, and a simple history-aware rewrite pipeline (VQRA) yields consistent Hit@1 gains. The manuscript includes history/candidate ablations, counterfactual and shuffle stress tests, multi-judge rubric checks, and an honest Limitations section.","tokens_in":34115,"tokens_out":1506,"duration_ms":26291,"significance":"If the main empirical pattern holds, APeB fills a genuine gap: existing shopping, recommendation, memory, and PPS benchmarks rarely combine raw underspecified queries, noisy heterogeneous histories, and closely competing user-viewed candidates. The construction pipeline (search-to-order disentanglement, non-trivial session filters, intent/refined localization, hard-candidate sets) is carefully documented, with privacy/release governance and multiple validity checks (human agreement on filters, purchase distinguishability diagnostics, contamination-style counterfactual histories). The constructive VQRA result is useful: it shows headroom from explicit history-utilization modules without claiming a full agent solution. Strengths include multi-model tables, ablations isolating history and candidate hardness, and rubric–Hit@K alignment analyses. These make the paper a credible testbed contribution for personalized agents, contingent on treating purchase labels with appropriate caution.","major_comments":[{"comment":"The central claim—that LLMs/agents specifically fail at early-stage personalization under intent + hard candidates due to ineffective history use—is measured almost entirely by Hit@1/5 against the observed purchase in Chard (Eq. 8; §2.2.3; §4.1.3; Table 2). The Limitations section correctly notes confounds from ranking exposure, stock, promotions, and platform mediation, and Appendix B.5.1 acknowledges residual commercial confounders after price/history diagnostics. Tables 8–9 show history improves pairwise purchase distinguishability, but they do not establish that the purchased item is the unique preference-aligned choice among near-miss viewed alternatives under fixed exposure. Because refined/random settings already demonstrate strong semantic competence, the distinctive “personalization gap” lives on this label. Please add load-bearing sensitivity analyses (e.g., exposure/rank-condi","section":null},{"comment":"Session retention and query localization rely on GPT-4.1 (with human audits) for intent–order alignment and for selecting distinct intent vs refined queries (§2.1.2; §2.2.2; Appendix B.3). Human agreement is high (~93–97% on audited samples), which is good, but the filter may preferentially retain trajectories where semantic alignment is LLM-recoverable, potentially understating harder personalization cases and slightly favoring models similar to the filter. Please report (i) how many sessions are discarded at each GPT step, (ii) agreement stratified by category/ambiguity, and (iii) a small human-only or alternative-model filter ablation on Hit@1 gaps, so the intent-vs-refined contrast is not partly an artifact of construction.","section":null},{"comment":"The agentic results (Table 2; §4.2–4.3; Appendix C.4–C.5) show ReAct helps on refined queries but yields limited or negative gains on intent, with Deerflow often worse. Rubrics attribute this to poor multi-step planning and weak history retrieval (low QH, limited spontaneous multi-round correction). This is informative, but the paper then concludes a need for “dedicated history-utilization modules” largely from VQRA and these agent setups. The agent configurations (prompted ReAct/Deerflow, BM25 title-heavy retrieval, limited memory baselines in the appendix) may not represent competitive personalization agents. Please either (a) strengthen the agent baseline suite with stronger memory/retrieval planners under a fixed compute budget, or (b) narrow the claim to “standard ReAct-style tool-use pipelines as evaluated here,” so the architectural recommendation is proportionate to the evidence.","section":null}],"minor_comments":[{"comment":"Table 1 is helpful; consider adding a row-level note on whether candidates are user-viewed vs randomly sampled, since that is a core differentiator of APeB.","section":null},{"comment":"In §2.1.1, the search-to-order window Δ = 30 minutes is stated without sensitivity; a short appendix table for Δ ∈ {15, 30, 60} on retained-session count and Hit@1 would reassure readers that the main pattern is not threshold-specific.","section":null},{"comment":"Figure 3 packs many rubric metrics; axis scales and “Low/High” color bars are hard to read in grayscale. Adding numeric tables for the same scores (even in the appendix) would improve auditability.","section":null},{"comment":"Notation: Hu/Su (Eqs. 1–2), Chard (Eq. 8), and Inputτ (Eq. 9) are clear, but τ is reused for query type and for thresholds τv/τa; consider distinct symbols.","section":null},{"comment":"Appendix C.3.2 category-prior history selection uses the intent-query category label; state explicitly in the main text that this does not use the purchased item (you say so in the appendix) to pre-empt leakage concerns.","section":null},{"comment":"Minor polish: “AgentPersonalizedBenchmark” spacing/hyphenation is inconsistent with “APeB”; “Causal shirts” in Figure 1 is likely “Casual”; a few arXiv-style citation glitches appear in the related-work block.","section":null},{"comment":"Release status is “planned for controlled release after compliance review.” For a benchmark paper, please state as concretely as possible what will be released (schema, sample size, access path) so reproducibility expectations are clear.","section":null}],"recommendation":"major_revision","confidential_remarks":"Solid benchmark paper with unusually careful construction and diagnostics for industry-log work. The purchase-label confound is the main scientific soft spot and is already partially acknowledged; I would not reject on that basis, but I would require stronger sensitivity analysis and claim tempering before acceptance. Scope fit is good for an AI/agents venue that values evaluation resources. Controlled-release data is a practical concern for reviewers and follow-up work—push for a clear access statement in revision."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that APeB is a real resource, not a re-skin of WebShop or JDSearch. They pair early-stage underspecified queries with heterogeneous product/media histories and user-viewed near-miss candidates, then measure whether LLM agents recover the observed purchase. That combination is what prior tables mostly avoid, and Table 1 is fair about it.\n\nWhat they do well: construction is careful (search-to-order disentangle, multi-query + extensive browsing + intent–order filters, GPT alignment with ~93–97% human checks, anonymization/release scope). Experiments are multi-model and multi-workflow (single-prompt, ReAct, Deerflow, UniSAR), with history and candidate ablations, shuffle/counterfactual history checks, and intermediate rubrics (QI/PQ/RQ/AR) that track Hit@K. The main empirical pattern holds: refined or random candidates look strong; intent + hard candidates collapse toward UniSAR, ReAct helps little or hurts, and VQRA’s history-grounded rewrite recovers Hit@1. That is a clean, actionable diagnosis for agent design—history utilization under ambiguity is the bottleneck, not raw semantic capacity.\n\nSoft spots, in proportion. Purchase-as-preference is the load-bearing label, and the Limitations section already says so: exposure, stock, promotions, platform mediation. Their own pairwise diagnostics (Tables 8–9) show history improves distinguishability, which supports treating the label as preference-correlated rather than pure noise, but it does not prove uniqueness among near-misses under fixed exposure. So the “personalization failure” story is partly recovery of platform-mediated outcomes. That weakens how hard we should treat APeB as a settled community standard; it does not erase the refined-vs-intent gap or the VQRA signal. Single-platform, static offline eval, and controlled release are ordinary resource-paper limits, not hidden landmines. Free thresholds (Δ, τ_v, τ_a, k) are documented; citation pattern looks normal for the area.\n\nWho it’s for: people building or evaluating shopping/personalization agents who need a harder offline test than refined-query retrieval. Worth a serious referee. I’d engage, cite the benchmark and the history-underuse finding, and keep the purchase confound in the footnote.","headline":"Solid new PPS agent benchmark that actually stresses vague intent + hard viewed candidates; the history-underuse diagnosis is useful, with purchase-as-preference as the main caveat rather than a collapse of the result.","tokens_in":34673,"tokens_out":575,"would_cite":true,"duration_ms":7834,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM agents handle clear product queries but fail early-stage personalization mainly because they underuse noisy user history.","keywords":["LLM agents","personalization","personalized product search","benchmark","query refinement","history utilization","ReAct","hard candidates"],"falsifier":"If, under the same intent queries and hard candidate sets, stronger history-selection or memory modules raised Hit@1 without first refining the query—or if VQRA stopped helping once confounds like price, stock, and exposure were controlled—then the claim that ineffective history use is the main bottleneck would be in trouble.","tokens_in":34552,"feed_emoji":"🛒","tokens_out":636,"duration_ms":6013,"temperature":0.7,"pith_summary":"This paper claims that personalization for LLM agents is not mainly about stronger general reasoning or multi-step tool use; it is about recovering latent intent and preferences from raw, underspecified queries and messy interaction histories, then choosing among close alternatives. Existing shopping and memory benchmarks largely miss that joint demand by using cleaned queries, simplified histories, or easy negatives. The authors build APeB, a personalized product-search benchmark from real platform logs that pairs early-stage vague intents with heterogeneous histories and hard candidates the user actually viewed. On that testbed, state-of-the-art models and ReAct-style agents do well when the query is already refined, but fall off when the goal is still vague; intermediate rubrics pin the drop on weak intent inference and poor history use rather than raw semantic matching. A simple history-conditioned query-refinement pipeline, VQRA, consistently lifts Hit@1, which the authors read as evidence that personalization agents need explicit history-utilization modules.","feed_headline":"LLM agents fail vague shopping queries, not clear ones","feed_subtitle":"A real-log benchmark shows the bottleneck is history use, and simple query rewrite helps","key_machinery":"APeB (Agent Personalized Benchmark): non-trivial search-to-order sessions from real logs, each pairing a long heterogeneous history, an intent vs refined query, a hard candidate set of user-viewed products, and an observed purchase; plus VQRA, a history-aware vague-query refinement pipeline that rewrites the early intent before ranking.","core_discovery":"Current LLMs and multi-step agent workflows handle explicit refined product queries well, but struggle on early-stage underspecified intents with hard user-viewed candidates; the performance gap is driven mainly by ineffective use of noisy history for intent and preference discovery, and explicit history-aware query refinement recovers part of that gap.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLM agents nail clear shopping queries but fail vague ones","Poor history use sinks agents on underspecified product intents","APeB: agents struggle early-stage queries needing preference discovery","History-aware rewrite closes gap on raw product search for LLMs","Explicit queries succeed; noisy histories expose agent personalization limits"],"cache_read_input_tokens":28416,"weakest_assumption_plain":"The observed purchase is treated as reliable ground truth for which hard candidate best matches the user’s preference, even though exposure, stock, promotions, and platform mediation can also decide what was bought.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents nail clear shopping queries but fail vague ones","Poor history use sinks agents on underspecified product intents","APeB: agents struggle early-stage queries needing preference discovery","History-aware rewrite closes gap on raw product search for LLMs","Explicit queries succeed; noisy histories expose agent personalization limits"]},"model":"grok-4.5","effort":"low","cost_usd":0.005434,"raw_usage":{"total_tokens":1439,"prompt_tokens":704,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":54340000,"prompt_tokens_details":{"text_tokens":704,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":668,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":704,"tokens_out":67,"duration_ms":5441,"temperature":1.0,"reasoning_tokens":668,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T04:25:19.789824+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"If, under the same intent queries and hard candidate sets, stronger history-selection or memory modules raised Hit@1 without first refining the query—or if VQRA stopped helping once confounds like price, stock, and exposure were controlled—then the claim that ineffective history use is the main bottleneck would be in trouble.","supporting_citations":[],"review_version":1}