{"id":"8fe07e44-2faf-4536-a22f-bcdc03626c36","arxiv_id":"2507.22213","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A data mining pipeline classifies e-commerce query rewrites into Same, Similar, and Inspired intent buckets, and a sequence-to-sequence model is trained to generate intent-conditioned reformulations, but the evaluation measures rewrite-type agreement rather than search relevance.","lead":"This paper from eBay describes a data mining pipeline that extracts buyer query reformulations from search logs and labels them into three intent buckets: Same, Similar, and Inspired. The authors train sequence-to-sequence models to generate intent-conditioned query rewrites, but the reported evaluation uses token-level metrics rather than the retrieval relevance metrics promised in the abstract.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 'precision-oriented relevance' gains rest on token-level agreement with a test set generated by the same mining pipeline; no retrieval-level or human-judged relevance evaluation supports the central claim.","rationale":"The reader's strongest-claim analysis correctly identifies that the paper rises or falls on whether intent-conditioned rewrites improve search relevance. My stress test agrees with the rejection: the reported metrics are token- and rewrite-type-level measures computed on a test set generated by the same mining pipeline, so the abstract's phrasing 'measurable gains in precision-oriented relevance metrics' is not supported by the evidence. I differ slightly from the reader's emphasis: the single most load-bearing problem is evaluation circularity rather than the intent-label assumption itself, although the two are related. If the co-click and one-hop labels are confounded, then even a perfect match to the mined test distribution would not imply better search relevance. The proposed end-to-end retrieval evaluation with independent queries and human or behavioral relevance outcomes would settle both the circularity concern and the label-validity concern. No formal verification, reproducibility artifacts, or error bars are present to offset this gap, so the reader's REJECT verdict stands unchanged.","tokens_in":7621,"tokens_out":3007,"duration_ms":35458,"concrete_test":"Generate candidate rewrites with θT/θT@5 for a held-out set of queries drawn after the 4-week mining window (or from fresh logs), then retrieve items through the same production retrieval stack used in deployment. Compare original-query retrieval against rewrite-expanded retrieval using precision@k, recall@k, and nDCG@k, scored with either human relevance judgments on a sampled subset or logged click/conversion as outcome, with bootstrap confidence intervals. If the intent-conditioned rewrites do not produce a statistically significant improvement over original queries and a standard baseline (e.g., [3] or a token-drop heuristic), the central claim fails as stated. To isolate label validity, also compare co-clicked-pair overlap against a popularity-matched random-pair baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that intent-aware reformulation yields measurable gains in precision-oriented relevance metrics—is load-bearing and unsupported as evaluated. Section 4 reports only rewrite-type frequency alignment (Table 2), token recall/precision, BLEU, ROUGE-L, and the new RATS/rtfw metrics (Table 3). All are computed against 'gold' reformulations mined by the same pipeline described in §3.1, so high scores measure agreement with the mining procedure, not retrieval precision. Table 2 itself shows that even the best model distributions diverge substantially from the test rewrite types: θT@5 produces 1.17% Same rewrites while the test set has 0.00%, and 17.91% Other versus 0.09% in the test set. The §4.3 claim that these results 'validate the effectiveness of the mined training data in capturing a representative sample of real-world reformulation behavior' is therefore an overreach. Moreover, the weakest link in the pipeline—co-clicked cross-session pairs and one-hop transitions as intent labels (§3.1)—is never validated against human judgments or downstream relevance, so the 'ground truth' targets themselves may encode popularity, merchandising, or positional confounds. Without retrieval-level precision@k/nDCG or engagement measurements on independent queries, the abstract's promise of 'measurable gains in precision-oriented relevance metrics' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an intent-aware query reformulation framework for e-commerce search. It introduces a mining pipeline that extracts query pairs from in-session transitions, cross-session co-clicked pairs, and one-hop co-clicked neighborhood transitions, then assigns these pairs to three intent buckets (Same, Similar, Inspired). The authors train intent-conditioned sequence-to-sequence models and evaluate them with rewrite-type frequency analysis, token-level precision/recall, BLEU, ROUGE-L, and two new metrics (RATS and rtfw). The central claim is that grounding reformulation in inferred intent yields measurable gains in precision-oriented relevance metrics.","tokens_in":7862,"tokens_out":3282,"duration_ms":40777,"significance":"If the central claim were supported, the proposed framework would be a valuable industrial contribution to e-commerce query reformulation, particularly for null/low-recall recovery and related-search recommendation. The paper has concrete strengths: it uses large-scale real buyer logs, defines a multi-bucket intent taxonomy, and proposes metrics (RATS, rtfw) that expose structural failure modes not captured by BLEU/ROUGE. However, the significance is currently limited because the evaluation only measures token-level agreement with a mining pipeline's own labels; no retrieval-level or human-judged relevance experiment is reported. The gap between the abstract's promise of relevance gains and the actual evidence is the main obstacle to acceptance.","major_comments":[{"comment":"The evaluation never measures retrieval precision or downstream engagement. All reported metrics are token-level agreement with gold rewrites generated by the same mining pipeline described in §3.1, not precision-oriented relevance metrics. The abstract's claim of 'measurable gains in precision-oriented relevance metrics' is therefore unsupported by the experiments. The authors should either add retrieval-level offline evaluation (e.g., precision@k or nDCG using independently judged query-item relevance) or substantially revise the abstract, §1, and §4.3 to claim only rewrite-fidelity improvements.","section":"§4, Tables 2–3; Abstract"},{"comment":"RATS and rtfw compare model outputs to labels produced by the same mining pipeline that created the training data, so the evaluation is partly self-referential. The sentence in §4.3 that these results 'validate the effectiveness of the mined training data in capturing a representative sample of real-world reformulation behavior' overreaches. To support this claim, the authors should validate a sample of the mined pairs with external human annotation or with independent behavioral outcomes, and report inter-annotator agreement.","section":"§3.1–3.2 and §4.3"},{"comment":"The label-generation assumptions for Similar and Inspired intent are not validated. Co-clicked items across sessions are assumed to indicate shared or similar intent, and one-hop transitions from co-clicked pairs are assumed to indicate exploratory intent. Co-clicks can instead reflect item popularity, merchandising, or positional bias. The paper provides no analysis controlling for these confounds, no qualitative audit beyond three examples per bucket, and no human judgment study. Because these assumptions are the source of the 'ground truth' labels, this is a load-bearing weakness that needs experimental support.","section":"§3.1"},{"comment":"The rewrite-type distribution of even the best-performing model deviates sharply from the test distribution. For example, θT@5 produces 1.17% Same rewrites while the test set contains 0.00%, and 17.91% Other versus 0.09% in the test set. This mismatch is not discussed quantitatively and weakens the claim that the model generalizes across rewrite types. The authors should analyze these discrepancies and either improve the model or temper the generalization claim.","section":"Table 2"},{"comment":"No error bars, significance tests, or multiple-seed variance are reported. The differences between models on RATS and rtfw may be within noise, especially given the small token-level effect sizes visible in Table 3. The authors should report standard deviations across training runs and perform significance tests for the headline comparisons, or explicitly state that the reported numbers are from a single run and should be treated as preliminary.","section":"§4"}],"minor_comments":[{"comment":"The figure is difficult to read in the provided PDF; the arrows and labels are too small. Please provide a higher-resolution version with legible font sizes.","section":"Figure 1"},{"comment":"The phrase 'two-Hop neighbors' has inconsistent capitalization. Define 'hop' precisely and use consistent terminology throughout.","section":"§3.1"},{"comment":"The meaning of 'NL recovery' is not defined at first use. If it stands for 'null and low' recall recovery, spell this out for the reader.","section":"§3.3"},{"comment":"Some example queries contain typos (e.g., 'sansung filp'). If these are verbatim user queries, state this explicitly; otherwise correct them.","section":"Table 1"},{"comment":"The distinction between Same intent and Similar intent is not crisp. The Same-intent example 'nike air jordan 4 → nike air jordan 11' changes the model number, while the Similar-intent example 'nike womens size 9 → nike womens air max size 9' adds a specific aspect. The authors should provide a rule or annotated criterion that separates these buckets, especially since the paper states that token count is not the deciding factor.","section":"§3.2"},{"comment":"The User Engagement score weights are described as a 'weighted combination' but the weights are never specified. Since this is a free parameter, provide the exact weighting scheme or state whether weights were tuned and how.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style industrial contribution with a promising pipeline but a substantial gap between claimed and demonstrated results. The lack of retrieval-level evaluation and the self-referential nature of the labels make the current acceptance risky. However, the central idea is not inherently unsound; adding a human-annotated sample and retrieval-based offline metrics, or rewriting the claims to match the actual evidence, could make it publishable. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's actually new. The cross-session one-hop co-click mining for Inspired Intent pairs is a genuine data-construction idea, and conditioning a seq2seq model on intent tags is a reasonable extension. The pipeline description is clear, the examples in Table 1 are helpful, and the paper is upfront that in-session pairs carry contextual bias. That part is worth reading.\n\nThe soft spots are in the evaluation, and they are load-bearing. The abstract promises measurable gains in precision-oriented relevance metrics, but the experiments never touch retrieval precision. Every metric in Table 3 is computed against gold rewrites mined by the same pipeline (§3.1) that produces the test set. RATS measures how often the model's rewrite type matches the pipeline's rewrite type. That is agreement with the labeler, not with any independent notion of relevance. Table 2 shows why this matters: the best model still produces a rewrite-type distribution far from the test set (e.g., 17.91% Other vs 0.09% in the test set; 1.17% Same vs 0.00%). So even the frequency-alignment story is weak.\n\nThe second soft spot is the intent labels themselves. Co-clicked items across sessions are treated as evidence of shared or similar intent, but co-clicks can reflect popularity, merchandising, or positional bias. The paper does not validate these labels against human judgments or downstream retrieval outcomes. That makes the ground truth itself questionable, and the model can only learn what the mining procedure encodes.\n\nThere are also minor issues: no error bars or significance tests, and the baselines are not the strong neural or counterfactual baselines cited in the introduction. For a workshop paper the scope is reasonable, but the central claim as stated outruns the evidence.\n\nWho is this for? Researchers in e-commerce search who want to see one concrete way to build intent-conditioned rewrite training data. The mining idea is worth citing in related work. But I would not cite it as evidence that intent-aware reformulation improves retrieval precision, because that is not what is measured.\n\nRecommendation: send it to review, with the clear expectation of major revision. A serious referee can push the authors to run a retrieval-level precision@k or nDCG evaluation on independent queries, and ideally to validate a sample of the mined pairs against human raters. The novel construction deserves that effort. Reject as it stands, but do not desk-reject.","headline":"The data-mining trick is new and worth engaging with, but the evaluation only measures agreement with that same trick, so the precision-gain claim does not land.","tokens_in":8376,"tokens_out":2368,"would_cite":false,"duration_ms":26108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims behavior-mined intent labels, not lexical overlap, should drive e-commerce query reformulation.","keywords":["e-commerce search","query reformulation","buyer intent","behavior mining","co-clicked queries","sequence-to-sequence model","neural machine translation","product search"],"falsifier":"Run an online A/B test on a search engine where low-recall queries are expanded by the intent-conditioned model's top rewrites against a baseline of lexical or in-session-only rewrites, and check whether the intent-conditioned rewrites raise search-results-page precision or conversion.","tokens_in":7404,"feed_emoji":"🔍","tokens_out":8696,"duration_ms":93605,"temperature":0.7,"pith_summary":"This paper tries to establish that query reformulation in e-commerce search improves when a rewrite model is trained on intent labels mined from buyer behavior rather than on lexical or synthetic signals. The authors build a dataset from four weeks of search logs by pairing queries three ways: in-session transitions that end in engagement, cross-session queries that share clicked items, and one-hop moves beyond those co-clicked pairs. Each pair is assigned to a Same Intent, Similar Intent, or Inspired Intent bucket, and a sequence-to-sequence neural machine translation model is trained with the intent tag attached to the source query. The paper reports that the full-pipeline Transformer comes closest to the real-world distribution of rewrite types and achieves the best coverage and rewrite-type agreement, which it takes as evidence that the mined data captures authentic reformulation behavior. A sympathetic reader would care because the same single model is positioned to recover null-recall queries, generate related-search suggestions, and populate bottom-of-page recommendations.","feed_headline":"Intent-tagged query rewrites beat lexical baselines","feed_subtitle":"Behavior-mined Same, Similar, and Inspired intent labels train one reformulation model for e-commerce search.","key_machinery":"The load-bearing mechanism has four pieces. First, the mining procedure: in-session n-hop query transitions, cross-session co-engaged query pairs connected by shared clicked items, and cross-session one-hop co-clicked neighborhood transitions. Second, the intent taxonomy: Same Intent preserves the goal, Similar Intent narrows or specifies it, and Inspired Intent pushes toward broader, exploratory products. Third, a sequence-to-sequence neural machine translation model trained with intent-type tags appended to each instance, so a single model learns intent-specific rewriting in a multitask setup. Fourth, the evaluation machinery: the Rewrite Type Agreement Score (RATS), which measures whether a prediction's structural rewrite type matches the gold rewrite's type, plus frequency-weighted recall and precision that reward models doing well on both common and rare rewrite patterns.","core_discovery":"On its own terms, the paper claims that one unified reformulation model can serve multiple e-commerce search applications if it is trained on query pairs whose intent is inferred from engagement rather than from text overlap. The mining starts with in-session n-hop pairs where the target query produced a click, purchase, bid, or cart action, then adds cross-session pairs whose queries share clicked items, then one-hop co-clicked neighborhood pairs that capture exploratory movement. After filtering on product category, recall similarity, and query length, the pairs are labeled Same Intent, Similar Intent, or Inspired Intent by domain-expert definitions. The paper's finding is that a Transformer trained on the full mixture, with the intent label appended to the source, outperforms heuristic token-drop, in-session-only, and knowledge-graph baselines on coverage, the Rewrite Type Agreement Score (RATS), and rewrite-type frequency-weighted recall and precision, and that its rewrite-type distribution is the closest to the test data. From this the paper concludes that the mined training data represents real-world reformulation behavior and that intent-conditioned rewrites improve retrieval relevance and downstream engagement.","pith_inferences":["Editorial inference: the same co-clicked and one-hop mining could be reused for session-aware query suggestion and personalization, because the dataset already encodes temporal and cross-user behavior.","Editorial inference: the paper's evidence stops at rewrite fidelity; the promised relevance gains would need an online A/B test of search-results-page precision, click-through, or conversion to be confirmed.","Editorial inference: a human-judgment audit of a random sample of mined pairs would separate genuine intent ties from popularity or merchandising artifacts in the Similar and Inspired buckets."],"forward_implications":["One intent-conditioned model can serve low-recall recovery, related-search suggestions, and bottom-of-page recommendations, replacing separate pipelines for each application.","Null- and low-recall queries gain multi-direction rewrites that broaden lexical coverage while staying connected to the original buyer goal.","The full-pipeline model reproduces the full spread of rewrite types, including supersets, subsets, replacements, and compound changes, rather than collapsing to token deletion.","Generating multiple rewrites, as in the top-five variant, reaches near-perfect coverage, which is what production systems need when several candidate queries are shown to users."],"supporting_citations":[{"why":"Supplies the sequence-to-sequence neural machine translation architecture used to generate reformulations.","marker":"[13]"},{"why":"Earlier behavior-driven reformulation framework whose direction this work extends with mined intent labels.","marker":"[2]"},{"why":"Empirical study of query reformulation in e-commerce search that motivates learning rewrites from buyer behavior.","marker":"[3]"},{"why":"Targets null and low search results, the flagship downstream application for the reformulation model.","marker":"[5]"},{"why":"Establishes statistical machine translation as a query-rewriting approach, the lineage of the neural model.","marker":"[4]"},{"why":"Describes the production search architecture into which the reformulation capability is integrated.","marker":"[1]"}],"fun_headline_variants":["Behavior-mined intent labels boost query rewrites","Intent-aware rewrites outperform lexical baselines","Unified intent model wins on e-commerce search","Search rewrites labeled by engagement beat text overlap","Neural reformulation trained on click intent improves retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole label scheme rests on one premise: two different buyers clicking on the same product during their searches means the two searches share the same or similar intent, and moving one step away from those shared clicks means the buyer is exploring; if clicks are driven by popularity, page position, or promotions, the labels teach the model the wrong mapping.","fun_headline_variants_meta":{"raw":{"variants":["Behavior-mined intent labels boost query rewrites","Intent-aware rewrites outperform lexical baselines","Unified intent model wins on e-commerce search","Search rewrites labeled by engagement beat text overlap","Neural reformulation trained on click intent improves retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2849,"prompt_tokens":944,"completion_tokens":1905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1834}},"tokens_in":560,"tokens_out":1905,"duration_ms":16479,"temperature":1.0,"reasoning_tokens":1834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:56:42.784222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an online A/B test on a search engine where low-recall queries are expanded by the intent-conditioned model's top rewrites against a baseline of lexical or in-session-only rewrites, and check whether the intent-conditioned rewrites raise search-results-page precision or conversion.","supporting_citations":[{"cited_title":"Hirsch, I","cited_arxiv_id":null,"evidence_quote":"Empirical study of query reformulation in e-commerce search that motivates learning rewrites from buyer behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Targets null and low search results, the flagship downstream application for the reformulation model."},{"cited_title":"Riezler, Y","cited_arxiv_id":null,"evidence_quote":"Establishes statistical machine translation as a query-rewriting approach, the lineage of the neural model."},{"cited_title":"Trotman, J","cited_arxiv_id":null,"evidence_quote":"Describes the production search architecture into which the reformulation capability is integrated."}],"review_version":1}