{"id":"1c7bf053-e384-4c4c-b1fc-fab1b69f2abc","arxiv_id":"2412.01269","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A continual pre-training framework combining query-item joint training, in-context pre-training on related queries/items, and teacher-generated reading comprehension data improves LLM relevance modeling in commercial search.","lead":"This paper presents CPRM, a three-part continual pre-training recipe that adapts a large language model to judge search relevance between queries and mini-app items. The authors report offline accuracy gains and a small but statistically significant click-through increase in an online deployment, suggesting the recipe works in a real commercial search engine.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline gains may be inflated by overlap between the continual pre-training corpora and the SFT test set; no disjointness check is reported.","rationale":"I read the paper in good faith: it is an industrial continual pre-training report with a coherent three-module framework, offline experiments, an online A/B test, and evidence of long-term deployment. The strongest claim is that CPRM yields convincing improvements over GLM-2B, and the load-bearing condition is that those improvements reflect transferable knowledge rather than evaluation-set contamination. The reader's weakest assumption focused on the undisclosed Contriever threshold sigma in ICP and on teacher-generated RCD data quality; those are legitimate tuning/quality concerns, but they affect the magnitude or attribution of individual module gains. The overlap/leakage issue is more fundamental: if the DKE/ICP/RCD corpora share queries or items with the SFT test set, every offline number in Table 2 is suspect, and the ablation logic that supports the framework's internal design collapses. This is not an accusation of intentional leakage; it is a missing control that is standard in empirical IR/LLM work and is especially urgent here because DKE explicitly uses the most frequently visited items and ICP explicitly uses click logs from the same scenario as the test data. The online A/B test is real evidence and partially mitigates the concern, but it does not repair the module-level attribution. I therefore agree with the reader's CONDITIONAL verdict: the claim is plausible, but independent verification is blocked by unreported overlap checks, undisclosed data, and no code release. My concern would push the conditionality specifically toward requiring a disjointness audit before the offline gains are treated as established; it does not justify rejection because the online A/B result and deployment record provide independent, if narrower, support. The concrete test above would settle the issue directly.","tokens_in":11532,"tokens_out":3375,"duration_ms":33666,"concrete_test":"Require the authors to quantify exact and near-duplicate overlap between each continual pre-training corpus (DKE, ICP, RCD) and the SFT test set, at the query level, item level, and query-item pair level. Then rerun the Table 2 evaluation after removing from all pre-training data every query or item that appears in the test set (or, alternatively, evaluate on a freshly sampled disjoint test split). If the Table 2 row 18 vs row 8 gaps (Acc +2.48, F1 +2.47, AUC +2.45) shrink materially, e.g., by more than 0.5 point in any metric, the headline offline claim is contaminated by memorization; if the gaps persist on a fully disjoint split, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central offline claim (Table 2, rows 8 vs 18: +2.48 Acc, +2.47 F1, +2.45 AUC for CPRM over GLM-2B) rests on the assumption that the continual pre-training data and the manually annotated test set are not contaminated by shared queries, items, or query-item pairs. Section 5.1 reports only dataset sizes: DKE uses the top 500K most visited items with 5 top queries per item from click logs, ICP is built from real-world search click logs, and RCD generates item-derived queries/summaries. The SFT test set (35,057 samples, 5,426 queries, 8,406 items) is also sourced from real mini-app search results. In such a commercial setting, the test items and queries are very likely to be among the most frequent, and therefore to appear in the DKE and ICP pre-training corpora. If a test query-item pair, or enough of its components, was seen during pre-training, the model can memorize co-occurrence patterns rather than learn transferable relevance reasoning. This is especially acute for ICP, whose training instances are ordered lists of click-related queries/items, so the model may simply learn to output 'relevant' for pairs it encountered in context. The paper never states that pre-training and SFT evaluation sets are disjoint, nor does it report overlap statistics. Without this, the module-level ablation gains attributed to DKE/ICP/RCD cannot be interpreted as evidence of generalization. The online A/B test provides some independent support, but it only reports a 0.32% PVCTR lift and does not validate the internal attribution of the three modules.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CPRM, a continual pre-training framework for LLM-based query-item relevance modeling in a commercial mini-app search engine. The framework combines domain knowledge enhancement via joint query-item masked language modeling, in-context pre-training on semantically related queries/items mined from click logs, and reading-comprehension distillation from a larger teacher LLM. Experiments on GLM-2B with a proprietary SFT dataset show additive gains of +2.48 Acc, +2.47 F1, and +2.45 AUC over GLM-2B (Table 2), and a two-week online A/B test reports a statistically significant 0.32% PVCTR lift and improvements in human-evaluated relevance metrics. The model is stated to have been serving production traffic for over nine months.","tokens_in":11755,"tokens_out":7035,"duration_ms":56687,"significance":"If the empirical claims hold up, this is a valuable industrial contribution: it provides a modular recipe for adapting open-weight LLMs to a specialized relevance domain, with a clear decomposition that lets practitioners see the contribution of each module. The paper also contains several good practices: it compares with a range of model sizes and architectures, includes an online A/B test, and reports long-query and training-step analyses. However, the entire case rests on self-reported, proprietary measurements. The absence of data-disjointness checks, threshold sensitivity analyses, and statistical details means the reader cannot currently distinguish a genuine transfer effect from memorization or from tuning to a specific filtering threshold. These concerns are addressable, but they are load-bearing for the central claim.","major_comments":[{"comment":"The offline gains in Table 2 (e.g., rows 8 vs 18: +2.48 Acc, +2.47 F1, +2.45 AUC) are presented as evidence of generalization, but the manuscript never establishes that the continual pre-training corpora and the human-annotated test set are disjoint. Section 5.1 reports that DKE data are built from the top 500K most visited items, ICP from real-world click logs, and RCD from item-derived generated queries, while the test set (35,057 samples, 5,426 queries, 8,406 items) is also drawn from real mini-app search results. In such a setting, exact or near-duplicate query-item pairs are likely to appear in both; models could then memorize co-occurrence rather than learn transferable relevance. Please report overlap statistics at the query, item, and pair level between each pre-training corpus and the test set, and, if feasible, re-run the ablation on a disjoint held-out set.","section":"Section 5.1, Table 2"},{"comment":"The claim in Table 2's caption and Section 5.3 that 'improvements over variants are statistically significant with p < 0.05' is not backed by any description of the test, the number of runs, or the variance across runs. No confidence intervals are reported for Acc., F1, or AUC in Table 2 or Figure 4, and Section 6 similarly states a 'statistically significant increase of 0.32% in valid PVCTR at a 95% confidence level' without giving the confidence interval, the test statistic, or whether the 0.32% is relative or absolute. Please specify the significance test (e.g., paired bootstrap, McNemar), the number of model runs, and report confidence intervals for the main offline and online metrics.","section":"Section 5.3, Table 2 and Section 6"},{"comment":"The ICP component, which provides the largest single-module gain (+1.54 AUC in Table 2, row 15), depends on the threshold σ in Eq. (4) and on the similarity-based reordering described in Section 4.2. The value of σ is never reported, and there is no sensitivity analysis or ablation of the fine-screening and reordering stages. If σ is chosen too loosely, semantically unrelated pairs remain; if too tightly, the constructed context loses diversity. Similarly, the choice of ascending order is asserted to 'implicitly indicate the strength of relevance' but is not compared with any alternative ordering. Please report σ, vary it over a reasonable range, and ablate the filtering and ordering choices to show that the ICP gains are not artifacts of a particular threshold.","section":"Section 4.2, Eq. (4), Table 2"},{"comment":"The RCD module contributes +0.42 AUC in Table 2 (row 16), but the teacher-generated queries, summaries, and explanations are used without any quality filtering. The paper does not report manual evaluation of the generated data, a filtering criterion, or an analysis of how generation errors affect downstream relevance. Since a teacher LLM can produce fluent but semantically incorrect content, please provide evidence that the generated instances are accurate (e.g., sample-based human evaluation, agreement with click labels, or ablation excluding low-confidence generations).","section":"Section 4.3, Table 2"},{"comment":"The online A/B test is the only out-of-sample support for the method, but the description is too brief to assess: no information is given on the number of users or queries in the 5% experiment traffic, the exact duration beyond 'two weeks,' how 'valid' clicks are defined, the baseline system's precise configuration, or the practical significance of a 0.32% lift. The human evaluation metrics (0.75% Badcase@10 reduction and 4.71% Error Filtering Rate decrease) also lack confidence intervals or sample sizes. Please expand this section with the missing experimental protocol details.","section":"Section 6"}],"minor_comments":[{"comment":"The title and abstract use 'A LLM-based'; the correct article is 'An LLM-based.' Similar grammar issues appear throughout the text (e.g., 'We conducts' in Section 5.3).","section":"Abstract and Title"},{"comment":"Rows 8 and 13 report identical GLM-2B results (91.16 Acc, 91.95 F1, 91.04 AUC). Please state explicitly whether row 13 is the same checkpoint used as the starting point for continual pre-training or a separately trained baseline.","section":"Table 2"},{"comment":"The mixing coefficient α in Eq. (3) is set to 0.7 without any sensitivity analysis. Since DKE contributes +1.11 AUC, please report the influence of α and of the segment-masking ratio on downstream relevance performance.","section":"Section 4.1"},{"comment":"The long-query analysis reports a 15.85% AUC improvement for queries longer than 15 characters, but the number of test queries in each length bucket is not given. Without confidence intervals and bucket sizes, this striking gain may be driven by a small subset of examples.","section":"Figure 4"},{"comment":"The y-axis label and metric for the training-step curves are missing. Please label the metric (presumably AUC) and add a legend identifying each series.","section":"Figure 5"},{"comment":"ChatGPT and GPT-4 are evaluated with 8-shot prompting but without fine-tuning, so calling them 'strong baselines' in the abstract is misleading. Please describe them as zero/few-shot reference points rather than trained baselines.","section":"Section 5.2"},{"comment":"The deployment description does not clarify whether the online GLM-0.3B model is also CPRM-pretrained or only receives cached scores from the CPRM-trained GLM-2B model. Please state how the evaluated CPRM model relates to the deployed two-tier scoring system.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an industrial systems paper with proprietary data, which is understandable, but the evaluation section does not yet meet the evidentiary standard for a journal. The missing items—data-disjointness statistics, threshold sensitivity, and significance-test details—are likely available to the authors and should be added without new experiments. I do not see a fundamental flaw in the approach, and the online deployment gives some credibility, but the central offline claim is currently under-specified. I have no concerns about citation ethics; the heavy use of Ant Group's prior work is in line with the domain, though the novelty statement in the contributions should be checked against the cited in-context pre-training literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CPRM is a sensible, well-engineered recipe for adapting an LLM to commercial search relevance, and the paper deserves a referee. But the evidence as presented cannot be fully audited, and one issue in particular needs a straight answer before the offline gains can be taken at face value: the paper never checks whether the continual pre-training corpora overlap with the human-labeled test set.\n\nThe most useful part of the paper is the combination. DKE, ICP, and RCD are each individually familiar — domain-adaptive pre-training, in-context pre-training, and teacher-student distillation — but putting them together for query-item relevance, with the click-derived related sets and the reordering-by-similarity, is a concrete and practical design. The ablations show that each module contributes, and the online A/B test, while small (0.32% PVCTR lift, statistically significant), is real evidence. The deployment details in Section 7 are also refreshingly concrete.\n\nThe soft spots are mostly about auditability, not about the architecture. The most serious is data leakage: DKE uses the top 500K most visited items, ICP is built from click logs, and the test set is sampled from real mini-app search results. In a commercial setting, the test items and queries are probably among the most frequent, so the overlap could be substantial. The paper doesn't report any disjointness statistics. If a test pair or its components appear in pre-training, the model can memorize co-occurrence rather than learn transferable relevance. That would inflate the module-level gains in Table 2.\n\nSecond, the ICP construction depends on an unstated threshold sigma in Eq. (4), and there is no sensitivity analysis for it or for the ordering choice. The module gains (+1.54 AUC) could hinge on that threshold, and we don't know. Third, the 'p<0.05' claim in Table 2 has no method attached, and there are no confidence intervals for the offline or online metrics. Fourth, the closest methodological baselines — standard DAPT and standard in-context pretraining without the click-derived sets — are absent; the comparison is against the untrained GLM-2B and generic older models.\n\nNone of this disproves the central claim, which is plausible. The paper is a solid industrial case study. But as it stands, the offline numbers are not independently verifiable, and the leakage risk is real enough that a referee should demand the overlap statistics and the sigma value before accepting the attribution.\n\nI would send it to peer review, but with a clear request for those fixes. For a reader, this is a useful reference for the recipe and for the online deployment lesson, and I'd be happy to see it in print after revision.","headline":"A plausible industrial recipe with a real data-leakage question mark; worth refereeing but the offline gains are not yet auditable.","tokens_in":12402,"tokens_out":3000,"would_cite":true,"duration_ms":25701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CPRM, a three-module continual pre-training recipe for LLMs, lifts commercial search relevance by 2.48 accuracy, 2.47 F1, and 2.45 AUC points over GLM-2B and delivers a 0.32% online click-through lift.","keywords":["continual pre-training","relevance modeling","commercial search","in-context learning","large language models","query-item matching","reading comprehension distillation","click logs"],"falsifier":"Run ICP with a sweep of σ values (including no filtering) and with randomly shuffled item/query order within instances; if AUC gains over the GLM-2B baseline do not shrink or vanish when the semantic filter or the similarity ordering is removed, the paper's explanation of ICP's mechanism is falsified.","tokens_in":11268,"feed_emoji":"🔍","tokens_out":5436,"duration_ms":38608,"temperature":0.7,"pith_summary":"This paper claims that a three-module continual pre-training recipe lets a 2-billion-parameter LLM match or beat much larger models on commercial search relevance. The recipe jointly pre-trains on queries and multi-field item text, pre-trains on sequences of semantically related queries or items built from click logs, and distills reading-comprehension outputs from a larger teacher LLM. In offline evaluation the full framework improves accuracy by 2.48 points, F1 by 2.47 points, and AUC by 2.45 points over the GLM-2B baseline, putting it on par with GLM-10B. Online A/B testing reports a statistically significant 0.32% lift in valid page-view click-through rate, and the model has served traffic for over nine months. The central claim is that domain-specific and task-structured pre-training data, rather than raw scale, drive relevance-modeling gains.","feed_headline":"A pre-training recipe lifts LLM search relevance by 2.48 points","feed_subtitle":"Offline accuracy, F1, and AUC all rise, and online A/B testing shows a 0.32% click-through lift.","key_machinery":"The load-bearing mechanism is the continual pre-training data construction. DKE joins queries and multi-field item text into a single pre-training sequence with 'startofpiece/endofpiece' segment tokens, trained with token-level and segment-level masked language modeling. ICP starts from click logs, builds Q2I and I2Q mappings sorted by click count, filters out pairs whose Contriever cosine similarity falls below a threshold σ (Eq. 4), and then orders the remaining items or queries by similarity before concatenating them into one pre-training example; the ordering is meant to teach the model graded relevance strength. RCD prompts a teacher LLM (Qwen2-72B) to summarize, paraphrase, generate background knowledge, and produce diverse queries with explanations from the item's structured fields, turning underused fields like description into pre-training text that transfers knowledge into the student.","core_discovery":"On the paper's own terms, the discovery is that relevance modeling in commercial search is improved more by how an LLM is continually pre-trained than by increasing model size alone. The authors build CPRM from three components: DKE jointly pre-trains a query with its top clicked item, using special segment tokens and a mix of token- and segment-level masked language modeling; ICP constructs training instances that concatenate a query with its semantically related items, or an item with its related queries, ordered by similarity; RCD uses a teacher LLM to generate summaries, background knowledge, and diversified queries from structured item text, then pre-trains the student on that generated data. When all three are combined on GLM-2B, the model reaches 93.64% accuracy, 94.42% F1, and 93.49% AUC, exceeding every baseline except the two largest GLM variants and matching GLM-10B's AUC within 0.13 points. The paper reports that each module contributes independently, with ICP giving the largest single AUC gain of 1.54 points, and that the full model maintains its advantage at every training step and on long queries.","pith_inferences":["The ICP ordering-by-similarity design implies a soft graded-relevance signal is being learned without explicit labels; a natural extension is to test whether the learned score correlates with human graded judgments.","Because the teacher LLM generates queries and summaries without filtering, the RCD gains may partly reflect data augmentation rather than distillation; a controlled test with randomly selected generated data would separate these effects.","The σ threshold in Eq. 4 is never reported nor ablated, so a sensitivity analysis across σ values is the clearest way to test how much of ICP's gain depends on the semantic filter versus the click-sourced co-occurrence."],"forward_implications":["Combining all three modules yields the best result: DKE+ICP+RCD tops every moderate-size baseline and matches GLM-10B, so the framework can substitute for parameter growth.","ICP alone contributes the largest single AUC gain (1.54 points), suggesting that context built from semantically ordered queries/items is the most powerful of the three signals.","The full CPRM model gains 15.85% AUC on queries longer than 15 characters, indicating that the pre-training recipe specifically improves long-query understanding.","The approach is stable across training steps: unlike the SFT-only baseline, which dips at 16K steps, all CPRM variants improve monotonically, implying the pre-training data eases domain adaptation.","Online, a 0.32% PVCTR lift at 95% confidence, a 0.75% Badcase@10 reduction, and a 4.71% error-filtering-rate decrease justify deployment; the model has been in production for over nine months."],"supporting_citations":[{"why":"Contriever is the semantic model used to filter click-derived query-item pairs in ICP.","marker":"Izacard et al., 2022"},{"why":"Supplies the token-level masked language modeling objective adopted by DKE.","marker":"Devlin et al., 2018"},{"why":"GLM is the base LLM architecture that all pre-training and SFT experiments build on.","marker":"Du et al., 2022"},{"why":"Establishes the LLM pre-training and in-context learning background that motivates ICP.","marker":"Brown et al., 2020"},{"why":"Prior in-context pre-training work that the paper extends to multi-domain query-item pairs.","marker":"Gu et al., 2023"},{"why":"Recent in-context pretraining method that ICP builds on by linking related documents.","marker":"Shi et al., 2024"},{"why":"Qwen2-72B is the teacher LLM used in RCD to generate summaries, knowledge, and queries.","marker":"Yang et al., 2024"},{"why":"Domain-adaptive pre-training baseline that motivates DKE's domain-knowledge injection.","marker":"Gururangan et al., 2020"}],"fun_headline_variants":["Continual pre-training boosts search relevance by 2.48 points","New framework enhances LLM relevance in commercial search","CPRM: pre-train LLMs with query-item context for better search","2.48-point lift from LLM pre-training recipe for search","Three-part pre-training improves LLM search relevance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The strongest single module, ICP, depends on an unstated threshold σ in Eq. 4 that decides which click-derived pairs count as semantically related and on the similarity-based ordering that teaches relevance strength; if the threshold or ordering does not reflect true relevance structure, ICP's AUC gain of 1.54 points would not transfer to other settings, and the paper does not report σ or ablate it.","fun_headline_variants_meta":{"raw":{"variants":["Continual pre-training boosts search relevance by 2.48 points","New framework enhances LLM relevance in commercial search","CPRM: pre-train LLMs with query-item context for better search","2.48-point lift from LLM pre-training recipe for search","Three-part pre-training improves LLM search relevance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001346,"raw_usage":{"total_tokens":5503,"prompt_tokens":1013,"completion_tokens":4490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":4405}},"tokens_in":629,"tokens_out":4490,"duration_ms":27147,"temperature":1.0,"reasoning_tokens":4405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:30:25.813140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ICP with a sweep of σ values (including no filtering) and with randomly shuffled item/query order within instances; if AUC gains over the GLM-2B baseline do not shrink or vanish when the semantic filter or the similarity ordering is removed, the paper's explanation of ICP's mechanism is falsified.","supporting_citations":[],"review_version":1}