{"id":"d3a36576-bfd8-464a-aa07-1e2e7b6b2813","arxiv_id":"2505.20966","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A personalized query auto-completion model with a learned [Reject] token improves both relevance and adaptive toxicity filtering, and is deployed at Kuaishou search.","lead":"Researchers at Kuaishou built a query auto-completion model that combines a user's long-term and short-term search interests and adds a learned 'reject' token that stops toxic suggestions from being shown. In online tests it lifted click-through rate by 4.08% and has been deployed to Kuaishou's main search traffic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The KSQAC detoxification metrics may be computed by the same Detoxification Expert that creates the RPO training signal, making the offline detoxification gains a possible self-measure; the paper never identifies the evaluation scorer.","rationale":"The strongest claim has two parts: offline state-of-the-art detoxification on KSQAC and online A/B gains. The online component is not affected by the scorer concern, and the AOL experiments with an external toxicity tool provide independent support, so a rejection is not warranted. The load-bearing weak point is the identity of the KSQAC toxicity scorer: the paper defines the Detoxification Expert as the model that creates the training signal but never states which model evaluates the reported UAmaxT/UProb metrics. If the same expert is used, the offline improvements could reflect overfitting to that expert's biases rather than genuine detoxification. The reader identified exactly this assumption, and I agree that it is the most important unresolved issue. The concrete test is feasible because the code is released: inspect the evaluation harness and, if needed, rerun with an independent toxicity classifier. The current conditional verdict is appropriate, and the proposed check would either confirm the conditional status or reveal a need to downgrade the offline claim.","tokens_in":19088,"tokens_out":5287,"duration_ms":56596,"concrete_test":"Check the released repository (https://github.com/JXZe/LaD) for the KSQAC evaluation script and identify the exact scorer used to compute UAmaxT/UProb and to construct the Toxic Test Set. If it is the Detoxification Expert or a model trained from it, re-run the toxicity metrics from Tables 2 and 3 using an independent classifier, such as Detoxify (already used for the AOL experiments in Appendix A.1) or a separately trained human-annotated toxicity model, while keeping the LaD checkpoint fixed. Then compare LaD against DAPT, Quark, PPO, and DAC. If LaD's UAmaxT/UProb advantage persists under the independent scorer, the self-measure concern is refuted; if it shrinks or reverses, the offline adaptive-detoxification claim needs re-benchmarking before the paper can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 describes a 48-layer Detoxification Expert that ranks generated completions and injects [Reject] during RPO training, while Section 4.1 reports UAmaxT/UProb for KSQAC without saying which model scores the generations. The only toxicity model named for the industrial dataset is that same expert. If the KSQAC evaluation uses it, and if the Toxic Test Set in Section 4.2 is also extracted with it, then the offline claim that LaD achieves the best detoxification metrics is partly a self-measure: the model is directly trained to satisfy the scorer and then judged by it. This is a reward-hacking/circularity risk, not a mere disagreement with consensus. It does not undermine the online A/B gain, since CTR and PV are independent of the toxicity scorer, nor the AOL appendix, which uses the external Detoxify tool. However, the core industrial detoxification claim rests on the unresolved scorer identity. This is compounded by ablations in Section 4.3 that select SL-37 and the rejection threshold on the test set without a validation split, so a confounded metric could also steer configuration choices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LaD, a generative query auto-completion (QAC) model deployed on Kuaishou search. It represents user interests hierarchically: long-term interests are encoded by a transformer encoder into coarse sentence-level vectors, while short-term interests are included as raw token sequences. These representations are fed, along with the typed prefix, into a generative language model. For detoxification, the paper introduces Reject Preference Optimization (RPO): a 48-layer Detoxification Expert scores sampled completions, a special [Reject] token is inserted below a learned threshold during training, and at inference any completion ranked below [Reject] is discarded. Experiments on a new industrial KSQAC dataset and the public AOL dataset report generation and detoxification metrics, and a 15-day online A/B test shows a 4.08% CTR increase over the production baseline. The paper claims this is the largest single-experiment metric improvement in nearly two years and that LaD has been deployed to primary traffic.","tokens_in":19331,"tokens_out":7000,"duration_ms":76313,"significance":"If the claims hold, LaD is a substantial industrial contribution: it demonstrates that a fully generative QAC model with hierarchical personalization can serve live traffic at 20 ms latency, and that a [Reject] token can implement adaptive refusal without a separate post-filtering model. The online A/B results and the AOL appendix, which uses the external Detoxify toxicity classifier, provide evidence that is independent of the in-house training signal. The code release is a further strength. The main risks are that the KSQAC detoxification metrics may be computed by the same Detoxification Expert used as the training signal, that configuration choices appear to be made on the test set, and that no error bars or significance tests are reported. These issues do not undermine the online business metrics or the AOL experiments, but they do affect the strength of the central offline detoxification claim.","major_comments":[{"comment":"The KSQAC toxicity evaluation is not specified as independent from the training signal. Section 3.2 introduces a 48-layer Detoxification Expert that scores generated completions and injects [Reject] during RPO training, while Section 4.1 reports UAmaxT and UProb for KSQAC without stating which model computes those toxicity scores. If the same Detoxification Expert is used, the large detoxification gains in Tables 2 and 3 are partly a self-measure, because the model is directly trained to satisfy that scorer. Please state explicitly what model scored the KSQAC generations for evaluation and, if it is the same expert, re-benchmark with an independently trained or public toxicity classifier (for example, the Detoxify tool used in the AOL appendix). Also specify how the Toxic Test Set in Section 4.2 was constructed; if it was filtered by the same expert, the evaluation subset is confounded as well.","section":"§3.2, §4.1, Tables 2-3"},{"comment":"The choice of L=7 and S=3 appears to be made on the KSQAC test set. Table 4 reports R@4, BLEU, and MRR for different S/L configurations on the test split, and SL-37 is then selected as 'optimal' from this table; no validation split is described. This test-set-based model selection can overstate the advantage of SL-37 over the alternatives and of LaD over baselines. Please add a validation split for configuration selection and report the selected configuration's test performance, or at least disclose that the reported test numbers are post-selection and provide standard deviations across multiple runs. Similarly, Section 4.1 states that epsilon=0.6 is calibrated to the Detoxification Expert's precision/recall; please clarify whether this calibration was performed on a held-out set and whether it influenced the reported test metrics.","section":"§4.3, Table 4"},{"comment":"The generation metrics on the Toxic Test Set are difficult to interpret across models with different rejection rates. R@4 and MRR are computed on the completions that survive after [Reject] filtering, so LaD's drop to 4.18% R@4, compared with 13.58% for LaD w/o AD, largely reflects refusal behavior rather than ranking quality. This is expected for an adaptive detoxification system, but the paper should either report recall/precision at a fixed output count for all models or explicitly state that these generation metrics are descriptive of the filtered output and are not comparable quality measures.","section":"§4.2, Table 3"}],"minor_comments":[{"comment":"There are several typographical errors, including 'futher' in the Abstract and Section 3.2, 'Tabel 7' in Section 4.5, 'imcomplete' in Section 1, 'inplemented' in the Appendix, and 'the length of long-term interests S is 3' in Section 4.1, which should read 'short-term'. These should be corrected.","section":"Throughout"},{"comment":"Please define N_gi and N_G precisely; as written, 'length of generations' is ambiguous (number of surviving completions versus total tokens). Also clarify how the ratio N_G / (1/N \\sum N_gi) penalizes fewer generations, and specify how the edge case of zero surviving generations is handled.","section":"§4.1, Eq. (12)"},{"comment":"The 'Copy' operation for short-term interests is the identity, so short-term interests are not learned representations. The paper should state this explicitly and explain why raw token sequences are preferred over learned encodings for short-term interests.","section":"§3.1, Eq. (2)"},{"comment":"The online A/B test reports relative improvements without confidence intervals or significance levels; some values are marked as 'not significant' in gray but the testing procedure is not described. Please report the statistical testing method and confidence intervals for all metrics in Table 7.","section":"§4.5, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's deployment claims and the 'largest single-experiment improvement in nearly two years' statement are not independently verifiable, but they are not scientific claims that require rejection. The key technical risk is scorer identity: if the KSQAC detoxification metrics are computed by the same Detoxification Expert used in training, the offline detoxification conclusions should be substantially softened. The AOL results with an external scorer and the online A/B test provide important mitigating evidence. The paper fits the applied data science scope of KDD, but the authors should be required to resolve the scorer-identity question before the offline detoxification claims are accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: LaD is a serious industrial QAC paper with one genuinely novel piece — the [Reject] token trained via RPO as an absolute rejection threshold during online training — and it reports a credible 4.08% CTR gain in a large-scale A/B test. But the paper's central offline detoxification claim on the KSQAC dataset is undercut by a possible self-measure: the scorer used to compute UAmaxT/UProb is never named, and the only toxicity model described for that dataset is the same Detoxification Expert that provides the RPO training signal. That needs to be fixed before the offline results can be taken at face value.\n\nWhat's new and good: the RPO objective is a real departure from DPO. Instead of requiring the chosen completion to be relatively better than a reference, the model learns an absolute threshold — the [Reject] token — that separates acceptable from unacceptable generations, and the token is injected during training and used at inference to filter below-threshold outputs. That's a clean and useful idea for safe generation in latency-sensitive settings. The hierarchical interest encoding (coarse long-term vectors, token-level short-term copy) is not revolutionary, but it's sensible and the 20ms online latency shows the engineering is solid. The AOL experiments use the external Detoxify tool, giving some independent evidence.\n\nThe soft spots are real but not fatal. The missing evaluation scorer is the big one — the paper needs to state which model computes UAmaxT/UProb on KSQAC and on the Toxic Test Set. If it's the Detoxification Expert, then the offline detoxification numbers are partly a measure of how well the model satisfies its own reward, and the comparison with DPO/Quark becomes less clean. The ablations in Section 4.3 also select L, S, and the rejection threshold on the test set, with no validation split and no error bars; that's a secondary but genuine methodological weakness. The online A/B test and the human evaluation are the strongest evidence, and they are not affected by the scorer issue.\n\nWho this is for: anyone working on query auto-completion, safe generation, or alignment with rejection mechanisms. It deserves a serious referee: the method is interesting, the deployment is real, and the issues are fixable with clearer reporting rather than fundamental flaws. I'd recommend accepting it for review with a request for the scorer identity and a held-out validation configuration.\n\nBest,\n[Your name]","headline":"Solid industrial QAC paper with a genuinely new [Reject]-token mechanism, but the offline detoxification numbers on KSQAC may be self-measured because the evaluation scorer is never identified.","tokens_in":19852,"tokens_out":2848,"would_cite":true,"duration_ms":29621,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LaD claims to cut toxic query completions to a 6.55% probability with comparable generation quality, and in an online A/B test it lifted CTR by 4.08%.","keywords":["personalized query auto-completion","detoxification","long-term and short-term interests","hierarchical user representation","Reject Preference Optimization","online generation","industrial deployment","KSQAC dataset"],"falsifier":"Re-run the KSQAC detoxification evaluation with an independent toxicity classifier, for example a publicly available toxic-comment model of the kind the paper's own public-dataset appendix uses, instead of the in-house Detoxification Expert; if the UAmaxT and UProb gaps between LaD and the stronger baselines shrink or disappear, the adaptive-detoxification claim would be a self-measure artifact.","tokens_in":18879,"feed_emoji":"🔍","tokens_out":10964,"duration_ms":92854,"temperature":0.7,"pith_summary":"Query auto-completion (QAC) must do two things that pull against each other: personalize suggestions to a specific user and refuse to surface toxic ones. This paper proposes LaD, a generative model that represents a user's long-term interests as coarse, cached sentence-level vectors and their short-term interests as fine-grained token-level behavior, then combines both with the typed prefix in a generation model. To handle toxicity, LaD trains with Reject Preference Optimization: a Detoxification Expert ranks candidate completions during training, a special [Reject] token is inserted below the safety threshold, and at inference anything ranked below [Reject] is hidden from the user. On an industrial-scale Chinese search dataset and in online A/B tests, the authors report that LaD cuts the probability of a toxic completion to 6.55% while keeping generation quality close to the no-detoxification model, and that the production deployment improved click-through rate by 4.08%.","feed_headline":"Toxic query suggestions fall 66% in a model that learns to refuse","feed_subtitle":"Deployed search combines long and short-term interests with a [Reject] token, lifting CTR by 4.08%.","key_machinery":"The load-bearing machinery is a pair of training-time mechanisms. A Long-term interests Transformer Encoder (LTE) compresses each of the user's past queries into a single embedding vector, so long-term interests enter the sequence as coarse tokens; short-term interests are copied verbatim as token sequences, preserving spelling and word boundaries. These are prefixed to the query prefix and fed to a Generative Language Model (GLM), which is trained with the standard cross-entropy generation loss. The second mechanism is Reject Preference Optimization (RPO): a Detoxification Expert—a 48-layer discriminative transformer trained on hundreds of millions of search logs plus tens of thousands of manual annotations—scores each generated completion; when a score falls below a threshold epsilon of 0.6, the special token [Reject] is inserted into the ranked list during training. The RPO loss pushes the probability of [Reject] above that of any rejected toxic completion and below the accepted ones, so at inference the model emits [Reject] itself and everything ranked below it is discarded. The online serving pipeline uses a daily-updated Memory Bank for long-term representations and a real-time module for recent queries, keeping latency at 20 ms, under the 30 ms budget.","core_discovery":"LaD's central claim is that the two failures of current QAC systems—flat, single-vector user representations and toxic generation—can be attacked in one architecture. The paper claims that a hierarchical interest representation, with a Long-term interests Transformer Encoder (LTE) compressing seven past queries into sentence-level embeddings and three recent queries kept at token level for direct copying, preserves enough personalization for online inference while outperforming single-representation baselines on recall, BLEU, and MRR. On detoxification, the paper claims that Reject Preference Optimization (RPO) makes the model adaptive: for a benign prefix (even one with typos like 'fruit strwab') it generates relevant clean completions, while for a sensitive prefix it places the [Reject] token first and returns nothing. The paper reports that LaD achieves the best UAmaxT and UProb scores among detoxified models on the KSQAC test set and on a public search-log test set, and that the full model deployed on the authors' production search system raised CTR by 4.08% over the production baseline—the largest single-experiment gain in nearly two years.","pith_inferences":["The [Reject]-token trick likely generalizes beyond QAC to any short-form generation task with a safety constraint, such as chat suggestions or headline generation, whenever a reliable quality judge is available at training time.","A cleaner test of RPO would separate the expert that ranks completions during training from the evaluator that scores test outputs; the paper does not state that separation, so re-benchmarking with an independent toxicity judge is a natural next step.","The reported 4.08% CTR gain combines hierarchical personalization and adaptive detoxification; an online ablation that switches off only RPO would reveal which component drives the uplift.","Cold-start users could be served by seeding the daily-updated Memory Bank with aggregate or demographic priors, since the current design depends on an individual's historical long-term representations."],"forward_implications":["If RPO works as described, generative systems can learn a default 'say nothing' response for unsafe inputs without an extra filter, cutting latency and maintenance cost.","The hierarchical representation—coarse cached long-term vectors plus fine-grained short-term tokens—offers a template for balancing personalization and online latency in other sequence-generation tasks.","Because the model's generations are used as a new recall source, it can surface completions that no index-based candidate pool contains.","The unbiased toxicity metrics UAmaxT and UProb, which penalize adaptive refusal that simply generates fewer queries, give a fairer way to evaluate detoxification in QAC."],"supporting_citations":[{"why":"Supplies the most-popular-completion baseline that LaD must outperform on recall and toxicity.","marker":"[3]"},{"why":"Represents the single-vector user-behavior personalization approach that LaD's hierarchical encoding is designed to beat.","marker":"[2]"},{"why":"Provides a trie-context augmentation baseline that adds user behavior to prefixes for personalized QAC.","marker":"[31]"},{"why":"Gives the multi-view multi-task personalized QAC method used as the nearline production baseline.","marker":"[45]"},{"why":"Defines the base BERT/BART generative query-suggestion architecture that LaD extends.","marker":"[32]"},{"why":"Supplies the domain-adaptive pretraining baseline that detoxifies by training on clean text.","marker":"[15]"},{"why":"Contributes the Quark reinforced-unlearning baseline for controllable generation.","marker":"[27]"},{"why":"Provides the PPO-based preference fine-tuning baseline for detoxification.","marker":"[47]"},{"why":"Offers DAC, the recent RL detoxification QAC baseline whose UAmaxT and UProb scores LaD claims to beat.","marker":"[28]"},{"why":"Is the source of the average-max-toxicity and toxicity-probability metrics that LaD adapts into UAmaxT and UProb.","marker":"[12]"}],"fun_headline_variants":["Hierarchical interests + adaptive detox: auto-completion with [Reject] token","Safer search: LaD models long and short-term interests, refuses toxic completions","Auto-completion learns to refuse toxic prefixes, boosting CTR by 4.08% on deployed search","[Reject] token enables adaptive detox in auto-completion; deployed with 4.08% CTR gain","Personalized auto-completion: hierarchical interests and a refuse switch for safer search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The detoxification claims rest on trusting the Detoxification Expert's own toxicity scores during training, and on the test-time toxicity metrics (UAmaxT, UProb) being computed by an independent judge rather than by that same expert.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical interests + adaptive detox: auto-completion with [Reject] token","Safer search: LaD models long and short-term interests, refuses toxic completions","Auto-completion learns to refuse toxic prefixes, boosting CTR by 4.08% on deployed search","[Reject] token enables adaptive detox in auto-completion; deployed with 4.08% CTR gain","Personalized auto-completion: hierarchical interests and a refuse switch for safer search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2780,"prompt_tokens":1100,"completion_tokens":1680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":1565}},"tokens_in":716,"tokens_out":1680,"duration_ms":15463,"temperature":1.0,"reasoning_tokens":1565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:42:40.420070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the KSQAC detoxification evaluation with an independent toxicity classifier, for example a publicly available toxic-comment model of the kind the paper's own public-dataset appendix uses, instead of the in-house Detoxification Expert; if the UAmaxT and UProb gaps between LaD and the stronger baselines shrink or disappear, the adaptive-detoxification claim would be a self-measure artifact.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the most-popular-completion baseline that LaD must outperform on recall and toxicity."},{"cited_title":"Search Intenion Network for Personalized Query Auto-Completion in E-Commerce","cited_arxiv_id":"2403.02609","evidence_quote":"Represents the single-vector user-behavior personalization approach that LaD's hierarchical encoding is designed to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a trie-context augmentation baseline that adds user behavior to prefixes for personalized QAC."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the base BERT/BART generative query-suggestion architecture that LaD extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the domain-adaptive pretraining baseline that detoxifies by training on clean text."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the Quark reinforced-unlearning baseline for controllable generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers DAC, the recent RL detoxification QAC baseline whose UAmaxT and UProb scores LaD claims to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the source of the average-max-toxicity and toxicity-probability metrics that LaD adapts into UAmaxT and UProb."}],"review_version":1}