{"id":"fb29c59c-296c-4dae-a1ae-f7a55d0274f3","arxiv_id":"2411.13045","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An explainable chain-of-thought LLM for relevance is distilled into small online models via score and reasoning supervision, improving e-commerce search ad relevance on Taobao.","lead":"This paper describes a framework in which a large language model teaches smaller e-commerce search models to judge whether a product matches a query, by first explaining its reasoning in steps. The authors report that the method improved click-through rate and relevance ratings in a live test on Taobao's search advertising system.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label-aligned CoT filtering guarantees label agreement, not reasoning validity; absent a faithfulness check, the CoT distillation mechanism and 'explainable' claim are unsupported.","rationale":"I agree with the reader's weakest assumption. The central empirical claim (MKD improves students by transferring reasoning knowledge) requires that the CoT used as teacher supervision actually encodes the evidence driving relevance decisions. The pipeline filters only on final-label agreement, which is a label-consistency filter, not a faithfulness guarantee. I considered whether the confounded use of 30M unlabeled examples in Table 2 is a more severe threat; however, the ablation 'w/o score&CoT' in Table 4 shows that some gain remains after removing score and CoT distillation, so the incremental contribution of the distillation dimensions is separable from the data-size effect. That makes CoT faithfulness the sharper unresolved issue. The sign error in the attention-regulation Lcot is real but likely a typo; it does not change the need for a faithfulness check. Therefore I would keep the CONDITIONAL verdict, adding a requirement for a CoT faithfulness evaluation.","tokens_in":17930,"tokens_out":6481,"duration_ms":67099,"concrete_test":"Take a random sample of 500 query-item pairs from the test set, have ELLM-rele generate full CoT outputs, and ask independent human annotators to (1) verify each extracted attribute and match/mismatch tag against the query and title, and (2) determine the final Good/Bad judgment from the CoT alone without seeing ELLM-rele's final judgment token. Measure attribute-level precision/recall and CoT-only judgment agreement with human labels. If CoT-only judgment accuracy or attribute precision falls below a pre-specified threshold (e.g., 90%), the token-level distillation target is not faithful and the CoT distillation mechanism is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.1 constructs CoT annotations by prompting Qwen2-72B and Llama3-70B with self-consistency, then selecting the sampled reasoning path whose final judgment matches the human label. This guarantees label agreement, not reasoning validity. The selected CoT is then used (a) to SFT the 7B ELLM-rele (Eq. 3) and (b) as supervisory signal in Section 3.3.2, where parsed evidence spans become BIO tags for interaction students and ±1 attention-regulatory factors for representation students. If the LLM's stated attributes are post-hoc justifications rather than the actual cause of its judgment, the student receives systematically mislabeled token-level supervision. The paper reports no human evaluation of CoT, no inter-annotator agreement, and no faithfulness metric (e.g., counterfactual or attribution tests). The ablation gains attributed to CoT distillation (Table 4, +0.4–0.5 ROC-AUC) could then reflect regularization or spurious correlations rather than transferred reasoning knowledge. Because the paper's stated novelty is explainable CoT knowledge and its reuse, this missing check is load-bearing for both the mechanism and the 'explainable' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ELLM-MKD, a framework that first builds an explainable LLM teacher (ELLM-rele) for e-commerce query-item relevance by recasting relevance judgment as chain-of-thought (CoT) reasoning over fine-grained aspects, and then distills this teacher into small deployable students (BERT, Sentence-BERT, PolyEncoder, ColBERT) through two complementary objectives: relevance score distribution distillation and token-level CoT knowledge distillation. Offline experiments report consistent ROC-AUC/PR-AUC gains over the baseline students, an ablation study attributes gains to the score and CoT components, and a two-week online A/B test shows CTR and Goodrate improvements, especially on long-tail queries.","tokens_in":18222,"tokens_out":5077,"duration_ms":48463,"significance":"The paper addresses a practically important problem—transferring LLM reasoning knowledge into efficient e-commerce relevance models—and the idea of distilling not just scores but also explicit CoT-derived token-level supervision is interesting and plausibly useful. Credit is due for reporting consistent gains across four student architectures, for the online deployment results, and for a fairly detailed ablation that attempts to separate the contributions of score and CoT distillation. The industrial-scale setting (Taobao search ads) gives the empirical claims real-world grounding. However, the headline comparison is confounded by the use of 30M unlabeled pseudo-labeled pairs only in the MKD students, the core KL distillation objective is not written as a valid mathematical expression, and the 'explainable' claim rests on CoT annotations that are filtered for label agreement but not validated for reasoning faithfulness. These issues are fixable but require additional experiments and corrections before the central claims can be accepted.","major_comments":[{"comment":"The main comparison in Table 2 is confounded: MKD students are trained with an additional 30 million unlabeled query-item pairs pseudo-labeled by the teacher, while the baseline students are trained only on the human-labeled set. The ablation in Table 4 shows that removing all unlabeled data ('w/o MKD') costs -2.8/-2.2 ROC-AUC for MKDColBERT/MKDBERT, which is larger than the gains attributed to score distillation (+0.9/+0.6) and CoT distillation (+0.5/+0.4). Thus the Table 2 differences over the baselines do not isolate the proposed distillation mechanism; they largely reflect the benefit of the extra training data. To support the claim that the multi-dimensional distillation itself improves students, please report a baseline trained on the same pseudo-labeled 30M pairs with hard labels only (e.g., the 'w/o score&CoT' variant) in Table 2, and report gains relative to that baseline. In addition, since the CoT and score contributions are small (0.4-0.9 ROC-AUC), please provide error bars over multiple runs or significance tests.","section":"Section 4.4, Table 2 vs Section 4.6, Table 4"},{"comment":"The KL divergence objective is not mathematically defined as written: s_T(Q_i,T_i) is a scalar in [0,1] after the projection in Eq. (4), while p_S(y|Q_i,T_i) is a probability distribution over the tokens Good/Bad. KL divergence requires two distributions over the same space. Please rewrite the objective as, e.g., KL([s_T, 1-s_T] || [p_S(Good), p_S(Bad)]) or explain an alternative construction. Additionally, Eq. (4) exponentiates probabilities rather than logits; this is an unusual choice that compresses the probability mass and should be justified or replaced with a more standard softmax over the Good/Bad logits.","section":"Section 3.3.1, Eq. (5)"},{"comment":"The CoT annotations are constructed by sampling multiple reasoning paths from larger LLMs and then selecting the path whose final judgment matches the human label. This guarantees label agreement but not the validity of the intermediate reasoning; the selected rationales could be post-hoc justifications that do not reflect the true basis of the relevance decision. Because the paper's 'explainable' claim and the CoT distillation mechanism (sequence tagging and attention regulation) assume these rationales are faithful evidence, the absence of any faithfulness check is load-bearing. The ablation shows CoT distillation adds only +0.4-0.5 ROC-AUC, so without a faithfulness study it is unclear whether the CoT signal transfers reasoning knowledge or merely acts as a regularizer. Please provide at least a human evaluation of CoT quality on a sample, an inter-annotator agreement measure, or a counterfactual/attribution test.","section":"Section 3.2.1 and Section 3.3.2"},{"comment":"The two CoT distillation losses are not stated consistently with the surrounding text. Eq. (6) is written as a sum of token-level log-probabilities, which is a cross-entropy objective, yet the text says a CRF layer is used; a CRF would include transition scores and a sequence-level partition function. Eq. (7) contains a leading minus sign and a normalization that do not correspond to a standard L2 loss (the L2 distance should be positive). Please correct these equations or clarify the notation so the reported objectives are reproducible.","section":"Section 3.3.2, Eq. (6)-(7)"}],"minor_comments":[{"comment":"There are several typos and inconsistent terms, e.g., 'aquire' (Abstract and Section 1), 'Warmart-LLM' vs 'Walmart-LLM' (Section 4.2.1 and Table 2), 'tempreaures' (Section 4.3.2), and 'trasnfers' (Section 2.2). A careful proofread is needed.","section":"Throughout"},{"comment":"The baseline list contains a numbering error: '3)' appears twice (for Poly-Encoder and ColBERT), so the enumeration is confusing and should be corrected.","section":"Section 4.2.1"},{"comment":"The row labeled 'Unlabeled' reports 'Good' and 'Bad' counts that are based on pseudo labels, but the column names are identical to those in the labeled splits; please make explicit that these are pseudo-label counts, and clarify what 'CoT term' counts mean for the unlabeled data.","section":"Table 1"},{"comment":"The hyperparameters λ1, λ2, λ3 and the KL temperature are given without sensitivity analysis, and the online A/B test reports only point estimates (+0.17% CTR, +0.89% Goodrate, +1.96% long-tail) with no confidence intervals or significance levels; please add at least a brief sensitivity discussion and, if available, statistical significance for the online metrics.","section":"Section 4.3.2 and Section 4.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industrial case study from Taobao with a practical distillation framework. The main technical claims are plausible but the current manuscript overstates the contribution of the distillation objectives because the comparison against baselines does not control for the 30M pseudo-labeled unlabeled data. The undefined KL divergence in Eq. (5) is a formal error that must be corrected. The missing faithfulness study for the CoT annotations is particularly important given the 'explainable' framing; however, this is addressable within the scope of a revision. I would not recommend rejection, but the paper needs another round of experiments and careful rewriting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about LLM distillation for e-commerce search. The core idea is sensible: decompose relevance into aspects (category, product, gender, etc.), have a large LLM generate a structured chain-of-thought plus a relevance score, then distill both the score distribution and the parsed CoT into small student models—BIO tagging for interaction-based BERT, attention-score regulation for representation-based models. That combination is genuinely new enough to count as a contribution, even though each ingredient exists in prior work. The paper does real engineering: consistent gains across four student architectures, ablations that decompose the contributions, an online A/B test, and the system is actually deployed on Taobao traffic. Inference time is unchanged; only training time grows. Good credit for that.\n\nThe soft spots are real, though not fatal. The headline comparison is confounded: the MKD students are trained with 30 million pseudo-labeled unlabeled pairs, while the baselines see only the human-labeled set. The ablation 'w/o MKD' removes the unlabeled data, so the +2.2–2.8 ROC-AUC gain attributed to the framework is partly just having 30M extra training pairs. A proper control would train baselines on the same pseudo-labeled data, or at least hold data volume fixed. Second, the stress-test note on CoT faithfulness is fair: Section 3.2.1 filters generated rationales by whether the final judgment matches the human label. That guarantees label agreement, not that the stated reasoning caused the judgment. If the rationales are post-hoc, the token-level tags and attention targets teach the student to mimic phrasing artifacts rather than true relevance reasoning. The paper gives no human evaluation of the CoT, no agreement statistics, no faithfulness test. The ablation gain from CoT distillation (+0.4–0.5) could be regularization. That matters because the paper's 'explainable' claim is load-bearing. Third, no error bars or significance tests anywhere; the 2–4 point gaps may be meaningful at this scale, but we can't judge. Fourth, the attention-regulation loss in Section 3.3.2 has a leading negative sign; as written it would push student attention away from the target. Likely a transcription typo, but it should be fixed.\n\nThe circularity concern from the reader is overblown: teacher supervised by human labels, student distilled from teacher, is standard. No equation-level circularity.\n\nWho is this for? Practitioners building industrial relevance models who want a concrete distillation recipe; also anyone interested in what 'explainable' LLM distillation actually delivers. It deserves a serious referee, but the referee should demand a fair baseline comparison and at least a minimal faithfulness check before accepting the interpretability claims.","headline":"A real, deployed industrial recipe for distilling LLM relevance knowledge into small rankers, but the headline gains are confounded by extra unlabeled data and the 'explainable' claim rests on an unevaluated CoT.","tokens_in":18699,"tokens_out":2161,"would_cite":false,"duration_ms":23912,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distilling an LLM's step-by-step relevance reasoning into small online models improves e-commerce search matching, with the largest gains on long-tail queries.","keywords":["e-commerce relevance","knowledge distillation","chain-of-thought","large language models","semantic matching","long-tail generalization"],"falsifier":"An experiment that swaps or scrambles the relevant/irrelevant span tags in the CoT distillation while keeping the final label fixed would distinguish faithful reasoning from superficial alignment: if the student's accuracy on long-tail queries stays unchanged, the reasoning trace is not the carrier of the distilled knowledge. Similarly, a human-annotation study rating whether the generated CoT steps actually reflect the attributes that determined the label would test the premise directly.","tokens_in":17737,"feed_emoji":"🛒","tokens_out":5558,"duration_ms":47245,"temperature":0.7,"pith_summary":"This paper claims that a large language model's relevance judgments can be made both explainable and transferable by treating the judgment as chain-of-thought reasoning over fine-grained product aspects. The authors build an explainable teacher LLM (ELLM-rele) that outputs step-by-step attribute matching before its Good/Bad verdict, then distill two kinds of knowledge into small online models: the calibrated probability of the verdict and the token-level reasoning trace. The distilled students keep their original architectures and inference speed while gaining relevance accuracy, especially on long-tail queries. Offline gains are 2.2 to 4.1 ROC-AUC points over four baselines; an online A/B test shows +0.17% CTR and +0.89% Goodrate. If true, this is a practical route to putting LLM-grade semantic understanding into deployable search systems.","feed_headline":"Distilling an LLM's chain-of-thought lifts e-commerce relevance models","feed_subtitle":"Score and reasoning knowledge transfer; up to +4.1 ROC-AUC offline, +0.89% online Goodrate.","key_machinery":"The load-bearing components are (1) ELLM-rele, a 7B LLM fine-tuned on CoT annotations generated by two larger models (72B and 70B) with self-consistency and filtered for label agreement, which produces structured reasoning of the form 'attribute in query; attribute in title; match/mismatch; final judgment'; and (2) the MKD head, which converts the teacher's Good/Bad token probabilities into a normalized relevance score and trains students with a Kullback-Leibler divergence objective, while simultaneously transferring the CoT via sequence tagging (CRF) for interaction models or attention-score regulation ($L_2$ distance on cosine attention) for representation models. The CoT trace is the key mechanism: it turns the teacher's latent matching process into explicit token-level supervision that tells the student which spans are relevant or irrelevant and how they align.","core_discovery":"The central claim is that decomposing e-commerce relevance into a chain of aspect-level match checks (category, product, brand, gender, and so on) and distilling both the resulting score distribution and the reasoning trace into existing student models improves relevance learning in both interaction-based and representation-based architectures. The teacher ELLM-rele outperforms a plain fine-tuned 7B LLM by 0.8% ROC-AUC, showing the CoT formulation itself adds accuracy, not just interpretability. The student models trained with the multi-dimensional distillation outperform their respective baselines by 2.2–4.1% ROC-AUC and 2.5–4.6% Neg PR-AUC, and the long-tail subset shows a larger gap, indicating improved generalization to low-frequency queries.","pith_inferences":["If the CoT traces are faithful, attention-score regulation may effectively encode hard aspect constraints (e.g., gender mismatch) into token-level attention, which could generalize to other structured matching tasks.","The paper does not measure whether the CoT rationales are causally responsible for the LLM's verdict; if they are post-hoc, the student may learn artifacts of phrasing. A controlled test would be to train students on rationales that are shuffled or generated after seeing the label and compare.","The multi-dimensional distillation may be portable to other LLM-as-teacher settings (e.g., question answering or retrieval) where decomposable, aspect-based reasoning is available.","Using two different large teacher models for annotation may provide a diversity benefit; a testable extension is whether single-teacher annotation underperforms the ensemble."],"forward_implications":["Deployed student models gain relevance accuracy without added inference latency or parameter growth, since distillation is applied only during training.","Long-tail queries benefit more than head queries, narrowing the frequency-driven performance gap that usually hurts low-data relevance models.","Both interaction-based and representation-based architectures can absorb the same teacher knowledge through two different CoT distillation losses.","The same CoT formulation can be reused for relevance annotation assistance and error analysis, as the paper notes.","The teacher's score probabilities carry fine-grained relevance information beyond hard labels, and distilling them adds accuracy over hard-label distillation."],"supporting_citations":[{"why":"Introduces chain-of-thought prompting, the reasoning format that the paper adapts to relevance judgment.","marker":"[53]"},{"why":"Provides the self-consistency strategy used to increase the accuracy of the CoT annotations generated by the larger teachers.","marker":"[50]"},{"why":"The strongest LLM-based baseline that directly generates relevance judgments, which ELLM-rele extends with CoT reasoning.","marker":"[27]"},{"why":"Specifies the Qwen2 model family, whose 7B variant is the backbone of ELLM-rele.","marker":"[54]"},{"why":"One of the two larger LLMs (70B class) used to produce the CoT annotations for supervised fine-tuning.","marker":"[8]"},{"why":"The interaction-based student architecture (BERT) that receives score and CoT distillation.","marker":"[7]"},{"why":"The representation-based student architecture (ColBERT) that receives attention-score regulation and is also the deployed online baseline.","marker":"[20]"},{"why":"Provides the CRF layer used for sequence-tagging-based CoT distillation in interaction models.","marker":"[22]"},{"why":"The prior knowledge-distillation baseline (ReprBERT) that the MKD results are compared against.","marker":"[55]"}],"fun_headline_variants":["CoT distillation turns LLM insights into better e-commerce relevance","Distill LLM reasoning for interpretable relevance gains","Explainable LLM distillation improves search relevance models","LLM's chain-of-thought distilled to boost relevance learning","Multi-dimensional distillation from explainable LLM lifts relevance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the chain-of-thought rationales generated by the larger LLMs — after discarding those that disagree with the human label — are genuinely the reasoning behind the relevance judgment, and not plausible post-hoc justifications.","fun_headline_variants_meta":{"raw":{"variants":["CoT distillation turns LLM insights into better e-commerce relevance","Distill LLM reasoning for interpretable relevance gains","Explainable LLM distillation improves search relevance models","LLM's chain-of-thought distilled to boost relevance learning","Multi-dimensional distillation from explainable LLM lifts relevance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1651,"prompt_tokens":1008,"completion_tokens":643,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":624,"tokens_out":643,"duration_ms":6976,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:53:48.524955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that swaps or scrambles the relevant/irrelevant span tags in the CoT distillation while keeping the final label fixed would distinguish faithful reasoning from superficial alignment: if the student's accuracy on long-tail queries stays unchanged, the reasoning trace is not the carrier of the distilled knowledge. Similarly, a human-annotation study rating whether the generated CoT steps actually reflect the attributes that determined the label would test the premise directly.","supporting_citations":[{"cited_title":"arXiv e-prints (2016)","cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting, the reasoning format that the paper adapts to relevance judgment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Specifies the Qwen2 model family, whose 7B variant is the backbone of ELLM-rele."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two larger LLMs (70B class) used to produce the CoT annotations for supervised fine-tuning."}],"review_version":1}