{"id":"23e4194a-5169-459b-bbe4-fe7cfbe57845","arxiv_id":"2501.18216","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DRP disentangles preference and relevance effects in e-commerce search via orthogonal representation editing plus dual-level adaptive fusion, improving click prediction without human-labeled relevance data.","lead":"This paper introduces DRP, a framework for e-commerce search that separates user preference from query-item relevance using a learned low-rank editing of neural representations, then adaptively fuses the two signals. The authors report consistent ranking improvements across two public datasets and one private e-commerce dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Preference editing's causal claim is unverified: O is trained only by behavior loss (Eq. 11), and the ablation shows adaptive fusion, not editing, drives nearly all gains.","rationale":"Good-faith reading: the paper is an applied method paper with extensive experiments across two public and one private dataset, meaningful ablations, and a plausible mechanism. The reader's conditional verdict is appropriate. The most load-bearing concern is exactly the bridge from the causal graph to the learned O: the paper asserts, via Geiger et al. [13], that an orthogonal low-rank space can be learned that matches the intervention space, but the method provides no supervision tying O to R→P. Because the only objective is behavior prediction, the model could learn a projection that merely improves prediction without disentangling causes. This is not an internal inconsistency—the equations are coherent—but it means the causal claim is an unsupported interpretation. The ablation data strengthen the concern: removing preference editing leaves performance almost unchanged, while removing adaptive fusion collapses performance toward baseline. So the headline empirical contribution is robust, but the 'untainted preference' framing should either be validated directly or softened to a learned representation adjustment. A synthetic causal-recovery test would settle whether O actually captures R→P.","tokens_in":18740,"tokens_out":5282,"duration_ms":55649,"concrete_test":"Build a semi-synthetic dataset with known ground truth: draw exogenous user preference U, query-item relevance R, set P = aR + U, and generate clicks B ~ Bernoulli(sigmoid(bR + cP + noise)). Train DRP on behavior B with the same architecture and loss as the paper. Then check (i) whether the learned O's row space matches the known direction of R→P (e.g., cosine similarity between O^T O and the linear map from R to P), and (ii) whether the edited representation e_pc predicts U better than the unedited e_p. If DRP attains high click AUC but fails these causal checks, the disentanglement interpretation is unsupported even though the method predicts well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Equation (6) removes the relevance effect from preferences, yielding 'untainted' preference predictions. For this to hold, the learned orthogonal low-rank matrix O must align with the true R→P intervention space. Nothing in the training objective enforces that. Section 3.4 states DRP has no auxiliary losses and is optimized only by the behavior loss (Eq. 11); Geiger et al. [13] identifies intervention spaces using known causal variables and supervised alignments, which are unavailable here. O is therefore free to learn any projection that improves behavior prediction, and e_pc = O^T(O e_p - O e_r) is not demonstrated to isolate preference from relevance. The paper's own ablation (Table 4, JDSearch/MLP) makes the problem concrete: DRP-2 (preference editing without adaptive fusion) gives AUC 0.6671 vs Base 0.6669, a ~0.0002 gain; DRP-5 (adaptive fusion without preference editing) gives 0.6813, a ~0.0144 gain; full DRP gives 0.6824. Thus almost all empirical improvement comes from adaptive fusion, not from the causal disentanglement mechanism. The 'untainted preference' narrative is an interpretation that is neither verified nor supported as the cause of the reported gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies e-commerce search behavior modeling, arguing that existing joint relevance/preference frameworks suffer from two problems: entangled relevance and preference effects (via the R->P causal path) and a collapsed modeling space caused by static fusion rules. It proposes DRP, which (i) edits the preference representation by subtracting a low-rank projection of the relevance representation (preference editing, Eq. 6) and (ii) replaces the fixed product fusion with a two-level adaptive fusion score (Eqs. 9-10). The method is evaluated on KuaiSAR, JDSearch, and a private dataset, under several preference and relevance backbones, reporting AUC, LogLoss, NDCG, and HR gains over joint-modeling baselines. The paper also provides a Venn-diagram analysis that divides the behavior space into six areas and argues that existing methods collapse these areas.","tokens_in":18975,"tokens_out":4195,"duration_ms":38455,"significance":"If the causal disentanglement claim were established, the paper would make a useful contribution: it offers a unified causal/Venn perspective on e-commerce search behavior, proposes a simple representation-editing mechanism that avoids human-labeled relevance data, and backs the proposal with unusually extensive experiments across three datasets and multiple backbone combinations. The derivation in Eq. (8) is correct, and the adaptive-fusion component appears to be a genuinely effective and practical contribution. However, the central claim that preference editing removes the relevance effect and yields 'untainted' preferences is not directly verified by the training objective or by the ablation results; as presented, the empirical gains are largely attributable to adaptive fusion rather than to the causal editing mechanism. The manuscript's contribution is therefore only partially supported in its current form.","major_comments":[{"comment":"The causal claim that Equation (6) removes the relevance effect from preference predictions is not verified. The orthogonal matrix O is trained only with the behavior prediction loss (Eq. 11); nothing in that objective ties the learned low-rank subspace to the causal intervention space of R->P. The cited basis, Geiger et al. [13], identifies intervention spaces using known causal variables or supervised alignments, which are not available in this setting. Consequently, e_pc = O^T(Oe_p - Oe_r) is a behavior-prediction-optimized projection, and the interpretation of it as 'untainted preference' remains an assumption. Since the abstract and contributions state this as a core mechanism, the paper needs a direct verification, for example in a synthetic setting with a known R->P intervention or via an external relevance/preference ground truth on at least one dataset.","section":"§3.2, §3.4, Eq. (6), Eq. (11)"},{"comment":"The ablation undermines the causal disentanglement claim. On JDSearch with MLP, DRP-2 (preference editing without adaptive fusion) yields AUC 0.6671 versus Base 0.6669, a gain of only 0.0002, while DRP-5 (adaptive fusion without preference editing) yields 0.6813, a gain of 0.0144; the full DRP yields 0.6824. Thus nearly all improvement comes from adaptive fusion, and the preference-editing module is not shown to deliver the claimed disentanglement benefit. Please also report DRP-1 (removal of the orthogonal constraint) in Table 4, since the variant list defines it but the table omits it, and provide a check that the edited representation actually reduces the relevance effect.","section":"§4.3, Table 4"},{"comment":"The model-visualization section defines Area#0-5 using the model's own top-20% relevance and preference scores, then uses the model's own predictions to demonstrate separation of these areas. This is circular as evidence of disentanglement: if the relevance and preference scores are entangled, the area labels are not ground truth. The visualization can illustrate the behavior of the fusion module, but it cannot substantiate the claim that preference editing removes the relevance effect. An external or synthetic ground truth for relevance/preference status, or at least a robustness check with different area-definition thresholds, is needed.","section":"§4.5, Figure 4"}],"minor_comments":[{"comment":"The matrix in Eq. (8) is interpreted as the joint probabilities for the six relevance-preference areas, but this implicitly assumes independence of p_c and r and is not normalized after the learnable alpha/beta replacement in Eq. (9). The paper should state this assumption explicitly.","section":"§3.3, Eq. (8)"},{"comment":"The sentence 'we incorporate an additional Area#0 following Area#6' is confusing: the paper defines six areas (Area#0-5), but the text refers to Area#6. Please clarify the labeling of the heatmap columns.","section":"§4.5"},{"comment":"The text says 'The fixed fusion in Equation 7 fails to differentiate Area#2&4 and Area#1&5,' but Equation (7) defines p_c, not the final fusion score. The reference should be to Equation (5) or (8).","section":"§4.5"},{"comment":"The initial values of alpha and beta, (1, 0.5), and the low-rank dimension D=16 are introduced as implementation details, but their selection is only tested on KuaiSAR. Please state how these hyperparameters were chosen and whether they transfer across datasets.","section":"§4.1.4"},{"comment":"Several DRP entries do not carry asterisks (for example, KuaiSAR LogLoss with the DSSM row and JDSearch LogLoss with the DSSM row), yet the text claims DRP outperforms the alternatives. Please clarify which differences are statistically significant and whether the claim refers to all metrics or only the starred ones.","section":"Table 3"},{"comment":"The notation for the low-rank projection matrix is inconsistent: the text defines O in Eq. (6) but later refers to 'D = 16 for R'. Please standardize the symbol.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript carries a WWW '25 DOI and ACM copyright notice, so it may already be part of the published literature; this report evaluates the manuscript as submitted. The main concern for the editor is that the headline causal claim (preference editing removes the relevance effect) is not supported by the training objective or ablations, while the adaptive-fusion component is well supported empirically. I recommend inviting a revision that either verifies the causal mechanism directly or substantially softens the causal interpretation; the paper is not fatally flawed because the adaptive-fusion contribution and the empirical comparison remain valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. DRP is a solid engineering contribution to e-commerce click prediction, and the adaptive fusion component is the real workhorse. The paper's headline claim—that preference editing removes the relevance effect to give 'untainted' preferences—is not verified, and the ablation data suggest the editing itself contributes almost nothing to the reported gains.\n\nWhat's actually new: combining a Geiger-style low-rank causal edit on the preference representation with a dual-level adaptive fusion, all trained without human-labeled relevance data. The identity in Equation (8) is a neat reframing of the fixed fusion rule, and the six-area Venn diagram is a clear way to talk about what joint models miss. The experimentation is extensive: three datasets (two public, one private), three relevance backbones, two preference backbones, several joint-modeling baselines, and an extra InvPref comparison in the appendix. The gains are modest but consistent, and significance tests are reported.\n\nThe soft spot is load-bearing. O in Equation (6) is learned only through the behavior loss in Equation (11). Nothing forces O to align with the true R→P intervention space. Geiger and collaborators identify intervention spaces using known causal variables or supervised alignments; neither is available here, so the edited representation is not demonstrated to isolate preference from relevance. The paper's own Table 4 makes this concrete: on JDSearch/MLP, preference editing alone (DRP-2) moves AUC from 0.6669 to 0.6671, while adaptive fusion alone (DRP-5) gives 0.6813, close to the full model's 0.6824. That is not a story about causal disentanglement driving the improvement. The visualization in Section 4.5 is also circular: areas are defined by the model's own top-20% relevance and preference scores, and then the model is shown to separate them. One more mild issue: the local fusion function F is under-specified—it is said to take only u, v, q during the feed-forward loop while using y_g and y to build its training objective, and it isn't clear what F actually computes at serving time.\n\nWho should read this: people building joint relevance-preference models for product search. The adaptive fusion component is worth borrowing, and the framing is legible. The causal editing idea should be treated as a plausible hypothesis until it gets a direct test—for example, synthetic data with known interventions, or a held-out set of human relevance judgments. As an editor, I would send this to review: the experiments are honest and reproducible in spirit (code link is given, though no commit hash), and the central claim is precisely the kind of thing referees should pressure-test.","headline":"Solid adaptive fusion gains for e-commerce search, but the causal disentanglement claim is not verified by the paper's own experiments.","tokens_in":19529,"tokens_out":3368,"would_cite":false,"duration_ms":29237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DRP improves e-commerce search prediction by editing preference representations to remove relevance's influence and fusing the two signals adaptively, without labeled relevance data.","keywords":["user behavior modeling","e-commerce search","relevance and preference disentanglement","preference editing","adaptive fusion","click-through rate prediction","causal graph","behavior modeling space"],"falsifier":"A reader could test the causal claim by constructing a dataset with known relevance labels and a planted relevance-to-preference influence, then checking whether the learned $\\boldsymbol{O}$ recovers the planted direction and whether edited preference scores stop shifting when relevance changes but preference is fixed; either failure would falsify the central claim.","tokens_in":18530,"feed_emoji":"🛒","tokens_out":5731,"duration_ms":54453,"temperature":0.7,"pith_summary":"This paper argues that behavior models in e-commerce search—which predict clicks by combining query–item relevance with user preference—are trained in a collapsed modeling space: relevance leaks into preference estimates, and a fixed multiplicative fusion formula cannot represent samples where relevance and preference disagree. To fix this, it introduces DRP, which edits the preference model's last-layer representation by subtracting a learned low-rank projection of the relevance representation, producing a calibrated preference score, and then combines relevance and preference with a dual-level adaptive fusion. If DRP is correct, search accuracy improves on click and ranking metrics without any human-labeled relevance judgments, which would make the approach much easier to deploy across e-commerce platforms. The claim is supported by consistent AUC and NDCG gains over the reported baselines on KuaiSAR, JDSearch, and a private dataset.","feed_headline":"Subtract relevance from preferences to improve e-commerce search","feed_subtitle":"Preference editing and adaptive fusion beat baselines on KuaiSAR, JDSearch, and a private dataset — no human relevance labels needed.","key_machinery":"The load-bearing device is the orthogonal low-rank projection matrix $\\boldsymbol{O}\\in\\mathbb{R}^{H\\times D}$. Because $\\boldsymbol{O}$ is orthogonal, $\\boldsymbol{O}^T$ is its inverse, so subtracting $\\boldsymbol{O}\\mathcal{e}_r$ inside the subspace and projecting back edits the preference representation exactly along the direction that best matches the causal influence of relevance on preference while preserving other information. The second device is the adaptive fusion formula, which rewrites the static product as a $2\\times 2$ probability table $P_{ij}$ for each relevance–preference status and then applies learnable vectors $\\boldsymbol{\\alpha},\\boldsymbol{\\beta}$ for global adaptation and a residual function $F$ for local adaptation, separating samples that were previously collapsed, such as irrelevant-but-clicked items.","core_discovery":"The central discovery is that the causal graph for e-commerce behavior has a path $R\\rightarrow P\\rightarrow B$, not only $P\\rightarrow B$ and $R\\rightarrow B$, and that existing joint models ignore it. Treating the last-layer representations $\\mathcal{e}_p$ and $\\mathcal{e}_r$ as neural counterparts of preference and relevance, DRP learns an orthogonal low-rank matrix $\\boldsymbol{O}$ whose row space approximates the relevance-to-preference intervention space, and computes $\\mathcal{e}_{p_c} = \\boldsymbol{O}^T(\\boldsymbol{O}\\mathcal{e}_p - \\boldsymbol{O}\\mathcal{e}_r)$. Then adaptive fusion replaces the fixed $\\hat{r}^{\\delta}\\hat{p}$ product with learnable global and local coefficients so the model can distinguish all six areas of the Venn diagram. The paper claims this yields disentangled, untainted preference predictions and a reconstructed modeling space, with no auxiliary supervision beyond click labels.","pith_inferences":["The same orthogonal-editing trick could decompose other confounded effects in search, such as position bias or exposure bias, wherever a causal path between two predictive signals exists.","If the learned $\\boldsymbol{O}$ truly isolates the relevance-to-preference influence, DRP's edited preference scores should be stable across queries that vary only in relevance; this is a testable prediction the paper does not run.","Since no labeled relevance data is used, the approach may transfer to smaller platforms, but only if behavior signal alone is rich enough to identify the intervention subspace; the paper's experiments do not directly verify this."],"forward_implications":["Click and ranking prediction in e-commerce search can be improved by removing the relevance-induced bias in preference representations while keeping the same backbone encoders.","The two components—preference editing and adaptive fusion—work across relevance backbones such as DSSM, QEM, and HEM and preference backbones such as MLP and DCN, so the improvement is not tied to one architecture.","Joint search models can be trained end-to-end from behavior signals alone, eliminating the need for expensive human relevance labels during training.","Samples where relevance and preference disagree, such as irrelevant-but-clicked items, receive distinct predictions instead of being collapsed into one score."],"supporting_citations":[{"why":"Supplies the technique of identifying a low-rank subspace that matches a causal intervention, which is used to define the preference editing operation.","marker":"[13]"},{"why":"Motivates aligning high-level causal variables with learned transformations of neural representations.","marker":"[12]"},{"why":"Provides the causality background for the claim that relevance influences preference in behavior modeling.","marker":"[15]"},{"why":"PRINT is the coarse adaptive-fusion baseline that DRP is designed to improve with finer-grained fusion.","marker":"[16]"},{"why":"CLK is the baseline that learns relevance from click data without human labels, representing the behavior-only relevance modeling line.","marker":"[54]"},{"why":"JDSearch is one of the two public datasets used to evaluate DRP.","marker":"[31]"},{"why":"KuaiSAR is the other public dataset, a unified search and recommendation log used for evaluation.","marker":"[41]"},{"why":"InvPref is the disentanglement baseline compared in the appendix, against which DRP reports gains.","marker":"[49]"}],"fun_headline_variants":["Untangle preference and relevance to fix e-commerce search","Subtract relevance from preferences for better search","Reconstruct behavior space to improve e-commerce search","Adaptive fusion and preference editing improve search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learned orthogonal low-rank matrix $\\boldsymbol{O}$ captures the true causal influence of relevance on preference, so subtracting $\\boldsymbol{O}\\mathcal{e}_r$ inside that subspace really removes the indirect relevance effect rather than merely reweighting the two representations.","fun_headline_variants_meta":{"raw":{"variants":["Untangle preference and relevance to fix e-commerce search","Subtract relevance from preferences for better search","Reconstruct behavior space to improve e-commerce search","Adaptive fusion and preference editing improve search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2816,"prompt_tokens":903,"completion_tokens":1913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1855}},"tokens_in":519,"tokens_out":1913,"duration_ms":14909,"temperature":1.0,"reasoning_tokens":1855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:16:14.991530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the causal claim by constructing a dataset with known relevance labels and a planted relevance-to-preference influence, then checking whether the learned $\\boldsymbol{O}$ recovers the planted direction and whether edited preference scores stop shifting when relevance changes but preference is fixed; either failure would falsify the central claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the technique of identifying a low-rank subspace that matches a causal intervention, which is used to define the preference editing operation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates aligning high-level causal variables with learned transformations of neural representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the causality background for the claim that relevance influences preference in behavior modeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PRINT is the coarse adaptive-fusion baseline that DRP is designed to improve with finer-grained fusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLK is the baseline that learns relevance from click data without human labels, representing the behavior-only relevance modeling line."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"JDSearch is one of the two public datasets used to evaluate DRP."}],"review_version":1}