Pith. sign in

REVIEW 3 major objections 6 minor 24 references

CCSS enforces known monotonicity between numerical features and recommender scores by contrasting counterfactual samples; five model architectures plus an online A/B test report gains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A plug-in contrastive loss that enforces monotonicity between numerical features and recommender outputs, via counterfactual sample synthesis, improves AUC, GAUC, and monotonicity.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A practical plug-in regularizer for monotonic numerical features in recommenders, with large reported gains; the interpretability metric is circular and the fixed monotonicity direction is unvalidated, but the core idea is solid. the 3 major comments →

arxiv 2509.03187 v1 pith:NJTN7FJM submitted 2025-09-03 cs.IR cs.LG

Enhancing Interpretability and Effectiveness in Recommendation with Numerical Features via Learning to Contrast the Counterfactual samples

Xiaoxiao Xu , Hao Wu , Wenhui Yu , Lantao Hu , Peng Jiang , Kun Gai This is my paper
classification cs.IR cs.LG
keywords recommender systemsnumerical featuresmonotonicitycounterfactual samplescontrastive learningCTR predictioninterpretabilityShapley value
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove a simple prior can be taught to any recommender: if a numerical feature such as likes, watch time, or click count is higher, the model's predicted score should be higher too. It argues that this monotonicity is central to interpretability and that enforcing it explicitly, rather than leaving it to chance, also improves prediction accuracy. The proposed framework, CCSS, synthesizes counterfactual and factual versions of each training sample by shifting one numerical feature in the expected direction or the opposite, then trains the network to rank the three samples accordingly. On a public benchmark and a 1.29-billion-sample industrial dataset, every tested backbone improves in AUC, GAUC, and a monotonicity-rate metric, and an online A/B test reports a 3.93% collect-rate gain. The load-bearing assumption is that each feature's monotonic direction is known and fixed.

Core claim

The central claim is that monotonicity between numerical features and the neural network output is a learnable, model-agnostic objective rather than a network-architecture constraint. CCSS does this in two stages: during training, for each sample it selects one numerical feature with probability proportional to its Shapley-value feature importance, then moves that feature to the center of an adjacent discretization bucket, producing a counterfactual sample (moved opposite to the expected effect) and a factual sample (moved in the expected direction). Hinge pairwise losses enforce the ranking factual greater than original greater than counterfactual for monotonic-increasing features, with the

What carries the argument

The central mechanism is pairwise contrastive ranking over locally synthesized samples. Given an original training sample, the Counterfactual Sample Synthesizer picks one numerical feature with probability proportional to its Shapley-value importance and moves that feature to the center of the neighboring discretization bucket: toward the expected direction for a factual sample (assigned the original label) and away from it for a counterfactual sample (unlabeled). Hinge losses then enforce, for positive labels under increasing monotonicity, that the factual sample scores higher than the original and the original higher than the counterfactual; the order reverses for negative labels or decrea

Load-bearing premise

The framework assumes the monotonic direction of every numerical feature is known in advance and fixed—in all experiments, larger input values should give larger predictions—and that moving a feature in that direction keeps the original label valid; if a feature is inverted or non-monotonic, the contrastive supervision pushes the model in the wrong direction.

What would settle it

Take a dataset with a numerical feature whose true effect is non-monotonic, such as a recommendation score that rises then falls with video length, or deliberately invert one feature's relationship with the label so the ground-truth effect is decreasing. Train the same backbone with and without CCSS assuming increasing monotonicity for that feature: if the monotonicity assumption is load-bearing, CCSS should lower AUC or GAUC and worsen Mono_rate on the violating feature relative to the backbone alone, whereas generic data augmentation would not produce that reversal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On both datasets, every backbone improves when CCSS is added; the paper reports relative-to-chance (RelaImpr) gains of at least 6.0% in AUC and 4.5% in GAUC.
  • Mono_rate, the fraction of factual/original/counterfactual triples ranked in the expected order, rises sharply for the top seven important numerical features, indicating improved interpretability of ranking decisions.
  • Ablation results show that removing any component (factual pointwise augmentation, factual contrastive loss, counterfactual contrastive loss) or replacing Shapley-weighted feature selection with equal-probability selection lowers AUC and GAUC relative to the full method.
  • Online A/B testing on a collect prediction model shows a 3.93% collect-rate gain when CCSS is added to an already optimized DCN model, suggesting the offline gains transfer to live traffic.
  • Because CCSS synthesizes samples locally for each training instance and only adds losses on those synthesized inputs, it can be added to an existing CTR or engagement model without changing the model's architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gains come from enforcing monotonicity rather than generic data augmentation, the same recipe should transfer to other ranking targets such as likes, follows, or dwell-time prediction whenever the numerical features have a defensible monotonic direction; this is a testable claim beyond the paper's collect-rate and CTR evidence.
  • The fixed global monotonicity direction is the framework's boundary: features with non-monotonic effects, such as engagement that peaks at mid-range values, would need per-feature direction learning or segment-wise monotonic constraints. A synthetic experiment with a known inverted feature would cleanly separate the monotonicity mechanism from generic augmentation.
  • Shapley-value importance sampling adds computational cost; a cheaper surrogate, such as approximate feature attribution or learned importance, could make CCSS practical in higher-throughput settings while preserving the contrastive ranking signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CCSS (Contrastive learning with Counterfactual Samples Synthesizing), a model-agnostic framework for imposing monotonicity between neural network outputs and numerical features in recommender systems. For each training instance, CCSS synthesizes a counterfactual sample and a factual sample by perturbing one numerical feature (chosen with probability proportional to Shapley importance) to the center of an adjacent bucket, assigns the factual sample the original label, and adds pairwise hinge losses to rank the original, factual, and counterfactual samples consistently with the assumed monotonic direction. The authors evaluate CCSS on KuaiRand-Pure and a large industrial dataset with five backbones (DNN, Wide&Deep, PNN, DCN, DeepFM), reporting consistent AUC/GAUC improvements and large Mono_rate gains, plus a 3.93% collect_rate improvement in an online A/B test.

Significance. The problem is practically important: numerical features are ubiquitous in CTR prediction, and enforcing sensible monotonic behavior can improve both effectiveness and interpretability. The proposed method is simple, model-agnostic, and demonstrates consistent offline gains across several architectures, which is a meaningful empirical result. The online deployment result is also valuable. However, the interpretability claim is currently supported only by Mono_rate, a metric that is essentially the training objective's own pairwise accuracy, and the method rests on a fixed and unvalidated monotonicity-direction assumption. If these weaknesses are addressed with independent evaluation and per-feature validation, the framework would be a solid contribution to industrial recommender systems.

major comments (3)
  1. [§3.3.1, §4.4.1, Eq. (13) vs Eqs. (7)–(9)] The Mono_rate metric is circular with respect to the training objective. Eq. (13) defines Mono_rate over exactly the (F,O) and (C,O) pairs that the pairwise hinge losses in Eqs. (7)–(8) and the augmented pointwise loss in Eq. (9) explicitly optimize. Thus the large Mono_rate gains in Table 5 are a direct measure of how well the model fits the training-time ranking constraints, not an independent evaluation of interpretability. To support the interpretability claim, please provide a separate evaluation, e.g., monotonicity checks over all numerical feature values (not only adjacent-bucket neighbors), or human/domain validation that the predicted scores respect the expected monotonic relationships for arbitrary feature perturbations.
  2. [§4.1, §3.2.2, Table 2] The method assumes a fixed monotonicity direction: 'the larger the input value, the larger the output result should be' for all numerical features in both datasets (Sec. 4.1). This direction is wired into sample synthesis (Table 2), the pairwise losses (Eqs. 7–8), and the label assignment for factual samples (Eq. 9). No evidence is given that every feature satisfies this monotone-increasing assumption. If any feature has an inverted or non-monotonic relationship, the losses provide incorrect supervision for all samples where that feature is perturbed, and since features are sampled with probability proportional to Shapley importance, the damage is concentrated on the most influential features. Please validate the monotonicity direction per feature, or extend the method to infer or adapt the direction from data, and report sensitivity to this assumption.
  3. [§4.5, Table 7] Table 7 appears to mislabel the full model as 'DNN'. The first row reports AUC=0.786 and GAUC=0.732, which match the DNN+CCSS values in Table 5 (0.7860/0.7320), whereas the actual DNN baseline in Table 5 is 0.7200/0.6560. As a result, the ablation does not compare the ablated variants against the true baseline, and the phrase 'confirm the advantages of our proposed random strategy' is not supported by the table as presented. Please correct the row labels and include the actual DNN baseline.
minor comments (6)
  1. [§2.3] The sentence is broken: '...removing phrases, which alThe generated counterfactual samples...' — likely a missing continuation or formatting error.
  2. [§4.3.1] The text says 'For all the three datasets' but only two datasets are used. Please correct to 'two datasets'.
  3. [§3.2.1] The Shapley value computation is not described. Please specify how feature importance q_i is calculated for numerical features (e.g., which background distribution, which model output, and how many feature coalitions are sampled).
  4. [§4.5 and §4.3.2] The ablation variants have inconsistent naming: 'CCSS(Only Data Augmentation)' in the text corresponds to 'DNN(Only Factual Pointwise loss)' in Table 7. Please unify the terminology.
  5. [References] References [2] and [3] are the same paper (Cheng et al., Wide & Deep). Please deduplicate.
  6. [§4.3.1] The hyperparameter α is set to 1.0 in all experiments, but Figure 3 shows sensitivity. Please state whether α=1.0 was chosen based on a validation set for each backbone or uniformly.

Circularity Check

1 steps flagged

Interpretability evidence is circular: Mono_rate counts the same (F,O)/(C,O) orderings that the CCSS training loss directly optimizes.

specific steps
  1. fitted input called prediction [Sec. 4.3.2, Eq. (13) vs. Sec. 3.3.1, Eqs. (7)–(9)]
    "Mono_rate: We define Mono_rate to evaluate the monotonicity (i.e., interpretability) between model output and numerical features: Mono_Rate = #Monotone_pairs(D) / #Comparable_pairs(D) (13) where Comparable_pairs(D) donates the entire set of the expected monotonic pairs, including all the (F,O) pairs and (C,O) pairs."

    The CCSS pairwise hinge losses in Eqs. (7)–(9) are defined over exactly the same ordering constraints that Mono_rate counts. For a positive original sample, Eq. (7) adds lP(F,O) requiring yhat_F > yhat_O and lP(O,C) requiring yhat_O > yhat_C; Mono_rate counts precisely those as valid monotone pairs. For a negative original sample, Eq. (8) imposes the mirrored constraints, and Eq. (13) counts the same mirrored pairs. Thus the reported Mono_rate improvements in Table 5 are not independent evidence that the model became more interpretable; they are a re-measurement of the training objective itself. The claim that 'Mono_rate improvements are clearly remarkable, which means the interpretability ... can be enhanced' therefore reduces by construction to the fact that the model was trained to sati

full rationale

The only substantive circularity is the interpretability evaluation: Mono_rate is defined as the fraction of (F,O) and (C,O) orderings that the pairwise losses in Eqs. (7)–(9) directly optimize, so the reported monotonicity improvements are essentially a training-objective report card. This affects one of the two headline contributions. The effectiveness claims (AUC/GAUC and online collect_rate gains) are measured on real labels and are not forced by the training construction; they give independent content to the paper. The assumed monotonic increasing direction ('the larger the input value, the larger the output result should be', Sec. 4.1) is an unvalidated modeling assumption and a correctness risk, but it is not circularity. The self-citations (e.g., [22] for GAUC/RelaImpr) are not load-bearing. Overall, the partial circularity in the interpretability metric warrants a score of 6, while the AUC/GAUC improvements remain non-circular evidence.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central claim rests on three free choices: the contrastive weight α, the Shapley-based sampling probabilities (whose computation is undisclosed), and the discretization buckets. It also rests on two domain assumptions about monotonicity direction and label preservation that are stated but not validated.

free parameters (3)
  • α (contrastive loss weight) = 1.0
    Hyperparameter controlling the trade-off between pointwise and pairwise losses; set to 1.0 for all offline and online evaluations without per-dataset tuning (Section 4.3.1).
  • Shapley value feature importances = not disclosed
    Used as sampling probabilities for selecting which numerical feature to disturb (Section 3.2.2); the computation method is not described, and the resulting probabilities depend on an unspecified model and approximation.
  • discretization bucket boundaries = not disclosed
    Counterfactual disturbance moves a feature to the center of a neighboring bucket, so bucket definitions fully determine the synthetic samples; the paper does not state how buckets are constructed.
axioms (3)
  • ad hoc to paper Monotonicity direction is known a priori (increasing for all features used in experiments)
    Assumed in Section 4.1 ('we expect that the larger the input value, the larger the output result should be') and used to decide the disturbance direction in Section 3.2.2. No validation per feature.
  • ad hoc to paper The factual sample retains the original label after disturbance
    In Section 3.2.2 step 4, a positive original sample yields a positive factual sample and a negative original yields a negative factual sample. This label propagation is assumed without evidence.
  • domain assumption Bucket neighbor centers represent meaningful monotonic steps
    The counterfactual/factual samples are generated by moving to the center of the adjacent bucket; the paper provides no analysis of bucket size or discretization sensitivity.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Interpretability and Effectiveness in Recommendation with Numerical Features via Learning to Contrast the Counterfactual samples." pith.science (2026). https://pith.science/paper/NJTN7FJM

@misc{pith2026250903187,
  author       = {Pith},
  title        = {Pith review of: Enhancing Interpretability and Effectiveness in Recommendation with Numerical Features via Learning to Contrast the Counterfactual samples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NJTN7FJM}},
  note         = {Machine review of arXiv:2509.03187}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We propose a general model-agnostic Contrastive learning framework with Counterfactual Samples Synthesizing (CCSS) for modeling the monotonicity between the neural network output and numerical features which is critical for interpretability and effectiveness of recommender systems. CCSS models the monotonicity via a two-stage process: synthesizing counterfactual samples and contrasting the counterfactual samples. The two techniques are naturally integrated into a model-agnostic framework, forming an end-to-end training process. Abundant empirical tests are conducted on a publicly available dataset and a real industrial dataset, and the results well demonstrate the effectiveness of our proposed CCSS. Besides, CCSS has been deployed in our real large-scale industrial recommender, successfully serving over hundreds of millions users.

Figures

Figures reproduced from arXiv: 2509.03187 by Hao Wu, Kun Gai, Lantao Hu, Peng Jiang, Wenhui Yu, Xiaoxiao Xu.

Figure 1
Figure 1. Figure 1: An illustration of the numerical features usage in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An illustration of our proposed CCSS: the left is the end-to-end learning framework with counterfactual sample [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of 𝛼 on (a) Mono_rate and (b) AUC. DNN is used as the network. (2) CCSS(Only Factual Contrastive Loss): only adopt the auxiliary hinge loss of Factual-Original sample pairs. (3) CCSS(Only Coun￾terfactual Contrastive Loss): only adopt the auxiliary hinge loss of Counterfactual-Original sample pairs. (3) CCSS(Equal Probability Random Strategy): select the numerical feature to be disturbed with equal p… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 14 canonical work pages

  1. [1]

    Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang

    work page

  2. [3]

    Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al

  3. [4]

    Yuan Cheng. 2022. Dynamic Explicit Embedding Representation for Numerical Features in Deep CTR Prediction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM ’22). Association for Computing Machinery, New York, NY, USA, 3888–3892. https://doi.org/10.1145/3511808.3557587

    arXiv 2022

  4. [5]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198

    2016

  5. [6]

    In Proceedings of the 1st workshop on deep learning for recommender systems

    Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems . 7–10

  6. [7]

    Daniel Fryer, Inga Strümke, and Hien Nguyen. 2021. Shapley values for feature selection: The good, the bad, and the axioms.Ieee Access 9 (2021), 144352–144360

    work page 2021

  7. [8]

    Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. 2021. An embedding learning framework for numerical features in ctr prediction. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining . 2910–2918

    work page 2021

  8. [9]

    Tom Fawcett. 2006. An introduction to ROC analysis. Pattern recognition letters 27, 8 (2006), 861–874

    2006

  9. [10]

    Malay Haldar, Prashant Ramanathan, Tyler Sax, Mustafa Abdool, Lanbo Zhang, Aamir Mansawala, Shulin Yang, Bradley Turnbull, and Junshuo Liao. 2020. Im- proving deep learning for airbnb search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 2822–2830

    work page 2020

  10. [11]

    Ninghao Liu, Yong Ge, Li Li, Xia Hu, Rui Chen, and Soo-Hyun Choi. 2020. Explain- able recommender systems via resolving learning representations. In Proceedings of the 29th ACM international conference on information & knowledge management. 895–904

    work page 2020

  11. [12]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction.arXiv preprint arXiv:1703.04247 (2017)

    Pith/arXiv arXiv 2017

  12. [13]

    Georgina Peake and Jun Wang. 2018. Explanation Mining: Post Hoc Interpretabil- ity of Latent Factor Models for Recommendation Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Min- ing (London, United Kingdom) (KDD ’18). Association for Computing Machinery, New York, NY, USA, 2060–2069. https://doi.org/1...

    arXiv 2018

  13. [14]

    Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang

  14. [15]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole- Jean Wu, Alisson G Azzolini, et al. 2019. Deep learning recommendation model for personalization and recommendation systems.arXiv preprint arXiv:1906.00091 (2019)

    Pith/arXiv arXiv 2019

  15. [16]

    Davor Runje and Sharath M Shankaranarayana. 2023. Constrained monotonic neural networks. In International Conference on Machine Learning . PMLR, 29338– 29353

    work page 2023

  16. [17]

    Ryotaro Shimizu, Megumi Matsutani, and Masayuki Goto. 2022. An explainable recommendation framework based on an improved knowledge graph attention network with massive volumes of side information. Knowledge-Based Systems 239 (2022), 107970. https://doi.org/10.1016/j.knosys.2021.107970

    work page arXiv 2022

  17. [18]

    In 2016 IEEE 16th International Conference on Data Mining (ICDM)

    Product-based neural networks for user response prediction. In 2016 IEEE 16th International Conference on Data Mining (ICDM) . IEEE, 1149–1154

    work page 2016

  18. [19]

    Why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining . 1135–1144

    work page 2016

  19. [20]

    Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. In Proceedings of the ADKDD’17 . 1–7

    work page 2017

  20. [21]

    Yongfeng Zhang, Xu Chen, et al. 2020. Explainable recommendation: A survey and new perspectives. Foundations and Trends ® in Information Retrieval 14, 1 (2020), 1–101

    work page 2020

  21. [22]

    Eunhye Song, Barry L Nelson, and Jeremy Staum. 2016. Shapley effects for global sensitivity analysis: Theory and computation. SIAM/ASA Journal on Uncertainty Quantification 4, 1 (2016), 1060–1083

    work page 2016

  22. [23]

    Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan E Hines, John P Dickerson, and Chirag Shah. 2020. Counterfactual explanations and algorithmic recourses for machine learning: A review. arXiv preprint arXiv:2010.10596 (2020)

    Pith/arXiv arXiv 2020

  23. [26]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 1059–1068

    work page 2018

  24. [2020]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Counterfactual Samples Synthesizing for Robust Visual Question Answer- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.