Certifying What Helps Customer-Return Timing: A Screen-and-Confirm Test for Conditioning Signals, and Why Decay Is Nearly Enough

Sang Su Lee; Shishir Dash; Vijay Raghavan; Vineeth Loganathan

REVIEW 2 major objections 3 minor 23 references

Certifying What Helps Customer-Return Timing: A Screen-and-Confirm Test for Conditioning Signals, and Why Decay Is Nearly Enough

T0 review · 2 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that once a decay-based inter-event clock is in a temporal point process, conditioning signals such as category, recency/frequency, calendar, and geography add no measurable timing-likelihood gain, and it offers a…

desk verdict A transparent, useful measurement study with a genuinely good validation protocol, but the 'certified null' is over-broad for category/RFM signals and the seed-std rule is weaker than 'statistically null.' read the letter →

arxiv 2608.11555 v1 pith:GLDFBUD3 submitted 2026-08-12 cs.LG stat.AP

Sang Su Lee , Vineeth Loganathan , Shishir Dash , Vijay Raghavan This is my paper · ORCID

classification cs.LGstat.AP

keywords temporalpointprocessescustomer-returntimingconditioningsignalsscreen-and-confirmprotocolpositivecontrolsmodel-freepredictabilityceilingcontinuous-timedecaynear-memorylessreturns

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the extra signals practitioners add to customer-return timing models—category, lifetime value, recency/frequency, calendar, geography—actually improve when a customer returns, and it answers no on four datasets once a continuous-time decay backbone is present. To make a null result meaningful it builds a screen-and-confirm protocol: plant a synthetic coupling of known strength, confirm the model recovers it monotonically, and only then read a flat real-data result as 'no signal' rather than 'weak method.' It also reports a model-free ceiling: no available covariate explains more than a single-digit percentage of inter-return gap variance, so returns are near-memoryless beyond the time since the last event. The central claim is that the inter-event clock—continuous-time decay—is nearly sufficient for return timing, and that conditioning on top of it is statistically null or mildly harmful. The contribution is not discovering that decay helps; it is turning 'conditioning doesn't help' into a checkable, certified statement.

What carries the argument

The central object is the screen-and-confirm test: a positive control that plants a feature-to-timing coupling of tunable strength in synthetic data, shows the conditioned model recovers it monotonically, then runs the identical pipeline on the real feature; a flat real result after a passing control certifies 'no signal in that encoding.' The second machinery piece is a model-free predictability ceiling: the fraction of log-gap variance explained by each covariate, which bounds any mean-shift signal available to any model. The third is the decay head, which lets the hidden state decay between events and relaxes the frozen-intensity assumption of an attention-based backbone. Together these separate the mechanism claim (the clock carries the timing signal) from the measurement claim (nothing else is there to find).

What would settle it

Take the same decay backbone and datasets, then use a bootstrap over customers or many more seeds to estimate the full sampling distribution of each conditioning delta; alternatively, plant a synthetic season-to-gap coupling of realistic strength into held-out data and check whether the screen fails to recover it at a comparable magnitude. A positive finding would be a strongly seasonal dataset where a calendar-conditioned decay model beats the decay-only backbone by more than twice the estimated uncertainty while the positive control still recovers.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a certified negative: in temporal point-process models of customer return timing, the conditioning components that are standard in practice and in the literature are redundant once the model contains continuous-time decay. On three public e-commerce/web benchmarks, every conditioning delta on top of a decay backbone moves temporal negative log-likelihood by at most about 0.06, within seed noise; on a proprietary marketplace dataset the same additions are null to mildly harmful, with category worsening likelihood by about 0.47 relative and large seed variance read as overfitting. The search for point predictions fails harder: a model-free regression ceiling shows that less than about 5 percent of inter-event gap variance is explainable by any measured covariate, with the previous gap itself the best predictor. Decay's large likelihood gains are distributional—correctly placing probability mass over when the next event occurs—not point-predictive. The paper is explicit that it does not claim to have discovered that decay helps; the certified claim is that the usual enrichments do not help on top of it, and the screen-and-confirm protocol is what makes that null interpretable.

Load-bearing premise

The main premise is that a model improvement is counted as real only if it is larger than twice the variation across three random training runs; if that seed variation is smaller than the true uncertainty from customer sampling, model choice, or encoding choice, the conclusion that no signal helps could be wrong.

Editorial extensions

If this is right

Adding category, LTV, RFM, or calendar/geography conditioning to a decay-equipped customer-return model will not improve timing likelihood; on public benchmarks any movement is within about 0.06 NLL of zero.
Point-timing predictions cannot be improved by measured covariates: predicting the global mean gap matches the neural TPPs once leakage is removed.
Decay's whole-nat gains in temporal NLL are distributional calibration effects, attainable even when point predictability is zero.
A flat conditioning result should not be trusted unless a positive control shows the model can recover a planted signal of comparable encoding.
On data with genuine exogenous timing drivers, the same screen does fire, so the customer-return null is data-specific rather than a method artifact.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The screen-and-confirm protocol could be adopted as a standard audit step for any proposed point-process feature: certify recoverability on synthetic couplings first, then interpret real-data gains or nulls.
The model-free ceiling suggests that datasets with strong external seasonality, such as holiday retail or weather-driven demand, may sit under a higher ceiling; re-running the ceiling and screen per dataset would settle when conditioning genuinely pays.
If the inferred mechanism that decay already encodes RFM and category is right, representation-probing or mutual-information tests should show near-zero additional information in those hand-crafted features given the decayed hidden state.
The passive-covariate null does not close the question of interventions such as marketing nudges; causal counterfactual data could reveal timing effects even when offline covariates are uninformative.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one. It's a genuinely useful measurement paper. The screen-and-confirm protocol is a good idea: plant a synthetic coupling, verify the model recovers it, then trust the real-data null. The authors validate it for categorical and continuous encodings, and the taxi hour-of-day real positive control is a nice touch. The model-free ceiling—single-digit percent of gap variance explained, previous gap always the best—is a clean, simple result that explains why conditioning fails. And the paper is unusually honest: they document leaked read-outs and retract the inflated MAE numbers. That alone earns respect.

The central finding—decay is nearly sufficient, conditioning is redundant—is plausible and mostly supported. On public benchmarks, every decay-regime delta is at most 0.06 NLL. The Thumbtack category delta is +0.47, which they read as overfitting.

But there are two soft spots. First, the certification claim is broader than what was actually screened. Section 5.5 says calendar/geography passed the planted-coupling screen; category/RFM/LTV nulls were "not separately screened—per-pathway positive controls are future work." So the headline that "conditioning is redundant" is certified only for the exogenous signals. For category/RFM, the null rests on direct measurement plus the model-free ceiling, not on a demonstrated ability to recover those couplings. Given that CatOnly-D on RetailRocket is −0.06 and the positive control only covers i.i.d. season, the category null is not fully validated. Second, the statistical rule for "null" is |Δ| < 2× seed std over three seeds. That treats seed variance as the full uncertainty and ignores sampling variability of the NLL estimate. For the public deltas near zero this may not change the practical conclusion, but it doesn't support the word "statistically null."

I don't think either flaw sinks the paper. The magnitude claims are small and the transparency is exemplary. For a practitioner, the message is: don't expect LTV/category/RFM to sharpen return-timing once you have a decay backbone. That's worth testing on your own data. The screen-and-confirm protocol is worth adopting.

Bring it to reading group, cite it. Send it to review with a request for clearer statistical language and either per-pathway positive controls or a scope statement that limits the certified null to calendar/geography.

Referee Report

2 major / 3 minor

Summary. The paper proposes a two-part methodology for evaluating conditioning signals in temporal point process (TPP) models of customer-return timing: (C1) a screen-and-confirm protocol using synthetic positive controls, and (C2) a model-free ceiling on the predictability of inter-event gaps. Applying these to three public benchmarks and one proprietary marketplace dataset, the authors report that continuous-time decay is the dominant timing signal and that additional conditioning (calendar, geography, category, RFM, proxy-LTV) is redundant or harmful on top of a decay backbone. The manuscript is unusually transparent, explicitly retracting earlier leaked readouts and flagging limitations, and it frames the contribution as a certification discipline rather than an architectural novelty.

Significance. If its claims hold, the paper offers a reusable template for interpreting null results in TPP conditioning, which is genuinely valuable to both practitioners and the evaluation literature. The authors deserve explicit credit for shipping a positive-control screen, for running a real-clock positive control (NYC taxi), for distinguishing point-prediction from distributional likelihood gains, and for honestly retracting artifacts that inflated earlier results. The model-free ceiling result is a useful sanity check on the field's tendency to add covariates. The main risks are that the claim of certification is broader than the experiments actually certify, and that the statistical language is not backed by a proper uncertainty analysis.

major comments (2)

[§5.5] The screen-and-confirm protocol is validated only for calendar/geography signals. As the manuscript itself states, 'the LTV/category/RFM nulls are measured under the same pipeline and bounded by the same ceiling but were not separately screened—per-pathway positive controls are future work.' The no-decay CatOnly baseline (Table 4) shows the model can recover a category signal when decay is absent, but it does not establish that the decay-conditioned model can recover a category-to-gap coupling when decay is present. Since the abstract and C3 claim to certify that 'the conditioning practitioners keep adding—LTV, category, RFM, calendar, geography—is redundant or harmful,' this is a load-bearing gap between evidence and claim. The manuscript should either run per-pathway positive controls for category, RFM, and proxy-LTV, or explicitly narrow the certified claim to calendar/geography and describe the other nulls as measured but not screen-certified.
[§4 and §5.3] The null rule 'we read a conditioning delta as null when |Δ| < 2× its seed std' ( §4) treats seed-to-seed optimization variance as the only source of uncertainty. With only three seeds, this ignores sampling variability across customers and does not correct for multiple comparisons across the several conditioning signals tested. Because the central claim uses the words 'statistically null' (abstract and §5.3), the current procedure does not support that statistical phrasing. The measured deltas of ≲0.06 NLL on public benchmarks are certainly small and may justify 'negligible' or 'practically null,' but the authors should either provide a proper uncertainty estimate (e.g., bootstrap over customers or a permutation test on the NLL delta) or replace the 'statistically null' wording with a magnitude-based statement.

minor comments (3)

[Fig. 1] The y-axis label 'ΔNLL (↓)' could be clearer, since negative values indicate improvement and the caption already explains the sign convention. Consider adding a zero line and an explicit arrow direction to help readers avoid confusion.
[Table 5 and §5.4] The values in Table 5 are percentages but the column entries like '1.4%' are sometimes followed by the notation '𝑟2' without specifying that these are R²×100. Adding a note in the caption that all entries are percentages of variance explained would improve readability.
[§7] The list of retracted bugs is valuable and should be preserved, but the paragraph is dense. A small table with columns such as 'artifact', 'cause', 'impact', and 'correction' would make the honest-evaluation account easier for readers to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are direct measurements with an independent positive control and a model-free ceiling.

full rationale

The paper's derivation chain consists of empirical measurements rather than analytic reductions. The decay contrast (THP vs. THP-D) and conditioning deltas are temporal NLL differences computed by one pipeline on fixed splits; the positive control is a synthetic dataset with a planted coupling whose strength is varied, and the model-free ceiling is an OLS/ANOVA variance decomposition of inter-event gaps. None of these quantities is defined in terms of the conclusion, and no fitted parameter is later renamed as a prediction. The seed-std null rule is a pre-specified noise threshold, so the word 'statistically null' is governed by a stated decision rule rather than by construction. The paper explicitly disclaims inventing decay ('We do not claim to discover that decay helps') and flags the unscreened LTV/category/RFM pathways as future work, which is a coverage limitation, not a circular step. The only role of external citations is to benchmark reimplementations or identify known mechanisms; the paper reimplements S2P2 and omits baselines it cannot reproduce, so no load-bearing result is imported by self-citation. Hence no circularity is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on measured deltas and a model-free regression ceiling. The free parameters are analysis choices, not fitted constants in a derivation. The axioms are the statistical and modeling assumptions that make the nulls interpretable. No new entities are introduced.

free parameters (2)

Null threshold = 2× seed std
The certification that conditioning is null uses this hand-chosen rule on all ΔNLL deltas (§4).
Customer filter = ≥3 paid requests
Thumbtack analysis includes only repeat customers with at least three paid requests, which excludes the first return and may influence the measured nulls (§4).

assumptions (5)

domain assumption Temporal NLL with the given compensator is a valid and comparable measure of event-timing quality across all models in the same-estimator contrasts.
The paper explicitly acknowledges the S2P2-F closed-form compensator is not directly comparable to Monte-Carlo estimates and rests its claims on the THP vs THP-D same-estimator contrast (§4, §5.2).
domain assumption Seed standard deviation over 3 random seeds adequately captures the uncertainty of the NLL deltas used to declare nulls.
The null rule |Δ| < 2× seed std is the basis for the central 'conditioning is null' certification (§4, §5.3). If seed variance underestimates true variability, the nulls may be false.
ad hoc to paper The planted synthetic coupling in the screen-and-confirm protocol is representative of how real calendar/geography signals enter the intensity model.
Interpretability of the real-data null as 'no signal' relies on the synthetic positive control being a faithful stand-in for real exogenous signals (§5.5).
domain assumption The fraction of log-gap variance explained by a covariate bounds the mean-shift predictability available to any model.
The model-free ceiling (Table 5) is used to conclude returns are near-memoryless and to explain why conditioning fails (§5.4).
domain assumption The decay head is a fair, leak-free realization of the inter-event clock; the retracted leakage was fully removed.
The claim that decay nearly suffices depends on the corrected pre-decay readout and the counterfactual regression test (§7).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Certifying What Helps Customer-Return Timing: A Screen-and-Confirm Test for Conditioning Signals, and Why Decay Is Nearly Enough." pith.science (2026). https://pith.science/paper/GLDFBUD3

@misc{pith2026260811555,
  author       = {Pith},
  title        = {Pith review of: Certifying What Helps Customer-Return Timing: A Screen-and-Confirm Test for Conditioning Signals, and Why Decay Is Nearly Enough},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLDFBUD3}},
  note         = {Machine review of arXiv:2608.11555}
}

read the original abstract

Practitioners enrich customer-return models with ever more signals (lifetime value, category, recency/frequency, calendar, geography), and the temporal-point-process (TPP) literature follows suit with covariate- and external-covariate-conditioned intensities. But does any of it improve the timing, and how would you know? A null ("feature X doesn't help") is only meaningful if the model could have found a signal. We make two contributions--a method and a measurement--to answer this credibly. (i) A screen-and-confirm protocol that certifies whether a candidate signal improves a TPP's event-timing likelihood: a positive control plants a coupling of known strength and confirms the model recovers it, so a real-data null can be read as "no signal" rather than "weak method." The control is validated for categorical and continuous encodings, and on a real clock-driven dataset (NYC taxi hour-of-day). (ii) A model-free ceiling quantifying how little of customer-return timing is point-predictable at all (a single-digit percentage of gap variance from any covariate; returns are near-memoryless). With these we certify a clean result on three public benchmarks (Amazon, Taobao, RetailRocket) and a real marketplace (Thumbtack): the inter-event clock--continuous-time decay, long known to beat frozen-intensity models--is nearly sufficient, and the conditioning the field keeps adding is redundant or harmful on top of it (statistically null on the public benchmarks, at most 0.06 NLL; null to mildly harmful on the marketplace). We do not claim to discover that decay helps; our contribution is the tools that turn "conditioning doesn't help" into a checkable, certified statement--plus an honest-evaluation account of the read-out/leakage pitfalls we hit and retracted.

Figures

Figures reproduced from arXiv: 2608.11555 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages

[1]

Du et al

N. Du et al. Recurrent Marked Temporal Point Processes.KDD, 2016

work page 2016
[2]

Mei and J

H. Mei and J. Eisner. The Neural Hawkes Process.NeurIPS, 2017

work page 2017
[3]

Zuo et al

S. Zuo et al. Transformer Hawkes Process.ICML, 2020

work page 2020
[4]

Zhang et al

Q. Zhang et al. Self-Attentive Hawkes Process.ICML, 2020

work page 2020
[5]

C. Yang, H. Mei, J. Eisner. Transformer Embeddings of Irregularly Spaced Events and Their Participants (AttNHP).ICLR, 2022

work page 2022
[6]

Shchur et al

O. Shchur et al. Intensity-Free Learning of Temporal Point Processes.ICLR, 2020

work page 2020
[7]

T. Omi, N. Ueda, and K. Aihara. Fully Neural Network based Model for General Temporal Point Processes.NeurIPS, 2019

work page 2019
[8]

Xue et al

S. Xue et al. EasyTPP: Towards Open Benchmarking Temporal Point Processes. ICLR, 2024

work page 2024

Show all 23 references

[9]

Chang, A

Y. Chang, A. Boyd, C. Xiao, T. Kass-Hout, P. Bhatia, P. Smyth, and A. Warrington. Deep Continuous-Time State-Space Models for Marked Event Sequences.NeurIPS, 2025

2025
[10]

Shchur et al

O. Shchur et al. Neural Temporal Point Processes: A Review.IJCAI, 2021

2021
[11]

Grob et al

G. Grob et al. A Recurrent Neural Network Survival Model: Predicting Web User Return Time.ECML PKDD, 2018

2018
[12]

Noorbakhsh and M

K. Noorbakhsh and M. Gomez-Rodriguez. Counterfactual Temporal Point Pro- cesses.NeurIPS, 2022

2022
[13]

Z. Meng, B. Li, X. Fan, Z. Li, Y. Wang, F. Chen, and F. Zhou. TransFeat-TPP: An Interpretable Deep Covariate Temporal Point Processes.ECAI, 2024

2024
[14]

B. Li, L. Zhang, F. Tsung, and X. Zhang. METP: Multi-Granularity Integration of External Covariates for Temporal Point Processes.AAAI, 2026

2026
[15]

Counting Your Customers

P. Fader, B. Hardie, K. Lee. “Counting Your Customers” the Easy Way: BG/NBD. Marketing Science, 2005

2005
[16]

Schmittlein, D

D. Schmittlein, D. Morrison, R. Colombo. Counting Your Customers: Pareto/NBD. Management Science, 1987

1987
[17]

X. Wang, T. Liu, J. Miao. A Deep Probabilistic Model for Customer Lifetime Value (ZILN). arXiv:1912.07753, 2019

1912 arXiv
[18]

A. G. Hawkes. Spectra of Some Self-Exciting and Mutually Exciting Point Pro- cesses.Biometrika, 1971

1971
[19]

Kapoor, M

K. Kapoor, M. Sun, J. Srivastava, and T. Ye. A Hazard Based Approach to User Return Time Prediction.KDD, 2014

2014
[20]

N. Du, Y. Wang, N. He, J. Sun, and L. Song. Time-Sensitive Recommendation from Recurrent User Activities.NeurIPS, 2015

2015
[21]

Bosser and S

T. Bosser and S. Ben Taieb. On the Predictive Accuracy of Neural Temporal Point Process Models for Continuous-time Event Data.TMLR, 2023

2023
[22]

Platzer and T

M. Platzer and T. Reutterer. Ticking Away the Moments: Timing Regularity Helps to Better Predict Customer Activity.Marketing Science, 2016

2016
[23]

Adebayo, J

J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity Checks for Saliency Maps.NeurIPS, 2018

2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Du et al

N. Du et al. Recurrent Marked Temporal Point Processes.KDD, 2016

work page 2016

[2] [2]

Mei and J

H. Mei and J. Eisner. The Neural Hawkes Process.NeurIPS, 2017

work page 2017

[3] [3]

Zuo et al

S. Zuo et al. Transformer Hawkes Process.ICML, 2020

work page 2020

[4] [4]

Zhang et al

Q. Zhang et al. Self-Attentive Hawkes Process.ICML, 2020

work page 2020

[5] [5]

C. Yang, H. Mei, J. Eisner. Transformer Embeddings of Irregularly Spaced Events and Their Participants (AttNHP).ICLR, 2022

work page 2022

[6] [6]

Shchur et al

O. Shchur et al. Intensity-Free Learning of Temporal Point Processes.ICLR, 2020

work page 2020

[7] [7]

T. Omi, N. Ueda, and K. Aihara. Fully Neural Network based Model for General Temporal Point Processes.NeurIPS, 2019

work page 2019

[8] [8]

Xue et al

S. Xue et al. EasyTPP: Towards Open Benchmarking Temporal Point Processes. ICLR, 2024

work page 2024