{"id":"992579a0-2ae4-4aec-9fcd-bfc8f400ce6a","arxiv_id":"1908.01207","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"JODIE predicts future user-item interactions by linearly projecting user embeddings in time, and reports at least 20% MRR improvement over six baselines on four datasets.","lead":"This paper introduces JODIE, a model that learns moving embeddings of users and items from timestamped interactions and projects user embeddings forward in time to predict the next interaction. It also contributes a batching method that trains the coupled model about nine times faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The trajectory-projection mechanism is never directly validated; the reported gains come only from downstream tasks, and the prediction layer's direct use of the previous item's embedding may drive much of the advantage.","rationale":"The reader identified as the weakest assumption that the future user embedding is obtained by elementwise linear scaling and is never directly validated; my independent reading converges on the same point. The load-bearing issue is not that the downstream numbers are false, but that the paper's headline contribution — learning and predicting embedding trajectories — rests entirely on an unvalidated projection ansatz. The strongest evidence for JODIE's empirical effectiveness is the consistent margin over six baselines on interaction prediction and state-change prediction, and the t-Batch speedup is a real algorithmic contribution. However, because the projection operator is trained only through a downstream item-embedding loss, high performance on repetitive datasets can plausibly be explained by the direct previous-item-embedding input in the prediction layer rather than by an accurate trajectory model. This does not warrant rejection: the method may still work well, and the code is publicly available for verification. It does warrant keeping the verdict conditional, because the central scientific claim about trajectory prediction is not established by the reported experiments. A clean ablation of the projection component and a direct comparison of projected versus observed future embeddings would settle whether the mechanism is real. The reader already reached a CONDITIONAL verdict with moderate confidence, and my analysis does not move that verdict; it sharpens the reason for conditionality.","tokens_in":15651,"tokens_out":4322,"duration_ms":48798,"concrete_test":"Run an ablation and a direct trajectory-fidelity check. (1) On the same data splits and hyperparameters, train three JODIE variants: the reported projection b_u(t+Δ) = (1 + Wp Δ) * u(t); a no-projection baseline with b_u(t+Δ) = u(t); and a non-linear projection MLP. Compare MRR and Recall@10 across Reddit, Wikipedia, and LastFM with multiple seeds. If the no-projection variant matches the reported projection within noise, the trajectory operator is not load-bearing. (2) Directly evaluate projection fidelity: for each held-out interaction (u, i, t+Δ), use the trained model to compute the projected embedding b_u(t+Δ) from u's last observed embedding at time t, and compare it with the observed updated embedding u(t+Δ) after the held-out interaction, via cosine similarity and L2 distance. Also compare against the constant baseline b_u(t+Δ) = u(t).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is the projection operator that 'models the future embedding trajectory' (Section 3.2), defined as b_u(t+Δ) = (1 + Wp Δ) * u(t). This operator is trained only through the downstream item-embedding loss in Eq. 1; no experiment compares projected embeddings against the actually observed future embeddings u(t+Δ) that the update operation produces when the user's next interaction occurs. Consequently, the claimed trajectory model is an untested ansatz, and every reported 20%+ improvement could in principle come from other components: the coupled mutually-recursive RNNs, the direct item-embedding output, or the prediction layer's explicit use of the previous item's dynamic embedding i(t+Δ−) (Section 3.3). The datasets are highly repetitive — 79% of Reddit and 61% of Wikipedia interactions are consecutive interactions with the same item — so feeding i(t+Δ−) directly into the predictor may capture most of the signal without any meaningful projection. The paper also states that a linear projection 'works the best' and that non-linearity 'reduce[s] the prediction performance,' but no supporting ablation or figure is provided. Thus, while the empirical results may be reproducible, the specific mechanism advertised in the title and abstract — predicting the embedding trajectory — is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JODIE, a coupled recurrent neural network model for temporal user-item interaction networks. Each user and item is assigned a static embedding and a dynamic embedding; at each interaction, two mutually-recursive RNNs update the dynamic embeddings of the interacting user and item. The paper's main novelty is an embedding projection operator that linearly scales a user's last dynamic embedding by a learned time-context vector to estimate the user's embedding at a future time. The projected embedding is fed into a linear prediction layer that outputs the embedding of the predicted next item, enabling near-constant-time inference via locality-sensitive hashing. The paper also introduces t-Batch, a batching algorithm for training the mutually-recursive RNNs in parallel while maintaining temporal consistency. Experiments on Reddit, Wikipedia, LastFM, and MOOC datasets report that JODIE outperforms six baselines by at least 20% in mean reciprocal rank for future interaction prediction and by at least 12% in AUC for user state-change prediction, along with a 9.2x speed-up over the closest baseline and robustness experiments.","tokens_in":16005,"tokens_out":7571,"duration_ms":69269,"significance":"If the empirical claims hold, JODIE represents a substantial advance in dynamic embedding recommender systems: it explicitly forecasts future user embeddings, directly predicts item embeddings rather than interaction probabilities, and introduces a principled batching scheme for mutually-recursive RNNs. The t-Batch algorithm is a practical contribution to scaling such models, and the release of code and datasets supports reproducibility. However, the central mechanistic claim in the title and abstract—that JODIE learns and predicts embedding trajectories—is not directly validated, and the user state-change comparison is confounded by an auxiliary supervision signal available only to JODIE. These issues are load-bearing for assessing the paper's contribution, and they require additional experiments or re-analysis before the significance of the reported gains can be fully evaluated.","major_comments":[{"comment":"The projection operator is never directly validated against observed future embeddings. Section 3.2 defines the projected embedding b_u(t+Δ) = (1 + Wp Δ) * u(t), but no experiment compares this projection with the actual embedding u(t+Δ) that is produced when the user's next interaction is observed, and no ablation isolates the projection's contribution from the coupled RNN updates or from the direct use of the previous item's embedding i(t+Δ−) in the prediction layer (Section 3.3). Consequently, the paper's claim of 'predicting dynamic embedding trajectory' is unsupported; the reported gains could be driven by other components. Please add a direct trajectory evaluation (e.g., L2 or cosine distance between projected and observed embeddings as a function of Δ) and an ablation that removes or replaces the projection with an identity or time-independent baseline.","section":"Section 3.2 and Section 4"},{"comment":"The user state-change comparison is unfair because JODIE receives additional label supervision during training. Section 3.3 ('Extending the loss for categorical prediction') describes adding a cross-entropy loss on state-change labels to JODIE's total loss, whereas Section 4.2 states that the baselines are trained only on interaction data and then fitted with a logistic regression classifier on the dynamic user embedding. This gives JODIE access to the state-change labels during training, which the baselines do not receive, potentially inflating the reported 12% AUC improvement. To support the claim, the baselines should be given the same auxiliary label signal (e.g., fine-tuned with state-change supervision) or JODIE should be evaluated in this experiment without the auxiliary loss.","section":"Section 4.2 and Section 3.3"},{"comment":"The headline performance claims ('at least 20% in MRR, 12% in AUC') are based on single runs with no error bars, standard deviations, or significance tests. Given that the paper's central contribution is empirical, the lack of uncertainty quantification is a load-bearing issue. Please report means and standard deviations over multiple random seeds and include a significance test (e.g., paired bootstrap or Wilcoxon signed-rank) for the key comparisons against the strongest baselines.","section":"Tables 3 and 4, Sections 4.1 and 4.2"}],"minor_comments":[{"comment":"The notation for the training loss is inconsistent with the preceding derivation: the text uses the previous item i and next item j (with target [j, j(t+Δ−)]), while Eq. (1) sums over interactions (u,i,t,f) with target [i, i(t−)]. This ambiguity makes the exact training objective difficult to reconstruct and should be clarified.","section":"Section 3.3 and Eq. (1)"},{"comment":"The statement that a linear projection layer 'works the best' and that adding non-linearity 'reduce[s] the prediction performance' is made without any supporting ablation or figure. Please provide the ablation or temper the claim accordingly.","section":"Section 3.2"},{"comment":"The claim that t-Batch generates the same embeddings as sequential processing is stated as 'trivial to verify,' but no formal argument or empirical verification is provided. A brief proof sketch or an experiment comparing batched and sequential training outputs would strengthen the reproducibility of the method.","section":"Section 3.4"},{"comment":"The abstract refers to the 'future embedding trajectory of a user/item,' but the projection operation is defined only for users; item embeddings are updated solely at interaction times. Please align the abstract and title wording with the actual model or extend the projection to items.","section":"Abstract and Section 3.2"},{"comment":"The legend for Figure 6 refers to the legend in Figure 5, which is not fully self-contained; adding a direct legend to Figure 6 would improve readability.","section":"Figure 6 and Section 4.5"},{"comment":"The text states that the Reddit dataset includes 1,000 subreddits as items, but Table 6 reports 984 items. Please reconcile this discrepancy.","section":"Table 6 and Section 4.1"},{"comment":"The runtime comparison reports the 9.2x speed-up over DeepCoevolve and an 8.4x speed-up from t-Batch alone, but the difference between these two numbers is not explained; clarifying the relationship would be helpful.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the projection operator is valid and should be a central part of the revision: the paper's title and contribution hinge on trajectory prediction, yet no direct or ablative evidence supports the mechanism. The most serious problem, however, is the unfair state-change comparison due to the auxiliary label loss granted only to JODIE; this should be addressed before the paper can be considered for acceptance. The paper is otherwise well-written and the t-Batch contribution is solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"JODIE is a well-executed empirical paper with a genuine practical contribution, but it oversells the trajectory-prediction mechanism. Read it for the t-Batch batching algorithm and the constant-time inference design; those are reusable and clearly explained.\n\nWhat's new: the mutually-recursive RNN structure is not new (DeepCoevolve already does that), but JODIE adds a projection operator that linearly scales the last observed user embedding with a learned time vector, predicts the next item's embedding directly instead of scoring all items, and introduces t-Batch to create independent, time-consistent batches so the coupled RNNs can be trained in parallel. The paper reports at least 20% MRR improvement over six baselines on three interaction datasets, releases code and data, and gives a clean temporal split for the interaction experiments. The t-Batch parallelization is a solid contribution and the reported 9.2x speedup over DeepCoevolve is credible.\n\nThe soft spots are real. The projection operator is never directly validated. No experiment compares the projected embedding against the user's observed future embedding, so the central claim in the title rests entirely on downstream interaction and state-change numbers. That matters because the prediction layer also feeds in the previous item's dynamic embedding i(t+Δ−), and with 79% of Reddit and 61% of Wikipedia interactions being consecutive repeats, that term could easily dominate. There is no ablation separating the projection's contribution from that shortcut. The state-change comparison is also asymmetric: JODIE is trained with the cross-entropy loss on labels, while baselines are trained on the interaction loss only and then a logistic regression is fit on their embeddings. That gives JODIE an extra label signal, so the 12% AUC improvement is not a fair apples-to-apples comparison. There are no error bars or significance tests.\n\nDespite those issues, the method is sensibly designed and the interaction prediction results are probably robust: the gains are large, the baselines are reasonable, and the temporal split is clean. I would trust that JODIE is a strong recommendation method. What I wouldn't trust is the specific narrative that the projection operator models the embedding trajectory. If I were refereeing, I'd ask for a direct trajectory validation (projected vs. observed embeddings), an ablation without the previous-item embedding, and a state-change protocol where all methods get the same label supervision.\n\nThis is a paper worth engaging with for anyone working on dynamic embeddings or temporal recommendation, especially for the batching technique. It deserves a serious peer review; it just needs an honest rewrite of what the projection actually contributes.","headline":"Well-executed empirical paper with a genuine practical contribution, but the trajectory-projection mechanism is oversold and the state-change comparison is skewed.","tokens_in":16474,"tokens_out":3930,"would_cite":true,"duration_ms":39917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a pair of coupled recurrent networks plus a learned projection operator can predict future user embeddings, and that these predicted trajectories improve next-item and state-change prediction by large margins.","keywords":["temporal interaction networks","dynamic embeddings","embedding trajectories","coupled recurrent neural networks","projection operator","future interaction prediction","state change prediction","t-Batch batching"],"falsifier":"Take a trained JODIE model and, for held-out interactions, compare the projected embedding $\\hat{\\mathbf{u}}(t+\\Delta)$ evaluated at the next interaction time with the actual updated embedding $\\mathbf{u}(t+\\Delta)$ computed by the update RNN. If the average distance from the projection to the actual embedding is no smaller than reusing $\\mathbf{u}(t)$ without projection, the trajectory mechanism is not doing the work attributed to it.","tokens_in":15482,"feed_emoji":"🔮","tokens_out":6540,"duration_ms":65522,"temperature":0.7,"pith_summary":"JODIE is a model for temporal interaction networks, records of who interacted with what and when, in which each user and item is represented by a static part plus a dynamic embedding that moves through time. The central claim is that, after each interaction, a learned projection operator can extrapolate a user's embedding to any future time, and that feeding these projected embeddings to a predictor improves next-item recommendation and user-state-change prediction. The paper reports that JODIE beats six baseline algorithms by at least 20% in mean reciprocal rank for future interaction prediction and by at least 12% on average in AUC for state change prediction, and that a batching scheme makes training 9.2 times faster than the closest comparable method. If true, the contribution is a practical way to make recommendations time-aware rather than frozen at the last observed action.","feed_headline":"Projected future embeddings lift next-item accuracy by 20%","feed_subtitle":"JODIE couples two RNNs to extrapolate user trajectories and predict next interactions and state changes","key_machinery":"The load-bearing object is the projection operator $\\hat{\\mathbf{u}}(t+\\Delta) = (1 + W_p\\Delta) * \\mathbf{u}(t)$, where $\\Delta$ is the time since the last interaction and $W_p$ is a learned linear map from elapsed time to a per-coordinate scaling vector. This converts a discrete sequence of observed embeddings into a continuous trajectory that can be evaluated at any future time. The update operation consists of two mutually-recursive RNNs, one for users and one for items, whose hidden states are the dynamic embeddings and which feed each other's latest states into their updates. The t-Batch algorithm supports the claim by constructing batches in which each user and item appears at most once, so batches can be processed in parallel while preserving temporal order. The projection supplies the predictive signal; the coupled updates supply the state that is projected; t-Batch makes the combined model trainable at scale.","core_discovery":"On the paper's own terms, the discovery is that future embedding trajectories are predictable and useful: a user's dynamic embedding at time $t+\\Delta$ can be estimated from the embedding at $t$ by an elementwise linear scaling $\\hat{\\mathbf{u}}(t+\\Delta) = (1 + W_p\\Delta) * \\mathbf{u}(t)$, and the estimated trajectory is accurate enough to lift downstream prediction. JODIE couples two recurrent networks so that the user update consumes the current item embedding and the item update consumes the current user embedding; the projection operation then turns elapsed time into a multiplicative attention vector. The model is trained to output the embedding of the next item directly, rather than a score over all items, which makes recommendation near-constant time. Alongside this, t-Batch groups interactions into independent, temporally ordered batches so the coupled RNNs can be trained in parallel without breaking time dependencies.","pith_inferences":["Beyond the paper: the projection ansatz could be tested directly against observed trajectories, and if the true drift is curved in embedding space, a second-order or nonlinear time transform should outperform the linear $W_p\\Delta$ scaling.","Beyond the paper: because JODIE outputs an item embedding rather than a score, the same architecture could be applied to temporal knowledge graphs by treating relation types as item-like entities and retrieving predicted tail entities by nearest neighbor.","Beyond the paper: the projection operator turns the last interaction time into a query-time feature, so an online recommender could serve different recommendations depending on how much time has passed since a user's last action, a property not present in the baselines tested, which keep embeddings frozen between interactions."],"forward_implications":["Directly outputting a predicted item embedding instead of interaction scores makes inference near-constant time using locality sensitive hashing, so recommendation scales to millions of items.","t-Batch makes mutually-recursive RNN training parallelizable while preserving temporal order, giving a 9.2x speed-up over the closest comparable method.","State-change prediction, such as predicting bans and student dropout, improves by at least 12% on average in AUC, supporting early intervention applications.","Performance stays stable across different training-data percentages and embedding sizes, suggesting the gains come from the model design rather than from more data or larger embeddings.","Because the model uses both static and dynamic embeddings, predictions retain long-term user identity while also capturing short-term drift."],"supporting_citations":[{"why":"Supplies the closest mutually-recursive RNN baseline that JODIE extends with projection, embedding output, and batching.","marker":"[11]"},{"why":"Recurrent recommender baseline using RNNs for user and item embeddings; it is the direct point of comparison for the reported MRR gains.","marker":"[45]"},{"why":"Baseline and source of the Hadamard-product time-context idea used in the projection operation.","marker":"[8]"},{"why":"Time-aware recurrent baseline that motivates modeling elapsed time and using one-hot static embeddings.","marker":"[52]"},{"why":"Temporal network embedding baseline that produces static embeddings and must be re-run per new edge.","marker":"[33]"},{"why":"Supplies locality sensitive hashing, used for near-constant-time retrieval of the closest item embedding from a predicted embedding.","marker":"[27]"}],"fun_headline_variants":["Future trajectory projection lifts next-item accuracy 20%","Coupled RNNs extrapolate user paths for 20% better prediction","Time-consistent batches give 9x training speedup, same accuracy","Direct next-embedding output makes recommendation near-constant time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central mechanism rests on the assumption that a user's future embedding is an elementwise linear scaling of the last observed embedding by a learned vector that grows linearly with elapsed time; the text reports no direct validation of this projection against observed future embeddings.","fun_headline_variants_meta":{"raw":{"variants":["Future trajectory projection lifts next-item accuracy 20%","Coupled RNNs extrapolate user paths for 20% better prediction","Time-consistent batches give 9x training speedup, same accuracy","Direct next-embedding output makes recommendation near-constant time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1698,"prompt_tokens":982,"completion_tokens":716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":654}},"tokens_in":598,"tokens_out":716,"duration_ms":7566,"temperature":1.0,"reasoning_tokens":654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:20:23.907632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained JODIE model and, for held-out interactions, compare the projected embedding $\\hat{\\mathbf{u}}(t+\\Delta)$ evaluated at the next interaction time with the actual updated embedding $\\mathbf{u}(t+\\Delta)$ computed by the update RNN. If the average distance from the projection to the actual embedding is no smaller than reusing $\\mathbf{u}(t)$ without projection, the trajectory mechanism is not doing the work attributed to it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recurrent recommender baseline using RNNs for user and item embeddings; it is the direct point of comparison for the reported MRR gains."},{"cited_title":"Beutel, P","cited_arxiv_id":null,"evidence_quote":"Baseline and source of the Hadamard-product time-context idea used in the projection operation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Time-aware recurrent baseline that motivates modeling elapsed time and using one-hot static embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Temporal network embedding baseline that produces static embeddings and must be re-run per new edge."},{"cited_title":"Leskovec, A","cited_arxiv_id":null,"evidence_quote":"Supplies locality sensitive hashing, used for near-constant-time retrieval of the closest item embedding from a predicted embedding."}],"review_version":1}