{"id":"5baae78a-50f2-406b-9455-ead70794eb98","arxiv_id":"2411.12179","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"M-GPT combines interaction-level graph convolution with multi-scale session attention and reports state-of-the-art next-purchase accuracy on three multi-behavior datasets.","lead":"This paper introduces M-GPT, a recommendation model that combines an interaction-level behavior graph with a multi-scale transformer to predict what a user buys next. It reports higher hit-rate and ranking scores than prior multi-behavior sequential recommenders on Taobao, IJCAI, and Retailrocket.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set hyperparameter selection in Appendix D and absent error bars leave the claimed universal superiority over all baselines unverified.","rationale":"Reader's verdict is CONDITIONAL and my concern is the same one: the empirical comparison is the load-bearing part, and the manuscript gives insufficient controls. I am not objecting to disagreement with established methods; the architecture description is a plausible integration, and the ablations show each component helps. The problem is that the comparison protocol cannot be audited: no validation split is mentioned, hyperparameters are selected on test metrics, no error bars are given, and code and dataset preprocessing are not released. This matters because the strongest claim is universal metric-wise superiority, and the smallest reported margins (Retailrocket 0.7–3.2%) are exactly where selection bias and variance dominate. A validation-based rerun is the single check that would settle it. I therefore leave the reader's CONDITIONAL verdict unchanged.","tokens_in":19493,"tokens_out":6692,"duration_ms":74254,"concrete_test":"Using the released anonymous code, create a validation split (for example, the last timestamp interaction of a random 10% of users) and repeat the Appendix D hyperparameter search for M-GPT and for the strongest baselines (MBHT, PBAT, MISSL, MB-STR) using validation HR@5. Then freeze each selected model and evaluate on the original leave-one-out test users across five seeds, reporting mean and standard deviation. If the validation-selected M-GPT margin over the best validation-selected baseline on Retailrocket is within one standard deviation, or the Taobao HR@5 gain drops below roughly 5%, the reported universal superiority is not supported; authors should also disclose whether ranking is over the full item set or a sampled candidate set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.2 reports leave-one-out evaluation and averages five runs, but no standard deviations or significance tests are provided. Appendix D selects M-GPT's mask ratio, dependency order, session-scale pair, and preference-granularity pair by the best HR@5/NDCG@5 on the same three datasets that appear in Table 2; no separate validation split is described. Because Table 2 is evaluated on the exact metrics used for selection, the M-GPT column is a post-selection maximum rather than an unbiased estimate. The searched grid (5 mask ratios × 4 orders × 4 session settings × 4 granularity settings) is large relative to Retailrocket's 11,649 users, where the reported gains over MBHT are only 0.7–3.2%. Section 4.1.4's statement that baseline parameters were refined is not accompanied by search ranges or a common validation protocol, so asymmetric tuning cannot be excluded. If M-GPT was tuned on test metrics while baselines were not, the headline 12.2% Taobao improvement and the smaller Retail gains could shrink or disappear. This directly affects the paper's central claim that M-GPT 'outperforms all the baselines in terms of all metrics.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M-GPT, a multi-behavior sequential recommendation model with two main components: an interaction-level dependency extractor (IDE) that builds a fully connected graph over historical interactions and applies multi-order graph convolution, and a multifaceted sequential pattern generator (MSPG) that combines linear self-attention with multi-grained, multi-scale session-based preference encoding. The model is evaluated on Taobao, IJCAI, and Retailrocket against a large set of baselines, with reported improvements of up to 12.2% in HR@5 over the strongest prior method. The central claim is that M-GPT outperforms all baselines on all metrics on these datasets.","tokens_in":19800,"tokens_out":3856,"duration_ms":40853,"significance":"If the empirical claim holds, M-GPT is a useful contribution to multi-behavior sequential recommendation: it addresses a genuinely underexplored modeling level (interaction-level dependencies), incorporates multi-grained temporal preference in a principled way, and is evaluated on three real-world datasets against 22 baselines with ablations and a public code link. The time-complexity analysis is also a strength. However, the current experimental evidence is not sufficient to support the headline claim of universal superiority, because the model's hyperparameters are selected on the same test metrics used for the final comparison and no measure of variance or significance is reported. The contribution is therefore plausible but not yet verified at the standard required for the claimed benchmark result.","major_comments":[{"comment":"The hyperparameters of M-GPT are selected by the best HR@5 and NDCG@5 on the same datasets that are then used for the headline comparison in Table 2. Appendix D reports choosing the mask ratio, dependency order, session-scale pair, and preference-granularity pair by best performance on Taobao, IJCAI, and Retailrocket, with no separate validation split described. As a result, the M-GPT column in Table 2 is a post-selection maximum rather than an unbiased estimate. This matters particularly on Retailrocket, where the reported gains over MBHT are only 0.7–3.2% depending on the metric. The authors should either evaluate on a held-out test set after selecting hyperparameters on a validation set, or report the selection procedure in a way that allows the reported numbers to be interpreted as unbiased estimates.","section":"Section 4.1.2 / Table 2 / Appendix D"},{"comment":"The paper reports that \"we select the average performance of five times\" but gives no standard deviations, confidence intervals, or significance tests for any of the results in Table 2. The claim that M-GPT \"outperforms all the baselines in terms of all metrics\" cannot be verified from averages alone, especially for the small Retailrocket margins. The authors should report variance across runs and perform statistical significance tests for the comparisons against the strongest baselines, or at minimum report per-run results so the stability of the differences can be assessed.","section":"Section 4.1.2 / Table 2"},{"comment":"The statement \"we refine the parameter setting of each model to get the best performance\" is not accompanied by search ranges, protocols, or the actual settings used for the baselines. Without a common validation criterion and per-baseline search budgets, asymmetric tuning cannot be excluded, and the fairness of the comparison in Table 2 is not established. The authors should specify the hyperparameter search space for each baseline and confirm that the same validation/test separation was used for baseline tuning as for M-GPT.","section":"Section 4.1.4"},{"comment":"The graph convolution definition is not reproducible as written. Equation (7) defines \\tilde D^{-1/2} as I + D^{-1/2} A D^{-1/2}, which is an additive normalization rather than an inverse square root of a degree matrix, while Eq. (6) uses \\tilde D^{-1/2} \\tilde A \\tilde D^{-1/2} without defining \\tilde A. This needs to be corrected or clarified, since the interaction-level dependency extractor is a core component of the proposed model.","section":"Section 3.3.3, Eqs. (6)–(7)"},{"comment":"The ablation study concludes that \"there exist a significant performance degradation\" when a component is removed, but no standard deviations or significance tests are reported for the ablation results, and Retailrocket is omitted from Table 3. Some ablation gaps are small (e.g., 0.358 to 0.369 in HR@5 on Taobao for the behavior-level variant), so the claim of significance is not supported. Reporting variance and significance, or at least softening the language, is necessary.","section":"Section 4.3 / Table 3"}],"minor_comments":[{"comment":"Table 2 is titled \"Experimental results on two datasets\" but contains three datasets, and Section 4.1.1 says \"we select two datasets\" before listing three. Please unify the wording.","section":"Title of Table 2 and Section 4.1.1"},{"comment":"The conclusion says \"we conduct comprehensive experiments on two public datasets,\" but the experiments use three public datasets. This should be corrected.","section":"Section 5"},{"comment":"In the loss definition, the summation index l is used both as the dependency-order index and as the upper limit of the sum, which is confusing. Using an explicit L for the maximum order and a distinct summation index would improve clarity.","section":"Section 3.5.2, Eq. (24)"},{"comment":"The text in Section 4.4 refers to \"figure 5\" when describing the behavior-pair matrices, but Figure 4 is the behavioral sequential pattern figure and Figure 5 is the multi-order dependency score figure. The figure references appear to be swapped and should be fixed.","section":"Section 4.4 and Figures 4–5"},{"comment":"There are recurrent typos and formatting issues, including \"mulit-behavior dependence\" in Section 4.5, \"we just we just\" in the ablation variant description, \"evovling\" in Appendix C, and the repeated \"NDGC@5\" instead of \"NDCG@5\" in Table 2 and Appendix D. These should be corrected in a final pass.","section":"Throughout the paper"}],"recommendation":"major_revision","confidential_remarks":"The central issue is empirical verification rather than the modeling idea itself. I would not ask for new model components, but the benchmark claim needs a proper validation protocol, variance reporting, and a documented baseline-tuning procedure. If the authors can provide those, the paper could be acceptable; without them, the claimed universal superiority is not established. The manuscript also appears to be an anonymized submission in a conference template, which is not itself a problem, but the missing significance testing and test-set hyperparameter selection are."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a credible engineering contribution to multi-behavior sequential recommendation, and the interaction-level dependency graph is a real incremental idea. The central claim that M-GPT beats every baseline on every metric, though, is not verifiable from the manuscript: the model's hyperparameters are chosen on the same test metrics that appear in Table 2, and no error bars are reported.\n\nWhat's actually new: unlike MB-GMN and MBHT, which model dependencies at behavior or item level, M-GPT builds a per-sequence graph over interactions with edge weights from the product of item-pair and behavior-pair similarities, then runs graph convolution over multiple orders. That is a reasonable way to capture cross-type interaction dependencies. The multi-grained session queries are borrowed from Atten-Mixer [42] and combined with two time scales and linear attention; the combination is new even if the ingredients are known. The paper ships an anonymous code link, reports three datasets, compares a lot of baselines, and runs several ablations. The ablations in Table 3 show each module contributes, which is genuine supporting evidence.\n\nThe soft spots are real. Section 4.1.2 says they average five runs but gives no standard deviations or significance tests, so the 0.7–3.2% Retailrocket improvements are hard to take at face value. Appendix D selects mask ratio, dependency order, session scales, and preference granularity by looking at HR@5/NDCG@5 on the same three datasets that produce the headline numbers; no validation split is described. That makes the M-GPT column a post-selection maximum, not an unbiased estimate. The baseline statement in 4.1.4 (\"we refine the parameter setting of each model\") is not backed by search ranges or a common protocol, so asymmetric tuning cannot be ruled out. The anonymous code was not inspected. Minor: the conclusion says \"two public datasets\" after using three, which is a sign of haste.\n\nProportionately, this is not a fatal flaw. The architecture is coherent, the improvements on Taobao and IJCAI are large and consistent, and even on Retailrocket the pattern is monotonic across most metrics. But the paper's strongest sentence — \"M-GPT outperforms all the baselines in terms of all metrics\" — is stronger than the evidence supports.\n\nFor a reader in recommender systems: worth one careful read as a modular recipe and as a cautionary example of test-set tuning in this literature. I'd send it to peer review, not desk reject, and ask for a validation split, error bars, a baseline tuning protocol, and code inspection before accepting. It could become a solid SOTA paper after that.","headline":"Plausible incremental architecture with a genuinely new interaction-level graph, but the universal-SOTA claim is unverified because the model is tuned on test metrics and no error bars are reported.","tokens_in":20281,"tokens_out":3093,"would_cite":true,"duration_ms":34760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M-GPT predicts next purchases by modeling interaction-level dependencies and multi-grained preferences, beating prior methods on Taobao, IJCAI, and Retailrocket.","keywords":["sequential recommendation","multi-behavior recommendation","interaction-level dependency","graph neural network","transformer","multi-grained preference","next-purchase prediction","temporal preference"],"falsifier":"Re-run the comparison on the same three datasets with a shared tuning budget and a fixed validation protocol for every model, and check whether M-GPT's Taobao Hit Rate@5 advantage over the second-best method remains near 12.2%; if a controlled re-implementation shrinks the gap to a few points, the paper's central claim of consistent state-of-the-art performance would not hold.","tokens_in":19311,"feed_emoji":"🛒","tokens_out":9794,"duration_ms":90467,"temperature":0.7,"pith_summary":"This paper proposes M-GPT, a model for multi-behavior sequential recommendation, and argues that next-purchase prediction improves when the model captures two things earlier methods miss: dependencies between individual interactions of different behavior types and items, and user preference at several time granularities. The authors build an interaction-level graph whose edges depend on both item and behavior embeddings, then pass it through multi-order graph convolution; a transformer with linear self-attention and multi-grained multi-head attention over sessions encodes the sequence. On Taobao, IJCAI, and Retailrocket, they report that M-GPT beats all compared baselines on every metric, with Hit Rate@5 gains up to 12.2% relative to the best baseline on Taobao. A reader should care because the recipe (learnable interaction graph plus multi-scale attention) is a transferable design for any recommendation setting where users act in multiple behavior types.","feed_headline":"Graph-transformer hybrid lifts next-purchase predictions by 12.2%","feed_subtitle":"Modeling cross-type interaction dependencies and multi-grained preference beats strong baselines on three shopping datasets.","key_machinery":"The load-bearing object is the interaction-level dependency graph, with incidence matrix entries built by combining item-level and behavior-level signals from each interaction, followed by symmetric-normalized graph convolution with a self-loop to model dependencies of increasing order. An entry $A_{i,j}$ is derived from element-wise products of item and behavior embeddings for interactions $i$ and $j$; repeated graph convolution produces dependency representations $H^{(l)}$ for orders $l=0,...,L$. The second mechanism is the multi-grained multi-head self-attention: within sessions at two time scales, queries are formed from the last one, two, up to $g$ interactions to capture different granularities of intent, pooled, and fused with the global linear-attention pattern. Together they let the model attend over both cross-type interaction structure and temporally localized preference, which the paper identifies as the two previously missing ingredients.","core_discovery":"M-GPT's central claim is that representing a user's history as a graph of interactions, rather than only behaviors or only items, and extracting multi-grained preferences from sessions at different time scales yields better next-purchase predictions than existing multi-behavior sequential recommenders. The incidence matrix of the graph is computed from paired item and behavior embeddings, and repeated graph convolution produces dependency representations at several orders; these are fed, together with global linear self-attention and session-level multi-grained attention outputs, into prediction. The authors report consistent wins over all baselines on Taobao, IJCAI, and Retailrocket, and their ablations attribute the gain to the interaction-level graph and the multi-grained preference module. In the paper's own framing, modeling interaction-level correlation and temporal multi-grained preference jointly is what closes the gap.","pith_inferences":["The interaction-level incidence-matrix construction is a general plug-in: it could be applied to other multi-behavior problems such as cart-abandonment prediction or session-level sales forecasting without changing the attention backbone.","Because the multi-grained preference module is largely behavior-agnostic, it may also improve single-behavior sequential recommenders; a testable check is to add multi-grained multi-head self-attention to a plain transformer on a single-behavior dataset and compare with strong single-behavior baselines.","The reported time complexity suggests the approach can scale to sequences longer than the $N=200$ used here; an extension would measure whether the Taobao gains persist when histories are truncated at 1,000 interactions."],"forward_implications":["On the three datasets tested, M-GPT reports higher HR, NDCG, and MRR than every baseline, including the strongest prior multi-behavior methods; the largest reported gain is a 12.2% relative improvement in Hit Rate@5 on Taobao.","Modeling interaction-level dependencies contributes beyond behavior-level and item-level dependencies: ablations that remove the interaction-level graph or replace the incidence matrix with behavior-only or item-only versions degrade scores on Taobao and IJCAI.","Capturing multi-grained preference at two time scales contributes beyond global sequential encoding: ablations that drop the multi-grained multi-head self-attention or keep only one granularity lower performance.","MaxPooling across dependency orders is the paper's chosen fusion; replacing it with attention-weighted summation hurts accuracy.","The linear self-attention formulation keeps time complexity at $O(N d^2)$ rather than $O(N^2 d)$, which matters for long interaction sequences."],"supporting_citations":[{"why":"Supplies the preprocessed Taobao, IJCAI, and Retailrocket datasets the evaluation follows, and is the strongest prior baseline (MBHT) that M-GPT must beat.","marker":"[36]"},{"why":"Defines the item-level multi-behavior dependency modeling that M-GPT contrasts with its interaction-level graph; MB-STR is also a direct baseline.","marker":"[40]"},{"why":"Provides behavior-level dependency modeling via a graph meta-network and serves as a baseline that M-GPT claims to outperform.","marker":"[34]"},{"why":"Establishes graph convolutional multi-behavior recommendation at the behavior level and serves as a core comparison point.","marker":"[12]"},{"why":"Source of the Cloze/masked-item training objective M-GPT adopts, and a sequential recommendation baseline.","marker":"[24]"},{"why":"Supplies the linear attention mechanism that M-GPT uses to encode the global sequential pattern at $O(N d^2)$.","marker":"[18]"},{"why":"Inspires the multi-grained query construction for user intent that M-GPT adapts into its multi-grained multi-head self-attention.","marker":"[42]"}],"fun_headline_variants":["Interaction-level graph and multi-grained preference boost next-purchase predictions","Multi-scale transformer with graph-captured cross-type dependencies wins in MBSR","M-GPT: modeling interaction-level dependencies and multi-grained preferences","Graph of interactions plus multi-grained attention sharpens sequential recommendation","New hybrid: interaction graph + multi-grained preference transformer for next-buy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed edge rests on a fair comparison: if the baselines were not tuned as intensively as M-GPT, or if M-GPT's hyperparameters were chosen using test-set feedback, the reported improvements could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Interaction-level graph and multi-grained preference boost next-purchase predictions","Multi-scale transformer with graph-captured cross-type dependencies wins in MBSR","M-GPT: modeling interaction-level dependencies and multi-grained preferences","Graph of interactions plus multi-grained attention sharpens sequential recommendation","New hybrid: interaction graph + multi-grained preference transformer for next-buy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2823,"prompt_tokens":932,"completion_tokens":1891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1800}},"tokens_in":548,"tokens_out":1891,"duration_ms":15681,"temperature":1.0,"reasoning_tokens":1800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:49:55.857298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on the same three datasets with a shared tuning budget and a fixed validation protocol for every model, and check whether M-GPT's Taobao Hit Rate@5 advantage over the second-best method remains near 12.2%; if a controlled re-implementation shrinks the gap to a few points, the paper's central claim of consistent state-of-the-art performance would not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides behavior-level dependency modeling via a graph meta-network and serves as a baseline that M-GPT claims to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes graph convolutional multi-behavior recommendation at the behavior level and serves as a core comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the multi-grained query construction for user intent that M-GPT adapts into its multi-grained multi-head self-attention."}],"review_version":1}