{"id":"e0ff7ae2-1184-4499-98b1-822ece8e642c","arxiv_id":"2607.19385","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"STN-TGAT, a Transformer plus graph attention model with an NMI relationship prior and learnable soft-threshold sparsification, shows higher backtested risk-adjusted returns than GRU, LSTM, and graph baselines on a Top-5 S&P 500 portfolio after transaction costs, but not higher ranking accuracy.","lead":"This paper proposes a deep learning model that combines a Transformer with a graph attention network, an information-based stock relationship map, and a learnable filter to pick a Top-5 portfolio from the 50 largest S&P 500 stocks each trading day. The model reports higher backtested profits and Sharpe ratios than several baselines, though not higher raw ranking accuracy than an LSTM.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The identical STN-TGAT configuration (ListNet+MSE) is reported with different IRR/Sharpe in Tables 1 and 2 (0.1807/2.9940 vs 0.2132/3.3130); until reconciled, the central profitability claim lacks a single reproducible result.","rationale":"The reader's CONDITIONAL verdict is based on statistical weakness and a missing market benchmark; those concerns are valid, but the table discrepancy is the most load-bearing because it is an internal inconsistency, not an external judgment about noise. The strongest claim is empirical: STN-TGAT outperforms baselines on net-of-fee IRR and Sharpe. If the same configuration yields materially different headline numbers in two tables, the reported superiority cannot be evaluated reliably. I considered other candidate concerns: the short 75-day test window, overlapping confidence intervals, absence of an S&P 500 buy-and-hold baseline, and the abstract's claim of consistent predictive-accuracy outperformance despite LSTM's higher MRR/RBO. These are real and should be addressed, but each is a robustness/interpretation issue. The Table 1/Table 2 discrepancy is a concrete reproducibility defect in the evidence itself. If the discrepancy is due to random seed variation, the paper should state the seed protocol and show per-run results; if it is a typo, the canonical table must be corrected. Either way, the central claim is unsupported by a single consistent set of numbers as currently printed. I therefore keep the reader's CONDITIONAL verdict unchanged, with the added explicit condition that the authors reconcile Tables 1 and 2 and report the S&P 500 benchmark.","tokens_in":10967,"tokens_out":7881,"duration_ms":66698,"concrete_test":"Re-run the exact STN-TGAT configuration from Table 1 (Transformer-GAT with NMI prior, learnable soft-threshold gating, top-weighted ListNet+MSE, same 8:1:1 chronological split, same seed protocol, same backtester with transaction costs) and recompute the Table 2 'ListNet+MSE' row under that identical protocol. If the recomputed row matches Table 1 (0.1807 IRR, 2.9940 Sharpe), Table 2's discrepancy is a reporting error; if it matches Table 2 (0.2132, 3.3130), Table 1's headline is not reproducible. The paper must state which table is canonical and why, and should report per-seed results to show that the difference is not just seed noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that STN-TGAT achieves the best net-of-fee Top-5 profitability among all baselines. That claim is supported only by the portfolio metrics in the empirical tables, so those numbers must be a stable representation of one well-defined configuration. Table 1 reports the final model (top-weighted ListNet + MSE, NMI prior, soft-threshold sparsification) as IRR = 0.1807, Sharpe = 2.9940. In Section 6.2, the loss-function ablation of the same framework reports the same 'ListNet+MSE' configuration as IRR = 0.2132, Sharpe = 3.3130. The text identifies ListNet+MSE as the final objective, so these two rows should be identical. They differ by 3.25 percentage points of IRR and 0.319 in Sharpe. No explanation is given (different seeds, different test split, or a typo). While the standard deviations in Tables 1 and 2 are large, the discrepancy affects the reported means, which are the basis of the headline 'highest IRR and Sharpe' claim. This internal inconsistency is more fundamental than the short-window/noise concern: it means the manuscript currently presents two incompatible realizations of its flagship result and the reader cannot determine which, if either, is canonical. I am not alleging misreporting; I am identifying that the evidence, as printed, does not yet support a single reproducible outperformance claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STN-TGAT, a hybrid Transformer/GAT model for Top-K portfolio construction. A temporal Transformer encodes daily features, an NMI-based prior graph with learnable soft-threshold sparsification guides a GAT module, and a head-weighted ListNet loss combined with MSE and graph regularization is used for training. The model is evaluated on 50 S&P 500 constituents over 752 trading days (2022-2024) with a Top-5 long-only net-of-fee backtest. The paper claims the highest IRR (18.07%) and Sharpe ratio (2.99) among baselines, while acknowledging that LSTM achieves higher values on some pure ranking metrics. Ablations study the loss components, the NMI prior, sparsification, and within-portfolio weighting.","tokens_in":11386,"tokens_out":3359,"duration_ms":27420,"significance":"If the reported gains are reproducible, the paper makes a useful contribution to decision-aware ranking for portfolio construction: the explicit Top-K formulation, differentiable ranking surrogate, adaptive graph sparsification, and net-of-fee evaluation are appropriate and practical. The paper's strengths include a clean problem formulation, multiple ablations isolating loss, prior, sparsification, and allocation choices, and an explicit treatment of transaction costs. The stress-test concern about circularity is correctly rejected: the NMI prior is computed from training returns and the threshold is learned, so no self-referential construction is present. However, the empirical foundation is weakened by an internal inconsistency between Table 1 and Table 2 for the same configuration, an unsupported 'consistently outperforms' claim in the abstract, and the absence of significance testing and an index benchmark.","major_comments":[{"comment":"The identical configuration 'ListNet+MSE' is reported with different results in Table 1 and Table 2: IRR 0.1807±0.1176 / Sharpe 2.9940±1.3287 in Table 1 versus IRR 0.2132±0.1309 / Sharpe 3.3130±1.5240 in Table 2. Since ListNet+MSE is identified as the final training objective, these two rows should be identical. The discrepancy of 3.25 percentage points in IRR and 0.319 in Sharpe directly affects the headline 'highest IRR and Sharpe' claim. Please reconcile the two tables, state which set of numbers is canonical, and explain the source of the difference (e.g., different seeds, different test split, or a typo).","section":"§6.1 Table 1 vs §6.2 Table 2"},{"comment":"The abstract states that STN-TGAT 'consistently outperforms benchmark models from predictive accuracy and investment profitability,' but Table 1 shows LSTM achieving higher MRR (0.2222 vs 0.1879) and RBO (0.0601 vs 0.0509) than STN-TGAT. The text in §6.1 acknowledges LSTM is higher on 'certain pure ranking metrics,' which contradicts the abstract's consistency claim. Please qualify the claim to reflect that the advantage is specifically in portfolio profitability, not in pure ranking accuracy.","section":"§1 Abstract and §6.1 Table 1"},{"comment":"No significance tests are provided, and the test window is approximately 75 trading days (10% of 752 days). The Sharpe difference between STN-TGAT (2.9940±1.3287) and GRU (2.6553±0.5632) overlaps within one standard deviation, so the claimed risk-adjusted outperformance may be noise. Additionally, no S&P 500 buy-and-hold baseline is reported, leaving the absolute returns uncontextualized. Please add hypothesis tests or bootstrap confidence intervals for the return differences, and include a broad index benchmark to make the profitability claim falsifiable.","section":"§5.3 Evaluation and Table 1"}],"minor_comments":[{"comment":"The heading 'Ranking Accuracy and Investment Performance Comparision' contains a typo: 'Comparision' should be 'Comparison'.","section":"§6.1 heading"},{"comment":"The caption contains the stray phrase 'check the left tables,' which appears to be an editorial note and should be removed.","section":"Table 1 caption"},{"comment":"The sentence 'tg∈(0,1) is a learnable threshold parameterized' is missing a word; it should read 'a learnable threshold parameter' or 'a parameterized learnable threshold.'","section":"Eq. (5)"},{"comment":"The text contains the typo 'LST Mattains' instead of 'LSTM attains'.","section":"§6.1"},{"comment":"The same symbol Aϕ is used both before and after symmetrization and max-scaling; please clarify the notation to avoid ambiguity about which matrix is used in Eq. (7).","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between Table 1 and Table 2 for the same configuration is the main technical barrier to acceptance. If the authors cannot reconcile these numbers and identify a single reproducible realization of the flagship result, the central profitability claim is not supported by the evidence as printed. The paper would also benefit from a data and code availability statement to support reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest, workmanlike ML-for-finance paper with a real internal inconsistency in its flagship numbers. The same STN-TGAT configuration (ListNet+MSE) is reported with IRR 0.1807 / Sharpe 2.994 in Table 1 and IRR 0.2132 / Sharpe 3.313 in Table 2. The paper never explains this, and it lands squarely on the headline “highest IRR and Sharpe” claim. Until that is reconciled, the central profitability result does not have a single reproducible form.\n\nWhat is genuinely new: the soft-threshold gating on an NMI prior graph, combined with a geometrically top-weighted ListNet objective, is a fair combination I haven't seen in the stock-ranking literature. The authors also did the unglamorous work of evaluating net-of-fee Top-5 portfolios with turnover costs, which is the right setup for the task. The method section is clear enough to reproduce in principle, though no code or data is provided.\n\nThe soft spots, in descending order of seriousness:\n1. The Table 1/Table 2 discrepancy is load-bearing. It is not a minor formatting issue; it changes the headline result by 3.25 points of IRR and 0.32 of Sharpe. A referee needs an explanation or a corrected table.\n2. The abstract claims the model “consistently outperforms” baselines on predictive accuracy and profitability, but Table 1 shows LSTM beats STN-TGAT on MRR (0.2222 vs 0.1879) and RBO (0.0601 vs 0.0509). That overclaim should be fixed.\n3. The test window is roughly 75 trading days. The Sharpe difference versus GRU (2.99 vs 2.66) is within one standard deviation, and no significance tests are reported. The results are suggestive, not conclusive.\n4. No S&P 500 buy-and-hold benchmark, so absolute returns are not contextualized.\n5. Minor: the sparsity regularizer in Eq. 14 is not ablated separately; the “no sparsity” variant in Table 3 removes the gating mechanism, which is related but not identical.\n\nWho this is for: people working on graph-based stock ranking will find the loss weighting and the thresholded NMI prior worth a careful look. It will not change how the field thinks about portfolio construction.\n\nRecommendation: send it to peer review, but with a clear request to reconcile the table discrepancy, add a market benchmark, and soften the abstract. If the discrepancy is a typo, this is a decent incremental contribution; if not, the profitability claim does not stand. Either way, the paper needs a serious referee, not a desk reject.","headline":"A workmanlike incremental stock-ranking paper whose headline profitability claim is currently undermined by an unexplained discrepancy between Table 1 and Table 2 for the same model configuration.","tokens_in":11799,"tokens_out":4147,"would_cite":false,"duration_ms":33259,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STN-TGAT beats stock-picking baselines by training on top-ranked picks with an adaptive NMI-prior graph.","keywords":["stock ranking","portfolio construction","graph attention network","transformer","normalized mutual information","soft-threshold sparsification","top-k selection","ListNet"],"falsifier":"Run the identical Top-5, net-of-fee backtest on a longer out-of-sample window after the paper's data cutoff (e.g., all of 2025), and compare the Sharpe ratio of STN-TGAT against the GRU baseline; if the gap shrinks below one standard deviation or reverses, the central claim of consistent outperformance fails. A simpler check: report the same backtest with an S&P 500 buy-and-hold portfolio to see whether the absolute returns represent market drift.","tokens_in":10739,"feed_emoji":"📈","tokens_out":6436,"duration_ms":51068,"temperature":0.7,"pith_summary":"The paper argues that for realistic Top-K portfolio construction, training a model to rank stocks correctly near the top of the list matters more than predicting returns accurately, and that the graph connecting stocks should be learned from a nonlinear dependence prior with adaptive sparsification rather than fixed or dense. It proposes STN-TGAT, which combines a temporal Transformer, a graph attention network guided by a Normalized Mutual Information prior, and a learnable soft-threshold gate that suppresses weak edges. In net-of-fee backtests on the 50 largest S&P 500 constituents, the model selects a Top-5 portfolio that beats all compared baselines on IRR and Sharpe ratio. The practical stakes: a decision-aligned ranking loss plus a sparsified relational prior can turn noisy market data into investable stock picks.","feed_headline":"Rank-first stock model hits Sharpe 2.99 in Top-5 backtest","feed_subtitle":"STN-TGAT combines a top-weighted ranking loss with an adaptive NMI graph to pick the S&P 500's best five.","key_machinery":"The load-bearing mechanism is the prior-guided, sparsified graph: an adjacency matrix $A_0$ from Normalized Mutual Information between stock return series is gated by $g_{ij} = \\sigma(\\beta((A_0)_{ij} - t_g))$ with learnable threshold $t_g$, yielding $A_\\phi = A_0 \\odot G$; this sparsified adjacency is injected into GAT attention logits as $\\lambda \\log((A_\\phi)_{ij} + \\varepsilon)$, so the data-driven attention is biased toward edges the NMI prior supports. Training uses a top-weighted ListNet loss that down-weights lower-ranked positions geometrically, an auxiliary MSE term that anchors score magnitudes, and a graph sparsity regularizer. The gating mechanism is what makes the graph adaptive: it smoothly attenuates weak dependencies rather than hard-thresholding them, and it is learned jointly with the prediction objective.","core_discovery":"On the paper's own terms, the central claim is that combining a head-weighted ListNet ranking objective with a graph attention network whose adjacency is an NMI-based prior refined by a learnable soft-threshold gate yields the best net-of-fee Top-5 portfolio among all tested models: 18.07% IRR and a 2.99 Sharpe ratio, versus 13.41% and 2.66 for the strongest recurrent baseline (GRU). The ablations support the mechanism: removing the NMI prior drops IRR from 18.07% to 8.63%, replacing NMI with GLASSO drops it to 3.81%, and disabling sparsification lowers Sharpe from 2.99 to 2.59. The paper concludes that decision-aligned training with adaptive relational modeling is a coherent and practical framework for data-driven portfolio construction.","pith_inferences":["If the ranking-loss results generalize, the design principle—optimize for the head of the ranking, not for full-list agreement—should transfer to other decision-centric learning tasks such as recommendation, search, or candidate screening, regardless of the encoder architecture.","The reported Sharpe gap between STN-TGAT and GRU (2.99 vs 2.66) sits within one standard deviation, and the test window covers only about 75 trading days; a longer or repeated out-of-sample evaluation could decide whether the gap is signal or noise.","The absence of a buy-and-hold S&P 500 benchmark leaves the absolute 18.07% IRR unanchored; adding such a baseline would clarify whether the model earns an excess return or mostly captures market drift in the test period.","The soft-threshold gating of an NMI prior is a general recipe for turning dense correlation matrices into adaptive sparse graphs, so it could be applied to other financial settings (e.g., sector rotation, credit networks) or other domains with noisy relational measurements."],"forward_implications":["A top-weighted listwise loss is the main driver of portfolio performance: swapping it for pointwise or pairwise losses cuts IRR roughly by a factor of two to three in the paper's ablations, so ranking objectives should be chosen with the investment decision in mind.","Learnable soft-threshold sparsification improves risk-adjusted returns over a fixed NMI graph (Sharpe 2.99 vs 2.59), so adaptive graph density is a transferable component for noisy relational data.","The nonlinear NMI prior outperforms a linear GLASSO prior by a wide margin (Sharpe 2.99 vs 0.75), suggesting that linear dependence measures may miss relations relevant to Top-K selection.","Score-based weighting within the Top-5 set beats equal weighting (IRR 0.181 vs 0.152), implying predicted score magnitudes carry investable information beyond the ordering."],"supporting_citations":[{"why":"Supplies the NMI-based relational graph construction that serves as the prior in the paper's graph module.","marker":"[5]"},{"why":"Provides the Transformer encoder used for temporal modeling of each stock's lookback sequence.","marker":"[24]"},{"why":"Provides the graph attention network operator that aggregates neighbor information with the injected prior.","marker":"[25]"},{"why":"Supplies the Rank-Biased Overlap metric and the listwise ranking perspective used to evaluate ranking quality.","marker":"[19]"},{"why":"Supports the use of Mean Reciprocal Rank as a ranking metric and motivates graph-based stock recommendation.","marker":"[6]"},{"why":"Provides the relational temporal graph convolutional ranking baseline and the MRR evaluation protocol.","marker":"[35]"}],"fun_headline_variants":["STN-TGAT delivers 2.99 Sharpe on Top-5 S&P 500 portfolio","Learnable soft-threshold NMI graph lifts Top-5 IRR to 18.07%","Graph attention with NMI prior beats GRU: Sharpe 2.99 vs 2.66","Sparsified NMI graph drives Top-5 portfolio to 18.07% IRR","NMI prior + sparsification: Top-5 picks yield Sharpe 2.99"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed outperformance rests on the assumption that the short 75-day test window is representative and that the observed differences in IRR and Sharpe ratio are economically meaningful rather than statistical noise, despite overlapping standard deviations and no buy-and-hold market baseline.","fun_headline_variants_meta":{"raw":{"variants":["STN-TGAT delivers 2.99 Sharpe on Top-5 S&P 500 portfolio","Learnable soft-threshold NMI graph lifts Top-5 IRR to 18.07%","Graph attention with NMI prior beats GRU: Sharpe 2.99 vs 2.66","Sparsified NMI graph drives Top-5 portfolio to 18.07% IRR","NMI prior + sparsification: Top-5 picks yield Sharpe 2.99"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2092,"prompt_tokens":910,"completion_tokens":1182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1060}},"tokens_in":526,"tokens_out":1182,"duration_ms":9701,"temperature":1.0,"reasoning_tokens":1060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:36:00.890628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical Top-5, net-of-fee backtest on a longer out-of-sample window after the paper's data cutoff (e.g., all of 2025), and compare the Sharpe ratio of STN-TGAT against the GRU baseline; if the gap shrinks below one standard deviation or reverses, the central claim of consistent outperformance fails. A simpler check: report the same backtest with an S&P 500 buy-and-hold portfolio to see whether the absolute returns represent market drift.","supporting_citations":[{"cited_title":"Relation-aware dynamic attributed graph attention network for stocks rec- ommendation.Pattern Recognition, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the NMI-based relational graph construction that serves as the prior in the paper's graph module."},{"cited_title":"Attention is all you need.Advances in neural information processing systems, 2017","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer encoder used for temporal modeling of each stock's lookback sequence."},{"cited_title":"Graphattentionnetworks","cited_arxiv_id":null,"evidence_quote":"Provides the graph attention network operator that aggregates neighbor information with the injected prior."},{"cited_title":"Stock ranking prediction using list-wise approach and node embedding technique.IEEE Access, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the Rank-Biased Overlap metric and the listwise ranking perspective used to evaluate ranking quality."},{"cited_title":"Graph-based stock recommendation by time-aware relational attention network.ACM Transactions on Knowledge Discovery from Data (TKDD), 2021","cited_arxiv_id":null,"evidence_quote":"Supports the use of Mean Reciprocal Rank as a ranking metric and motivates graph-based stock recommendation."},{"cited_title":"Relational temporal graph convolutional networks for ranking-based stock prediction","cited_arxiv_id":null,"evidence_quote":"Provides the relational temporal graph convolutional ranking baseline and the MRR evaluation protocol."}],"review_version":1}