{"id":"d4e276bb-76e8-4761-b002-3da7a96c99b7","arxiv_id":"1908.07999","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HATS selectively aggregates corporate-relation information and is reported to outperform its stated baselines for stock and index movement prediction, with caveats about experimental rigor.","lead":"This paper introduces HATS, a two-level graph attention network that weighs different corporate relationships when predicting whether stocks and market indexes will rise or fall. It reports gains over baselines on S&P 500 data, but the evidence is weakened by duplicated result rows and missing statistical tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported outperformance is not statistically substantiated: duplicate table rows, missing error bars, and an unsupported 19.8% Sharpe claim leave the headline margin within noise.","rationale":"The key claim of the paper is a quantitative performance advantage. The graph-construction limitation the reader flags is real and acknowledged by the authors, but since all compared methods use the same graph, it does not invalidate the head-to-head comparison. The more load-bearing concern is whether HATS's margins over baselines are real or noise. This is decided by the experimental tables, and they have serious integrity problems: duplicate rows, no variance information, and an abstract number that doesn't match the tables. These issues directly undermine the central claim. My recommendation remains CONDITIONAL: the paper's architecture is plausible and the code is available, but the empirical evidence must be repaired before the claim can be trusted. I agree with the reader's overall conditional verdict, but disagree that the graph completeness is the single weakest point; the statistical substantiation of the headline numbers is more critical.","tokens_in":19091,"tokens_out":6790,"duration_ms":65794,"concrete_test":"Recompute the aggregated metrics after removing the duplicate Phase 7 and 8 rows, and report standard deviations and paired significance tests (e.g., paired t-test across the 10 unique phases for HATS vs. each baseline) for both F1 and Sharpe. Also, trace the abstract's 19.8% Sharpe improvement: specify the baseline and formula (e.g., (SR_HATS - SR_baseline)/SR_baseline or difference in Sharpe ratios); if the number cannot be reproduced, correct the abstract. If after removing duplicates the F1 margin and Sharpe advantage are not significant at p<0.05, the claim of outperformance should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'Sharpe ratio and F1 score 19.8% and 3% higher than the existing baselines'—rests on Tables 2 and 3, but the evidence has three interlocking problems. First, no error bars or significance tests are reported. The paper states experiments were repeated five times and averaged, but individual runs are never shown; the F1 margin over GCN-TOP20 is only 0.0069 on average (≈2.1%), and the Sharpe ratio fluctuates wildly across phases (HATS ranges from 0.41 to 4.39). Without variance estimates, this margin could easily be noise. Second, Table 2 contains exact duplicate rows: Phase 1 and Phase 7, and Phase 2 and Phase 8, are identical for all seven models. Since phases are different market windows, identical results are implausible and indicate a reporting or data-handling error that undermines the integrity of the whole table. Third, the abstract's 19.8% Sharpe improvement does not match any baseline comparison in Table 3: HATS's average Sharpe (1.9914) is 72.8% higher than the best baseline LSTM (1.1523), 107% higher than the best relational baseline GCN-TOP20 (0.9589), and 98.6% higher than TGC (1.0026). No comparison yields 19.8%. Unless a specific baseline and computation are clarified, the headline number is unverifiable. Together these issues mean the quantitative superiority of HATS over existing methods is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HATS, a hierarchical graph attention network for stock movement prediction. Node representations are initialized from an LSTM/GRU feature extractor over price change rates; HATS then applies two levels of attention—state-level attention over neighbors within each relation type and relation-level attention over relation types—to produce updated node representations, which are used for individual stock classification (node classification) and market index movement prediction via mean pooling (graph classification). The authors construct a corporate graph from Wikidata, using direct relations and two-hop meta-paths among 431 S&P 500 companies, and evaluate over 12 phases (text says 8) comparing against MLP, CNN, LSTM, GCN, GCN-TOP20, and TGC. They report that HATS outperforms all baselines in F1, accuracy, average daily return, and Sharpe ratio, and they analyze which relation types are useful.","tokens_in":19434,"tokens_out":6636,"duration_ms":61452,"significance":"The contribution is potentially useful: it is among the first to explicitly model relation-type selection for stock prediction and to cast index prediction as graph classification, with code and data publicly released. The relation-level analysis (Table 1 and Figure 5) is a valuable empirical observation that not all corporate relations help prediction. However, the central claim of consistent superiority is not currently supported by the evidence as presented: the experimental reporting contains internal inconsistencies, an unverified headline performance number, a baseline constructed with test-set leakage, and no uncertainty quantification. With corrected analysis the work could be a solid contribution, but the quantitative claims in the current manuscript are not yet established.","major_comments":[{"comment":"In Table 2, the F1 rows for Phase 1 and Phase 7 are identical to four decimal places for all seven models, as are the F1 rows for Phase 2 and Phase 8, even though the corresponding accuracy rows differ. Exact equality across all models for two different market windows is implausible and indicates a copying or data-processing error. Since Table 2 is the primary evidence for the F1 improvement claimed in the abstract, the table must be corrected and the source of the duplication explained.","section":"Section 5.1, Table 2"},{"comment":"The text states 'we divided our entire dataset into 8 smaller datasets' and that 'each phase consists of 250 days of training, 50 days of evaluation, and 100 days of testing,' yet Tables 2 and 3 report 12 phases. With 1,174 trading days, eight non-overlapping 400-day phases are impossible; the degree of overlap or alternative splitting scheme is not described. The experimental timeline is therefore not reproducible as written, and the text/table mismatch must be resolved.","section":"Section 5.1, Figure 3, Tables 2-3"},{"comment":"GCN-TOP20 is constructed by selecting the 20 best-performing relation types based on F1 scores on the Phase 4 test set (Table 1 is described as 'test set of Phase 4'). Using test-set labels to select the baseline's relation set leaks Phase 4 test information into that baseline's configuration, and also means GCN-TOP20 is not an 'existing method' as claimed. Relation selection should be performed on training/validation data only, or GCN-TOP20 should be re-framed as an oracle-style upper bound; as presented, the comparison is biased in GCN-TOP20's favor and cannot be used to support the claim that HATS outperforms all existing methods.","section":"Section 5.2, Table 1"},{"comment":"The paper states all experiments were repeated five times and averaged, but no standard deviations, confidence intervals, or significance tests are reported for any metric. The average F1 advantage of HATS over GCN-TOP20 is 0.0069 (approximately 2.1%), and per-phase Sharpe ratios for HATS range from 0.41 to 4.39, so without variance information the reported margins may be within noise. Report per-run results with error bars and test whether HATS's advantages over LSTM, TGC, and GCN-TOP20 are statistically significant.","section":"Section 5.1, Tables 2-3"},{"comment":"The abstract and contribution list state that HATS's Sharpe ratio was 19.8% higher than existing baselines. This number does not match any comparison in Table 3: HATS's average Sharpe (1.9914) is about 73% higher than LSTM (1.1523), 99% higher than TGC (1.0026), and 108% higher than GCN-TOP20 (0.9589). The 3% F1 figure also does not exactly correspond to any shown comparison (HATS vs. GCN-TOP20 is 2.1%; vs. LSTM is 5.8%). The baseline and computation behind the headline numbers must be specified, or the numbers corrected.","section":"Abstract, Section 1, Table 3"}],"minor_comments":[{"comment":"Equation numbering is duplicated: Eq. (3.9) is used for both the node-classification loss in Section 3.3 and the mean-pooling formula in Section 3.4; renumber the equations.","section":"Section 3.4"},{"comment":"The text refers to 'table 5' for the market index prediction results, but those results appear in Table 4; the appendix Table 5 contains relation definitions, so the cross-reference should be corrected.","section":"Section 5.4"},{"comment":"There is a duplicated word in 'which we we leave for future work'; this should read 'which we leave for future work.'","section":"Section 4.2"},{"comment":"Table 1 lists 'Country-Country of origin' twice with different F1 scores (0.2948 and 0.2851); one entry is presumably 'Country of origin-Country' and should be corrected.","section":"Table 1"},{"comment":"The sentence beginning 'With only five assets' starts with a lowercase 'With' after a period; this should be capitalized.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The combination of duplicated table rows, inconsistent phase counts, and an unmatched headline number suggests that the experimental results need careful re-verification before publication. I recommend asking the authors to provide per-seed results, a precise description of phase construction, and corrected tables; the referenced GitHub repository may help verify the numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read HATS with a skeptical eye and came out with two feelings at once. The central idea is genuinely reasonable: give the model two levels of attention, one over neighbors within a relation type, one over relation types, so it can down-weight noisy relations like country or stock exchange. That is a sensible extension of the relational-stock-prediction line, and to my knowledge the explicit comparison of which relation types help or hurt is new. The graph-pooling approach to index movement prediction is also a legitimate small extension. I credit the authors for running the analysis on 75 relation types and showing some relations are pure noise—that is actionable for anyone building corporate graphs.\n\nNow the soft spots, and they are mostly in the evidence. The duplicate rows in Table 2 (Phase 1 = Phase 7, Phase 2 = Phase 8 for all seven models) are not plausible and need an explanation; they undermine trust in the whole table. No error bars or significance tests are reported despite five repeated runs, and the F1 margin over GCN-TOP20 is about 0.007—within noise for this kind of task. The abstract's '19.8% higher Sharpe ratio' matches no comparison in Table 3; the average Sharpe for HATS is 1.99 versus 1.15 for LSTM, which is ~73% higher, not 19.8%. One number is simply wrong, and it is the headline number. I would also flag the GCN-TOP20 baseline: it is built by picking the top 20 relations using test-phase F1 from Phase 4, which is test-set peeking and gives that baseline an unfair disadvantage or advantage depending on the phase. The labeling thresholds and portfolio construction are under-specified, though not fatal once you read the text.\n\nThe hierarchical attention mechanism is structurally close to the Heterogeneous Graph Attention Network (HAN), which is not cited. That is a substantial oversight for a paper whose novelty is largely the application domain, not the architecture. The authors do acknowledge the graph construction can be much improved, which is honest, but it also means the reported gains might be specific to this particular Wikidata meta-path graph.\n\nWho is this for? People working on stock prediction with relational data, and anyone building corporate knowledge graphs for finance. The paper is worth a serious referee because the idea is timely and the empirical questions are important, but it needs a major revision: fix the table, report variances, reconcile the abstract, and redo GCN-TOP20 without test-set selection. I would not desk-reject it, but I would not trust the headline numbers in the current form.","headline":"The empirical headline doesn't survive contact with the tables, but the core idea—learned per-relation weighting for stock prediction—is worth a careful look.","tokens_in":797,"tokens_out":2699,"would_cite":false,"duration_ms":37573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a hierarchical graph attention network which learns to weight corporate relations can improve stock movement prediction for both individual stocks and market indices, beating existing baselines on Sharpe ratio and…","keywords":["stock movement prediction","graph neural network","attention mechanism","corporate relations","market index prediction","Sharpe ratio","Wikidata meta-path","hierarchical attention"],"falsifier":"Train HATS on a graph whose relation type labels are randomly permuted, or whose edges are rewired while preserving degree; if the improvement in Sharpe ratio and F1 score over the baselines persists unchanged, then the gains come from the attention mechanism's extra capacity rather than from selectively aggregating meaningful relation types, and the paper's central claim would be falsified. A second check: if the learned relation attention weights show no correspondence to the per-relation predictive performance measured in Section 5.2, the selection mechanism is not doing the work.","tokens_in":18880,"feed_emoji":"📈","tokens_out":4698,"duration_ms":42584,"temperature":0.7,"pith_summary":"The paper argues that when predicting stock movements from a corporate relational graph, different types of company relations carry very different predictive value, and indiscriminately aggregating them hurts performance. It proposes HATS, a hierarchical attention network that first weights neighbors within each relation type and then weights relation types themselves, so the model learns which relations matter for the prediction task. The authors show that this selective aggregation outperforms existing graph-based and non-graph baselines on both individual stock classification and market index classification, with 19.8% higher Sharpe ratio and 3% higher F1 score on average. The central message is that relation selection should be learned, not fixed, because the usefulness of a relation depends on the market phase.","feed_headline":"Attention network lifts stock prediction Sharpe ratio by 19.8%","feed_subtitle":"A two-level attention model learns which company relations matter, beating static graph and sequence baselines on S&P 500 stocks.","key_machinery":"The central mechanism is the two-layer hierarchical attention: a state attention layer computes weights over neighboring companies within each relation type, producing a summarized representation per relation, and a relation attention layer then weights these relation summaries based on the target company's current state, with the weighted summaries added to the node's own representation. The state- and relation-dependent weights allow the model to filter noisy relations and focus on informative ones, in contrast to static adjacency-matrix methods like GCN or the temporal graph convolution of TGC. This machinery carries the paper's claim that automatic relation selection improves prediction.","core_discovery":"On its own terms, the paper establishes that stock movement prediction improves when a model is able to selectively aggregate information from different corporate relation types. It demonstrates that the choice of relational data changes prediction performance dramatically: some relations, such as parent-organization and subsidiary ties, help, while dense geographical relations such as country of origin degrade accuracy. The paper's contribution is HATS, a two-level attention module that summarizes neighbor information per relation and then weights the relation summaries, adding the result to each company's representation. On S&P 500 data, HATS outperforms existing methods, including a manually-selected-top-20-relations GCN, on F1 score, and achieves the highest average Sharpe ratio among all tested models. The same module is adapted to predict market index movements through graph pooling, outperforming baselines there as well.","pith_inferences":["The finding that dense, semantically weak relations hurt performance suggests a testable preprocessing rule: filtering or down-weighting high-degree relation types could improve even non-attention baselines like GCN.","If HATS's gains persist on graphs built from richer or alternative knowledge bases, or from text-extracted relations, the method would become a general-purpose relational modeling module for financial graphs.","The learned relation attention weights could be used as interpretable indicators of which corporate connections matter at a given time, potentially serving as features in fundamental or risk analysis."],"forward_implications":["Using all available relations without selection can degrade prediction; models that learn to weight relations adaptively achieve the best results.","The same relational modeling module transfers from node classification (individual stock movement) to graph classification (market index movement) when combined with graph pooling.","HATS removes the need for manual selection of relation types, since its attention weights learn which relations matter per market phase.","The profitability gains, measured by average daily return and Sharpe ratio, are stronger for HATS than for any baseline, indicating that attention-based relation selection improves risk-adjusted returns."],"supporting_citations":[{"why":"Supplies the GCN baseline that must be beaten and the precedent of using LSTM as a feature extraction module.","marker":"[7]"},{"why":"Supplies the TGC baseline, the main relational modeling competitor, and the Wikidata meta-path idea that HATS builds on.","marker":"[13]"},{"why":"Supplies the price-change-rate features and the neutralized portfolio trading strategy used to measure profitability.","marker":"[14]"},{"why":"Supplies the attention mechanism that HATS extends into a two-level hierarchy.","marker":"[25]"},{"why":"Supplies the Wikidata knowledge base from which the corporate relation graph is built.","marker":"[26]"},{"why":"Supplies the GCN convolutional layer used in the baseline graph models.","marker":"[17]"},{"why":"Supplies the multi-phase experimental design used to test performance across volatility regimes.","marker":"[3]"}],"fun_headline_variants":["HATS: hierarchical attention selects useful relations for stocks","Relation-aware attention outperforms static graphs in stock prediction","Stock movement prediction improved by selective relation aggregation","HATS network learns which company ties matter for future prices","Selective relation attention beats baseline graphs on S&P 500"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result rests on the assumption that the Wikidata-derived corporate graph, using meta-paths of at most two hops and excluding companies with no relations, captures the relationships that actually drive stock prices.","fun_headline_variants_meta":{"raw":{"variants":["HATS: hierarchical attention selects useful relations for stocks","Relation-aware attention outperforms static graphs in stock prediction","Stock movement prediction improved by selective relation aggregation","HATS network learns which company ties matter for future prices","Selective relation attention beats baseline graphs on S&P 500"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3269,"prompt_tokens":968,"completion_tokens":2301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2234}},"tokens_in":584,"tokens_out":2301,"duration_ms":18132,"temperature":1.0,"reasoning_tokens":2234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:43:18.155033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train HATS on a graph whose relation type labels are randomly permuted, or whose edges are rewired while preserving degree; if the improvement in Sharpe ratio and F1 score over the baselines persists unchanged, then the gains come from the attention mechanism's extra capacity rather than from selectively aggregating meaningful relation types, and the paper's central claim would be falsified. A second check: if the learned relation attention weights show no correspondence to the per-relation predictive performance measured in Section 5.2, the selection mechanism is not doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GCN baseline that must be beaten and the precedent of using LSTM as a feature extraction module."},{"cited_title":"Tem- poral relational ranking for stock prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the TGC baseline, the main relational modeling competitor, and the Wikidata meta-path idea that HATS builds on."},{"cited_title":"Deep learning with long short-term memorynetworksforfinancialmarketpredictions","cited_arxiv_id":null,"evidence_quote":"Supplies the price-change-rate features and the neutralized portfolio trading strategy used to measure profitability."},{"cited_title":"Wikidata: a free collaborative knowledge base","cited_arxiv_id":null,"evidence_quote":"Supplies the Wikidata knowledge base from which the corporate relation graph is built."},{"cited_title":"A deep learning framework for fi- nancial time series using stacked autoencoders and long-short term memory","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-phase experimental design used to test performance across volatility regimes."}],"review_version":1}